2016 Intersemiotic Translation As Resemiotisation - A Multimodal Perspective

12/8/2019 Intersemiotic Translation as Resemiotisation: A Multimodal Perspective
Signata
Annales des sémiotiques / Annals of Semiotics
7 | 2016
Traduire : signes, textes, pratiques
Dossier
3. Transpositions between verbal and non-verbal semiotics
Intersemiotic Translation as
Resemiotisation: A Multimodal
Perspective
K L. O’H ,S T P W
p. 199-229
Abstracts
English Français
Intersemiotic translation is viewed as the basis of cultural communication through which ideas
are circulated, translated and explained using language, images and other semiotic resources.
From this perspective, intersemiotic translation is conceptualised through the principle of
resemiotisation which is concerned with how semiotic choices are translated over time as social
practices. Key issues which arise from viewing intersemiotic translation as resemiotisation are
investigated in this paper: (a) how can shifts of meaning be conceptualised across semiotic
resources which are fundamentally different in nature?; (b) what meanings are retained and
changed as a result of resemiotisation?; and (c) how can such meanings be modelled theoretically
and tracked analytically? In order to explore these issues, a multimodal social semiotic approach
based on Michael Halliday’s systemic functional theory is presented. In this approach, semiotic
resources are conceptualised as systems of meaning which differentially fulfil various functions in
society: to structure experience and make logical connections in the world, to enact social
relations and create a stance towards the world, and to organise these meanings into multimodal
messages. The multimodal approach to intersemiotic translation is illustrated through examples
which explore shifts of meaning which take place across theses strands of meaning as semiotic
choices are resemiotised within and across multimodal texts. The focus of this discussion is
intersemiotic translation across language (spoken and written), images (photographs, film tracks
and graphs) and mathematical symbolism in a range of different text types (i.e. a news report, an
infographic, a video and a mathematical graph). These examples illustrate the principles of
intersemiotic translation through which semantic expansions occur as semiotic choices are
resemiotised using different semiotic resources, which result in access to a new meaning
potential.
La traduction intersémiotique est considérée comme la base de la communication culturelle, par

le biais de laquelle les idées sont diffusées, traduites et expliquées à l’aide du langage, d’images, et
d’autres ressources sémiotiques. Dans une telle perspective, la traduction intersémiotique est
conceptualisée par le principe de resémiotisation qui considère comment les choix sémiotiques se
https://journals.openedition.org/signata/1223?lang=en 1/25
traduisent en pratiques sociales au fil du temps. Cet article étudie les questions clés soulevées par
le fait de traiter la traduction intersémiotique comme une resémiotisation: (a) comment les
glissements de signification peuvent-ils être conceptualisés à travers des ressources sémiotiques
qui sont de nature fondamentalement différente ? (b) quelles significations sont conservées et
modifiées en raison de la resémiotisation ? et enfin (c) comment ces significations peuvent-elles
être modélisées théoriquement et suivies de façon analytique ? Pour explorer ces questions, une
approche socio-sémiotique multimodale est présentée, basée sur la théorie systémique
fonctionnelle de Michael Halliday. Dans cette approche, les ressources sémiotiques sont
conceptualisées comme des systèmes de significations qui remplissent des fonctions variées dans
la société, pour structurer l’expérience et établir dans l’univers des connexions logiques, pour
établir des relations sociales et une attitude vis-à-vis du monde, et pour organiser toutes ces
significations en messages multimodaux. L’approche multimodale à la traduction
intersémiotique est illustrée par des exemples qui explorent les glissements de signification qui se
produisent à travers la toile des significations, lorsque les choix sémiotiques sont resémiotisés au
sein de textes multimodaux, et entre ceux-ci. Au cœur de cette discussion est la traduction
intersémiotique entre langage (parlé et écrit), images (photos, films et graphiques) et symbolisme
mathématique, dans une gamme de différents types de textes (par exemple reportage, document
infographique, vidéo et graphique mathématique). Ces exemples illustrent les principes de la
traduction intersémiotique au moyen de laquelle des expansions sémantiques se produisent,
lorsque les choix sémiotiques sont resémiotisés à l’aide de différentes ressources sémiotiques,
donnant ainsi accès à un nouveau potentiel de signification.
Index terms
Mots-clés : traduction, multimodalité, intersémioticité, semiosis, image
Keywords : translation, multimodality, intersemioticity, semiosis, image
Full text
1. Introduction to intersemiotic
translation
1 Translation is typically thought of as involving language, in particular written
language. However, translation extends beyond linguistic translation and the
interpretation of linguistic signs through rewordings to “intersemiotic translation”,
which, in its original sense, was seen as involving the interpretation of linguistic signs
by means of non-verbal resources (Jakobson 1959, p. 114). Jakobson (Ibid., p. 233)
distinguishes three ways of interpreting verbal signs: intralingual translation
(translation into other signs of the same language); interlingual translation (translation
into another language); and intersemiotic translation (translation from language into
another, nonverbal system of symbols). For Jakobson, intersemiotic translation
involved language. He does not discuss translation from one non-verbal semiotic
system to another non-verbal semiotic system or the translation of multisemiotic texts.
2 Today, Jakobson’s (1959) definition of intersemiotic translation has been broadened
to include translations across non-linguistic semiotic resources (e.g. Kourdis and Yoka,
2014). This development seems inevitable, given the proliferation of different forms of
multimodal texts in today’s digital environment, where semiotic resources (e.g.
language, image and sound resources) “coexist, cooperate, and get translated” (Kourdis
2015, p. 311) on a regular basis. In this regard, intersemiotic translation, the constant
translation of signs into other signs, forms the basis of cultural communication, as Eco
(1979) explains:
[…] culture continuously translates signs into other signs, and definitions into
other definitions, words into icons, icons into ostensive signs, ostensive signs into
new definitions, new definitions into propositional functions, propositional
functions into exemplifying sentences, and so on; in this way it proposes to its
members an uninterrupted chain of cultural units composing other cultural units,
and thus translating and explaining them. (Eco 1979, p. 71)
3 Eco’s (1979) view of intersemiotic translation as the basis for cultural communication
is adopted here. From this perspective, intersemiotic translation is conceptualised as
“resemiotisation”, which is concerned with “how semiotics are translated from one into
the other as social processes unfold” (Iedema 2003, p. 29). Key issues which arise from
viewing intersemiotic translation as resemiotisation include: (a) how can shifts of
meaning be conceptualised across semiotic resources which are fundamentally different
in nature?; (b) what meanings are retained and changed as a result of resemiotisation?;
and (c) how can such meanings be modelled theoretically and tracked analytically?.
4 The phenomenon of intersemiotic translation remains “virtually unexplored in terms
of its conceptual modelling, especially from a semiotic perspective” (Aguiar and Queiroz
2009, p. 1) despite it being theoretically relevant and commonly practised. While
Jakobson’s view of intersemiotic translation always included language, Aguiar and
Queiroz (Ibid., p. 1) extend the principle to include “translations of texts of all kinds”.
Aguiar and Queiroz also propose that the main methodological difficulty in
intersemiotic translation is the comparison of different semiotic resources, and they
critique current models as being mainly descriptive, lacking explicative models and
being “dissociated from results produced in the area of general semiotic and translation
studies” (Ibid., p. 7). They attempt to resolve this methodological difficulty by adopting
a conceptual framework based on Peirce’s model of sign process, as a starting point
(Ibid., p. 2).
5 Perez-Gonzalez (2014) cites conflicting views of intersemiotic translation in the
translation literature, with some scholars (e.g. Gottlieb 1997, p. 111; Remael 2001,
pp. 13-14) regarding intersemiotic translation as the transfer of meaning across
different media, while others (e.g. Fine 1984) understand it as shifts between two
different variants of the same sign system, such as changing from spoken to written
language in film subtitling. Perez-Gonzalez (2014) also cites a lack of consensus on the
definition of terms such medium, mode, and sign system, which “exposes the need for a
more comprehensive and sophisticated understanding of the semiotic fabric of
translated and interpreted texts” (Ibid., p. 120).
6 O’Sullivan (2013, p. 5) notes that translation studies has “struggled at times with the
concept of multimodality”, which is concerned with the study of the various semiotic
resources involved in communication (see different approaches to multimodality in
Jewitt, Bezemer and O’Halloran 2016). O’Sullivan (2013) explains that while some
progress has been made in developing methodologies for research on the translation of
multimodal texts, a number of technical and logistic challenges remain (Ibid., p. 6). She
goes on to say that “it makes sense then that the saturated multimodality of many texts
today would require both a new, or at least a rethought, critical and analytical toolbox,
and potentially also new approaches to translation” (Ibid., p. 6).
7 At least some of the difficulties in developing a theoretical framework for
intersemiotic translation arise from attempting to find a “direct” translation between
meanings made through choices from semiotic systems that are fundamentally
different in nature (e.g. language and image). As Iedema (2003, p. 47) points out,
translations between different semiotic resources inevitably introduce discrepancies.
While an exact intersemiotic rendition is highly improbable, if not impossible (except in
certain cases, see discussion of mathematics below), a reasonable approximation is
certainly likely. The sticking point, as O’Sullivan points out, is in finding “a critical and
analytical toolbox” (O’Sullivan 2013, p. 6) which is capable of theorising and modelling
the shifts of meaning which take place through intersemiotic translation.
8 In this paper, intersemiotic translation is conceptualised through the principle of
resemiotisation, which is concerned with “how meaning shifts from context to context,
from practice to practice, or from one stage of a practice to the next” (Iedema 2003,
p. 41). Iedema first used the term to account for “the origin and dynamic emergence” of
those shifts in meaning (Ibid., p. 40). Jewitt (2014, p. 467) summarises Iedema’s
(2003) use of the term resemiotisation as the phenomenon “in which a particular set of
meanings is transformed from one semiotic system (and configuration of media and
modes) to another as social processes unfold”. The examples Iedema (2003) cites
illustrate changes in context over time and trace the stages in a social process, such as
how what is said and agreed on at meetings is summarised into a written report, which
is then resemiotised as an architectural design. This view of resemiotisation implies

successive stages over time but it also implies multimodality. Each stage involves taking
meanings from a source(s) and reconstructing them in a different mode or medium,
which itself might be multisemiotic.
9 While Iedema (2003) focuses on the processes or dynamics of resemiotisation, it is
also possible and productive to examine the actual products of resemiotisation (i.e. the
multimodal texts) which are circulated and resemiotised by members of a culture. For
people who did not have access to the process, the products are the basis of cultural
communication through which social processes are (re-) configured. From this
perspective, the view of intersemiotic translation adopted in this paper is that any
attempt at translation of meanings made though choices in one semiotic resource into
meanings made though choices made from another semiotic resources involves
resemiotisation in terms of process and product. In this paper, the focus is
intersemiotic translations which take place within and across the semiotic products or
artefacts resulting from resemiotisation processes: that is, the focus is intersemiotic
translation in the multimodal texts themselves.
10 To address the issues discussed above, the aim of this paper is to present a
multimodal approach to intersemiotic translation which provides a common theoretical
platform for conceptualising different semiotic resources and exploring the shifts of
meaning which take place through resemiotisation. The approach is based on Michael
Halliday’s (1978) social semiotics, where language and other semiotic resources are
viewed as resources for making meaning. Halliday’s (1978) social semiotic approach,
developed as systemic functional theory (SFT), views semiotic resources as systems of
meaning which fulfil a range of functions in human communication. The systems of
meaning are unique to each semiotic resource, but configurations of system choices
work together in order to construct thought and reality in specific ways in any culture.
This theoretical approach is discussed in detail in Section 3.
11 Before introducing the systemic functional approach to intersemiotic translation,
other semiotic and multimodal approaches to intersemiotic translation are reviewed.
Following this, the basics of Halliday’s systemic functional theory (SFT) are presented.
The multimodal approach to intersemiotic translation is then developed and illustrated
through examples which explore shifts of meaning which take place as semiotic choices
are resemiotised. The focus of this discussion is language (spoken and written), images
(photographs, graphs and a film track) and mathematical symbolism in a range of
different text types (i.e. a news report, an infographic, a video and mathematical
graphs) found on an Internet website, in this case, the World Health Organization
Ebola website.1 The wide range of examples chosen demonstrates that the systemic
functional approach can be applied to different types of intersemiotic processes, which
in this case, occur within the context of a single website. The examples analysed below
also demonstrate that new methods for investigating intersemiotic translation are
required: in this case, purpose-built software applications for text, image and video
analysis are demonstrated to handle the complexity and multi-level nature of
multimodal semiosis.
2. Multimodal approaches to
intersemiotic translation
12 Many text types involve language in combination with other semiotic resources, such
as still and moving images, diagrams, graphs, music and typography. Multimodality can
be defined as “the use of several semiotic modes [i.e. resources] in the design of a
semiotic product or event” (Kress and van Leeuwen 2001, p. 20, cited in Jewitt 2014,
p. 1). Multimodal texts involve more than language. They can include meanings made
from choices from any semiotic system and do not necessarily need to incorporate
language. Any text which utilises more than one semiotic resource is a multimodal text.
13 O’Sullivan (2013, p. 2) points out that multimodal meaning-making is deployed for
many purposes, such as promotional, political, expressive and informative purposes
and that signifying elements other than language must be understood and accounted
for by professionals who work with text, such as technical translators, literary
translators, copywriters, subtitlers and publishers. The list could be expanded to
include people who work with the design and creation of multimodal texts, such as web
designers and graphic designers. O’Sullivan also points out that, while multimodality
has long been present in texts, it “has become increasingly conspicuous” with the
development of the World Wide Web and new forms of communication and
entertainment (Ibid., p. 5).
14 Taylor (2013, p. 98) comments on the growing importance of multimodality,
especially for audiovisual translation, and says that “a limited number of scholars have
in fact ventured into this field”. Taylor (2013) also comments that “the role of
translation in multimodality studies … seems to have attracted little or no interest”
from outside the field of translation. Taylor (2013) does cite the work of a number of
scholars influenced to varying degrees by Halliday’s (1978) ideas on language as a social
semiotic as opening doors “to a growing interest in how different modes worked
together to create meaningful texts” (Taylor 2013, p. 98).
15 One such study is Borodo’s (2015) multimodal analysis of the relationship between
the verbal and the visual in the translation of comic books. Borodo’s (2015) work
focuses on how the verbal and visual modes interact and jointly contribute to creating
meaning and is influenced by the work of Kress and van Leeuwen (1996, 2001), Jewitt
(2014), Kress (2009) and Royce (2007). Borodo (2015, p. 23) argues that language is
one element within a larger semiotic framework that includes images, gesture, posture,
gaze and colour, which should be viewed as possessing equal (but not equivalent)
meaning-making potential. Borodo (2015, p. 40) proposes that “investigating comics
from a multimodal perspective may be another step towards a more complete
understanding of the nature of this still largely unexplored sphere within Translation
Studies”.
16 The assumption so far appears to be that intersemiotic translation involves
translation between texts, taking meanings from one text and transposing them as
accurately as possible into another text. While this is probably the most important
aspect of intersemiotic translation for professional translators, there is also the
phenomenon of intersemiotic translation within a text to consider. That is, in the same
text meanings encoded by one semiotic resource are often re-encoded, or resemiotised,
through another semiotic resource: for example, information in a graph could be re-
expressed in language or a photograph could be resemiotised as an infographic.
Phenomena such as these are important for people designing and creating these texts
and for the people who read and view them. Beyond this, as intersemiotic translation
forms the basis for explaining and circulating ideas in society and culture, it has
implications that extend well beyond translation and multimodal text design.
3. A multimodal systemic functional

approach to intersemiotic translation
17 The approach to intersemiotic translation adopted here is derived from multimodal
social semiotic theory, based on Halliday’s systemic functional theory (SFT). In SFT,
semiotic resources are conceptualised as inter-related systems which together
constitute and manifest culture (Halliday, 1978, 2008; Halliday and Hasan, 1985).
While Halliday studied language, he always understood that language was one semiotic
resource among the many (e.g. images, gesture, dress and sounds) which constitute
culture and the fundamental principles of the approach are applicable for the study of
other semiotic resources. The view of multimodal semiosis, in which recognizable
configurations of semiotic choices constitute social practices, provides the basic
foundations of the approach. Significantly, the SFT approach is capable of handling the
multi-dimensional complexity of intersemiotic translation as resemiotisation, as
explained below.
18 Semiotic resources are viewed as having a meaning potential which is described in

terms of interconnected systems of meaning. Systemic functional approaches to
multimodal discourse analysis (SF-MDA) (e.g. see Chapter 3 in Jewitt, Bezemer and
O’Halloran 2016; O’Halloran, Tan and Marissa 2015a; Tan, Smith and O’Halloran
2015) are concerned with the “grammatics” of semiotic resources, which involves
formulating the systems of meaning in order to understand the functions of different
semiotic resources and the meanings which arise when semiotic choices combine in
multimodal phenomena over space and time (O’Halloran and Lim 2014). Following
Halliday’s SFT, the systems are organised according to the functions (called
“‘metafunctions”) which the resources serve in society (e.g. Halliday and Matthiessen
2014; Martin and Rose 2007):
Ideational meaning which consists of experiential meaning: to structure

experience of the world and logical meaning: to make logical connections in the
world;
Interpersonal meaning: to enact social relations and create a stance towards
the world;
Textual meaning: to organise experiential, logical and interpersonal meanings
into messages
19 The meanings of multimodal processes and texts are characterised in terms of

options selected from the systems which realise the metafunctions (i.e. ideational,
interpersonal and textual). For example, systems of meaning have been formulated for
language (Halliday and Matthiessen, 2014), static images (Kress and van Leeuwen,
2006; O’Toole, 2011), music (van Leeuwen 1999), action (Martinec 2001, 2004) and
film resources (Bateman and Schmidt 2012). The systems are typically organised
according to different ranks of constituency (e.g. discourse semantics, lexico-grammar
and phonology/graphology for language; and work, episode and figure for image) (see
Section 4 below).
20 While the metafunctional principle that each semiotic resource is organised to realise
four different strands of meaning simultaneously (i.e. experiential, logical,
interpersonal and textual) is applied to any semiotic system, the underlying systems
and choices are different for each semiotic resource. That is, semiotic resources have
differential capacities with regards to the different metafunctions and the subsequent
meanings which are made. Images, for instance, do not structure and order the world in
the same way as language does. Images order human experience by situating
happenings in relation to other happenings, as parts of a whole, unlike language where
happenings are constructed in sequential order. For example, in a photograph, painting
or a scientific diagram many happenings and actions are taking place in relation to each
other simultaneously. However, certain aspects of the image are made salient through
semiotic choices such as gaze, light, colour and framing and immediate features of the
context of the situation (e.g. instructions to view parts of the image, or captions). That
is, images are ‘read’ in particular ways, depending on the semiotic choices which are
made within the image and the context. In this regard, multimodal semiosis results in
an expanded meaning potential derived from the integration of different
metafunctional capabilities. As illustrated by the examples considered below,
intersemiotic translations permit semantic expansions which extend beyond those
possible with one resource alone.
21 The metafunctional principle plays an important role in SF-MDA for understanding
the functionalities and underlying organisation of semiotic resources and investigating
the ways in which semiotic choices interact to create meaning. As O’Halloran and Lim
(2014, p. 140) explain, the metafunctional organisation of meanings is particularly
useful for SF-MDA,
[…] because it provides a common set of fundamental principles to compare

semiotic resources and the meanings which arise when semiotic choices integrate
in multimodal text. That is, the organisation of metafunctional meanings offers a
unifying platform for studying semiotic resources and their inter-semiotic
relations. (O’Halloran and Lim 2014, p. 140)
22 In SF-MDA, the focus on the metafunctional organisation of semiotic resources is

critical, but the actual choices in multimodal texts and processes are interpreted in
relation to the context using the concepts of register and genre (e.g. Eggins 2004;
Martin 1992, 2002; Martin and White 2005). That is, choices from multimodal systems
of meaning form more or less stable (but evolving) configurations, which are socially
and culturally recognisable. That is, while the meaning potentials of semiotic resources
are diverse, the actual options selected in any context are conditioned by previous
choices within that culture. These dimensions are described using register theory
(Halliday 2002 [1977]; Matthiessen 2009, Martin, 1992, Martin and Rose, 2003),
which is concerned with three key dimensions: field—the nature of the social activity
(realised through experiential and logical choices); tenor—the social relations which are
enacted (realised through interpersonal choices); and the mode—spoken, written and
visual forms of representation (realised through textual choices). The genres found in
any culture are realised through the various configurations of register variables (i.e.
field, tenor and mode). In this case, genre is defined as “the system of staged
goal‑oriented social processes through which social subjects in a given culture live their
lives” (Martin 2002, p. 56). Therefore, while multimodal semiosis “multiplies” the
potential meanings which can be made, in reality meaning is constrained according to
context and culture (e.g. Lemke 1998).
23 The multi-level systemic model, in which semiotic resources in multimodal texts are
theorised in terms of metafunctionally organised systems of meaning and semiotic
choices are analysed according to register and genre, opens up space for an SF-MDA
approach to be applied in order to trace the shifts in meaning which take place through
intersemiotic translation. That is, the application of a unified theoretical model which
applies equally well to different semiotic resources permits the effects of intersemiotic
translation to be calibrated, as illustrated in the examples which follow.
4. Examples of SF-MDA approach

applied to intersemiotic translation
24 The SF-MDA approach to intersemiotic translation is demonstrated by analysing the
meanings which arise in a news report with linguistic text and photographs, an
infographic, a video and a mathematical graph. In each case, intersemiotic translation
is explored using the concepts of metafunctionally organised systems of meaning,
register and genre. The systems are organised according to different ranks of
constituency for each resource. For example, the text and image systems in Table 1 are
organised according to discourse semantics, lexico-grammar and
phonology/graphology for language; and work, episode and figure for image. These
systems are based on Halliday’s (Halliday and Matthiessen 2014) and Martin’s (Martin
and Rose 2007) systems for language and O’Toole’s (2011) framework for images. Other
systems are also considered, for example, cinematography in video and film (see Table
2) and mathematical symbolism (see Table 3). In what follows, the text and image
systems in Table 1 are used to analyse the intersemiotic translation of meanings
between text and photograph in a new report from the WHO Ebola website. Following
this, intersemiotic translations which occur in a video and a mathematical graph from
the same website are explored.
25 In order to handle the complexity of such analysis, purpose-built software tools,
Multimodal Analysis Image2 and Multimodal Analysis Video3, which are specifically
designed to explore semiotic interactions in static (e.g. written texts and images) and
dynamic media (e.g. videos) respectively, are used. The software applications permit
media files to be imported and analysed using different systems which are entered into
the software, and the results are stored in a database for further data processing (see
O’Halloran, Tan and Marissa 2015b).
Table 1
Language Image
Rank System Rank System
EXPERIENTIAL MEANING
Ideation & Taxonomic relations

Text & Narrative Theme; Representation;
Discourse Organizing experience into activity Work Setting
semantics sequences and class/sub-class and Nature of the scene
whole/part relations
Processes; Participant Roles; and

Circumstance
Episode
Clause Processes; Participant Roles; Visual happenings, actions and
(lexico- Circumstance relations
grammar) Happenings, actions and relations
Posture; Dress
Figure
Characteristics of the participants
INTERPERSONAL MEANING
Exchange Structure
Speech Function
Text &
Angle; Camera Distance; Lighting
Discourse Exchange of information (e.g. Work
semantics statements and questions) and Visual effects
goods & services (e.g. commands
and offers)
Proportion in Relation to the

Whole Image: Focus; Perspective
Mood Episode
Happenings, actions and relations
Clause Declarative or interrogative (for
with respect to the whole image
(lexico- exchange of information)
grammar) Imperative and modulated Gaze-Visual Address
interrogative (for exchange of
goods and services) Figure Direction of participant’s gaze as
internal to image or external to
viewer
TEXTUAL MEANING
Compositional Vectors; Framing

Text & Reference and Retrieval
Discourse Identifying and tracking participants Work The organisation of the parts as a
semantics in text whole, with the visual marking
(e.g. framing) of certain parts
Relative Placement of Episode;

Framing
Episode Position of the happenings,

actions and relations in relation to
the whole image, and the visual
Clause Information Focus marking of certain aspects
(lexico- Organisation of information, with Relative Placement of the Figure
grammar) points of departure for what follows within the Episode; Arrangement;
Framing
Figure Position of figures in relation to
happenings, actions or relations,
and the visual marking of certain
aspects of those figures
Text and image systems
4.1. Intersemiotic translation between photograph

and text
26 This example discusses intersemiotic translation between two photographs and

accompanying text in a news story “Ebola—a test too far for one little girl”4 from the
WHO Ebola website displayed in Fig. 1a. Fig. 1b shows an excerpt of the story as
analysed in Multimodal Analysis Image software, using the text and image systems
displayed in Table 1. The story has been selected to demonstrate the principles of
intersemiosis and resemiotisation between the linguistic text and the photographs. The
section of the story used for illustrative purposes consists of both photographs and the
first four paragraphs of linguistic text following the introductory paragraph, displayed
in Fig. 1b. The photographs and linguistic text realise ideational, interpersonal and
textual choices, where the choices are made from different sets of options from different
systems, as displayed in Table 1. The results of these choices, when combined, realise
meanings which are different from those realised by each semiotic resource. This
intersemiosis is multi-directional. Whether the text is read after viewing the
photograph or the photograph is viewed after reading the text, the meanings which can
be made from each, both individually and together, are different from the meanings
which are made by each resource in isolation. This transference of meaning is also not a
single occurrence. Each time a viewer/reader moves from image to text or the other way
around, something new emerges that changes how both are perceived.
27 The photographs show two groups of people involved in discussion or negotiation.
From the photographs alone a viewer cannot tell what the people are talking about or
what relationships there are among them. For the purposes of this discussion, it is
useful to start with textual or compositional meaning. Textual meaning is organised
differently in the linguistic text and in the photographs.
Figure 1a
News story ‘Ebola – a test too far for one little girl’4 as displayed on the WHO Ebola website
28 In linguistic texts, information flows from the beginning to the end of the text in
waves of different scales (Martin and Rose 2007 pp. 175-205) and the position of that
information tells the reader whether or not it is prominent as thematic or new
information. In the photograph all the information is presented simultaneously and
different resources such as Foregrounding, Parallelism and Relative Position and
Proportion in the work as a whole are used to identify points of prominence (O’Toole
2011, pp. 23-29). For example, in both photographs compositional choices guide the
viewer to see the man in the white T-shirt and dark sleeveless jacket as prominent
(outlined in red in the photographs in Fig. 1b). He is the focus of the other people’s
attention. Even though he is not foregrounded in either photograph he is still textually
the most prominent figure in both. While his Gaze-Visual Address is directed at what he
is holding in his hands and he is absorbed in a telephone conversation in the first
photograph, the gaze vectors of the other participants are directed at him (indicated by
blue directional arrows in Fig. 1b). These gaze vectors draw the viewer’s attention to
him even though he is not in the foreground. In the second photograph the upper torso
and head of the same man are framed by the window behind. This draws attention to
the parts of him that are framed. He is also facing the viewer in both photographs while
most of the other participants have their backs to the viewer.
29 The same man is also made thematically prominent in the parts of the linguistic text
that relate most directly to the photographs. The man is identified in the text as Dr
M’Bemba Camara and as “a member of WHO’s surveillance team in Guinea”. By name
and by role he has thematic prominence in three clauses in the first paragraph following
the story’s introductory paragraph. In the linguistic text discourse semantic resources
of reference and retrieval are deployed to establish Dr M’Bemba’s identity and track
him through pronouns. As well as being identified by name, Dr M’Bemba is identified
as him, he and his five times in the four paragraphs (outlined in shades of red in the text
in Fig. 1b). In the photographs the viewer can identify him visually. As the photographs
depict episodes in a single instance of time and he appears once in each photograph, it
is relatively simple to track his identity across the two photographs. By combining
information in the text and photographs the viewer/reader can specifically identify who
Dr M’Bemba is among the participants in the photographs. This is done largely through
his clothing in both photographs and through the medical apparatus (the contactless
thermometer and checklist) he is holding in the second photograph. Of the other
participants in the photographs four are identified. The girl, Mariam, is identified by
name and can be identified in the second photograph by inference since she is the only
girl in the photograph. The other three can be identified by inference and by drawing
connections between the photographs and the linguistic text. The older man, wearing
the cap in the first photograph and seated in the second photograph is, by inference,
Mariam’s father. The viewer can also infer, less strongly, that the woman in the first
photograph is Mariam’s mother. (She could also be part of WHO’s surveillance team.)
The other male participant in the first photograph is most likely to be the social
mobilizer mentioned in the story. The roles of the other two male participants in the
second photograph cannot be determined. Choices from two different systems of
semiotic resources combine to both identify him and reinforce his prominence.
Prominence of participants in the photographs tends to parallel and reinforce
prominence in in the linguistic text and vice versa.
Figure 1b
Intersemiotic translation between photograph and text as analysed in Multimodal Analysis Image software
30 Interpersonally, at the rank of discourse semantics, the photographs and linguistic
text both function to present information to the reader/viewer. In the linguistic text the
whole text grammatically is in declarative mood, the default mood choice for the speech
function of giving information. In terms of the register variable of tenor, the linguistic
text establishes Dr M’Bemba’s professional and institutional status because he is a
doctor and he is working for WHO. This status is reinforced in the photographs where
he is the focus of the gaze vectors of all the other participants while in each photograph
he is looking elsewhere. The function of the photographs to present the viewer/reader
with information is also reinforced by the absence of any gaze from within the
photographs directed externally towards the viewer. There are no overt choices made to
engage the viewer, who is positioned as an observer of the scene.
31 Experientially, the photographs and linguistic text complement and reinforce each
other. Both photographs are also primarily experiential: they present episodes as they
were in life. At the rank of genre, while the text is a news report, its generic structure is
that of a narrative (e.g. Eggins 2004). The photographs represent two stages of that
narrative (the orientation and the complication): the orientation, Mariam’s case coming
to the doctor’s attention; and the complication, the doctor reporting his conclusions
and recommendations to Mariam’s father. The rest of the narrative is carried by the
linguistic text. The experiential information contained in each photograph is situated at
the rank of episode. The language related to these episodes is encoded in clause
complexes and conjunctively related clause complexes. What happens at the rank of
episode in the photograph aligns more or less with what happens in clause complexes
and in larger pieces of text.
32 There are parallels between what happens at the rank of figure in the photograph and
at clause rank in the story. In the linguistic processes unfold one at a time and one per
clause as the text develops. For example the doctor is the actor in a number of material
processes (verbs shaded in red in the text in Fig. 1b): he has been “working”, he “sets
out”, he “arrives”, he “begins”. He is also the receiver in verbal processes: people “bring”
him information the monitoring team “tell” him things and he “is” also well accepted.
Other participants in the text also engage in different processes. The key point here is
that, in the linguistic text, participants can only engage in one process per clause: one
process at a time, in a series of happenings which unfold. On the other hand, in a
photograph the same person or thing can be a participant in a number of processes
simultaneously. For instance, in the second photograph, Dr M’Bemba is simultaneously
engaged as the ‘doer’ in looking, standing, holding and probably speaking. He is also
the ‘done to’ in that he is being looked at and listened to. The key difference between
the visual action and linguistic action is that, in photographs, participants can be
engaged in multiple processes simultaneously, although the images are viewed in

particular ways, given the compositional and interpersonal choices as illustrated here.
In this regard, a key function of language is to explicitly order happenings in the world
as a logically connected series of events, as highlighted in this example. In the
photographs and in the section of the linguistic text analysed, however, Dr M’Bemba is
the main participant. He engages as the in more processes than the other participants
and he is the focus of the actions of other processes.
33 In summary the combinations of choices across metafunctions and across language
and photograph build and reinforce the position of Dr M’Bemba, and by extension
WHO as the central participant in the whole text. Linguistically, at the rank of genre,
the narrative is used as an illustration of WHO’s role in combatting Ebola. Dr M’Bemba,
as a representative of WHO, is highlighted experientially as the central participant in
that narrative. Likewise, the photographs embedded in the narrative highlight Dr
M’Bemba (and WHO) as the major participant. Interpersonally Dr M’Bemba is shown
as having status (as a doctor) and authority (as a representative of WHO). Textually he
is also highlighted through thematic choices in language and through compositional
choices in the photographs.
34 The discussion above has demonstrated that the principle of metafunctions can be
applied to sets of choices from different semiotic systems to show that, although the
choices are different, they realise complementary and compounding meanings. Choices
made from one set of semiotic resources reinforce metafunctionally aligned choices
made from a different set of semiotic resources, resulting in semantic expansions which
extend beyond those possible with either language or image alone. The resemiotisations
have the effect of, by using a specific example, highlighting the role of WHO in fighting
the Ebola epidemic.
4.2. Resemiotisation in an infographic

35 The same principle can be illustrated through discussion of resemiotisation in
another multisemiotic text. Fig. 2 is an infographic from the home page of the WHO
Ebola website. The infographic exemplifies resemiotisation between language and
graphics. The linguistic text and the graphics appear simultaneously on the website as
one multimodal text. The focus of the discussion is directed towards resemiotisation
through the textual and experiential metafunctions.
36 The infographic consists of five separate parts, each of which is concerned with a
different aspect of Ebola. The parts are separated from each other by bold horizontal
lines and by a straight blank space vertically down the centre. Despite being separated
they are also unified compositionally by having much the same general layout of a bold,
large heading in the upper left, smaller supporting text and prominent, stylised images
which use the same colours and basic shapes.
37 Textually, the position of the headings suggests a reading/viewing path based on
written English. These headings are thematic and serve as the topic for their respective
parts. The accompanying text presents new information related to its topic. Each image
resemiotises some of the information in the written text, reinforcing and highlighting
its function as salient new information. The combination of linguistic elements and
images in the composition of the individual parts and in all of the parts together creates
a visual cohesion which assists the reader/viewer in following the flow of information.
Figure 2
Resemiotisation in an infographic
38 Some of the features noted in the textual organisation of the infographic are also
echoed in its ideational organisation. Notably, participants in the main processes in the
language of the infographic are also encoded in the images. For example, in the top left
part, the text lists “headache”, “fever” and “vomiting” among the symptoms of Ebola.
The symptoms also share taxonomic lexical relations as co-hyponyms and as hyponyms
of “symptoms”. Three of these symptoms (“headache”, “fever” and “vomiting”) are
resemiotised as stylised, but quite congruent, images (see Fig. 3). In this way, entities in
the linguistic text (i.e. “headache”, “fever” and “vomiting”) are intersemiotically
translated into visual processes with human participants as well as a relational process
with a material object (thermometer). This intersemiosis creates a visual connection
from the abstract entities in the heading and written text to the concrete material world
of human life which is visually depicted in order to explain how the effects of the Ebola
virus can be identified.
Figure 3
Ebola symptoms resemiotised as images
39 The other parts in the larger infographic also resemiotise linguistic participants as
visual processes and participants. This creates a logico-semantic relationship where the
meaning of the images depends on language: that is, the written text in the infographic
is necessary to interpret the images. The text, on the other hand, would still make sense
without the images but its message would be much more abstract without the
accompanying images. However, unlike in the photographs, where the images depict
processes and participants in real life, the images in the graphics are highly stylised and
experientially pared down to the bare minimum required to be unambiguous for the
purpose of providing information to the reader/viewer.
4.3. Resemiotisation between different media

40 This example demonstrates how different semiotic resources have been chosen to
realise similar meanings in two different media using visual resources only. The focus
here is on the resemiotisation of experiential and interpersonal meaning in examples
are taken from different parts of the WHO Ebola website as they appeared on 22 June
2015.
41
Figure 4
Resemiotisation between different media – graphic and photograph

42 The stylised image of the three figures in Fig. 4 with their arms raised (left image)
appears in an infographic and the photograph (right image) is from the main visual
display of the website and also from a news story which is linked to the main visual
display. The photograph has been cropped to show just the three central figures. In
both images, the experiential content is similar in terms of the gender and age balance.
In the photograph the gender and relative ages of the participants is clear, because they
are shown as they appeared in real life. This option is not available to the designer of
the infographic so other options are chosen in terms of stylised dress and relative size.
Compositionally the figures in the photograph and in the infographic are similarly
placed, with the adult male in the centre, the female figure to his right and the smaller
male figure to his left, although the larger male figure is more foregrounded in the
infographic. The foregrounding emphasises the relatively larger size of the male figure.
43 Interpersonal meaning is also resemiotised across the two images. The figures in the
photograph are facing the viewer, as are the figures in the infographic. The gaze of the
participants in the photograph is directed at the viewer. While the viewer cannot see the
faces of the figures in the infographic, the white space under their heads indicates chins,
which would only be visible if they were facing the viewer. The figures in the
photograph are all smiling, showing positive affect. Happiness cannot be shown in the
same way in figures without faces so alternative choices are used. In the infographic the
figures appear to be jumping for joy: they have their arms upwards and outwards in
symbolic happy gestures. A further indicator of happiness can be seen in the blue cloud
shapes above the three figures in the infographic. The colour blue here symbolises blue
sky, a typical sign of positive affect, even though it is the clouds which are blue rather
than the sky. The combination of blue sky and positions of the arms realises a similar
interpersonal effect through different sets of choices.
Figure 5
Resemiotisation between different media – graphic and video
44 A similar intersemiotic translation can also be found in the connection between the
stylised image of disembodied hands washing in the infographic and the still frame of
the WHO video entitled “Hand hygiene in Ebola care facilities”, as featured under
“Latest videos” in the “News” section of the web page, which shows a woman washing
her hands (Fig. 5). Here the same action is replicated in two different media. The
graphic highlights the hands by showing them just as hands, with no body attached.
Blue coloured circles representing soap bubbles create a contrast with the solid black of
the hands. The simple graphic is minimalist with respect to the experiential information
required to convey the message. The video still frame, on the other hand, is realistic,
showing a woman washing her hands in real time. The translation between the two
depends on the purpose of what is being translated. For example, the graphic shows
only what is crucial to the act of hand-washing, hands and something to wash them
with, so the translation focuses only on those essential elements. The video, on the
other hand shows the whole process of hand-washing in moving images, with narration,
captions and background conversation, as discussed in the next example.
4.4. Resemiotisation in a video

45 Tracking intersemiotic relations in dynamic media, such as films and videos, for
example, which “incorporate both visual and acoustic modes of signification, as well as
different graphic sign systems” (O’Sullivan 2013, p. 6), is inherently more complex than
unpacking the intersemiotic relations between text and images. In order to facilitate the
transcription of multi-semiotic resources in audio-visual texts, Taylor (2003, 2013)
proposes a page-based multimodal transcription technique that involves inserting
images of still frames and meta‑textual descriptions into a table (with rows and
columns), so as to establish “where meaning was being created in a multimodal text…”
(Taylor 2013, p. 102). Another way of handling this complexity is using software such as
Multimodal Analysis Video (O’Halloran and Lim 2014; O’Halloran, Tan and Marissa
2015b), which includes facilities for importing and organizing video files; creating and
editing catalogues of system frameworks and system choices for video annotation;
storing and consolidating projects of analyses; annotating and analysing videos by
creating time-stamped annotations; visualizing combinations of multimodal choices;
and exporting data from the analyses to Excel spread sheets for further data processing
and visualisation. In this case, the analysis of the WHO video “Hand hygiene in Ebola
care facilities” performed with Multimodal Analysis Video software, aids in the
identification of intersemiotic relations and transformations that occur as information
is resemiotised across on-screen captions, voice-over narration, and the WHO official’s
internal dialogue. The systems used for the analysis of the video are displayed in Table
2.
46 Table 2 shows that, in dynamic texts, combinations of semiotic choices work together
to realize multiple meanings simultaneously (although one or the other metafunction
might predominate at a given time), resulting in constant shifts of meanings as the
video unfolds (see also Figs. 7 and 8). The analysis reveals, for instance, that textually,
the on-screen caption “Hand hygiene in Ebola care facilities” (Fig. 6, left) is the first
caption that viewers see when they watch the video. Indeed, it is similar in wording to
the title shown in the still-frame featured under “Latest videos” in the “News” section of
the Ebola webpage (see Fig. 5). Although probably added last in terms of post-
production processes, the initial caption functions like a title in an introductory phase
that provides the context for the whole video. This function also appears to be
manifested visually, as it is displayed in a much larger font-size than the rest of the on-
screen captions (see Fig. 6). The on-screen captions clearly fulfil different functions: the
first on-screen caption summarises the whole video, whereas the other on-screen
captions only resemiotise certain parts of the WHO official’s dialogue, as discussed
below.
Table 2
Semiotic
Metafunctional
Resource & System Description
Orientation
Rank
The experiential component of Register

Language: Processes,
Experiential describes the way language is used in
Participant
Meaning Register (Field) context, i.e. what the text is all about,
Roles
what is happening (Field).
Linguistically, verbal interactions fulfil four

Language: primary speech functions for exchanging
Interpersonal Speech
Discourse information (e.g. statements, questions),
Meaning Function
Semantics or goods and services (e.g. commands,
offers).
Textual Language: Depicts how much time (and space) is

Meaning Dialogic Space Participants allocated to the verbal interactions of
(spoken) individual participants.
Language: On-screen Depicts how much time (and space) is

Visual Text Captions allocated to on-screen captions.
(written)
Visual:
Textual Composition Organisation of visual information into
Cinematographic
Meaning (Phase, Shot) coherent units of meaning.
Resources
Camera
Distance
Realizes interpersonal relations such as
(Long shot,
social distance.
Medium shot,
Close shot)
Camera
Movement
Visual: (Stationary, Realizes interpersonal involvement by
Interpersonal
Cinematographic Pan, Tilt, channelling the viewer’s gaze.
Meaning
Resources Zoom-in,
Zoom-out)
Vertical
Viewing
Perspective Realizes interpersonal power relations
(High angle, through camera angle and perspective.
Eye-level,
Low angle)
Systems for analysing intersemiotic relations in dynamic videos
Figure 6
On-screen captions
47 The extra-diegetic voice-over narration [00:00:07 – 00:00:32], which immediately
follows the introductory phase, but precedes any internal dialogue, in turn summarises,
and hence resemiotises, the actions and events that are shown on screen, as well as
those that are yet about to unfold in the video, thus adding another contextual layer.
The voice-over narration, which is scripted and formal in terms of register, also
resemiotises certain aspects of the WHO official’s dialogue, which uses a more informal
lexis (see Fig. 7). For example, in the voice-over narration, participants are represented
in discourse as impersonalised and collectivised (“WHO teams”, “health care workers”;
Fig. 7 left). As such, they provide a contrast to the WHO official’s utterances, which are
rendered more interpersonally engaging, partly through the use of first and second
person plural pronouns (“we”, “you”; Fig. 7 centre). In this particular example, the
process of intersemiotic translation is not unidirectional, or bidirectional, but
multidirectional, as part of the WHO official’s dialog is again translated into a more
formal, institutional register in the on-screen captions, which coincide with some of the
WHO official’s utterances. In this case, the on-screen captions are all clauses in passive
voice (e.g. see van Leeuwen 2008) which omit the agent or doer of an action altogether
(Fig. 7 right). This has the effect that the experiential content is resemiotised in even
more abstract terms than in the voice-over narration, as the “deagentialised” actions
described in the on-screen captions are no longer identifiable with any particular
participant(s).
48 As illustrated in Fig. 7, these multiple resemiotisations across verbal and visual
resources, across time and space, thus frame the WHO official’s informal,
interpersonally engaging face-to-face dialogue within more authoritative, institutional
registers, with a heightened focus on experiential meaning, resulting in a change in
metafunctional orientation.
49 The overall meaning of a video, however, is achieved through the complex

co‑deployment of textual, experiential and interpersonal elements which are designed
to make some meanings more prominent, and which guide the viewer to arrive at
particular interpretations. In dynamic audio-visual texts, these interpretations are
likely to result from choices made in the deployment of cinematographic resources and
post-editing devices. Whilst these resources fulfil textual meaning in that they structure
the unfolding of the video, they simultaneously realise interpersonal meanings such as
Social Distance, Interpersonal Involvement,
Figure 7
Intersemiotic relations and translations across verbal and visual resources in a video
50 Power Relations and Subjectivity or Point of View, as conveyed by Camera Distance
(e.g. close shot, medium shot, long shot), Camera Angle (e.g. high angle, eye-level, low
angle), Camera Movement (e.g. stationary, pan, tilt, zoom-in, zoom-out), Vertical
Viewing Perspective (e.g. high angle, eye-level, low angle).
51 As illustrated in Fig. 8 (which shows a snapshot of the analysis in the Multimodal
Analysis Video GUI), the combination of multimodal semiotic resources, such as the
dialogic space accorded to participants and on-screen captions, and the shifts in
cinematographic choices that occur at the time the WHO official delivers her speech, all
work together so that the viewer focuses on the WHO official and what she has to say at
that moment, and the statements that are resemiotised in the form of on-screen
captions. For example, as the WHO official begins to address the “gaps” in the hand-
washing demonstration performed by the health care workers, the camera reduces the
social distance between the WHO official and the viewer by zooming in and changing
from a long shot to a medium shot. The placement of participants (dressed in dark
colours) to the left and right of the WHO official (clad in a white top) additionally work
to channel the viewer’s perspective to focus on the WHO official and by extension, the
on-screen captions. While she delivers her speech, the camera remains stationary in a
single Shot (with no editing cuts), and at eye-level, giving prominence to the points
made in the on-screen captions. The overall meaning created would have been different
if the camera had remained at long distance (as shown in the frame displayed at the
top-left in Fig. 8), or if it had executed a pan or tilt which, together with high or low
camera angles and changed depth perspectives, could have created a distancing or even
disorientating effect. As illustrated here and discussed in the previous examples,
intersemiotic translation can be modelled theoretically and analytically using the
concept of metafunction, which reveals which the nature of the shifts of meaning which
take place within and across multimodal texts.
Figure 8
Multi-semiotic combinations in a video
4.5. Resemiotisation in mathematics

52 One of the most sophisticated and exact forms of intersemiotic translation occurs in
mathematics which has specialised semiotic resources in the form of scientific
language, mathematical images (e.g. graphs and diagrams), and mathematical symbolic
notation to (re-)construct the material world (e.g. O’Halloran 2015a, 2015b). As
O’Halloran explains:
[…] mathematics is considered to be a specialized tool for thinking, specifically

designed to move beyond our everyday experience of the world to an abstract
semiotic realm for restructuring thought and reality. Natural language also
functions to organize and structure human experience on an abstract semiotic
plane (e.g. Halliday, 1978; Whorf, 1956) but, unlike mathematics, it lacks the
“meaning potential” (Halliday, 2003 [1985]) to effectively model and predict
events in the physical world. (O’Halloran, 2015b, p. 288)
53 Mathematics achieves its unique functionality by drawing upon the three semiotic
resources which work closely together to bridge textual representations (i.e. language
and mathematical symbolism) and visual representations (i.e. graphs and diagrams),
with a specific focus on experiential and logical meanings. Each semiotic resource has
its own functionality in this regard: language to contextualise the mathematics problem
and results; images to provide a perceptual account of the mathematical relations
where the parts are viewed in relation to the whole; and mathematical symbolism which
mathematical relations are configured and rearranged to solve problems. Significantly,
these three semiotic resources have systems of meaning which specifically designed to
fulfil these functions, while seamlessly integrating so that it is possible to shift between
each resource with ease. Moreover, mathematical symbolism has a specialised grammar
for encoding meaning economically and unambiguously (as do mathematical graphs
and diagrams) so that intersemiotic translations across language, image and the
symbolism are precise and exact in nature. This is achieved through sophisticated
compositional organisation of mathematical texts and backgrounding the interpersonal
metafunction (i.e. through consistent semiotic choices within a narrow semantic field)
so that the focus is experiential and logical meaning.
54 The “semantic circuit” across language, image and symbolism in mathematics

(O’Halloran 2005) is critical for achieving the semantic expansions that are required for
rewriting the physical world in mathematical terms. The semantic circuit permits the
meaning potential of three different resources to be accessed and, in addition, the
intersemiotic translations of semiotic choices across text, image and symbolism create
semantic expansions which extend beyond those possible for each resource. For
example, the three graphs from the WHO website in Fig. 9a depict the number of cases
of Ebola in Guinea, Liberia and Sierra Leone respectively. Upon mouse over, the
numbers of cases on the “Patient database” and “Situation report” for each day are
visible, as displayed on Fig. 9b. While the mathematical graphs are contextualised by
the surrounding language, as displayed in Fig. 9a, the focus of the following discussion
is the intersemiotic translation which takes place across mathematical relations (i.e. the
reported number of cases each day) and the visual representations of this numerical
data.
Figure 9a
The Ebola crisis: mathematical graphs and text.
Figure 9b
The Ebola crisis: mathematical graphs and text. (The situation in Guinea)
55 The number of Ebola cases recorded in the “Patient database: and the “Situation
report” are recorded on a daily basis. This results in a series of identifying mathematical
relations between time (x) and number of cases (y): for example, on June 21 2015 there
were 12 cases in both databases (i.e. Patient database and Situation report), as
displayed in Fig. 9b. The data has the form of a mathematical relation where x (date)
corresponds to y (number of cases): i.e. x ↔ y. This mathematical relation (x ↔ y) is
translated visually into a point on the graph in Fig. 9b. In this case, there is
intersemiotic translation between a mathematical relation x ↔ y (at the rank of clause)
to an entity, the point (at the rank of visual entity), as shown in Table 3 which displays
the systems for experiential meaning for language, image and mathematical symbolism.
From here, the series of visual entities (i.e. the points representing the mathematical
relation) form a visual relationship at the rank of episode, and the points are joined to
form a graph at the rank of work, as displayed in Table 3.
56 The example illustrates how intersemiotic translation involves shifts of meaning
where semiotic choices at one rank for one semiotic system are resemiotised at a
different rank in another semiotic system, with the result that the semiotic choice is re-
contextualised within a new semantic field so that meaning potential of the new
resource can be accessed, leading to further semiotic expansions. From there, other
semiotic translations take place, as for example, when the complete work of the graph is
semiotised as the word “graph”, in this case opening up the vast meaning potential of
language to resemiotise the Ebola situation further, as illustrated in Fig. 9a. Such
intersemiotic translations and resemiotisations form the basis for rewriting the
material world in mathematical and scientific terms, consequently changing the nature
of life on earth today.
57
Table 3
Experiential meaning: language, image and symbolic systems in mathematics
5. Conclusion
58 Intersemiotic translation is the basis of cultural communication through which
thought and reality are structured using a variety of semiotic resources which function
to construct our experience of the world and interpersonal relations through
resemiotisation processes. As illustrated above, systemic functional theory (SFT)
provides the necessary theoretical tools for modelling semiotic resources and analysing
the meaning arising from semiotic choices within and across multimodal texts. In this
case, semiotic resources are modelled as systems of meaning with various strands (i.e.
experiential, logical, interpersonal and textual meanings) from which choices are made
in multimodal texts and processes at different levels. In this way, it is possible to
conceptualise shifts of meaning across semiotic resources which are fundamentally
different in nature to reveal which meanings are retained and changed as a result of
resemiotisation. That is, systemic functional theory provides comprehensive
foundations for modelling intersemiotic translations theoretically and for
demonstrating the semantic expansions that occur as a result, as illustrated in the
examples discussed above.
59 However, even with this multi-level theoretical platform, modelling and analysing
intersemiotic translations within and across multimodal processes and texts and
tracking the resultant effects are complex and time-consuming tasks. For this reason
digital tools in the form of multimodal annotation software are employed, as discussed
in the examples above. Even so, intensive close multimodal analysis is still labour-
intensive and moreover it is difficult, and in some cases impossible, to track semantic
patterns within and across multimodal texts. For this reason, mathematical modelling
(e.g. clustering techniques and temporal logic) and visualisation techniques (e.g. state
transition diagrams) have been used in order to capture semantic patterns in
multimodal data which otherwise would not have been easily detected (e.g. O’Halloran,
Marissa and Tan 2014; O’Halloran, Tan and Marissa 2015a; Tan, Smith & O’Halloran
2015). These modelling and visualisation techniques have only been applied to a limited
number of multimodal texts, however. In this regard, the latest challenge is to combine
close multimodal analysis with data mining and information visualisation to analyse
large datasets of multimodal texts (e.g. O’Halloran, Tan, Pham, Bateman & Vande
Moere, 2016). Interdisciplinary collaboration is required to develop new digital
techniques and approaches to develop the science of intersemiotic translation for big
data analytics. Such a science has implications in a range of fields where there is a need
to understand patterns of human discourse (e.g. national security and defense,
terrorism studies, disaster management, healthcare) in a world where communication
has been radically transformed through advances in digital technology. In this respect,
semioticians have a major role to play with regards to modeling, tracking and
understanding intersemiotic translation as the basis for cultural communication.
Appendix
Websites
http://www.who.int/csr/disease/ebola/en/
http://multimodal-analysis.com/products/multimodal-analysis-image/
http://multimodal-analysis.com/products/multimodal-analysis-video/
http://www.who.int/features/2015/guinea-ebola-resources/en/
A , Daniella & Q , Joao (2009), “Towards a Model of Intersemiotic Translation”, The
International Journal of the Arts in Society, 4(4), pp. 203-210.
B , John & S , Karl-Heinrich (2012), Multimodal Film Analysis: How Films Mean,
London and New York: Routledge.
B , Michał (2015), “Multimodality, Translation and Comics”, Perspectives, 23:(1), pp. 22-
41, DOI: 10.1080/0907676X.2013.876057.
E , Umberto (1979), A Theory of Semiotics, Bloomington: Indiana University Press.
E , Suzanne (2004), An Introduction to Systemic Functional Linguistics, 2nd edition,
London and New York: Routledge.
F , Elizabeth (1984), The Folklore Text, Bloomington: Indiana University Press.
G , Henrik (1997), Subtitles, Translation and Idioms, Copenhagen: University of
Copenhagen.
H , Michael A.K. (1978), Language as Social Semiotic: The social Interpretation of
Language and Meaning, London: Edward Arnold.
H , Michael A.K. (2002 [1977]), “Text as Semantic Choice in Social Contexts”, in W
(ed.), Linguistic Studies of Text and Discourse: Volume 2 in the Collected Works of M.A.K.
Halliday, London and New York: Continuum, pp. 23-81.
H , Michael A.K. (2003 [1985]), “Systemic Background”, in W (ed.), On language
and Linguistics: the Collected Works of M.A.K. Halliday volume 3, London and New York:
Continuum, pp. 185-198.
H , Michael A.K. (2008), Complementarities in Language, Beijing: Commercial Press.
H , Michael A.K. & H , Ruqaiya (1985), Language, Context, and Text: Aspects of
Language in a Social-semiotic Perspective, Geelong, Victoria: Deakin University Press
[Republished by Oxford University Press 1989].
H , Michael A.K. & M , Christian M.I.M. (2014), Halliday’s Introduction to
Functional Grammar, 4th ed., revised by M (ed.), London and New York: Routledge.
I , Rick (2003), “Multimodality, Resemiotisation: Extending the Analysis of Discourse as
Multi-semiotic Practice”, Visual Communication, pp. 29-57. DOI:
10.1177/1470357203002001751.
J , Roman (1959), “On Linguistic Aspects of Translation”, in B , (ed.), On
Translation, Cambridge, MA: Harvard University Press, pp. 232-239.
J , Carey (2014), “An Introduction to Multimodality”, in J (ed.), The Routledge
Handbook of Multimodal Analysis, 2nd ed., London: Routledge, pp. 15-30.
J , Carey (ed., 2014), The Routledge Handbook of Multimodal Analysis, 2nd ed., London
and New York: Routledge.
J , Carey, B , Jeff & O’H , Kay L. (2016), Introducing Multimodality,
London: Routledge.
K , Evangelos (2015), “Semiotics of Translation: An Interdisciplinary Approach to
Translation”, in T (ed.), International Handbook of Semiotics, Dordrecht: Springer,
pp. 303-320.
K , Evangelos & Y , Charikleia (2014), “Intericonicity as Intersemiotic Translation in a
Globalized Culture”, in W & J (eds.), Our World: A Kaleidoscopic Semiotic Network,
Proceedings of the 11th World Congress of the IASS/AIS, 5-9 October 2012, Hohai University
Press, pp. 162-176.
K , Gunther (2009), “What is a Mode?”, in J (ed.), The Routledge Handbook of
Multimodal Analysis , London: Routledge, pp. 54-67.
K , Gunther & L , Theo (2001), Multimodal Discourse: The Modes and Media of
Contemporary Communication, London: Edward Arnold.
K , Gunther & L , Theo(2006), Reading Images: The Grammar of Visual Design,
2nd ed., London: Routledge.
L , Jay L. (1998), “Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text, in
M &V (eds.), Reading Science: Critical and Functional Perspectives on Discourses of
Science, London: Routledge, pp. 87-113.
M , James (1992), English Text: System and Structure, Amsterdam: Benjamins.
M , James (2002), “Meaning beyond the Clause: SFL Perspectives”, Annual Review of
Applied Linguistics, 22, pp. 52-74.
M , James R. & R , David (2007), Working with Discourse: Meaning beyond the Clause,
2nd ed., London: Continuum.
M , James R. & W , Peter R.R. (2005), The Language of Evaluation, Appraisal in
English, London and New York: Palgrave Macmillan.
M , Radan (2001), “Interpersonal Resources in Action”, Semiotica, 135(1/4), pp. 117-145.
M , Radan(2004), “Gestures that Co-concur with Speech as a Systematic Resource: The
Realization of Experiential Meanings in Indexes”, Social Semiotics, 14(2), pp. 193-213.
M , Christian M.I.M. (2009), “Multisemiotic and Context-based Register Typology:
Registerial Variation in the Complementarity of Semiotic Systems”, in V &M (eds.),
The World Told and the World Shown: Multisemiotic Issues, Hampshire: Palgrave Macmillan,
pp. 11-38.
O’H , Kay L. (2005), Mathematical Discourse: Language, Symbolism and Visual
Images, London and New York: Continuum.
O’H , Kay L. (2015a), “The language of learning mathematics: a multimodal perspective”,
The Journal of Mathematical Behaviour, 40 Part A, pp. 63-74.
O’H , Kay L. (2015b), “Mathematics as Multimodal Semiosis”, in D and D (eds.),
Mathematics, Substance, and Surmise, Berlin: Springer, pp. 287-303.
O’H , Kay L., & L -F , Victor (2014), “Systemic Functional Multimodal Discourse
Analysis”, in N & M (eds.), Texts, Images and Interactions: A Reader in
Multimodality, Berlin: Mouton de Gruyter, pp. 137-154.
O’H , Kay L., M K.L.E. & T , Sabine (2014), “Multimodal Analytics: Software and
Visualization Techniques for Analyzing and Interpreting Multimodal Data”, in C. J (ed.),
The Routledge Handbook of Multimodal analysis (2nd edition), London: Routledge, pp. 386-
396.
O’H , Kay L., T , Sabine & M K.L.E. (2015a), “Multimodal Semiosis and
Semiotics”, in W (ed.), The Bloomsbury Companion to M.A.K. Halliday, London:
Bloomsbury, pp. 386-411.
O’H , Kay L., T , Sabine & M K.L.E. (2015b), “Multimodal Analysis for Critical
Thinking”, Learning, Media and Technology, DOI: 10.1080/17439884.2016.1101003.
O’H , Kay L., T , Sabine, P , Duc-Son, B , John & V M , Andrew
(2016), “A digital Mixed Methods Research Design: Integrating Multimodal Analysis with Data
Mining and information visualization for big Data Analytics”, Journal of Mixed Methods
Research. DOI: 10.1177/1558689816651015.
O’T , Michael (2011), The Language of Displayed Art, 2nd ed., London and New York:
Routledge.
O’S , Carol (2013), “Introduction: Multimodality as Challenge and Resource for
Translation”, The Journal of Specialised Translation, 20, pp. 2-14.
P -G , Luis (2014), “Multimodality in Translation and Interpreting Studies”, in
B &P (eds.), A Companion to Translation Studies, Chichester: Wiley-Blackwell,
pp. 119-131.
R , Aline (2001), “Some Thoughts on the Study of Multimodal and Multimedia
Translation”, in G &G (eds.), (Multi) Media Translation: Concepts, Practices and
Research, Amsterdam: John Benjamins, pp. 13-22.
R , Terry (2007), “Intersemiotic Complementarity: A Framework for Multimodal Discourse
Analysis”, in R &B (eds.), New Directions in the Analysis of Multimodal Discourse,
T. Mahwah NJ: Lawrence Erlbaum Associates, pp. 63–110.
T , Sabine, S Bradley, A. & O’H , Kay L. (2015), “Online Leadership Discourse in
Higher Education: A Digital Multimodal Discourse Perspective”, Discourse & Communication,
9(5), pp. 559-584, DOI: 10.1177/1750481315600302.
T , Christopher J. (2003), “Multimodal Transcription in the Analysis, Translation and
Subtitling of Italian Films”, The Translator, 9(2), pp. 191-205, DOI:
10.1080/13556509.2003.10799153.
T , Christopher J. (2013), “Multimodality and Audiovisual Translation”, in G &
D (eds.), Handbook of Translation Studies, Volume 4, John Benjamins, pp. 98-104.
L , Theo (1999), Speech, Music, Sound, London: Macmillan.
L , Theo (2008), Discourse and Practice: New Tools for Critical Discourse Analysis,
New York: Oxford University Press.
W , Benjamin L. (1956), Language, Thought and Reality: Selected Writings of Benjamin Lee
Whorf, MA: MIT Press.
List of illustrations
Title Figure 1a
4
Caption News story ‘Ebola – a test too far for one little girl’ as displayed on the
WHO Ebola website
URL http://journals.openedition.org/signata/docannexe/image/1223/img-1.jpg
File image/jpeg, 196k
Title Figure 1b
Caption Intersemiotic translation between photograph and text as analysed in

Multimodal Analysis Image software
Title Figure 2
Caption Resemiotisation in an infographic
Title Figure 3
Caption Ebola symptoms resemiotised as images
Title Figure 4
Caption Resemiotisation between different media – graphic and photograph
Title Figure 5
Caption Resemiotisation between different media – graphic and video
Title Figure 6
Caption On-screen captions

Title Figure 7
Caption Intersemiotic relations and translations across verbal and visual

resources in a video
Title Figure 8
Caption Multi-semiotic combinations in a video
File image/jpeg, 2.6M
Title Figure 9a
Caption The Ebola crisis: mathematical graphs and text.
Title Figure 9b
Caption The Ebola crisis: mathematical graphs and text. (The situation in Guinea)
Title Table 3
Caption Experiential meaning: language, image and symbolic systems in

mathematics
References
Bibliographical reference
Kay L. O’Halloran, Sabine Tan and Peter Wignell, « Intersemiotic Translation as Resemiotisation:
A Multimodal Perspective », Signata, 7 | 2016, 199-229.
Electronic reference
Kay L. O’Halloran, Sabine Tan and Peter Wignell, « Intersemiotic Translation as Resemiotisation:
A Multimodal Perspective », Signata [Online], 7 | 2016, Online since 31 December 2017,
connection on 08 December 2019. URL : http://journals.openedition.org/signata/1223 ; DOI :
10.4000/signata.1223
This article is cited by

Constantinou, Maria. (2018) Charlie Hebdo’s controversial cartoons in question:
stances, translational narratives and identity construction from a cross-linguistic
perspective. Social Semiotics. DOI: 10.1080/10350330.2018.1521356
About the authors

Kay L. O’Halloran
Kay O’Halloran is Professor in the School of Education, Faculty of Humanities at Curtin
University. Her areas of research include multimodal analysis, social semiotics, mathematics
discourse, and the development of interactive digital media technologies and visualization
techniques for multimodal and sociocultural analytics.
Sabine Tan
Sabine Tan is a Research Fellow in the School of Education, Faculty of Humanities at Curtin
University. Her research interests include critical multimodal discourse analysis, social semiotics,
and visual communication. She is particularly interested in the application of multidisciplinary
perspectives within social semiotic theory to the analysis of institutional discourses involving
traditional and new media.
Peter Wignell
Peter Wignell is a Research Fellow in the School of Education, Faculty of Humanities at Curtin
University. Peter’s current research interests are in Systemic Functional Linguistics, especially in
its application to the analysis of multimodal texts. His research has also focused on the role of
language in the construction of specialised knowledge.
Copyright
Signata - PULg

2016 Intersemiotic Translation As Resemiotisation - A Multimodal Perspective

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2016 Intersemiotic Translation As Resemiotisation - A Multimodal Perspective

Uploaded by

Copyright:

Available Formats

12/8/2019 Intersemiotic Translation as Resemiotisation: A Multimodal Perspective

La traduction intersémiotique est considérée comme la base de la communication culturelle, par

is then resemiotised as an architectural design. This view of resemiotisation implies

3. A multimodal systemic functional

18 Semiotic resources are viewed as having a meaning potential which is described in

Ideational meaning which consists of experiential meaning: to structure

19 The meanings of multimodal processes and texts are characterised in terms of

[…] because it provides a common set of fundamental principles to compare

22 In SF-MDA, the focus on the metafunctional organisation of semiotic resources is

4. Examples of SF-MDA approach

Rank System Rank System

Ideation & Taxonomic relations

Processes; Participant Roles; and

Proportion in Relation to the

Compositional Vectors; Framing

Relative Placement of Episode;

Episode Position of the happenings,

Text and image systems

4.1. Intersemiotic translation between photograph

26 This example discusses intersemiotic translation between two photographs and

engaged in multiple processes simultaneously, although the images are viewed in

4.2. Resemiotisation in an infographic

Ebola symptoms resemiotised as images

4.3. Resemiotisation between different media

Resemiotisation between different media – graphic and photograph

Resemiotisation between different media – graphic and video

4.4. Resemiotisation in a video

The experiential component of Register

Linguistically, verbal interactions fulfil four

Textual Language: Depicts how much time (and space) is

Language: On-screen Depicts how much time (and space) is

Systems for analysing intersemiotic relations in dynamic videos

49 The overall meaning of a video, however, is achieved through the complex

Multi-semiotic combinations in a video

4.5. Resemiotisation in mathematics

[…] mathematics is considered to be a specialized tool for thinking, specifically

54 The “semantic circuit” across language, image and symbolism in mathematics

The Ebola crisis: mathematical graphs and text.

Experiential meaning: language, image and symbolic systems in mathematics

Caption Intersemiotic translation between photograph and text as analysed in

File image/jpeg, 332k

Caption Intersemiotic relations and translations across verbal and visual

Caption Experiential meaning: language, image and symbolic systems in

This article is cited by

About the authors

You might also like