Critical Discourse Studies

ISSN: 1740-5904 (Print) 1740-5912 (Online) Journal homepage: https://www.tandfonline.com/loi/rcds20

‘Doing critical discourse studies with

multimodality: from metafunctions to materiality’
by Per Ledin and David Machin

Kay O’Halloran, Peter Wignell & Sabine Tan

To cite this article: Kay O’Halloran, Peter Wignell & Sabine Tan (2018): ‘Doing critical discourse
studies with multimodality: from metafunctions to materiality’ by Per Ledin and David Machin,
Critical Discourse Studies, DOI: 10.1080/17405904.2018.1556173

To link to this article: https://doi.org/10.1080/17405904.2018.1556173

Published online: 14 Dec 2018.

Submit your article to this journal

Article views: 92

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rcds20
CRITICAL DISCOURSE STUDIES
https://doi.org/10.1080/17405904.2018.1556173

RESPONSE

‘Doing critical discourse studies with multimodality: from

metafunctions to materiality’ by Per Ledin and David Machin
Kay O’Halloran, Peter Wignell and Sabine Tan
School of Education, Faculty of Humanities, Curtin University, Bentley, Western Australia
ARTICLE HISTORY Received 21 November 2018; Accepted 3 December 2018

In their article, Ledin and Machin (2018) consider core concepts and models from systemic
functional (SF) approaches to multimodality in relation to how suitable they are for analys-
ing multimodal forms of communication and for answering questions in Critical Discourse
Studies (CDS). Ledin and Machin (2018) define CDS as
the critical investigation of diverse social phenomena, with language as its core focus, aiming
at challenging what is usually taken for granted, in order to draw out buried discourses which
support the interests of specific ideologies and dominant groups in society. Ledin and Machin
(2018)

discuss perceived problems with the notions of ‘text’ and ‘context’ in systemic theory and
suggest ways to deal with the ‘materiality’ of multimodal communication.
The focus on SF approaches to multimodality is timely, given that systemic functional
theory and the forms of multimodality built around it are ‘hugely inspiring’ and have
‘transformed the landscape of visual communication analysis’ (Ledin & Machin, 2018).
Beyond this, systemic functional theory provides the foundations for major theoretical,
methodological and analytical trends in multimodality (Tan, O’Halloran, & Wignell, 2019).
It is possible that these and other research efforts (including those in CDS) might,
through collaboration, lead to the establishment of the discipline of multimodality
(O’Halloran et al., 2019). In this context, Ledin and Machin’s (2018) concerns are under-
standable, given the research trajectory of multimodality where systemic theory is
expected to continue to play a major role for the reasons provided in this response.
Before addressing Ledin and Machin’s (2018) concerns, we clarify some issues of
terminology.

Terminology
Michael Halliday (1978) devised the term ‘social semiotic’ to describe language and other
semiotic resources (e.g. images, gestures, music, clothing, architecture and so forth) as
defining a culture: ‘Indeed, we can define a culture as a set of semiotic systems, as sets
of systems of meaning, all of which interrelate’ (Halliday & Hasan, 1985, p. 4). Halliday
developed systemic functional theory in order to formulate language as a social semiotic
system, resulting in systemic functional linguistics and systemic functional grammar.
Therefore, we use the following terms in this discussion:

CONTACT Kay O’Halloran kay.ohalloran@curtin.edu.au

© 2018 Informa UK Limited, trading as Taylor & Francis Group
2 K. O’HALLORAN ET AL.

Systemic functional (SF) theory (aka SFT): Theoretical concepts which can be applied to
language and other semiotic resources (e.g. metafunction, system, context) (e.g. Halli-
day, 1978, 2003 [1985]).
Systemic functional linguistics (SFL): Development of systemic functional (SF) theory for
language, based on the notion of language as a social semiotic resource (e.g.
Martin & Rose, 2007).
Systemic functional grammar (SFG): The part of systemic functional linguistics (SFL) which is
concerned with the grammatical description of language (e.g. Halliday & Matthiessen,
2014).

Multimodality as a ‘Grand Theory’

Ledin and Machin (2018) argue that ‘SFL based multimodality’ has become ‘a kind of grand
theory of all forms of communication’. We clarify the two terms ‘SFL based multimodality’
and ‘grand theory’ before investigating this claim. In the first case, ‘SFL based multimod-
ality’ suggests that the approach is based on SFL, rather than on SF theory (e.g. O’Halloran,
2007; O’Halloran & Lim, 2014; O’Halloran, Tan, & Wignell, 2019). Therefore, we use the term
‘SF based multimodality’ to capture how systemic functional (SF) theory is used to theorise
language and other semiotic resources in multimodal research. That is, SF theory provides
the foundations for systemic functional linguistics (SFL), systemic functional grammar
(SFG), and SF based multimodality.
Second, Mills (1959) used the term ‘grand theory’ to ‘refer to the form of highly abstract
theorising in which the formal organisation and arrangement of concepts takes priority over
understanding the social world’1 [emphasis added]. Schwandt (2007, p. 131) explains:
In the social sciences, grand theory refers to those efforts devoted to abstract, analytical
theory building. It stands in contrast to empiricist approaches that emphasise that knowl-
edge of society is best acquired by accumulating empirical generalisations through
improvements in methodology as well as scientific approaches that encourage the devel-
opment of lawlike propositions and axiomatic theory.
Halliday developed SF theory as a platform for describing how language is organised
and used to fulfil different functions in society. Therefore, SFL and SFG contain descriptions
of the grammatical and discourse systems through which the various functions are
realised using language in relation to the context. The abstractions are applied to
different instances of language use to investigate the meanings which are made and
the strategies which are used. The abstractions have proved extremely useful in edu-
cational linguistics, forensic linguistics, clinical linguistics, genre-based literacy pro-
grammes, translation studies, and descriptive work across languages and language
families (e.g. Martin, 2016).
Based on the definition above, SF theory is not a ‘grand theory’. On the contrary, SF
theory is designed to understand how language and other semiotic resources are organ-
ised and used in different contexts. In doing so, SF theory provides a unifying platform for
analysing the meanings arising from the interaction of language with other semiotic
resources in multimodal texts, interactions and events. The underlying principles (meta-
functions, systems, register, genre, realisation, stratification and constituency) can be
‘explicated and extended to other semiotic resources’ providing a platform for how
 CRITICAL DISCOURSE STUDIES 3

‘meaning arises through combinations of semiotic choices – that is, from semiotic
interactions within and across different resources rather than from individual system
choices – and how these meanings can be modelled, analysed and interpreted
(O’Halloran, Tan, & Wignell, 2019) (see also O’Halloran & Lim, 2014). As such, SF based mul-
timodality contains abstract concepts and models to describe how semiotic resources are
organised to create meaning and the strategies through which meanings are made in mul-
timodal texts. The approach provides a useful platform for analysing how multimodal dis-
courses are resemiotised and recontextualised as social practices unfold (Iedema, 2003;
O’Halloran, Tan, & Wignell, 2016; van Leeuwen, 2008).
Different semiotic resources have a different underlying organisation through which
meaning is made (i.e. language is different from image, sculpture, architecture) so SF
based multimodality does not involve a direct mapping of language systems on to
image and other semiotic resources (e.g. see systemic frameworks in O’Toole, 2011).
The clause is a basic semantic unit in SFL and SFG because it is the fundamental seman-
tic/grammatical unit in language and the three metafunctions are realised at that level.
Due to the sequential nature of language production a clause-based grammar is the
most suitable approach to grammatical analysis. Many other semiotic artefacts, such as
images, do not share this property. They are perceived, initially at least, as a whole (O’Hal-
loran, Tan, & Wignell, 2016; O’Halloran, Tan, & Wignell, 2019). Multimodal texts which, for
instance, contain language and images share both of these properties. Therefore, multi-
modal discourse analysts are typically, as a starting point, interested in the text itself
(however ‘text’ is defined) and recontextualizations of that text. The aim of multimodal
analysis is therefore to develop abstractions which describe the underlying organisation
of semiotic resources, the processes through which meaning is made through the inter-
action of semiotic choices, the unfolding of those semiotic interactions throughout the
whole text in relation to the context, and changes in those configurations over time.

The notion of context

The two levels of context in SF theory are ‘the context of situation’ and ‘the context of
culture’, derived from Malinowski (1923). The context of situation is concerned with the
context in which the communication take places (e.g. university lecture versus conversa-
tion at a bar). The term refers as much to the semiotic context as it does to the physical
context. For example, if we consider a piece of abstract academic prose, the physical
context in which the text was written (or typed) is (generally) not as important as the situ-
ation of the text as part of a broader discourse. The context of situation is construed by the
semiotic selections which construct the material unfolding of any text, interaction and
event. The field (the content), tenor (the social relations between participants in terms
of social distance, power and affect) and the mode (the nature of communication e.g.
face to face, written, digital and the semiotic distance between a text and its physical
context) are realised through the systems for ideational, interpersonal and textual meta-
functions respectively.
The context of culture preconditions the semiotic configurations which are made in any
situation, as Ledin and Machin (2018) point out. In SF based multimodality, the context of
culture accounts for the traditions of semiotic use that impact on the kinds of semiotic
resources and choices found in contexts such as photojournalism (Caple, 2013). The
4 K. O’HALLORAN ET AL.

context of culture impacts on the ‘canons of use’ which account for different configur-
ations found in any instance of semiosis. The challenge for SF based multimodality is to
document those semiotic configurations and track changes over time. SF based multimod-
ality can potentially account for such semiotic happenings, as discussed below.

Texts as meaningful wholes

SF based multimodality is concerned with the analysis of semiotic artefacts and happen-
ings. There are several levels of description for capturing the various dimensions of the
analysis: the material plane: the actual materiality of the semiotic artefact and/or happen-
ing; the meaning arising from the semiotic interactions; the register configuration (i.e. in
terms of field, tenor and mode); and the generic description of the whole text, interaction
or event (e.g. webpage, video, news report, shampoo bottle, news photograph). It is poss-
ible to provide descriptions of two shampoo bottles and a press photograph, both are con-
ceptualised as cultural objects. However, the strength of SF based multimodality lies in the
use of abstractions which may be applied to different instances so that patterns and trends
can be discerned and mapped. This relates to the notion of ‘system’ and ‘choice’ which, as
Ledin and Machin (2018) point out, is a key element of SF based multimodality. Such
abstract concepts make it possible to systematically analyse and compare multimodal
semiotic acts beyond consideration of representative examples.

System and choices

Mathematics and linguistics both developed abstract and generalised knowledge which is
applied in different real-life contexts for understanding the physical world and the human
world of language respectively. These abstract structures hold, regardless of the context in
which they are applied and used. In a similar manner, SF based multimodality is concerned
with the development of abstract context-based frameworks which can be used for ana-
lysing semiotic phenomena in order to understand and map the human world (O’Halloran
et al., 2019). In other words, SF multimodality is a scientific view of multimodal semiosis,
with the required abstractions for analysing semiotic phenomena in relation to context.
As Ledin and Machin (2018) point out, system networks are often used to model semio-
tic systems, as exemplified by Kress and van Leeuwen (2006 [1996]). However, other
methods have been explored, for example, van Leeuwen (1999, 2009) uses parametric
systems to map the experiential meaning potential of material resources. Djonov (2013)
explains:
A parametric system presents those physical qualities, or affordances, of a given resource
which people have taken up in communication. These qualities are always gradable and
together define the meaning potential of a given signifier such as a certain type of voice,
texture or colour. A particular voice for example can be described as a combination of
degrees of each of several parameters: tension; roughness; breathiness; loudness; vibrato;
and nasality.

More recently, digital approaches are being developed to handle the complexity of multi-
modal analysis, including mixed methods approaches which are designed to map and
understand patterns of human communication today (O’Halloran, Tan, & Wignell, 2016;
O’Halloran, Tan, Wignell, Bateman et al., 2016; O’Halloran et al., 2018; Tan, O’Halloran,
 CRITICAL DISCOURSE STUDIES 5

Wignell, Chai, & Lange, 2018). These new methodologies involve the integration of multi-
modal theory with computational techniques and visualisation (O’Halloran et al., 2019),
heralding a new era in SF based multimodality aimed at addressing key problems in
society today, in particular those arising from the rapid advance of digital technology.
Bateman, McDonald, Hiippala, Couto-Vale, and Costetchi (2019) provide an overview of
computational approaches to SF based multimodal research. In this new wave of multimo-
dal research, the system network is being replaced with computational techniques such as
neural networks and deep learning. In other words, SF based multimodality has moved
well beyond the original formulations presented in early works such as Kress and van
Leeuwen (2006 [1996]).

Semantics, context and stratification

The notion of stratification was developed in SF theory to show how meanings were
realised contextually through wordings and sound – and in the case of SF based multi-
modality, the material basis of the semiotic artefact (see O’Toole, 2011). That is, ‘language
[and other resources] is a semiotic system: not in the sense of a system of signs, but a sys-
temic resource for meaning – what I have often called a meaning potential’ (Halliday, 2003
[1985], pp. 192–193). Halliday explains:
The organising concept is not structure, but system. Since language is a semiotic potential, the
description of language is a description of choice. The various levels, or strata, of the semiotic
‘code’ are interrelated networks of options. The constituent structure is the realisation of these
options, and hence plays a derivative role in the overall interpretation. (2003 [1985], p. 193)

In this view, structural features of language have their origin in the semantics; that is, it has
some function in the expression of meaning. Moreover, ‘the different types of structure
tend to express different kinds of meaning, as embodied in the metafunctional hypothesis
(Halliday, 2003 [1985], pp. 193–194). Although Halliday believed that ‘the heart of
language is the abstract level of coding that is the lexicogrammar’, it ‘is not a closed, deter-
minate system’ (Halliday, 2003 [1985], p. 194). Rather, language and other semiotic
resources are viewed as open, dynamic systems which are subject to change over time
(Lemke, 2000). Thus, as Ledin and Machin (2018) explain, SF based multimodality necess-
arily takes into account micro-level and macro-level analyses to account for the context of
the situation and the context of culture in order to account for social practices (van
Leeuwen, 2008). A major aim of SF based multimodality is to capture the nature of
these configurations and the interests which are served (e.g. Wignell, O’Halloran, & Tan,
2018).

Instantiation or canons of use

SF based multimodality views multimodal semiosis as a constituent of social practices,
realised as semiotic configurations (material practices) in a culture. The goal of SF based
multimodality is to map these configurations (i.e. the canons of use) over space and
time. This involves moving beyond the analysis of some representative samples to the
analysis of large datasets of multimodal texts and happenings. In order for this to
happen, however, abstract context-based frameworks are required, such as those being
6 K. O’HALLORAN ET AL.

developed in SF based multimodality. Such work (O’Halloran et al., 2018) offers an exciting
opportunity for collaboration with CDS and scholars from other disciplines.

Concluding comments
Halliday envisaged variations of SF theory:
The value of a theory lies in the use that can be made of it … Since there are so many tasks for
which one needs a theory of language, any particular theory is likely to be, or very quickly to
become, a family of theories … This is why there is no orthodox or ‘received’ version of sys-
temic theory. (Halliday, 2003 [1985], p. 192)

Halliday (2003 [1985], p. 192) explains that systemic theory is like a language itself; that is
‘a system whose stability lies in its variation’. This does not mean that we cannot charac-
terise SF theory like Ledin and Machin (2018) have attempted to do, but the character-
isation has to incorporate this feature of variability. In much the same way, there are
variations in SF based multimodality which we have tried to capture in this response.
Indeed, the goal is to ‘engage with other kinds of theories and models’ (Ledin &
Machin, 2018) in order to develop robust approaches aimed at addressing issues in
the age of digital technology.

Note
1. https://en.wikipedia.org/wiki/Grand_theory.

Disclosure statement
No potential conflict of interest was reported by the authors.

Notes on contributors
Professor Kay O’Halloran is Professor and leader of the Multimodal Analysis Group in the School of
Education at Curtin University, Western Australia. She has a background in mathematics, multimodal
discourse analysis, and linguistics. She has extensive experience in establishing and leading interdis-
ciplinary research teams to develop and make widely available new digital tools and techniques for
analysing images, videos, and 360 videos. Kay is currently developing mixed methods approaches
that combine multimodal analysis, data mining and visualisation for big data analytics in areas
which include online extremism, political rhetoric and 360 video. Author postal address: School of
Education, Faculty of Humanities, Curtin University, Kent Street, Bentley Western Australia 6102.
Email: kay.ohalloran@curtin.edu.au
Dr Peter Wignell is a Senior Research Fellow and a member of the Multimodal Analysis Group in the
School of Education at Curtin University, Western Australia. He has an academic background in Sys-
temic Functional Linguistics, discourse analysis and multimodal analysis. His research has been both
theoretical and applied. For example, his work on the role of language in the construction of special-
ised knowledge systems has informed literacy theory and pedagogy. Recent research has led to pub-
lication in a diverse range of applications such as brand semiotics, performance studies and
translation. Peter’s current research focus is on the application of a Systemic Functional Multimodal
analysis approach to violent extremist discourse and political discourse. Author postal address:
School of Education, Faculty of Humanities, Curtin University, Kent Street, Bentley Western Australia
6102. Email: peter.wignell@curtin.edu.au
 CRITICAL DISCOURSE STUDIES 7

Dr Sabine Tan is a Senior Research Fellow and a member of the Multimodal Analysis Group in the
School of Education at Curtin University, Western Australia. She has a background in critical multi-
modal discourse analysis, social semiotics, and visual communication. She has applied multidisciplin-
ary perspectives for the analysis of institutional discourses involving traditional and new media. She
has worked on interdisciplinary projects involving the development of interactive software for the
multimodal analysis of images, videos and 360 videos for research and educational purposes.
Sabine’s current work involves developing multimodal approaches to big data analytics in the
fields of online terrorism, political discourse, and 360 video. Author postal address: School of Edu-
cation, Faculty of Humanities, Curtin University, Kent Street, Bentley Western Australia 6102. Email:
sabine.tan@curtin.edu.au

References
Bateman, J., McDonald, D., Hiippala, T., Couto-Vale, D., & Costetchi, E. (2019). Systemic functional lin-
guistics and computation: New directions, new challenges. In G. Thompson, W. L. Bowcher, L.
Fontaine, & D. Schönthal (Eds.), The Cambridge Handbook of systemic functional linguistics.
Cambridge: Cambridge University Press.
Caple, H. (2013). Photojournalism: A social semiotic approach. Basingstoke: Palgrave Macmillan.
Djonov, E. (2013). Semiotic profile: Theo van Leeuwen. SemiotiX: A Global Information
Magazine. Retreived from https://semioticon.com/semiotix/2013/12/semiotic-profile-theo-
van-leeuwen/
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and
meaning. London: Edward Arnold.
Halliday, M. A. K. (2003 [1985]). Systemic Background. In J. Webster (Ed.), On language and linguistics:
The collected works of M. A. K. Halliday (Vol. 3, pp. 185–198). London: Continuum.
Halliday, M. A. K., & Hasan, R. (1985). Language, context and text: Aspects of language in a social-semio-
tic perspective. Victoria: Deakin University Press.
Halliday, M. A. K., & Matthiessen, C. M. I. M. (2014). Halliday’s introduction to functional grammar (4th
ed., revised by C. M. I. M. Matthiessen). London: Routledge.
Iedema, R. (2003). Multimodality, resemiotization: Extending the analysis of discourse as a multise-
miotic practice. Visual Communication, 2(1), 29–57.
Kress, G., & van Leeuwen, T. (2006 [1996]). Reading images: The grammar of visual Design (2nd ed.).
London: Routledge.
Ledin, P., & Machin, D. (2018). Doing critical discourse studies with multimodality: From metafunc-
tions to materiality. Critical Discourse Studies, doi:10.1080/17405904.2018.1468789
Lemke, J. L. (2000). Opening up closure: Semiotics across scales. In J. Chandler & G. v. d. Vijver (Eds.),
Closure: Emergent organizations and their dynamics (Vol. 901, pp. 100–111). New York: New York
Academy of Science Press.
Malinowski, B. (1923). The problem of meaning in primitive languages. In C. K. Ogden & I. A. Richards
(Eds.), The meaning of meaning (Vol. Supplement 1, pp. 296–336). London: Kegan Paul
(International Library of Psychology, Philosophy and Scientific Method).
Martin, J. R. (2016). Meaning matters: A short history of systemic functional linguistics. WORD, 62(1),
35–58. doi:10.1080/00437956.2016.1141939
Martin, J. R., & Rose, D. (2007). Working with discourse: Meaning beyond the clause (2nd ed.). London:
Continuum.
Mills, C. W. (1959). The sociological imagination. Oxford: Oxford University Press.
O’Halloran, K. L. (2007). Systemic functional multimodal discourse analysis (SF-MDA) approach to
mathematics, grammar and literacy. In A. McCabe, M. O’Donnell, & R. Whittaker (Eds.), Advances
in language and education (pp. 75–100). London: Continuum.
O’Halloran, K. L., & Lim, F. V. (2014). Systemic functional multimodal discourse analysis. In S. Norris, &
C. Maier (Eds.), Texts, images and interactions: A Reader in multimodality (pp. 137–154). Berlin:
Mouton de Gruyter.
O’Halloran, K. L., Tan, S., Pham, D.-S., Bateman, J., & Vande Moere, A. (2018). A digital mixed methods
research design: Integrating multimodal analysis with data mining and information visualization
8 K. O’HALLORAN ET AL.

for big data analytics. Journal of MIxed Methods Research, 12(1), 11–30. doi:10.1177/
1558689816651015
O’Halloran, K. L., Tan, S., & Wignell, P. (2016). Intersemiotic translation as resemiotization: A multimo-
dal perspectives. Signata 7(1), 199–229. Special Issue on Translating: Signs, Texts, Practices.
O’Halloran, K. L., Tan, S., & Wignell, P. (2019). SFL and multimodal discourse analysis. In G. Thompson,
W. L. Bowcher, L. Fontaine, J. Y. Liang, & D. Schönthal (Eds.), The Cambridge Handbook of systemic
functional linguistics. Cambridge: Cambridge University Press.
O’Halloran, K. L., Tan, S., Wignell, P., Bateman, J., Pham, D.-S., Grossman, M., & Vande Moere, A. (2016).
Interpreting text and image relations in violent extremist discourse: A mixed methods approach
for big data analytics. Terrorism and Political Violence. doi:10.1080/09546553.2016.1233871
O’Halloran, K. L., Tan, S., Wignell, P., Wang, R., Chai, K., & Lange, R. (2019). Multimodality: A new dis-
cipline. In J. Pflaeging, C. -I. Tseng, J. Wildfeuer, J. A. Bateman, & O. Seizov (Eds.), Multimodality:
Towards a new discipline. Berlin: De Gruyter.
O’Toole, M. (2011). The language of displayed art (2nd ed.). London: Routledge.
Schwandt, T. A. (2007). Grand theory. The SAGE dictionary of qualitative inquiry. Thousand Oaks, CA:
Sage.
Tan, S., O’Halloran, K. L., & Wignell, P. (2019). Multimodality. In A. D. Fina, & A. Georgakopoulou (Eds.),
Handbook of discourse studies. Cambridge: Cambridge University Press.
Tan, S., O’Halloran, K. L., Wignell, P., Chai, K., & Lange, R. (2018). A multimodal mixed methods
approach for examining recontextualisation patterns of violent extremist images in online
media. Discourse, Context and Media, 21, 18–35. doi:10.1016/j.dcm.2017.11.004
van Leeuwen, T. (1999). Speech, music, sound. London: Macmillan.
van Leeuwen, T. (2008). Discourse and practice: New tools for critical discourse analysis. Oxford: Oxford
Univerity Press.
van Leeuwen, T. (2009). Parametric systems: The case of voice quality. In C. Jewitt (Ed.), The Routledge
Handbook of multimodal analysis (pp. 68–77). London: Routlege.
Wignell, P., O’Halloran, K. L., & Tan, S. (2018). Semiotic space invasion: The case of Donald Trump’s US
presidential campaign. Semiotica. doi:10.1515/sem-2017-0109