GRAMMAR IN

MULTIMODAL DISCOURSE
LECTURER
Nurul Afiyattena, M.Pd.

STKIP La Tansa Mashiro

 Definition of Multimodal Discourse
• Discourse
Discourses are socially constructed knowledges of (some aspect of) reality. By 'socially constructed, we mean that
they have been developed in specific social contexts, and in ways which are appropriate to the interests of social actors
in these contexts, whether these are very broad contexts or not (a particular family), explicitly institutionalized contexts
(newspapers) or not (dinner- table conversations), and so on.
Discourses are socially situated forms of knowledge about (aspects of) reality. This includes knowledge of the events
constituting that reality (who is involved, what takes place, where and when it takes place, and so on) as well as a set of
related evaluations, purposes, interpretations and legitimations.
• Multimodal
Multimodal is one branch of the study of Systemic Functional Linguistics (SFL) developed by Kress and Van
Leeuwen in Reading Images (2006). This multimodal study applies all interactions, both verbal and visual
interactions. The term multimodality views communication and representation as more than language. It places
more attention on other communication modes, such as the visual mode of images, gesture, gaze, posture, color,
typography, or composition (Kress & Leeuwen, 2006). Kress and Leeuwen certainly offer grammar, which is a
structure for the study of visual modes. The three Meta-functions of visual grammar are called as symbolic, interactive,
and compositional.
The social semiotic approach to image analysis is based on Halliday’s approach to language as a social semiotic.
Language is not seen in this approach from the ideational meaning: the definition of the circumstance, operation, or state
of affairs, but it is seen as one of the semiotic structures representing a culture. These Meta-functions are the
ideational ,interpersonal, and textual function (Halliday & Matthiessen, 2004).
• Multimodal discourse is defined as the use of several semiotic
modes in the design of a semiotic product or event, together with
the particular way in which these modes are combined they may
for instance reinforce each other (say the same thing in different
ways), fulfil complementary roles, or be hierarchically ordered, as
in action films, where action is dominant, with music adding a
touch of emotive colour and sync sound a touch of realistic
presence (Kress G. Van Leeuwen, T, 1996, 2001 Batesman et al.,
2002)
• It is clear that a mode (gesture, colour, taste) may be quite fully
articulated and yet not have a correspondingly articulated set of
labels in language, spoken or written
Establishing the right modality
• Naturalistic modality,
Naturalistic modality produced
with sound, images, audio-visual
texts or the combination of
all these.
• Abstract modality Abstract
modality with regard to images
• Sensory modality
Sensory modality based on the
sensory pleasure or displeasure
represented by an audio-visual
text