ADVANCED

APPLIED
LINGUISTICS

Multimodality for ELT Instructor: Prof. Dr. Siusana Kweldju

Multimodality in ELT

Actually we have long used mutimodal dimension in ELT instruction, but with
new digital technologies multimodality comes explicitly in its totality, instead
of the presentation of texts and pictures which complement each other, or
audiovisual materials using CD Rom, as the third stage of CALL. The teacher is
a cognitive guide who provides support to the learner’s cognitive processing
using CALL.

The concept of multimodality is based on the use of sensory modalities by

which humans receive information, such as tactile, visual, and auditory.
Sometimes the terms multimodality and multimedia are used interchangeably
by laypersons, but the term multimedia is the favored term in non-academic
field or industry for production and distribution, and a hybrid between industry
and higher education, while multimodality is used within rhetoric and
composition field, as contrasted to Multimedia. In other words, multimodality is
the way an idea is expressed or perceived, or the interpretation and regeneration
of information presented in different media. Multimodality is the manner of
using gestures, language, etc. To perform an action; it is a means to persuasion,
while multimedia is the technological form or the medium of presentation. In
education, multimedia has been used for more effective learning and
understanding compared to presentation solely in one format. Thus,
multimodality can be defined as (a) an approach/(b) a field of inquiry/ (c) a field
of research that understands representation and communication as relying on a
multiplicity of modes, all of which have been socially developed as resources to
make meaning. Modes such as gesture, sound, image, colour, or layout, for
examples, are conceived as sets of organized resources that societies have
developed.

In our daily life of digital multimedia today, communication is accomplished by

using a number of meaning-making modes; that is, the combination of written
texts, pictures, moving images and sound mediated through digital devices such
as computers and mobile phones. This trend gives a new light to the concept of
text. Text today is multimodal. This is the reason that multimodal literacies are
important to develop in educational setting, and it has also been discovered that
the use of multimodality in classroom activities helps students learn concepts
more easily, as human beings have separate channels to process visual and
auditory information.

Multimodality promotes enough noticing which will eventually promote the

basic degree of retention in learners’ cognitive structures, assuring a certain
stability for future automatic use. The presentation of verbal text and visual
images, redundantly even, enables repeated exposure that learners need to a
new item, and to compensate for the lack of formal repetitive work in the
foreign language classroom

Multimodality in Reading Instruction

Bao (2017) has applied multimodality in teaching English reading and attempts
to answer the question whether the application of multimodality to teaching
English reading effective. He discovered that multimodal teaching has been
demonstrated to be effective in activating classroom atmosphere, inspiring
students’ motivation to read after class and building up their confidence in
learning English, especially English reading.

According to the top-down model, reading is a process of psycholinguistic

guessing game (Goodman, 1996) for constructing meaning in response to text
and that reading requires interactive use of graph-phonic, syntactic and semantic
cues to construct meaning. Readers are required to take advantage of the
intelligence and experience to predict and comprehend the text in order to get
the general idea. However, according to Rumelhart (1977) the interactive
reading model is the most effective process of reading in which the bottom-up
model and the top-down model work interactiely, which means readers can use
a top-down method to predict the probable meaning and then move on to the
bottom-up method to check whether it is really the author’s intention.

Multimodality, which is nearly always present in communication, is the mixture

of several semiotic signs used in discourse, within a socio-cultural domain
causes a semiotic activity, especially the use of modes of meaning other than
linguistic, including visual meanings (images, page layouts, screen formats);
audio meanings (music, sound effects); gestural meanings (body language,
sensuality);spatial meanings (the meanings of environmental spaces,
architectural spaces)etc. In language teaching and learning, the selection of
different multimodalities will bring out different effect in classroom and
teachers should take it into consideration.
Godhe and Magnusson’s Report from Sweden

According to Godhe and Magnusson (2017) as communication becomes

multimodal the content of the subject has also been expanded when it
incorporates movies, blogs and digital techniques for producing communication
in the curricula. ). Also, more changes take place in society that needs to be
taken into account, including an increasing flexibility in the labor market,
globalization, individualization, and increasingly multicultural societies.
However, the revisions that are currently being made in the national Swedish
curricula in order to strengthen digital competences in different subjects largely
denote digital competence as concerning the use of digital tools and do not
address changes in content to any great extent. Also, the concept of
multimodality is not clearly stated, and replaced by the partial concept of the
configuration of text and illustration.

Today’s teachers need to have new competence to meet the need of a large
proportion of the meaning-offerings that the students come into contact with on
a daily. The use of digital devices also shapes a new mindset (Lankshear and
Knobel (2008) of being more collaborative and participatory, because subject
traditions have affected both teachers’ and students’ perception for the content
of the subject, expecially how and to what degree digital tools are incorporated
into instruction. Knowledge is considered to be collective and distributed
rather than centered on individual expertise. Furthermore, a new mindset
means that technology is used to do something different, as compared to an old
mindset, where well-known things are done but with more, or other,
technology. Formal learning in Swedish school is still dominated by the reading
and writing of traditional texts, whereas informal learning, often connected to
activities outside of school, encourages young people to develop other skills and
competences in meaning-making which involves producing and consuming
multimodal and multimedial texts (Kress, 2010). This change cannot be
ignored by formal education; it is a challenge for formal education to address, in
order to provide adequate education for the future. Changing this mindset
should also change the curriculum, as classroom practices are governed by
rules, including curricula and syllabi, and how the subject is conceptualized in
the curricula, and the assessment. In fact, Hjukse (2010) has pointed out the
difficulties in assessing texts containing several modes and stresses the need for
teachers to address these issues.
For them, multiliteracies are not only multimodal, but also the multimedial
aspects of communication. In this context, they prefer to use the term meaning-
offering, instead of meaning making. The idea of meaning offering is connected
both to the intended and the perceived meaning (Selander & Kress, 2010).
In their argument, Godhe and Magnusson state their questions:
(a) What does it mean to be able to express meaning in contemporary
societies?
(b) What competences will be needed in the future to be an active citizen?
(c) Which subjects in school should be responsible for developing the
students’ wider literacy competences?
(d) How can new media technology be incorporated in curricula and syllabi?

The Theoretical Framework

of the Current Education and Teaching

The meaning of education and teaching in the media landscape of contemporary

societies can be discussed based on two pespectives: multimodal and socio-
cultural:

Multimodal Meaning-making is possible in different modes and media in a

perspective non-hierarchic, ecological way. All modes (audio, visual,
gestural, tactile, spatial, verbal spoken, verbal written) have full
meaning potential which means that a meaning-offering can
convey ideational, interpersonal and textual meaning in its own
right. This non-hierarchic approach to understand meaning-
making can be used for discussions of how and to what extent
different modes are used and recognized as learning in formal
education both in, for example, teachers’ actual planning and in
assessment.
Sociocultura In a socio-cultural perspective, learning is social, situated and
l perspective mediated (Säljö, 2005). With a multimodal approach, the
mediation applies to the meaning potential of all modes, i.e.
meaning is made, distributed, interpreted and remade through
many communicational resources (cf. Jewitt, 2008, Kress
& Van Leeuwen, 2001).

Multimodality in Bao’s Research

Bao’s multimodality design was adopted from the multiliteracies model by the
New London Group (NLG), a group of ten researchers, educators, and
visionaries, in 1994 in New London, New Hampshire, USA. His reason to adopt
the model was because, according to him, the model offered a more complete
system of modality elements. The design elements are classified into five
categories, as follows:
No. Categories of Subcategories
Designing
Multimodality
1 Visual design 1. images,
elements (for 2. color,
drawing the 3. fonts,
viewers’ 4. bar graphs,
attention) 5. texts,
6. icons symbols, and
7. geographical location
2 Audio design 1. background music,
elements (for 2. listening materials
making the 3. (volume),
reading process 4. (pitch),
more colorful) 5. (pacing), and
6. (rhythmic)

3 Linguistic 1. vocabulary design,

design elements 2. the design of collocation,
3. lexicalization,
4. word meaning,
5. the designs of information structures and coherence
6. relations,
7. modality and transitivity design, and
8. the design of features of intonation, stress, rhythm, accent,
9. etc.
4 Gestural design 1. behavior,
elements 2. gestures,
3. feelings,
4. affect,
5. kinesics and
6. proxemics.
5 Spatial design 1. design of classroom setting, and
elements 2. the settings of the multimodal products.