Multimodal literacy and learning in school

subject areas: A functional semiotic perspective

on textbook design and reading to learn.  

Professor Len Unsworth

Australian Catholic University, Sydney

Professor Len Unsworth

Faculty of Arts and Education
Australian Catholic University,
Sydney AUSTRALIA
Len.Unsworth@acu.edu.au
An agenda for discussion:
 Research on students’ reading comprehension of image-
language relations in national literacy assessments.
 A metafunctional framework for considering image-
language relations in textbook design – a systemic
functional semiotic approach.
 Research on image-language relations in textbooks for
teaching English as a Foreign Language (TEFL) in Vietnam.
 Image selection in text book design: Examples of implicit
influences on student learning in history and science.
 Conclusion: The essential role of images in reading to
learn in all curriculum areas and the importance of
semiotic analyses in informing textbook design.
 Results of an ARC study1 indicating the
relative difficulty of different types of
image/language relations in the NSW BST.
Unsworth, L., & Chan, E. (2008). Assessing integrative reading of images and text in
group reading comprehension tests. Curriculum Perspectives, 28(3), 71-76.

Unsworth, L. (2013). Interfacing comprehension of image-language interaction in

state-wide reading texts and semiotic accounts of image-language relations. In C.
Gouveia & M. Alexandre (Eds.), Languages, Metalanguages, Modalities, Cultures:
Functional and socio-discoursive perspectives. (pp. 177-198). Lisbon: Books on
Demand/Instituto de Linguística Teórica e Computacional (ILTEC)

Unsworth, L. (2014). Multimodal Reading Comprehension: Curriculum

Expectations and Large-scale Literacy Testing Practices Pedagogies: An
International Journal, 9, 26-44.
1
An Australian Research Council (ARC) Linkage Grant in conjunction with the Educational Measurement and Schools Assessment
Directorate of the New South Wales (NSW) Department of Education and Training (DET), 2006-2008, entitled New dimensions of
group literacy tests for schools: Multimodal reading comprehension in conventional and computer-based format.
Equivalence of meaning in image and text

91%
Correct How many legs do
hatchlings have?
Complementarity – augmentation of
meaning in image and text
Which element of nature is represented by the Serpent shown in the painting?
Complementarity – augmentation of
meaning in image and text
Which element of nature is represented by the Serpent shown in the painting?

13%
correct
A metafunctional framework for
considering image-language
relations in textbook design – a
systemic functional semiotic
approach.
3 simultaneous types of meaning in images

Representational (experiential)
representing participants, events, circumstances and concepts

Interactive (interpersonal)
viewer-image relations – social distance, power, involvement

Compositional (textual)
layout of image and text – emphasis etc
 Genre relations 173 and 179

A typical environmental science text book.

Martin, J. R., & Rose, D. (2008). Genre relations: Mapping culture. London: Equinox Pub.
 Note the complete interconnectedness of image and language in
constructing representational meaning. This design facilitates learning.

Martin, J. R., & Rose, D. (2008). Genre relations: Mapping culture. London: Equinox Pub.
The image and its correspondingly numbered caption text seem
to emphasise different causes of the death of aquatic animals
3 simultaneous types of meaning in images

Representational (experiential)
representing participants, events, circumstances and concepts

Interactive (interpersonal)
viewer-image relations – social distance, power, involvement

Compositional (textual)
layout of image and text – emphasis etc
Aboriginals (sic)
More daunting than the climate was
the History: “Going West”
“inhabit”
but passive Transactional action
construction and reaction -
purposeful activity

Settlers - Actors Non-transactional

(Doers) in material reaction
(action) processes Stillness, isolation
and inactivity - close
Sight of smithy is to being a
‘thematized’ - conceptual image -
Aboriginals ‘typical Aborigine’
“watching” in
non-thematic,
non-finite, The only example of
clause Aboriginals as Actors
in effective material
Clauses constructs
Coburn, P. (1982) Going West. Sydney: Bay Books them as acquiescent
Van Leeuwen, T. (2001). Semiotics and iconography. In T. VanLeeuwen & C. Jewitt (Eds.),
Handbook of visual analysis (pp. 92-118). London.

Note how the

portrayal of the
women presents
them as “other” to
the mainstream
Dutch community

Van Leeuwen p.93

3 simultaneous types of meaning in images

Representational (experiential)
representing participants, events, circumstances and concepts

Interactive (interpersonal)
viewer-image relations – social distance, power, involvement

Compositional (textual)
layout of image and text – emphasis etc
 Greenhouse effect
explanation in
diagram and caption
only
Main text
relates to
image at the
top of the
page and not
the one
immediately
above
Masters of the Night Sky (Woodside, 1993).
Starting the Lesson Plan Seeing with sound
Microbats release a special sequence of highly
structured sounds and, when the sound
strikes an object or insect, it reflects back to the bat as
an echo. The bat then decodes
the echo like a secret message, because only the bat
that sent the signal has all the
information needed to make sense of the information.
What is the relationship between these two texts?
What does this suggest about how the texts might
be used in a lesson?
How might the image and caption be used in
relation to the texts?
Echolocation in microbats enables them to take a ‘sound picture’ of their environment,
take measurements of each obstacle, and navigate in their habitat. They can use sound
to locate insect prey and even identify the kind of insect from its texture, size and
hardness. The bat can also calculate the speed of the insect compared with its own flight
speed, so that it can take aim and swoop the insect up.
Each species of bat has its own sound ‘signature’ and every young bat must learn its call
and how to use it. Echolocation signals are mostly so high in pitch that scientists can
only tell which bats are flying by using special listening devices for eavesdropping.
Note:
•the repetition of ‘information’ in the main text;
•the technical word ‘sonar’ appears in the caption only

The bat then decodes the echo like a secret message,

because only the bat that sent the signal has all the
information needed to make sense of the information.

A long-eared bat sends out a sonar signal (dotted

lines). And the echo returns from its target prey
(solid lines).
 Research on image-language
relations in textbooks for
teaching English as a Foreign
Language (TEFL) in Vietnam.
 The role of images in Vietnamese textbooks
for the teaching of English as a Foreign
Language
Since an ever increasing range of texts in paper and electronic media are including images as
a means of conveying meaning independently and in interaction with language, developing
literacy as a part of language learning can no longer be considered to be concerned with
words alone.
In some textbooks images were principally resources for arousing motivation, invoking
students’ background knowledge of a topic or as or as visual cues to accessing the meaning
of vocabulary in texts.
In another kind of textbook images were included within the reading materials, and learning
experiences involved students in deriving meaning from the images and from the
relationship between the images and the language, which served as much richer  language
learning resources.
This kind of integration of image and language reflects the multimodal nature of a large
proportion of authentic texts in English and hence, for all language learners, whether in
contexts of first language learning (L1), English as a second language (ESL) or English as a
foreign language (EFL), learning to construct meaning from the interaction of image and
language is a crucial aspect of language competence.
Data analysed in a study of the role of images
in secondary school textbooks for teaching
English as a foreign language (TEFL) in
Vietnam
 Represe
ntationa
y lm te s an eaning
i n d ica
a ge s ir e ct
e se im a ny d
of t h at o r
n a ny l t hre d
ng i e nta p i c te
nothi environm nimals de actional image
f a
kind o to the two vectors represent walking
t
threa

reactional process
animal looking

conceptual analytic image

Trees constituting the rainforest
 re o f a ny
y the n atu m a ls
e c if e a n i
u age to sp e nt of th
th e lang da n g e rm
in e n
nothing ntal threat or
vi ronm e
en

a n d d o
an s w ers
p os s i b le n .
n of r vati o
or m u l atio o f c o n se
e i n t h ef e i ss u es
ve n o rol ts w i th th
t ure s ha e s t ud en
the pic tly engage th
ec
not dir

These images simply provide background and do not contribute directly

and independently to communicating a need for conservation of habitat
and wildlife.
Since the main issue in the text seems to be the impact of animal
habitat destruction, one wonders why a conceptual image of an
apparently pristine, intact rainforest is depicted.

Alternative Images
From an interpersonal or interactive perspective the depicted animals
are in distant view, their frontal plane is not parallel to that of the
viewer and they are ‘observe’ images rather than ‘contact’ images since
the animals gaze is not directly at the viewer.
If the purpose of the images were to engage the readers’ emotional
response and empathy for the animals then close up, contact images
with their frontal planes parallel to the viewer would have been more
effective choices.
Compositionally the arrangement of the images seems quite
dysfunctional. The circular framing of the monkey image and its
positioning bring a certain salience to this image in relation to the other
two, and yet, it is not at all clear why this image would be salient.
 g a nd l e a rn i n g.
to e nh a n c e teachin
age s strate gically
Selecting im
 Advanced Textbook
Tiếng Anh 10 nâng cao
 Th
e im
ho
w t age
na
tur his o of th
al e ccu e d
ven rre est
t, s d. It roye
e uch m df
th i
as ght ores
o f af we t d
a e
g n. o res ll be oes
im luti o t fi t no
he o l re h e t in
t p . resu dic
out ter lt o ate
a b wa
i ng ts fa
oth gges
i s n t su
ere tha
Th ach
be f th e de ad e le ph an t contains no
The im a ge o
aching.
clue to connect this with po
 Similarly the image of the forest
destruction can be matched
with the ‘solution’ without
engaging the students in any
only the label conveys the actual language learning
meaning of pollution. relevant to the issues of
students can answer conservation
correctly if they know the
the word ‘beach’ and match
this with the picture

To match the image of the

dead elephant with the
‘solution’, all the students
need to do is to match the the
word ‘wildlife’ with the same
word on the image – they do
not need to know what
‘wildlife’ means.
 ll l on g -di sta n ce
e , t he i m a ge s are a
e rs o n al pe rs p ectiv ge s , s o the
te r p e i m a
From an in n o hu m a n s in a n y of th
re a d er.
h e re a re f ro m t h e
views and t is to c re ate re m o te ness
rso na l e ff e ct
interpe

Compositionally,
the solutions are
background of th more salient bec
e print, the star- ause of the whit
and the location fl ash in which the e
of this in the cen language appear
tre of the compo s
sition.

On the elephant image the yellow

background makes the pink text
almost unreadable.
In the case of the advanced textbook,
there are no images to accompany the
feature reading text on the Nam Cat
Tien National Park (p.127). The
inclusion of one or two images with
this article would approximate the
normal presentation of such a piece in
another form of publication and would
also create opportunities for enhanced
language learning. For example, the
article mentions that the national park
has seven different habitat types but
does not indicate what they are. This
could be done by including a
conceptual classificational image
illustrating the seven habitats and a
map locating where these occurred.
Such images would then facilitate
discussion relating text information, for
example, about peacocks dancing, to
habitat type and location.
In the case of the advanced textbook,
there are no images to accompany the
feature reading text on the Nam Cat
Tien National Park (p.127). The
inclusion of one or two images with
this article would approximate the
normal presentation of such a piece in
another form of publication and would
also create opportunities for enhanced
language learning. For example, the
article mentions that the national park
has seven different habitat types but
does not indicate what they are. This
could be done by including a
conceptual classificational image
illustrating the seven habitats and a
map locating where these occurred.
Such images would then facilitate
discussion relating text information, for
example, about peacocks dancing, to
habitat type and location.
Supplementary Textbook
Solutions
(Upper-Intermediate Student’s Book)
Our Planet (pp 81-89)
Representationally,
the chimney is a
narrative image,
which is actional as
the emissions are The image of the wind
flowing out into turbines is conceptual
Ththe
is im
atmosphere. sinc age since there is no action.
e it s distinc The image illustrates the
is de ugge tion
picti sts t is sig parts of the turbine and
whil ng a h a n i
e th e pollu t the ch ficatower nt and that a very
wind ting imne
turb proc ylarge number of these
ine i e ss
s no make up the power
t.
generation plant.
However,
representing
the wind
turbines in this
way means that
the sound of
the rotating
blades cannot
be shown and
also the
cropped image
cannot show
the vast area of
land required
for the very
large number of
towers.
Here the images are not being used to simply match language items.
The images with the language of the task description provoke learning
of language relations  beyond those listed in the textbook and do so in
the context of learning about pollution and conservation issues.
For example, the wind turbine is never named in the book.
The superordinate term ‘renewable energy’ is used in the task
description and the naming of the image provides just one subordinate
of this superordinate term.
Class discussion can add other subordinates such as ‘hydro power’ and
‘solar power’ -hence the images provoke building taxonomic relations
in language extending the students’ language about this topic.
Similar potential exists with the coal-fired chimney as being one
subordinate of ‘fossil fuel energy’.
It is the contrastive nature of the images that provokes also the
building of the language of comparison of the advantages and
disadvantages of the various forms of energy production. There is no
comprehensive visual provocation of this comparison but sufficient is
depicted visually to give impetus to the discussion.
Representationally this is an image of an adolescent
holding what appears to be candles or torches and is
located in a street or lane. From an interpersonal or
interactive perspective, the image is shown from a high
angle, with the frontal plane of the person aligned in
parallel with the frontal plane of the viewer and with
the person making contact with the reader by looking
directly at the reader. The representation of a human is
important in visually linking the discussion of the
destruction of the natural environment and the need
for conservation to people, which does not happen in
the standard and advanced textbooks. The
interpersonal aspects of the image are significant in that
they construct a pseudo  interpersonal interaction
between the represented adolescent and the reader (it
is almost as if the reader is relating to the participant in
the image as s/he would if the image were a person) –
the gaze of the eyes of the person in the image directly
towards the reader makes contact and engages the
student interpersonally. The high angle of the image is
important in positioning the represented adolescent as
not having power and suggesting his vulnerability.  
Surprisingly, in the
standard textbook the
‘Mass Media’ topic
contained no images at all
Advanced Text Book
 im age s.
te d in the
s d ep ic
e w hat wa
a s to n am
o nl y task w
The
On the one occasion
when images were
included with a text to be
read by students, the
images were not
necessary to comprehend
the text and were not at
all involved in the
learning task for the
students
Supplementary Textbook
Solutions
(Upper-Intermediate Student’s Book)
Communication (pp 71 – 80)
In the supplementary
textbook there were
multiple images and in
many cases these
images were integrally
related to the relevant
language segments and
the learning tasks
required the students to
pay close attention to
the meanings
constructed by the
images and their
relationship to the
language.
In the supplementary text there are more than
twice the number of images than in the standard or
advanced text and they are distributed functionally
throughout the unit as part of the various text
resources across the unit. Several of the images
involve humans and some of these make direct eye
contact with the reader so that the student reader
is positioned as an interactant and is not positioned
remotely from the images through distance,
observe, oblique angle views. The images in this
text are not simply to provide some background
context or opportunities for matching vocabulary
with pictures and they are not peripheral to the
learning tasks for students.. These images are
active in provoking language activity and language
development through the meanings that they
communicate, which are complementary to the
meanings constructed in the language.
 Implications for textbook design
The teaching of English as a foreign language needs to take account of the nature
of English language education and the expectation of learners of English where
this is the official language. This means that TEFL pedagogic practice and TEFL
textbooks need to incorporate learning experiences that prepare those learning
English as a foreign language to be able to negotiate the meanings made by
images and image-language interaction in English texts. We have shown that
existing TEFL textbooks such as Solutions are productively oriented to supporting
such learning experiences. While the other textbooks we have examined are
clearly not currently designed to support the inclusion of images as an integral
aspect of textual meaning, we have suggested some ways in which more strategic
choices and use of images could be adopted as part of an improved textbook
design process in order for textbooks to support bringing TEFL practices into
alignment with the multimodal nature of English language learning in countries
where English is the official language. On the basis of what it has been possible to
illustrate from the modest initial study outlined in this paper, it is clear that the
improvement of textbook design would benefit from more extensive studies of
this kind.
Image selection in text book design:
Examples of implicit influences on
student learning in history and
science.
 Time Visuals
Research in education and educational psychology shows
that even adolescents experience difficulties in handling
chronological order and in understanding and representing
the duration of historical periods (Carretero et al., 1991:
35; Stow and Haydn, 2000; Wood, 1995).
Derewianka, B., & Coffin, C. (2008). Time visuals in history textbooks:Some pedagogic
issues. In L. Unsworth (Ed.), Multimodal Semiotics: Functional Analysis in Contexts of
Education (pp. 187-200). London: Continuum.
Pupils’ chronological understanding can be accelerated
through clearly targeted teaching and learning strategies.
According to Thornton and Vukelich (1988),
the more explicitly and systematically teachers address
time concepts the more likely are students to develop an
understanding of its complexity.

To what extent and in what ways do time visuals help

(teachers) develop students’ chronological understanding?

Derewianka, B., & Coffin, C. (2008). Time visuals in history textbooks:Some pedagogic
issues. In L. Unsworth (Ed.), Multimodal Semiotics: Functional Analysis in Contexts of
Education (pp. 187-200). London: Continuum.
Functions of Time Visuals

Sequencing time
Segmenting time
Setting in time
Duration
Phasing It is important to distinguish the
different types of time visuals and
their respective functions in the
text. Then it is important to teach
students quite explicitly how to
interpret these visuals.
Sequencing
time
Segmenting
time
Setting in time
Phasing
Theoretical Diagrams: The cause of the seasons
 The frame of reference is the northern hemisphere, although this
is not made explicit. The bottom half of the diagram means that
there is more light in the southern Hemisphere when it is Winter in
the northern Hemisphere.
 8

What is not made clear here is why the earth appears to be titled in
different directions on its axis in the two diagrams.
 These
diagrams
could be read
as if the earth
had a vertical
orbit around
the sun.
 Whereas in this diagram it
seems clearer that the earth has
a horizontal orbit around the
sun. What makes the difference
is the 3D effect and perspective
created by the light and the
variation in the thickness of the
arrow lines.
The same is
true for this
diagram.
Compare the
perception of
orbit in these
diagrams.
Implications for textbook design and multiliteracies education

• The design of textbooks can influence and kinds of learning

that are developed .
• Images and language interact to construct knowledge in
textbooks.
• Images can elaborate or complement the meanings conveyed
through language.
• Meanings are sometimes not constructed by language or image
alone but at the intersection of image and language.
• Students need to be taught how to reading language and
images interactively.
• Textbook designers and teachers need to understand how
images make meanings and how images interact with language
to make meanings – they need functional semiotic knowledge to
optimize textbook design.
http://www.australiancurriculum.edu.au/
Questions
and
discussion