A taxonomy of the discourse relations between

Information words14(3),
Design Journal and 207–230
visuals 207
© 2006 John Benjamins Publishing Company

Kenneth Kong
A taxonomy of the discourse relations
between words and visual

Keywords: multimodality, visual analysis, word-image

interface, systemic-functional linguistics, rhetorical
structure

A document is mainly composed of words and images, ‘What is the use of a book’, thought Alice, ‘without
but the complex relationship that binds these two pictures and conversation?’
completely different semiotic resources is usually taken Alice’s Adventures in Wonderland (Lewis Carroll)
for granted as transparent. The simple relations between
‘Of course the abstract idea must be occasionally
words and images – ‘anchorage’ and ‘relay’, identified by
explained – paraphrased, as it were – by the aid of
Barthes almost 30 years ago – are unable to deal with the
pictures; but discreetly, cum grano Salis’
complexity of their bond, made even more complex by
(Arthur Schopenhauer)
current printing and computer technology. This paper
aims to identify the potential relations that bind texts and
images together by arguing for a multilevel description
of their logico-semantic relationships. The multiple,
evaluative and metaphorical functions of the relations
will also be discussed. The data generated from the
proposed framework can form an empirical corpus for
quantitative analysis. Examples from a variety of sources
will be used as examples to show how the framework can
be operationalized.
208 Kenneth Kong
Introduction
Schopenhauer’s cautionary statement about the use of
pictures is rarely heeded these days, particularly because
the use of visuals has almost been stretched to its limit
through current technology and the popularity of the
Internet. Visuals are seen as essential to the communica-
tion of messages, at least from the perspective of web-
based communication. Of course, visuals are not new;
they were used long before the invention of words in any
language. But it has been through the Internet that the
use of visuals has reached a new stage of development in
which our past concepts and notions of communication
seem outdated or even irrelevant. It is a mistake, however,
to assume that visual language has replaced or will
replace verbal language as the prime means of commu-
nication. The history of verbal languages has proved
their importance and adaptability, despite threats posed
by other semiotic means, including visuals. While it is a
prime concern to explore the many possibilities of visuals
as semiotic tools in new media, it is equally important
to understand how visuals can work with other semiotic
elements, particularly verbal elements, in communicating
messages. As Barthes (1978, p. 195) argues:
the two structures (words and images) of the message
each occupy their own defined spaces, these being
contiguous but not ‘homogenized’, as they are for
example in the rebus which fuses words and images in a
single line of reading.
Although there has been an increasing amount of work
done on the use of visuals in a general sense (cf. Kress
and van Leeuwen, 1996; Oyama, 1999; van Leeuwen &
Jewitt, 2000), there has been a lack of systematic atten-
tion paid to the links between visual and linguistic
elements. Scholars have advocated that visual literacy
should not be seen as an obstacle to written literacy;
instead, they are complementary to each other (Kress
and van Leeuwen, 1996; Still, 2001; Stroupe, 2000;
Walker, 2001). Nevertheless, recent studies of verbal-
visual connections have been slanted towards linguistics
or visual communication, without fully integrating the
insights of both, and there remains an unsolved question
of how the two important codes connect and relate to
each other.
To trace the study of verbal-visual connections,
we need to revisit Roland Barthes’ Elements of Semiol-
ogy (1967), which argues that verbal text is the central
anchorage of human meanings and perceptions.
Barthes identifies the relationship between verbal and
visual codes along two dimensions – elaboration and
extension. Verbal codes can elaborate or extend visual
elements. Elaboration can come in two forms – either
language coming first and visuals coming afterwards
to become what is traditionally labelled ‘illustration’, or
visuals coming first and language afterwards, which is
known as ‘anchorage’. A verbal code can also extend or
add new information to the visual code, which Barthes
labels ‘relay’.
Barthes’ notions of ‘anchorage’ and ‘relay’ are
important because they touch on the heart of the issue,
i.e. how the two important semiotic tools converge.
However, Barthes presupposes linearity and does not
allow the reality that any code, whether visual or verbal,
can anchor or relay the other (van Leeuwen, 2005). This
is not surprising because Barthes reveals that his focus is
on images by suggesting a historical reversal in which ‘the
image no longer illustrates the words; it is now the words
which, structurally, are parasitic on the image’ (Barthes,
1978, p.204). In his terms, images are polysemous
– unstable and subject to interpretation – and words are
used to fix the ambiguous meaning of images.
The three relations (‘anchorage’, ‘relay’ and the
traditional illustrative function of visuals to verbal
language) are not sufficient to deal with the complex
relationship between words and images, particularly
in web-based texts. Although there has been a
revolutionary shift from the prime focus on written

language to the mixed mode of written and visual
language, it is not to assume the complex phenomenon
can be understood by the existing theories of language
or visuals alone. As Kress (1998, p. 65) points out, ‘we
seem to have a new code of writing and image [author’s
italics], in which information is carried differentially
by the two modes… simpler syntax does not mean that
the text – the verbal and visual elements together – is
less complex’. In fact, Bonsiepe made a similar point
more than 30 years back by arguing that a visual/verbal
rhetorical figure is a combination of two types of signs
whose effectiveness in communication depends on the
tension between their semantic characteristics. The signs
no longer add up, but rather operate in cumulative recip-
rocal relations. (1966, p. 171)
While it is possible to borrow insights from
linguistics and visual design, it is too risky to take these
for granted and apply them to either mode without
careful consideration. While visuals have taken up some
of the functions that written language used to perform,
whereas written language is mainly used for reporting
and narrating (Kress, 1998), the ‘display’ function of
visuals needs to be explored, especially in relation to the
narrating and reporting function of words. The most
important issue in the research of multimodal docu-
ments is whether they can communicate meanings that
traditional documents cannot (see Kaltenbacher, 2004).
In other words, we need a more systematic and sophisti-
cated network that can allow practitioners and analysts
to understand the increasingly complicated verbal-visual
connections.
Word-image connections
Visuals have been under close examination in a diverse
discipline of what is traditionally known as semiotics
that is ‘concerned with everything that can be taken as
A taxonomy of the discourse relations between words and visuals 209
a sign’ (Eco, 1976, p. 6). A sign can include an image,
gesture, sound and of course a word. One of the most
important concepts in semiotics is the distinction of
signifier (the sign itself) and signified (the referent),
(Saussure, 1974), which implies that signs are arbitrary
entities that vary from culture to culture. Another
important distinction is between iconic and symbolic
signs (Peirce, 1931-58). Iconic signs are the direct repre-
sentations of the referents whereas symbolic signs make
use of another image to make reference to the referent.
Iconic signs are less subject to convention but may have
more constraints in terms of use; on the other hand,
symbolic signs are more subject to convention but have
fewer constraints in terms of use and manipulation. One
of the most salient manipulative functions of signs is
to persuade people to take action, or, in brief, signs can
have a persuasive function. Although rhetoric has been
mainly directed towards the study of words, it has been
argued that visuals can perform similar persuasive func-
tions. The focus on the rhetorical/persuasive dimension
of signs has been labeled ‘visual rhetoric’ (Bonsiepe,
1966; Marcus, 1979). For example, Marcus (1979) argued
that many classical rhetorical devices (such as devices
building in climax and devices of intensification) can
be applied to images. Verbal and visual signs cannot
be isolated when studying their persuasive functions,
because, as Bonsiepe (1966) noted decades ago, there is a
strong correlation of words and images in (for example)
advertisements. This correlation has intensified in the
age of the Internet.
As mentioned earlier, recent studies of the connec-
tions between words and visuals have focused on the
supporting functions of language for visuals and vice
versa, without fully considering the interactive interplay
between the two. A good example of studies that focus
on the supportive functions of visuals to body text
would be Pegg’s (2002) classification of images into
ancillary, correlative or substantive types:
210 Kenneth Kong
[Ancillary images are] usually placed in the
neighbourhood of relevant text (except in the case
of title-page illustrations, front-pieces, and thematic
colophons). The exact relationship of the image to the
text however is left for the reader to determine. (p. 170)
Correlative illustrations are usually associated with
technical and scientific discourse and are characterised
by keying illustrations to the text in a variety of ways.
(p. 172)
In […] substantive (illustrations), there is no need for
the reader to build bridges between image and text, as
with ancillary illustrations, nor for a writer to construct
elaborate and misunderstanding-prone bridges out of
callout lines, numbering, or labels, as with correlative
illustrations. If words (or numbers) used with
substantive illustrations are laid out two-dimensionally
on a page so as to display structurally the relationships
between them, then text and image are one. (p. 174)
By focusing on how images can ‘illustrate’ the verbal
content, Pegg’s study can only capture the degree of
integration between words and images. While this is
an important index of connection, it does not explain
the semantic relations that bind the two. In the same
vein, Eckkrammer (2004) identified four relations in
word-image connections: transmedial relationships
(such as photos in novels), multimedial discourse (such
as book illustrations), mixed discourse (such as comic)
and syncretic discourse (such as visual poetry). Again
with a focus on the degree of connections, this does
not touch on how the connections are made possible.
Similarly, Royce’s study (2002) of the intersemiotic
systems of words and visuals in academic texts can
only partly identify the semantic relationships between
words and images. Drawing on Halliday and Hasan’s
ideas of lexical cohesion (1976), Royce argued that
visual and linguistic elements can be realized through
similarity relations (such as synonyms in verbal codes),
opposition relations (such as antonymy in verbal codes),
class:sub-class relations (such as hyponymy in verbal
codes) and expectancy relations (such as collocation
in verbal codes). While Royce is able to identify some
of the important word-image relations, the four types
of cohesion are not adequate in capturing the complex
relationship. Indeed, the four ‘cohesive devices’
identified by Halliday and Hasan are superficial links
that connect words and clauses in verbal texts. There
are more ‘subtle’ or ‘external’ links that combine units
in a text, which are known as coherence in discourse
analysis. For example, relations such as ‘explanation’,
‘sequence’ and ‘cause’ cannot be adequately dealt with by
cohesive devices even though they may contribute to the
overall coherence of a text.
Through a review of 24 previous studies, Marsh and
White (2003) could identify 46 relations between image
and text, which they then classified according to their
degree of integration or closeness as (a) functions that
have little relation to the text, such as those that decorate
or elicit emotion, (b) functions that have a close relation
to text, such as those that reiterate or exemplify, and (c)
functions that go beyond the text, such as those that
interpret, develop and contrast. Although this taxonomy
was based on extensive review, a large number of labels
are either overlapping or difficult to apply in analysis.
For example, relations such as ‘control’, ‘relate’, ‘sample’,
‘motivate’ and ‘translate’ are ambiguous terms that need
to be elaborated and re-defined. However, Marsh and
White highlight the importance of close integration
between words and images. An image merely placed
beside text cannot make the connection clear; what
is needed is a framework that can help analysts or
document designers to understand the complex
relationship between the two.
Amongst recent studies, Horn’s idea of ‘visual
language’ (1998) is closer to providing an understand-
ing of this relationship. Although visual language
can be analyzed from linguistic perspectives, it ‘has

distinct properties that make it different from natural
languages of words and from purely artistic languages.
It has a more complex syntax and requires more
diverse and complex analysis’ (p. 13). Horn classifies the
relations between words and images along three main
dimensions: semantic categories, classical rhetorical
devices and temporal relations. Semantic categories
refer to the linking of two modes through their potential
meanings and include relations such as substitution,
disambiguation, labelling, example, reinforcement and
completion. Classical rhetorical devices are very similar
to the cohesive devices of Halliday and Hasan (1976) and
include synecdoche (part-whole relation), metonymy
(association relation) and metaphor (suggestion of anal-
ogy or likeness). Temporal relations are ‘different sets of
relationships between verbal and visual elements as seen
in … process communications [rather than] in static
displays’. Although Horn’s framework offers important
insights into the ways in which words and images work
together to create meanings, its classifications can be
refined and expanded. For example, the ‘metaphor’
relation can be an inherent characteristic of any relation
and is not necessarily an individual relation. In other
words, a multi-layered framework that can consider the
multiple functions of relations is a better solution.
Delin, et al. (2003), in an attempt in providing multi-
layered framework of studying multimodal documents,
argues that a detailed analysis of a document should take
into consideration at least five levels: content structure,
rhetorical structure, layout structure, navigation struc-
ture and linguistic structure. The rhetorical structure, i.e.
‘how the content is argued’ (p. 56) is most relevant to the
issue of verbal-visual connections, but their framework
is based on the linguistic model of Mann and Thomp-
son (1988) without modifications. Besides, the linguistic
structure should be ideally incorporated into the rhetori-
cal analysis because no information can be without rheto-
ric; language without rhetoric is ‘a pipe-dream’ (Bonseipe,
1966). Multimodality and its relevance in different
A taxonomy of the discourse relations between words and visuals 211
disciplines such as speech synthesis and automation,
medicine, hypertext design, education and entertainment
industry has been comprehensively reviewed by Kalten-
bacher (2004), who also argued that many such studies
’lack empirical evidence to support many of the claims
made’ (p. 202) and a corpus (a larger compilation of data)
is an ideal solution to this problem. This paper can be
regarded as an important step towards a more systematic
and quantitative approach to the issue.
The proposed taxonomy
Logico-semantic relations
The two text-image relations identified by Barthes are
‘anchorage’ and ‘relay’, which come very close to what
Halliday (1994) calls ‘elaboration’ and ‘extension’, are two
of the many ‘logico-semantic relations ... which may
hold between a primary and a secondary member of
a clause nexus’ (p. 219). Although the idea is based on
language clauses, the framework is equally applicable to
verbal-visual connections, although with modifications.
There are two categories of relations in Halliday’s
framework: expansion and projection. Expansion
refers to how a unit expands the other by extension and
elaboration, and projection refers to how a unit projects
another unit, which can be a locution or an idea. The
relation of projection is more straightforward, and is
usually found where the drawing of a character (a visual
element) may have a projected speech or thought (a
linguistic element). The relationship is usually signaled
by a connecting line between two units and the placing
of the projected speech or thought inside a balloon. The
expanded meaning is more complex and will receive
more treatment below. A third category – decoration
– will also be added to the framework.
The idea of expansion is useful when applied to
word-image connections. The three types of expansion
212 Kenneth Kong
can be compared to elaborating an existing building,
extending a building and enhancing the environment.
In the case of elaboration, one unit (a clause, in
Halliday’s sense) ‘elaborates on the meaning of another
by further specifying or describing it’ (p. 225). The new
unit ‘does not introduce a new element into the picture
but rather provides a further characterization of one
that is already there, restating it, clarifying it, or adding
a descriptive attribute or comment’ (p. 225). There are
three sub-categories under elaboration: exposition,
exemplification and clarification. ‘Exposition’ is
equivalent to ‘in other words’ in linguistic terms.
‘Exemplification’ is equivalent to ‘for example’ and
‘Clarification’ is equivalent to ‘to be precise’. I tend to
use terms that are user-friendly, and replace ‘exposition’
and ‘clarification’ with ‘explanation’ and ‘specification’,
following van Leeuwen’s terminology (2005). Moreover,
there is a relation that Halliday neglects but which is
extremely important in multimodal discourse. A word
or text can be used to identify a particular image and
vice versa. ‘Identification’ is an important function in
many genres, such as academic textbooks, technical
manuals and travel guidebooks. Hence, one unit can
elaborate on another by explaining, exemplifying,
specifying or identifying.
The second category of expansion is extension. Like
an extension of a building, a unit with new meaning
can be added to the original unit. Based on language
data, there are two main ways of extending a unit
– addition and variation (Halliday, 1994). In the case of
addition, one unit is ‘adjoined to another; there is no
implication of causal or temporal relationship between
them’ (p. 230). In linguistic terms, this is like the use of
the conjunction ‘and’. In the framework proposed here,
the term ‘collection’ will be used instead, to underscore
the constitutive function of semiotic resources. In the
case of variation, one unit is regarded as ‘being in total
or partial replacement of another’, which is similar to
the function of ‘comparing and contrasting’. Although
collection and variation are important in verbal texts,
there are other equally important meanings present
in word-image extension but missing in Halliday’s
framework. The first of these is the idea of sequentiality.
In Halliday’s model, sequence is classified as a relation
of enhancement (which will be discussed below) of a
main unit. In other words, the unit with the meaning of
sequence is subordinated to another unit by quantifying
it. In the word-image world, both pictures and words
can convey ideas in chronological ways, without one unit
being subordinated to another. A good example is comic
strips, which are composed of spatially arranged panels
that may contain words or pictures only. These units are
self-sufficient and constitute the comic as a whole, and
cannot be regarded as subordinate in any sense.
Another important meaning of extension in word-
image relationships is alternation, which is different
from variation in the sense that one of the entities can
completely replace another without losing any meaning.
Response, as proposed by Grimes (1975), is embedded
in the mode of question and answer. Although derived
from another unit, the ‘response’ unit also has its own
information.
The third category of expansion is enhancement
(Halliday, 1994), which tries to expand a main idea unit
by specifying circumstances. This is based on the idea of
symmetry of relations, which will be further elaborated
below. Basically the enhancement unit qualifies another
unit by specifying time, purpose, condition, goal etc.
These three categories of expansion are significant
in linking words and images, as is a fourth connection,
which is more diffuse but less constrained by convention
and more subject to interpretation. This is the deco-
rative function of images. Although words can also
do the same to images, this is less likely because we
seldom relate words to decorative functions when
we see words and pictures together. Although this
function is subjective, it is important enough to be
labeled as another relation because while pictures can
 A taxonomy of the discourse relations between words and visuals 213

  Explanation
 (in other words)

  Exemplification
 Elaboration  (for example)
   Specification
 (No new

 information  (to be precise)
   Identification
 
 (namely)

 


  Collection
Extension   Variation
 
 (New   Sequence
 information)   Response
 
 Expansion    Alternation
  
 


  Spatio-temporal
    Manner
    Cause
  


  Effect
    Condition
  Enhancement   Means
  
 (New information   Purpose

  by specifying   Justification
Types   circumstances)   Concession
 
of    Motivation
relation    Enablement
   Restatement
 
   Summary
 Projection 
 
(New  Speech

 information 
 
usually in Thought
 linguistic forms) 
 

 Decoration (New 
 but omissible 
 Various forms
  and types
 information) 

Figure 1.  Logico-semantic relations between words and images

214 Kenneth Kong
simply decorate accompanying messages, they can
also ‘elicit emotion-laden reactions that may precede
cognitive awareness and influence interpretation of
messages’ (Richards and David, 2005, p. 31). Conse-
quently, decoration, together with our existing two
main relationship binders – expansion and projection
– constitute what I mean by the three logico-semantic
relations of words and images.
Symmetry/Asymmetry of relations
The symmetrical relations of verbal language have
attracted enormous attention in linguistics (cf. Mann
and Thompson, 1988). Dealing with the schematic
structure of language, studies in textual relations have
always had strong implications for pedagogy (Bhatia,
1993) and computer-based text generation (Mann and
Thompson, 1988). One very useful notion of textual
linguistics that can be applied to word-image relations is
that of unit hierarchy, namely that relations which bind
units together may or may not be of equal status. There
can be at least three possibilities. First, a unit can be
subordinate to the main unit. This is what linguists call a
hypotactic relationship or hypotaxis1:
Although he woke up late, he could still catch the bus.
A unit subordinate to another unit is found in the
logico-semantic relations having supporting functions,
such as elaboration, enhancement and decoration.
Second, two units can be of equal status and linked
in coordinate fashion:
Mary was a teacher and her husband was a nurse.
This is known as a paratactic relationship or parataxis,
and is found in logico-semantic relations which are of
equal standing, without one modifying the other. The
logico-semantic relations of extension and projection
are good examples of this type of relationship. Although
this is a useful classification, there are occasions in
which these relationships do not work. This is why
Grimes (1975) proposed a third type of relationship that
can take either form. Known as the ‘neutral predicate’,
Grimes argued that it was most common in discourse.
The best examples of neutral relations are collection
and sequence, in which a unit may take a dominant
role (with a bigger size or a more central position). In
fact, I would like to go a step further by arguing that all
relations can be neutral, depending on the layout, posi-
tioning (to be discussed below) and intended purpose.
Symmetrical and asymmetrical relations can be illustrat-
ed by the diagrammatic analogy of nucleus (main unit)
and satellite (subordinate unit) (Mann and Thompson,
1988). The two basic relations also give rise to other
possibile combinations (see Figure 2).
More examples of how these can be applied to word-
image relations can be found in Section 3.
Spatial arrangement of relations
Kress and van Leeuwen’s concept of ‘integrated text’ is
particularly useful in understanding how the arrange-
ment of relations can be linked to their discourse func-
tions. As they argue, verbal and visual codes should not
be seen as separate; instead they should be ‘looked upon
as interacting with [and] affecting one another’ (1996, p.
183). This view of integrated text allows us to see how the
‘representational’ and ‘interactive’ meanings are interwo-
ven into three interrelated systems:
Information value: The placement of elements …
endows them with the specific informational values at-
tached to the various zones of the image: left and right,
top and bottom, center and margin.
Salience: The elements are made to attract the viewer’s
attention to different degrees, as realized by such factors
as placement in the foreground or background, relative
size, contrasts in total value (or colour), difference in
sharpness, etc.
 A taxonomy of the discourse relations between words and visuals 215

Figure 2.  Diagrammatic illustrations of various symmetrical and asymmetrical patterns

216 Kenneth Kong

Framing: The presence or absence of framing devices As all document designers know, the arrangement of
(realized by elements which create dividing lines, or by elements in a document is more than a random choice,
actual frame lines) disconnects or connects elements of and reflects rather complicated cultural and individual
the image, signifying that they belong or do not belong expectations. Kress and van Leeuwen touch on the issue
together in some sense. (p. 183) by identifying what is regarded as ‘given’ on the left-hand
side and what is regarded as ‘new’ on the right-hand
In other words, information value is linked to the
side, following how people read English sentences:
relationship between how information is spatially
‘for something to be Given means it is presented as
arranged and its inherent meanings and implications.
something the viewer already knows, as a familiar and
Salience and framing are related to how information
agreed-upon … For something to be New means that it
is highlighted and how it is divided. Since these three
is presented as something the viewer must pay special
elements are important in creating text coherence
attention.’ (1996, p.187). Other specific meanings may
through combining and textualising semiotic resources,
be created by other spatial arrangements. Centre posi-
they are treated under the ‘textual’ meta-function in
tion usually denotes a more important role of the unit
systemic-functional framework of Halliday. Two other
compared with the marginal position, the top position
meta-functions are ‘ideational’ and ‘interpersonal’,
means ‘ideal’, and the bottom position is usually related
which will be dealt with in detail below.
to more real and down-to-earth particularities. This is
Particularly relevant to our discussion here is the
why attractive models (which take up most of the space)
idea of information value, which pays attention to how
are always put in the central or ‘ideal’ position in an
different verbal and visual elements can be arranged
advertisement and disclaimers (which aim at reducing
in such a way that specific meanings can be created.
legal responsibilities) are placed at the very bottom.

Figure 3.  Example of a prototypical left-right and given-new arrangement

 A taxonomy of the discourse relations between words and visuals
It is certainly useful to apply these concepts in
analyzing the different logico-semantic and symmetrical
relations in a document. For example, the ‘identifica-
tion’ relation, as a subordinate unit, is usually put on
the right hand side (as NEW information2), and what is
being identified is put on the left hand side (as GIVEN
information).
In other words, what is designated as GIVEN on
the left hand side usually coincides with the NUCLEUS
element (the key element), whereas what is presented
as NEW is closely related to the SATELLITE unit (the
subordinate unit) on the right hand side. This is basically
the same as linguistic information.
He arrived in Hong Kong When he was 12.
Nucleus Satellite
(Enhancement: Time)
Given/Familiar New/Unfamiliar
Satellite: Identification
(New/Unfamiliar
Information)
Figure 4.  An example in which the given-new pattern is violated
217
There are, in fact, empirical findings that support
these observations. Biber et al. (2002) argue, based
on a large computerized language corpus, that most
satellite clauses are posed after nucleus clauses and that
only about 30% of satellite clauses are positioned at
the beginning of a sentence or in the medial position.
There are at least two reasons for this deviation. Firstly,
satellite clauses can function as the ‘bridge’ between
previous and subsequent discourse (Givon, 2000).
Interestingly, this principle can be equally applicable to
the relationship between verbal and visual elements.
In the diagram below, the identification satellite in
the middle is put on the left hand side of the diagram
being illustrated, instead of the usual right position.
This is mainly because the function of the identification
satellite is to serve as a link between the two panels
(those of the recorder and the remote control) to show
the corresponding buttons. Secondly, from a linguistic
point of view, in addition to their coherence function,
218 Kenneth Kong
pre-posed satellite clauses have been shown to have
significant interpersonal functions of mitigating or
softening requests that are put after the satellite.
Another important factor in word-image integration in
terms of spatial arrangement is page layout. As Bateman
et al. (2004) point out, the relevant unit of analysis may
not be the page in its entirety. The example they use
is newspapers, in which spaces have been allocated as
so-called ‘newsholes’ (Lie, 1991), the areas for advertising
and other fixed elements, and where another important
issue is whether the combination of visuals and words
can give a balanced layout from an aesthetic point of
view. In other words, the GIVEN-NEW sequence or
the NUCLEUS-SATELLITE sequence may not apply.
In some cases the opposite may even be true. Hence,
the interplay of conventionality and other factors such
as coherence, interpersonal, production and aesthetic
considerations should be further explored in the
relationship between visuals and words, as visual and
verbal elements may clash and contradict each other
(Kress and van Leeuwen, 1998). Because directionality
is ‘now a variable, a matter of choice in many genres’
(Kress and van Leeuwen, 2001, p. 125), it is important
to examine how and why document designers violate
conventional patterns to create new meanings.
Meta-functions of relations
From systemic functional perspectives, language
has three simultaneous functions: its ideational,
interpersonal and textual meta-functions. The ideational
meta-function is its role in giving and presenting
information; the interpersonal meta-function is its role
in negotiating and regulating human relationships; and
the textual meta-function of language is its power to
interweave ideational and interpersonal resources into
a coherent piece. These functions can be applied to texts
and images individually, as well as their relationships.
Following Mann and Thompson (1988), who deal with
Ideational
Expansion:
Elaboration (explanation, exemplification, specification,
identification)
Extension (collection, variation, alternation, response)
Enhancement (spatio-temporal, manner)
Projection
Interpersonal
Expansion:
Enhancement (justification, motivation, concession,
enablement, cause, effect, condition, means, purpose)
Decoration
Textual
Expansion:
Enhancement (restatement, summary)
Figure 5.  Meta-functions of word-image relations
verbal texts only, the relations between words and
images can be classified into these three important
functions.
Projection and most of the expansion relations
are ideational because their main functions are to
present, clarify or emphasize information. However,
some of the expansion relations, such as justification
and motivation, are mainly important in negotiating
stance with readers or an audience. Decoration is an
obvious example that has no explicit ideational value
and is there mainly to catch audience attention or for
aesthetic purposes. Some of the expansion relations,
such as summary, are related to organizing messages
and making a text more coherent. In other words,
all relations can be assigned a value in terms of their
semantic and pragmatic functions. However, these three
meta-functions are not meant to be clear-cut: on many
occasions more than one function is at stake. What is
important is our awareness of the functional value that
may exist between images and words.

Evaluative nature of relations
Some relations may have additional evaluative functions
in addition to the logico-semantic ones outlined above.
Evaluation is closely related to the ideational and
interpersonal meta-functions mentioned previously.
It is related to the ideational meta-function because
evaluation is used to give additional information
about something in question,3 and to the interpersonal
meta-function because firstly evaluation can mitigate
the degree of certainty in a proposition, as in the case
of a hedge (the epistemic nature of evaluation), and
secondly it can influence a reader’s attitude towards
ideational content (the attitudinal nature of evaluation).
Evaluation, hence, is closely related to the function of
persuasion, which has been argued to be an important
issue in effective document design (Fogg, et al. 2001).
The following shows how a visual can be anchored or
elaborated by words through identification (‘Beloselsky-
Belozersky Palace facade’) and at the same time its
embedded evaluation (‘distinctive’).
Figure 6.  An example of an evaluative relation between a
picture and words
[Source: Eastern Europe (2001). Lonely Planet Press]
A taxonomy of the discourse relations between words and visuals 219
Evaluation is a focus in linguistic studies (see Hunston
and Thompson, 2000) and recently in the study of
visuals in political cartoons (Lemke, in progress),
webpages (Lemke, 2002) and in a more general manner
(Kress and van Leeuwen, 2001). Apprisal theory
(Martin, 1992, 2000; Martin and White, 2005; White,
2001) is one of the recent attempts to study evaluation.
The theory identifies three main elements of evaluation:
affect, judgment and appreciation. The framework
avoids a simplistic treatment of evaluation and considers
a large range of evaluative resources. Another strength is
its consideration of the interface of discourse and finer
linguistic resources. However, the framework has some
ambiguous and overlapping classifications which should
be subject to fine-tuning in future. Another framework
was developed by Greenbaum (1969) and Lemke (1998)
and has been recently modified to study evaluation
more holistically (Kong, forthcoming). The framework
provides a tidier treatment of evaluation and is more
suitable to study the connections between words and
images. In the model, a total of 7 evaluative categories
are identified and divided into large groups of categories:
epistemic and attitudinal (see Figure 7 below). Epistemic
evaluative categories express doubt or certainty about
a proposition or an idea, whereas attitudinal evaluative
items tend to express more personal judgment about
what is said or shown in a picture.4 Warrantability and
comprehensibility can be regarded as belonging to the
first group because they are both related to the degree of
conviction or doubt. For example, warrantability items
in linguistic mode such as ‘definitely’ and ‘undoubtedly’
express conviction, and warrantability items such as
‘possibly’ and ‘allegedly’ express a certain degree of
doubt. When comprehensibility is also related to the
expression of epistemic elements, it has an additional
element of ‘observation or perception of a state of affairs’
(Greenbaum 1969, p. 204). Again comprehensibility
items can be used to express conviction or some degree
of doubt. For example: expressions such as ‘evidently’
220 Kenneth Kong

Warrantability
(Doubt or certainty)
Epistemic
Comprehensibility
(Observation or Perception in
Evaluative type addition to doubt or certainty)

Desirability
Normativity
Attitudinal Usuality
Importance
Humorousness

Figure 7.  Classification of evaluative types

and ‘obviously’ tend to express the writer’s conviction,
whereas words such as ‘seemingly’, ‘apparently’ express
some degree of doubt.
Other categories are expressions of more personal
judgment about a statement or visual. Desirability
assesses the extent to which something causes
satisfaction or otherwise. Usuality expresses an attitude
that something is expected or unexpected. Normativity
assesses the appropriateness of an entity. Lastly,
humorousness assesses the extent to which something
can cause entertainment or has a humorous effect. The
above figure summarizes the semantic relationships of
the evaluative categories.
Metaphorical nature of relations
Like evaluation, metaphor may not be an inherent
feature of all word-image relations but is important
in various genres. To define it briefly, metaphor is
a figurative expression that is transferred from one
semantic domain to another. ‘Metaphor’ here includes
all kinds of figures of speech such as similes, metonomy,
synecdoche, hyperbole, and apostrophe, since the term
‘metaphor’ has been so widespread that it can be used as
an umbrella term for all related terms (Chandler, 2004).
Metaphor has been studied mainly from cognitive
and pragmatic perspectives, focusing on verbal data
alone. The most notable exception is Marcus (1998)
who studied the use of visual metaphors in user-inter-
face design in computer documentation. The cognitive
approach to metaphors argue that metaphors reflect
not only superficial linguistic patterns, but also the
existence of certain conceptual patterns in our minds
(Lakoff and Johnson, 1980; Lakoff and Turner, 1989).
This can be illustrated by the way cognitive linguists
study metaphors. They identify the linguistic resources
that are used to express a metaphor with small letters.
For example, the metaphorical expression ‘I am quite
rusty in Spanish’ is represented by small letters. This
is known as a linguistic metaphor. When cognitive
linguists refer to the conceptual frame that underlies
the linguistic metaphor, which is known as a conceptual
metaphor, they will use capital letters such as ‘MIND
IS MACHINE’ (A = B). ‘A’ denotes the target domain
(mind) and ‘B’ is the source domain (machine).
Metaphors have been frequently used to convince
and persuade others. As Gibbs and Gerring (1989, p. 156)
argue, metaphors play a crucial role in maintaining social
and interpersonal relationships through the common
ground of the speaker and the listener. The listener has
 A taxonomy of the discourse relations between words and visuals 221

to work out the reasons why certain metaphors are

used and the conceptual framework underlying them.
However, this cannot explain why certain metaphors are
used more frequently than others in certain situations,
which is why a pragmatic approach has been advocated.
As Charteis-Black (2004, p. 11) argues,

“the only experiment of metaphor motivation [from a

cognitive perspective] is with reference to an underlying
experiential basis. This assumes that metaphor is an
unconscious reflex, whereas a pragmatic view argues that
speakers use metaphors to persuade by combining the
cognitive and linguistic resources at their disposal”.
Figure 8.  An example of metaphorical relation (Adapted
from Horn, 1998, p. 119) ©Robert Horn. Reproduced with
permission.

Again, previous studies on linguistic metaphors can be This figure can be illustrated diagrammatically as follows,
fruitfully applied to the study of visual metaphors. By together with the concepts of logico-semantic relations,
focusing on printed advertisements, Forceville (1996) symmetry/asymmetry and nucleus-satellite pair.
identified four types of pictorial metaphors: those with
one pictorially present term, those with two pictorially Specification
present terms, pictorial similes and verbo-pictorial
metaphors. Obviously, the last category is most relevant
to the discussion here. According to Forceville, in verbo-
pictorial synthesis, one of the domains (target or source)
is realized verbally, and the other is realised pictori-
Picture: Plane and the label Sentence: ‘This project is really
ally. The removal of either element may result in the
taking off.’
disappearance of a metaphorical relationship. Focusing (Inherently metaphorical)
on the use of metaphors in computer documentation,
Marcus (1998) argues that metaphors are a significant
component of user-interface designs (for example, the Identification (Metaphorical)
use of trash bin to represent a file folder for discarded
computer files) because metaphors can increase the
level of familiarity of new concepts to readers and can

also increase ease of learning, memorization and use.
Horn (1998) points out that metaphorical relationship The Bradford Plane (Visual)
between words and images can be elaborated by words Project
at a higher level. For example, a picture that shows a (Words)
plane taking off with the label ‘the Bradford project’ is
a metaphorical relationship, which is anchored by a In this diagram, the plane is the nucleus (the most
speaker’s sentence with a similar metaphorical meaning: important proposition in a message to which other
‘this project is really taking off ’. more peripheral propositions are referred), not only
222 Kenneth Kong

because it takes a central position in the picture but also An example of analysis (full text in appendix)
because the sentence ‘this project is really taking off ’
clarifies or specifies the metaphorical relationship. It This classification can be put into practice by examining
should be noted that words and images may not have an extract from a travel guide. Travel guides exhibit a
any metaphorical relationship binding them, but as with range of styles, from content that is extremely packed
other classification values, metaphor can overlap with with words without any pictures to content that is fully
other values. In the example above, the relationship illustrated with images and pictures. This reflects the
between the visual plane and the label (the Bradford history of the genre – texts produced more than 10
Project) is metaphorical and linked by the logico- years ago tend to be more word-based, whereas those
semantic relation of identification. The following chart produced in the last 5 years are usually illustrated
summaries the different possible combinations of text- with more pictures and images, although the degree
image relations or ‘network’ of relations in Halliday’s of integration between words and images can vary
terms (1994): considerably. The example selected puts words and
   Expansion images in a rather integrated fashion and is what Horn

 Logico-semantic   Projection (1998) calls ‘visual language’.
 
relations   Decoration The text is about a scenic spot in central India called

  Fatehpur Sikri. The text, excluding the page numbers
  and label at the top, can be divided into six blocks of
   Coordination
  (Nucleus-Nucleus) information. The name of the place ‘Fatehpur Sikri’, as the
 Hierarchy   Subordination smallest piece of information, is the first block (Block 1).
 
 (Nucleus-Satellite) The second block serves as an introduction and is located


  Neutral at the top left-hand corner, immediately underneath the
 title (Block 2). It consists of an image of ‘Fretwork jali’
   Right and a paragraph that introduces readers to the history
 
Text-image    Left of the place. The most prominent aspect of this extract
Relations  Information   Centre is obviously the drawing of the place together with a
 
value   Top synthesis of words and photographs that branch out from

   Bottom the drawing (Block 3). The fourth block, squared into a
 box, is located at the top right-hand corner and is entitled
   Ideational
  ‘Visitors’ Checklist’ (Block 4). The last two blocks are
 Meta-functions   Interpersonal located at the bottom on the right-hand side. One is about
   Textual
  the sights that visitors should not miss (Block 5) and the
  other is a plan of Fatehpur Sirki (Block 6). For the sake
   Evaluative
Evaluative of better organization, I will initially focus on the logico-
   Non-evaluative
 semantic relations, the symmetrical status and the meta-
  functional distribution of these relations in the extract.
   Metaphorical
Metaphor I will then move on to the spatial arrangement of these
  Non-metaphorical
relations, and finally the evaluative and metaphorical

Figure 9.  Network of relations nature of a number of the relations.

Logico-semantic relations and their hierarchical represen-
tation and meta-functional distribution
At the macro-level, the three-dimensional picture
with its elaboration can be regarded as the nucleus of
information. All other blocks are satellites to it. But
what are their binding relationships? The first block is
the IDENTIFICATION of the nucleus by a satellite. The
second block, giving background historical information
to the main block, MOTIVATES the readers to read
the rest of the page by conveying the importance
of the scenic spot. Of course, readers can skip this
information and jump to any part of the extract, but
that is the intended function because it is put in the top
left-hand corner, orienting readers to it as the first piece
of information by following the normal reader sequence
from left to right in English. This is also qualified as
an enhancement relation because it does not elaborate
Figure 10.  Logico-semantic relations in the extract
A taxonomy of the discourse relations between words and visuals 223
but adds new information to the nucleus. Similar to the
MOTIVATION block, the fourth block, the ‘Visitor’s
Checklist’ adds further information about the site
(instead of simply elaborating), such as exact location,
address and telephone number of the information
centre, and other important information about the site.
Hence, this qualifies as a MEANS relation, also under
the category of enhancement. The last two blocks do
not give new information, and mainly elaborate what
is already there in the nucleus. The plan of Fatehpur
Sikri EXPLAINS the spatiality of the site, whereas the
‘Star Sights’ block reinforces what is also highlighted
in the nucleus by SPECIFYING which spot should not
be missed. This also has an IDENTIFYING function
by telling the reader exactly what a star sight is. The
following diagram shows the logico-semantic relations
found in the extract:
224 Kenneth Kong

The foregoing analysis has only identified the nucleus
and satellites of the larger blocks of information, and
the logico-semantic relations that bind them together.
Obviously each block of information has it own
internal structure that is at the same time marked by a
hierarchical layer (or layers) of relations. To explain
how this internal hierarchical structure works, I will
refer to the central block (Block 3) as an example. The
three-dimensional map with a centrifugal elaboration
of words and images can be seen as a ‘composite’ of
different sub-blocks linked by COLLECTION, as a
type of extension that adds new information. Each
element in this composite is also supported by a number
of IDENTIFICATION satellites, which label the
individual spots in the place. What about these satellites
in relation to the photographs and short paragraphs that
go along with them? The photographs can be regarded
as nuclei that invite readers to COMPARE (i.e. the
VARIATION relation) them with the corresponding
section in the map. The short paragraph that is usually
placed underneath each photograph is its anchor or
EXPLANATION. The complicated relationship of the
main block (Block 3) can be illustrated as follows.
The following table summarizes the distribution of all
the relations found in the extract.
Meta-functions Logico-semantic Number Percentage
relations
Ideational Elaboration: Identification 18 32.7%
Extension: Collection 12 21.8%
Elaboration: Explanation 9 16.4%
Extension: Variation 7 12.7%
Elaboration: Specification 5 9.1%
Sub-total 51 92.7%
Interpersonal Decoration 2 3.4%
Enhancement: Motivation 1 1.8%
Enhancement: Means 1 1.8%
Sub-total 4 7.3%
Textual 0
Total 55 100%
Figure 12.  Distribution and percentage of logico-semantic
relations

Central Block
Extension: Collection

Elaboration: Identification

   Elaboration: Explanation   Extension: Compare

……………………………
Text Photograph Corresponding
part in the 3-D
map
Figure 11.  Complicated relationship of the main block

It is clear from the table above that most of the relations
are elaborative, particularly that of IDENTIFICATION.
The second most frequent relation is COLLECTION,
under the category of extension. EXPLANATION is
the third most frequent relation. This is not surprising
at all because the main communicative purpose
of travel guides is to identify where to find certain
places and then explain why they are worth visiting.
COLLECTION is frequently used to organize aspects of
the explanation. COMPARISON is used in this extract
because photographs are used to show images of the
spot to increase the readability of the document. The
less frequent relations found are SPECIFICATION,
DECORATION, MOTIVATION and MEANS. In
other words, the types and distribution of relations are
consistent with the overall purpose of the document
in question. In terms of the meta-functions of the
relations, most of the relations are ideational, that is,
to convey core information about the place. Although
interpersonal meta-functions constitute only 7.3% of the
total relations, they are important in positioning readers
and constructing the image of a professional writer who
understands the needs of a traveller.
Spatial arrangement, and evaluative and metaphorical
functions
In terms of the information value of spatial
arrangement, the nucleus is in the central position.
To put this another way, the picture is regarded as the
nucleus of all relations mainly because it is in the central
position and takes up most of the space. What I am
trying to argue is that the information value of a relation
is both its constructive and constitutive feature. As I
explained above, the NUCLEUS-SATELLIE pattern
usually coincides with the GIVEN-NEW pattern, but
there are always other considerations – regardless of
whether they are ideational, interpersonal or textual
– when a document designer makes a decision. This
A taxonomy of the discourse relations between words and visuals 225
extract can illustrate some of these considerations. The
most obvious example is the picture. The specific spots
in the picture are identified by text and photographs
on every side of it, instead of its right hand side, which
is supposed to be the position for a satellite. However,
the document designer still follows what is regarded as
the usual way of an identification relation in the block
about the floor plan (Block 6). What is being labelled
is put on the left hand side and the labels are put on
the right hand side. Also note the placement of the
background information (as a MOTIVATION satellite
relation) at the top left hand corner, while the ‘Visitor’s
Checklist’ (as a satellite relation of MEANS) is in the top
right hand corner. In sum, the information value of a
document is rather complex and can only be understood
in terms of the specific layout in question, in addition
to the conventional understanding of what we mean by
information value.
Evaluation is not an inherent feature of all relations
and is usually embedded in a logico-semantic relation.
In the extract, evaluation is frequently found in the
verbal elaboration of the real pictures. Some of them are
related to how beautiful or desirable a certain feature is
(desirability):
lie within this lavishly decorated ‘Chamber of Dreams.’
Akbar’s queens and their attendants savoured the cool
evening breezes.
The fine dado panels and delicately sculpted walls of
this ornate sandstone pavilion make the stone seem like
wood.
Some linguistic expressions are used to anchor the visual
with highlights of its uniqueness (Usuality/unusuality,
not Normativity, which usually refers to the idea of
appropriateness):
This hall for private audience and debate is a unique
fusion of different architectural styles and religious
motifs.
226 Kenneth Kong
It is topped with an unusual stone roof of imitation clay
tiles.
The evaluative category of warrantability is also found in
the verbal elaboration:
Its decorative screens were probably stolen after the city
was abandoned.
Sometimes identified as the treasury…
These are the main evaluative categories in the extract,
and are consistent with the communicative functions
of a travel guide. Readers expect a travel guide to
be informative, elaborate and accurate. At the same
time, writers of the text also have to convince readers
that a certain place is worth visiting, which explains
why desirability and usuality/unusuality are most
frequently used. This also highlights the persuasive
function of travel guides, in addition to the commonly
acknowledged function of giving factual information.
To summarise, evaluative meanings that are usually
embedded in a larger segment of a logio-semantic
relationship can be used to anchor pictures or words
(words anchoring pictures in the extract) by supporting
the broader communicative function of a text. This is
subtle, but can be argued to be more powerful.
The last characteristic to be discussed here is meta-
phor. Similar to evaluation, metaphor is NOT the
inherent feature of a relation. It is possible to classify a
relation into a logico-semantic type or call it a nucleus
or satellite, but it may not be possible to identify a
word-image relation of any metaphorical nature. In the
sample extract, a relation that is related to metaphorical
meaning can be found in the information block about
the star sights. In fact, the heading ‘Star Sights’ is
metaphorical in itself. Star belongs to the domain of
‘universe’ and does not have any direct connection to
scenic spots. The associated meaning of stars is always
attached to importance in everyday discourse, and the
same meaning is invoked here. Star sights are places that
tourists should not miss. This inherently metaphorical
heading is used to identify the two important sights: the
Turkish Sultana’s House and Panch Mahal, which are
preceded by a star ‘*’. The relationship between the ‘star’
and the names of the places is one of identification, and
is also metaphorical.
Conclusion
Although the importance of turning visual information
into empirical data has been highlighted in the recent
literature on multimodality, there remains a gap
between what has been said and what can be done to
study multi-modal discourse. Kaltenbacher (2004, p.
203) succinctly argues for a need “to develop systematic
methods for theoretically developing, systematically
analyzing and empirically testing the semiotic unfold-
ing of resources and modes and their combinations
in all aspects of our daily lives.” Some recent attempts
(Bateman et al., 2004; Taylor, 2003) have been made to
transcribe or annotate multi-modal discourses so they
can be stored and further studied. However, a systemic
classification of the schematic or functional relationship
between text and images is still unexplored, as language
has its verbal, pictorial and schematic modes (Walker,
2001, p. 177).
Current studies either take an over-simplified view
of the issue or use existing text-based models (such as
that of Mann and Thompson, 1988) as their framework
without any modification, ignoring the potential
differences between words and images. Further studies
in word-image relations will not only be important in
testing assumptions about these two modes (such as
‘only words are used to anchor pictures’), but will also
be important in two other respects. First, they will offer
practical guidelines to document designers on how to
use words and visuals in a more effective way. Second,
they will offer an understanding of how new meanings

are created in this word-picture fusion. Although
increasing attention is being paid to meaning making
and negotiation in verbal language or conversational
discourse, this important – subtle, but more powerful
– meaning-making device has not received enough
consideration.
Hence, this paper identifies the potential relations
between words and images. Most importantly, it argues
for a multiple level of relations and the need for empiri-
cal analysis. The framework, shown using different
types of data, can be applied to other types of discourse
including web-based genres and visual-based genres.
Owing to space limitations, many concepts, particularly
the various logico-semantic relations, have not been
examined here in great detail, and only a limited set of
examples has be used as illustrations. The potentially
subjective and ambiguous nature of some verbal-visual
relationships will need to be followed up in future
studies.
Acknowledgements
*  I would like to thank the Research Grant Committee of
Hong Kong SAR for their generous support to the project on
which this paper is based (RGC research project # HKBU2162/
03H). Every attempt has been made to obtain permission to
reproduce copyright material. If any proper acknowledgement
has not been made, or permission not received, I would be glad
to hear from the copyright holders.
Notes
1.  It should be highlighted that the idea of subordination and
coordination or hypotaxis and parataxis is also related to the
structural hierarchy of clauses. One of the clauses may have
a higher standing and can stand alone, and the other is lower
and cannot exist alone. Although this may be applicable to
some of the relations identified in this study, the hierarchical
A taxonomy of the discourse relations between words and visuals 227
units examined in this paper are based on meaning as the sole
criterion for distinction, i.e. whether a unit is subordinate to
another in terms of their meaning. This is more useful since
visuals do not have some of the structural characteristics of
verbal language. Moreover, in the case of verbal examples, the
units of analysis can be a word, a clause, a sentence or even a
paragraph although clauses are used as examples.
2.  It should be noted, however, that the ‘new’ element in
the GIVEN-NEW pair is not the same as new information in
the classification of logico-semantic relations in which new
information is related to the presence of information new to
the nucleus. New information in terms of information value
concerns the degree of familiarity to the readers/viewers. This
is the reason why alternative phrasing such as ‘Given-Un-
familiar’ or ‘Familiar-Unfamiliar’ might be more suitable in
order to distinguish the two classification systems.
3.  In systemic-functional linguistics, evaluation is usually
regarded as an interpersonal resource only, through which an
evaluator is used to consciously influence a reader towards
some ideational content. I have to thank an anonymous re-
viewer for pointing this out.
4.  The epistemic function of evaluation is closely related to
modality in systemic-functional linguistics and is for adjusting
a writer’s stance towards a proposition. Modal verbs such as
‘may’ fall into this category.
References
Barthes, R. (1967). Elements of Semiology. London: Cape.
Barthes, R. (1978). Image-Music-Text. New York: Hill and
Wang. Reprinted in S. Sontag (Ed.) (1993). A Barthes
Reader. London: Vintage.
Bateman, J., Delin, J., & Henschel, R. (2004). Multimodality
and empiricism: Preparing for a corpus-based approach to
the study of multimodal meaning-making’ In E. Ventola,
C. Charles & M. Kaltenbacher (Eds.), Perspectives on
Multimodality (pp. 65–87). Amsterdam: John Benjamins.
Bhatia, V. K. (1993). Analysing Genre: Language Use in
Professional Settings. London: Longman.
Biber, D., Conrad, S., Leech, G., & Longman (2002). Longman
Student Grammar of Spoken and Written English. Harlow
Essex: Longman.
228 Kenneth Kong
Bonsieppe, G. (1966). Visual-verbal rhetoric. Dot Zero, 2, 37–38.
Chandler, D. (2004). Semiotics: The Basics. London: Routledge.
Charteris-Black, J. (2004). Corpus Approaches to Critical Meta-
phor Analysis. New York: Palgrave Macmillan.
Delin, J., Bateman, J., & Allen, P. (2003). A model of genre in
document layout. Information Design Journal, 11(1) 54–66.
Eckkrammer, E. M. (2004). Drawing on the theories of inter-
semiotic layering to analyse multimodality in medical self-
counselling texts and hypertexts. In E. Ventola, C. Charles
& M. Kaltenbacher (Eds.), Perspectives on Multimodality
(pp. 211–226). Amsterdam: John Benjamins.
Eco, U. (1976). A theory of semiotics. Indiana University Press:
Bloomington.
Fogg, B. J., Marshall, J., Laraki, O., Osipovich, A., Varma, C. et
al. (2001). What makes websites credible? A report on a large
quantitative study. SIGCHI, March 31-April 4, 2001, Seattle,
WA, USA.
Forceville, C. (1996). Pictorial Metaphors in Advertising.
London: Routledge.
Gibbs, R.W., & Gerring, R. J. (1989). How context makes
metaphor comprehension seem ‘special’. Metaphor and
Symbolic Activity, 4(3), 145–158.
Givon, T. (2000). Syntax Vol. 2. Amsterdam: John Benjamins.
Greenbaum, S. (1969). Studies in English Adverbial Usage.
London: Longman.
Grimes, J. (1975). The Thread of Discourse. The Hague: Mouton.
Halliday, M. A. K. (1994). An Introduction to Functional Gram-
mar. London: Arnold.
Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English.
London and New York: Longman.
Horn, R. E. (1998). Visual Language: Gobal Communication
for the 21st Century. Bainbridge Island: Washington:
MacroVU, Inc.
Hunston, S., & Thompson, G. (2000). Evaluation in Text:
Authorial Stance and the Construction of Discourse. Oxford:
Oxford University Press.
Kaltenbacher, M. (2004). Perspectives on multimodality: From
the early beginnings to the state of art. Information Design
+ Document Design, 12(3), 190–207.
Kong, K. C. C. (forthcoming). Linguistic resources as
evaluators in English and Chinese research articles.
Multilingua.
Kress, G. (1998). Visual and verbal modes of representation in
electronically mediated communication: the potentials of new
forms of text. In I. Snyder (Ed.), Page to Screen: Taking Lit-
eracy into the Electronic Era. London, New York: Routledge.
Kress, G., & Van Leeuwen, T. (1996). Reading Images: The
Grammar of Visual Design. London: Routledge.
Kress, G., & Van Leeuwen, T. (1998). Analysis of newspaper
layout. In A. Bell & P. Garret (Eds.), Approaches to Media
Discourse. Oxford: Blackwell.
Kress, G., & Van Leeuwen, T. (2001). Multimodal Discourse:
The Modes and Media of Contemporary Communication.
London: Arnold.
Lakoff, G., & Johnson, M. (1980). Metaphors We Live By.
Chicago: Chicago University Press.
Lakoff, G., & Turner, M. (1989). More than Cool Reason: A Field
Guide to Poetic Metaphor. Chicago: University of Chicago
Press.
Lemke, J. C. (1998). Resources for attitudinal meaning:
evaluative orientations in text semantics. Functions of
Language, 5(1), 33–56.
Lemke, J. C. (2002). Travels in hypermodality. Visual
Communication, 1(3), 299–325.
Lemke, J. C. (work in progress). Visual and Verbal Resources
for Evaluative Meaning in Political Cartoons.
Lie, H. K. (1991). The Electronic Broadsheet: All the News that
Fits the Display. Master’s Thesis, Boston, School of Archi-
tecture and Planning, MIT. http://www.bilkent.edu.tr/pub/
WWW/People/howcome/TEB/www/hwl_th_1.html.
Mann, W., & Thompson, S. A. (1988). Rhetorical Structure
Theory: Toward a functional theory of text organization.
Text, 8(3), 242–281.
Marcus, A. (1979). Visual rhetoric in a pictographic-ideo-
graphic narrative. In T. Borbe (Ed.), Semiotics Unfolding:
Proceedings of the Second Congress of the International As-
sociation for Semiotic Studies Vienna, July 1979.
Marcus, A. (1998). Metaphor design in user interfaces. Journal
of Computer Documentation, 22(2), 43–57.
Marsh, E. E., & White, M. D. (2003). A taxanomy of
relationships between images and text. Journal of
Documentation, 59(6), 647–672.
Martin, J. R. (1992). English Text: System and Structure. Am-
sterdam: John Benjamins.
Martin, J. R. (2000). Beyond exchange: appraisal systems in
English. In S. Hunston & G. Thompson (Eds.), Evaluation
in Text: Authorial Stance and the Construction of Discourse
(pp. 142–175). Oxford: Oxford University Press.
 A taxonomy of the discourse relations between words and visuals 229

Martin, J. R., & White, P. R. R. (2005). The Language of Evalua- Sources of examples
tion: Appraisal in English. New York: Palgrave Macmillan.
Oyama, R. (1999). Visual semiotics in a cross-cultural perspec- Eastern Europe (2001). Lonely Planet Press, 6th edition.
tive: a study of visual images in Japanese and selected British Horn, Robert E. (1998). Visual Language: Gobal
advertisements. Unpublished PhD dissertation, University Communication for the 21st Century. Bainbridge Island,
of London. WA: MacroVU, Inc.
Pegg, B. (2002). Two dimensional features in the history of India Travel Guide (2002). London: Dorling Kinderly (Mark
text format: how print technology has preserved linearity. Warner © Dorling Kindersley).
In N. Allen (Ed.), Working with Words and Images.
Westport, CT: Ablex Publishing.
Peirce, C.S. (1931-58). Collected Writings (Vols. 1–8). In C.
Hartshorne, P. Weiss & A.W. Burks (Eds.). Cambridge,
about the author
MA: Harvard University Press.
Richards, A.R., & David, C. (2005). Decorative color as a Kenneth Kong is an associate professor of linguistics in the
rhetorical enhancement on the world wide web. Technical English department of Hong Kong Baptist University. His
Communication Quarterly, 14(1), 31–48. academic interests include discourse analysis, multimodal
Royce, T. (2002). Multimodality in the TESOL Classroom: analysis, intercultural pragmatics and language for spe-
Exploring Visual-Verbal Synergy. TESOL Quarterly, 36(2), cific purposes. He has published extensively in the areas of
191–205. discourse analysis and pragmatics.
Saussure, F. de ([1916] 1974). Course in General Linguistics
(trans. Wade Baskin). London: Fontana/Collins. Contact
Still, J. M. (2001). A content analysis of university library Web
Kenneth Kong
sites in English speaking countries. Online Information
Department of English Language and Literature
Review, 25(3), 160–164.
Hong Kong Baptist University
Stroupe, C. (2000). Visualizing English: Recognizing the
Waterloo Road
hybrid literacy of visual and verbal authorship on the Web.
Kowloon
College English, 62(5), 607–632.
Hong Kong
Taylor, Ch. (2003). Multimodal Transcription in the Analysis,
e-mail: kkong@hkbu.edu.hk
Translation and Subtitling of Italian Films. The Translator,
9(2), 191–205.
Van Leeuwen, T. (2005). Introducing social semiotics. New
York: Routledge.
Van Leeuwen, T., & C. Jewitt (Eds.) (2000). Handbook of Visual
Analysis. London: Sage.
Walker, S. (2001). Typography and Language in Everyday Life.
London: Longman.
White, P. (2001). Appraisal website. Retrieved October 10, 2001
from http:///www.grammatics.com/appraisal/Appraisal
Guide
230 Kenneth Kong

Appendix 1.  Sample analysis of a multimodal document

Block 1: Identification Block 2: Motivation Block 4: Means

Block 3: Nucleus Block 5: Specification/ Block 6: Explanation

Identification

Mark Warner© Dorling Kindersley. Reproduced with permission, except for the top left photo on the left page (© Ashok
Dilwaki) and the top right and top left photos on the right page (© DN Dube).