Available online at www.sciencedirect.

com

Journal of Pragmatics 44 (2012) 2067--2084

www.elsevier.com/locate/pragma

Representing emotive meaning in visual images:

A social semiotic approach
Dezheng Feng a,b,*, Kay L. O’Halloran b
a
Department of English, The Hong Kong Polytechnic University, Hong Kong
b
Multimodal Analysis Lab, Interactive Digital Media Institute, National University of Singapore, Singapore
Received 15 July 2011; received in revised form 4 October 2012; accepted 7 October 2012

Abstract
This study examines how emotive meaning is represented in visual images using the semiotic resources of facial expression, touch
and body orientation. Complementing the cognitive metaphorical interpretation, the visually represented emotive behaviors are theorized
as partial iconic representations of real life behaviors, which are in turn indexes of emotion. Adopting a social semiotic approach, this
paper formulates facial expression, touch, and body orientation as inter-related systems of meaning, the selection and combination of
which constitutes a systemic functional ‘‘lexico-grammar’’ for analyzing and interpreting meaning making in visual imagery. The systems
are then used to explain how emotive meanings are represented in comic books. The analysis shows that cartoonists’ systemic choices
for representing emotions are largely consistent with the coding accuracy of facial behaviors on the one hand, and style guidelines
advocated in comic handbooks on the other. Through the examination of both American and Japanese comics, it is demonstrated that the
social semiotic lexico-grammatical approach is not only effective in explaining the emotion resources in nonverbal behavior, but also
useful for investigating cultural differences in the visual depiction of emotion. The systemic choices also provide artists with a framework
to ‘‘design’’ emotion in creative visual media such as comics, and possibly in computer vision, game design and related domains.
© 2012 Elsevier B.V. All rights reserved.

Keywords: Emotive meaning; Emotive behavior; Visual representation; Social semiotic approach; Comics

1. Introduction

In visual art forms such as comics and graphic novels, emotion is not only constructed by the more or less
conventionalized signals such as pictorial runes, pictograms and balloons (see Forceville et al., 2010), but also by stylized
versions of bodily behaviors such as facial expression, body posture, and touch. Cognitive theorists, most notably Charles
Forceville (e.g. 2005, 2011), interpret these nonverbal behaviors as visual manifestations of the Idealized Cognitive Model
(ICM) of emotion, drawing upon the cognitive metaphor theory (e.g. Lakoff and Johnson, 1980; Kövecses, 1986, 2000). As
an alternative to the cognitive approach, the present study argues that a large number of visually represented emotive
behaviors are direct simulations of those in real life. Premised on this assumption, this study aims to systematically model
visually represented emotive behaviors from a social semiotic perspective. The social semiotic approach, specifically
Michael Halliday’s systemic functional theory (e.g. Halliday, 1978; Halliday and Matthiessen, 2004), emphasizes the
significance of interpersonal meaning (i.e. enactment of social relations) as the gateway to the experiential basis of
meaning making. However, in studying visual images, social semioticians have tended to concentrate on interpersonal

* Corresponding author at: Room AG 415, Department of English, The Hong Kong Polytechnic University, Hung Hom, Kowlong, Hong Kong.
Tel.: +852 65808797; fax: +65 6774 2647.
E-mail addresses: dezhengfeng@gmail.com (D. Feng), kay.ohalloran@nus.edu.sg (K.L. O’Halloran).

0378-2166/$ -- see front matter © 2012 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.pragma.2012.10.003
2068 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

resources such as camera angle, modality, and gaze, which negotiate image-viewer relations (e.g. Kress and van
Leeuwen, 2006; O’Toole, 2011), rather than visual realizations of emotive meaning in communicative acts.
This paper attempts to systematically analyze the visual representation of emotion, focusing on the embodied
resources of facial expression, touch and body orientation, which are collectively called emotive behaviors. Martinec’s
(2001) systemic descriptions of facial actions are refined and extended in order to elucidate the complex interaction of
emotion resources. It should be emphasized that from the social semiotic perspective, emotive behaviors are not seen as
accurate ‘‘semiotic codes’’ for encoding emotive meaning, but as ‘‘semiotic resources’’ which are organized to create
meaning in social contexts. Therefore, like Martinec (2001), we are not arguing that these resources are closed semiotic
systems which are able to signify emotions independently; rather, our aim is to systemically model the choices of facial
expression, touch and body orientation which are available for meaning making in visual art forms. It follows that our
framework does not constitute a ‘‘prescriptive grammar’’ for the construction of emotive meaning, but rather serves as a
‘‘descriptive tool’’ which provides a systematic account of the available semiotic choices. The social semiotic description
complements Forceville’s (2005) cognitive approach, which interprets visually represented emotive behaviors as
belonging to the ICM of emotion concepts.
The systemic frameworks are used to analyze the patterns of emotive resources in the (nonverbal) construction of
emotion in visual media, more specifically comics, where artists depict emotion through exaggerated facial expressions
and other embodied actions (McCloud, 1993; Ozawa, 2005). Following Forceville (2005), the analysis is undertaken in a
top-down manner, where emotions are identified first based on the multimodal expressions of the characters and the
narrative context. Following this, the facial expression, touch, and body orientation of the characters which display
unambiguous emotions are analyzed to see what choices are made from the paradigmatic systems developed for these
three semiotic resources. That is, as in the case of Forceville (2005), rather than judging an emotion from a single
expression (e.g. wide open mouth), our aim is to systematically describe the semiotic choices that are made for the
representation of a given emotion. The focus on how a given emotion is semiotically constructed, rather than the meaning
of a single nonverbal behavior, avoids the problem of interpreting ‘‘what is felt’’ through ‘‘what is expressed’’ subjectively.
This paper unfolds in the following manner. Section 2 is concerned with the semiotic status of emotive behaviors,
addressing the questions ‘‘what type of signs are they?’’ and ‘‘how do they signify meanings?’’. Following this, a social
semiotic framework for emotion resources, that is, a systemic functional ‘‘lexico-grammar’’ which describes the choice
systems for portraying facial expression, touch and body orientation is developed in section 3. The usefulness of the social
semiotic framework for interpreting emotive behaviors is demonstrated through the analysis of cartoonists’ choices for
representing emotions in comics in section 4. The analyses are also related to handbooks for comic creators, such as Blair
(1994) and McCloud (2006) (see Eerden, 2009 for an effort of this kind). The approach provides analysts with an explicit
framework to explain the emotion resources in nonverbal behavior and a method to investigate cultural differences in the
visual depiction of emotion. Furthermore, the systemic choices may serve as a meta-language for teaching about the
design of comics, and related fields such as game design and computer vision.

2. Emotion and the body: the semiotic status of emotive behavior

The signifying power of emotive behaviors is premised on psychological findings that nonverbal expressions of
emotion can be recognized with a high degree of accuracy (e.g. Ekman and Friesen, 1975; Izard, 1977). An early attempt
to scientifically relate emotion to nonverbal behavior is Charles Darwin’s (1872) description of behavioral expressions and
possible emotions. For example, Darwin links blushing with shame, body contact with affection, clenching fist with anger,
and so forth. Ekman and colleagues (e.g. Ekman and Friesen, 1975, 1976) further demonstrate that certain combinations
of facial muscle movement can unambiguously encode basic emotions across cultures. As a result, nonverbal behaviors
are able to function as ‘‘actional semiotic resources’’ for the creation of meaning (Thibault, 2004:79).
Before exploring the emotion resources in nonverbal behavior, it is necessary to clarify the nature of nonverbal emotive
behavior as a sign. According to Charles Peirce’s (1931--1958) trichotomy of icon (representing the object through similarity),
index (representing the object through cause--effect relations) and symbol (representing the object through convention),
emotive behavior is indexical because it is related to emotion through causal continuity relations (i.e. emotion causes emotive
behavior) (Sebeok, 2001:53; Forceville, 2005:73). However, we are not concerned with actual physical emotive behaviors
(and their biometric measures) at this stage, but rather the ‘‘representation’’ of emotion in visual images.
Regarding the visual representation of emotion, Forceville (2005) distinguishes two types of signs in comics: indexical
signs and pictorial runes. The first type refers to the physical symptoms that we recognize in everyday life, such as frowns
and smiles. The second type, following Kennedy (1982), refers to signs that are not perceptible in real life, like the use of
smoke or fire above a character’s head to indicate anger. Forceville’s (2005) formulation of these two types of visual signs
warrants further investigation.
First, Forceville (2005) rightly points out that emotion defies iconic representation because it is an abstract concept.
Therefore, he concludes that visual representations of emotion with nonverbal behavior are indexical in nature. However,
 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2069

Indexical Iconic
Emotion Bodily effects Visual representation

Fig. 1. The semiotic status of visually represented emotive behavior.

Indexical Iconic
Emotion Bodily symptoms Visual representation
Partial

Fig. 2. The semiotic status of visually represented emotive behavior revised.

the represented facial expressions and gestures in images are not just indexical signs, but should be viewed as ‘‘icons’’ of the
indexes. That is, the visual images are ‘‘signs of signs’’. Rene Magritte’s (1928--1929) ‘‘Treachery of Images’’, the famous
painting of a pipe with the caption ‘‘this is not a pipe’’, makes the point that the painting is not a pipe per se, but rather a
‘‘representation’’ of a pipe (McCloud, 1993:25). In the same way, the facial expression in real life is an index of emotion, but
the facial expression in visual media is an icon of the real life expression. The signifying process is illustrated in Fig. 1.
The second step, which is alluded to in Forceville (2005), is the process of abstraction and transformation in visual
representation. This important step, often involving partial representations of real life emotion expressions, cannot be
neglected. For example, facial expressions of happiness include corners of lips drawing back and up, cheeks raising,
lower eyelid showing wrinkles, and so on (Ekman and Friesen, 1975:112), but the icon ‘‘ ’’ only displays the (stylized) lips.
The partial utilization of real life action has further implications for gesture and touch, in which visual representations only
provide a ‘‘snapshot’’ of the movement to represent the whole action (see Fig. 6). Accordingly, the semiotic status of
visually represented emotive behavior is revised in Fig. 2.
Second, Forceville (2005, 2011) argues that indexical signs in visual representations are motivated by metaphors. In
analyzing the representation of anger in comic strips, Forceville (2005:83) suggests that ‘‘the signs are commensurate
with (or indeed motivated by) the ANGER IS HOT FLUID IN A PRESSURED CONTAINER metaphor’’. For example, in analyzing the sign
of tightly closed mouth, he explains that ‘‘the aperture of the mouth is emphatically closed so as to prevent the anger from
coming out of the body-container’’ (Forceville, 2005:81). However, while pictorial runes are metaphorical in nature,
indexical signs are literal. Therefore, we agree that ‘‘smoke (also often fire) is clearly an effect of the heating up of the fluid
or gas in the body-container, exemplifying the ANGER IS FIRE metaphor’’ (Forceville, 2005:82). However, the indexical signs
are direct simulations of bodily expressions (e.g. smile), and thus do not involve metaphors. For example, a drooping body
posture signifying sadness is not an instantiation of the metaphor SAD IS DOWN, but rather a mimicry of the typical physical
body posture of sadness. In other words, such visual representations are approximations of physical symptoms which in
turn motivate metaphors, rather than vice versa. This is in accordance with the function of metaphor, which is to construe
abstract concepts in terms of concrete ones. The bodily symptoms, however, are concrete physical phenomena which
can be directly represented, and thus do not require metaphorical forms. Metaphorical expressions are used when the
representation is not a direct mimicry of the real bodily expression, such as the use of fire or smoke to represent anger.
We can therefore identify two types of visual signs for the representation of emotion, metaphorical symbolic and literal
iconic, corresponding to Forceville’s (2005) runes and indexes respectively. Symbolic signs refer to conventionalized
visual symbols, such as the small stars which represent pain in comics. They are metaphorical in nature because,
as Forceville (2005, 2011) argues, they are not completely arbitrary (e.g. ‘‘seeing stars’’ due to brain injury). In literal
iconic representation, the visual signs are direct simulations of emotion expressions, as argued above. For example,
the smiley ‘‘ ’’ and the more realistic smiling faces in comics are iconic representations of the real life smile. In this paper,
we investigate the iconic representations of embodied behavior, which we argue, are simple mimicry of bodily behaviors
rather than visual manifestations of abstract ICMs.
The two points above thus open a dialog with Forceville’s (2005, 2011) influential cognitive approach. It follows that our
task is to provide an alternative description of the visually represented emotive behaviors to complement the insights
provided by the cognitive theory. Adopting the social semiotic approach, the nonverbal resources are formulated as inter-
related systems of options from which choices are made and combined to represent emotions. In what follows, systemic
functional models for facial expression, touch and body orientation are developed. The systems map out the meaning
making resources in the three types of semiosis and provide a ‘‘lexico-grammar’’ for analyzing and interpreting emotive
meaning in visual images.

3. A social semiotic (systemic functional) model for emotion representation

Numerous attempts have been made to theorize the meaning making resources in nonverbal behavior, including
Birdwhistell’s (1970) widely known and much debated linguistic approach. Birdwhistell believes behaviors can be broken
into parts (like sentences or words) which can be categorized. Based on this, Birdwhistell formulates a ‘‘kinegraph
2070 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

system’’ to transcribe the entire kinetic behaviors, which include allokines, kine, kinemes and kinemorphemes, analogous
to linguistic units such as phonemes and morphemes. Opponents of Birdwhistell’s approach contend that while language
is typological and segmented, kinetic behaviors are typically continuous and thus cannot be broken down in this manner
(e.g. Richmond et al., 2008).
Conversely, Ekman and Friesen (1975) focus on micro-behaviors which are visually discernible, and their descriptions
include the molecular units of behaviors required for the analysis of nonverbal interaction. This ‘‘external variable
approach’’ excludes non-visible units like allokines and kines because they have no significance in relation to social
meaning and communication (Richmond et al., 2008:51). Following this approach, we focus on the system variables that
can signify emotion and the parameters that can distinguish one behavior from another.
To investigate how ‘‘external variables’’ create emotive meaning, we draw upon the systemic functional theory
(Halliday and Matthiessen, 2004) which conceptualizes grammar as ‘‘meaning potential’’ in the form of inter-related
system networks, and meaning as paradigmatic choices from the available options in the system networks. The systemic
functional grammar for English language includes systems which function to realize ideational, interpersonal and textual
meanings. The grammar has been extended to non-linguistic resources, such as visual images (Kress and van Leeuwen,
2006; O’Toole, 2011), mathematical symbolism (O’Halloran, 2005) and action (Martinec, 2001). Adopting the systemic
functional model, nonverbal behaviors are formulated as inter-related paradigmatic systems in which meaning is made
through the selection and combination of systemic options. In what follows, theoretical frameworks are developed to
provide a systematic description of facial action, touch and body orientation for the analysis and interpretation of the visual
representation of emotion.

3.1. Facial action as a semiotic system

As McCloud (2006:81) points out, we all know how to read and express facial emotions, but few of us can consciously
reproduce them with style and grace in art. McCloud explains that ‘‘the only way to understand the face is to go beneath
the surface and see its parts in action’’ (McCloud, 2006:91). These ‘‘parts in action’’ are extensively studied by
psychologists who have developed a variety of facial behavior coding systems (e.g. Ekman et al., 1971; Ekman and
Friesen, 1978). The best-known system is the Facial Action Coding System (FACS) developed by Ekman and Friesen
(1978). The descriptions are based on muscle combinations which produce visibly distinguishable facial actions, in
parallel to morpheme/letter combinations which form meaningful words. Ekman and Friesen (1978) derive 46 action units
which alone or in combination account for visible facial movements. FACS allows precise specification of the morphology
and the dynamics of facial movement. However, the major impediment to its widespread use is the time required to train
human experts and to manually score the videotape (Bartlett et al., 1999:254).
McCloud (2006) demonstrates how the psychological findings of Ekman and Friesen (1976, 1978) and others can be
applied to comic design. Reducing the complexity of FACS, McCloud (2006:92) identifies twelve facial muscles, including
the squinter, the brow lifter, the corrugator and so on, the combination of which denotes the basic emotions. Following
McCloud (2006), the present study proposes a more elaborated framework using the descriptive tool of system network to
analyze the exact choices made in the representation of basic emotions in visual art.
Working within the systemic functional paradigm, Martinec (2001) proposes a simplified system of facial actions
displayed in Fig. 3. Basic emotions are classified into two broad groups based on two generalized dimensions of
realization: action and value. Martinec’s (2001) coding system, with the dimensions of open and closed, backward and
forward, and up and down, is designed to account for the expression of basic affect.
Martinec’s (2001) system poses several problems. First, the division between action-based and value-based
dimensions of facial action means that action-based anger and fear do not have positive/negative values. However,

Active
Closed, forward Anger
Action
Open-closed
Reactive
Forward-backward
AFFECT Open, backward Fear

Positive
Up Happiness
Value
Up-down
Negative
Down Sadness

Fig. 3. Martinec’s (2001:127) coding system for facial action.

 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2071

horizontal (1)
wrinkled
Forehead vertical (2)
0
raised (3)
whole brow
lowered (4)
Eyebrows raised (5)
lowered (6)
inner corners
drawn together (7)
0
The Face wide open (8)
Eyes
narrow/closed (9)

wrinkled (10)
nose
0
raised (11)
cheeks
0
Lower face open (12)
tensed
closed (13)
open (14)
relaxed
mouth closed (15)

corners up (16)
corners down (17)
Fig. 4. Emotion resources in facial expression.

emotions such as fear and anger do have values, and the up/down dimension plays an important role in their identification.
In addition, the up/down and close/open combinations need to be associated with specific facial/body parts (e.g. inner/
outer eyebrows, bottom/lower mouth) to produce meaning, because different areas of up/down and close/open variations
result in different emotive meanings. For example, ‘‘eyebrows up’’ constructs surprise, ‘‘inner eyebrows up’’ constructs
sadness, and ‘‘upper lip up’’ constructs disgust (Ekman and Friesen, 1975). Therefore, the exclusion of the facial area can
potentially lead to inaccurate modeling of meaning. Furthermore, the associations between the dimensions and the
emotions themselves may be inaccurate. For example, active emotion (anger) is labeled ‘‘closed’’ in the system, but
psychological studies have shown that the mouth in anger may be open or closed. As we shall see, most anger
expressions in the comic data select for open mouth (see also Forceville, 2005:76).
In order to develop a system that is capable of describing and comparing complex expressions of emotion, we draw
upon the scientific theories of facial expression. The proposed system of facial expressions, displayed in Fig. 4,1 includes
the facial dimensions in Ekman and Friesen (1975) and Izard (1977), except for the tension of eyelids, which is difficult to
recognize in human beings and does not appear in our corpus of comics. Following Ekman and Friesen (1975), the face is
divided into three areas---the brow/forehead, the eyes/lids, and the lower face which mainly includes the cheeks, the
mouth and the nose. These facial areas are represented as simultaneous entries in the system, which means that they are
obligatory members of the face, rather than that their combination is obligatory in any instances of representation. That is,
the system maps out the total choices and combinations available in the face, but not all facial areas may be chosen in the
actual representation of emotion.
In this way, facial actions are formulated as choices from the system network. The choices are represented as [A: B]
+ [C: D] + . . ., in which A and C are first level choices and B and D are second level choices. Sometimes third or fourth level
choices are needed, which are simply labeled by adding more colons. For example, anger can be represented by
[forehead: wrinkled: vertical] + [eyebrow: inner corners: lowered] + [eyebrow: inner corners: drawn together] + [mouth:
tensed]. The nodes of the system network are numbered for the convenience of annotation and further analysis. For

1
As a convention in Systemic Functional Theory (Halliday and Matthiessen, 2004), squared brackets means either-or choice, curly brackets
means both-and choice and ‘‘0’’ means the item is not a meaningful choice.
2072 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

Table 1
Emotion and touch.

Emotions Types of touch

Affection, friendliness, and satisfaction Stroking, kissing, licking, holding (hands, arms, etc.), linking arms,
shaking hands, embracing, patting
Anger, hatred Slapping, punching, kicking, pinching

example, anger expression may involve choices from (2) (6) (7) (13) in Fig. 4. It should be reiterated that our aim is not to
prescribe what choices signify what emotions, but to provide a descriptive tool for visual analysis. The system enables us
to describe exactly what choices are made in representing particular emotions. The selection and combination of system
nodes for particular emotions is discussed in section 4, where patterns of choices are revealed through the annotation and
analysis of comic images.
According to Ekman and Friesen (1975:88), most basic emotions can be shown unambiguously in one facial area. For
example, happiness can be recognized by ‘‘lip corners up’’ and anger by ‘‘inner corners of eyebrow lowered’’. This makes
the partial utilization of facial expression possible in emotion representation (cf. partial representation in Fig. 2). In visual
representation, the artists also have the freedom to combine choices from the three facial areas to amplify an emotion or to
create emotion blends (Ekman and Friesen, 1975). In terms of amplification, surprise, for instance, can be constructed by
the combination of (1) (3) (8) (14) in Fig. 4. In terms of blending, happy surprise, for instance, can be created by choices (8)
+ (16).

3.2. Touch

Touch is an effective means to communicate many of our emotions (Richmond et al., 2008:135). Argyle (1975)
identifies 16 types of touch and suggests their emotive meanings, as shown in Table 1. However, the meaning of touch is
not as clear and rich as that of facial expression. That is, differential patterns of touch cannot characterize discrete
emotions unambiguously (Planalp, 1998:34). Therefore, it is reasonable to consider these resources as continuous
expressions of the underlying dimensions of emotion, such as activation (i.e. the arousal level) and valence (i.e. positive
and negative) (see Russell, 1980 for an overview of the dimensional approach to emotion). Our aim, then, is to investigate
how the underlying dimensions of touch are related to dimensions of emotive meaning.
Based on Burgoon et al.’s (1996:86) five dimensions of touch, which include intensity, duration, the touched body part,
the touching body part, and the frequency of contact, we propose the system of touch in Fig. 5.
Contact part produces the semiotic resources of body distance and reciprocity. Body distance indicates emotional or
social distance. Reciprocity refers to whether the same body parts of the agent (the person doing the touching) and the
patient (the person being touched) are used. Generally, although with exceptions, negative emotions are non-reciprocal,
such as slapping and punching, while positive emotions tend to be reciprocal, such as kissing, embracing and handshake.
For the vector of movement, movements towards the agent which draw each other closer typically encode positive

agent (hand, mouth, etc.)

patient (hand, mouth, etc.)
Contact part body distance
reciprocal
reciprocity
non-reciprocal
Touch toward agent
vector toward patient
0
Movement velocity (from low to high)
strength (from low to high)

duration (from short to long)

Fig. 5. Emotion resources in touch.

 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2073

Interpretant 1 Interpretant 2

Raised palm Slapping Anger

Sign1 Object1 and Sign 2 Object 2

Fig. 6. The signifying relation of pre-touch.

emotions such as love and friendliness (e.g. hugging) while movements towards the patient which push each other away
usually encode negative emotions such as anger and hatred (e.g. punching). Some touches, such as kiss and stroking, do
not have a clear vector and are coded ‘‘0’’. Velocity and strength encode the activation of emotions. Strength also encodes
the intensity of emotions, as does the duration of movement. For a specific touch, for example hugging, the more strength
which is used and the longer it lasts, the more intense the emotion is. It should be noted that these dimensions do not stand
alone; rather, they are always instantiated simultaneously in meaning making. For example, slapping includes the
combination of [hand] + [face] (entailing distance and reciprocity) + [vector: towards patient] + [velocity: high] + [strength:
high]. The meaning resources in these choices configure to signify a negative, high activation emotion, like anger.
It should also be pointed out that these dimensions are not accurate signifiers of the valence or activation of emotions in
many cases. For example, an angry person may draw the offender towards him/her by grabbing the collar, in which
[vector: toward agent] encodes negative emotions. Such disparate association between signifier and signified is a
common problem in the semiotization of non-linguistic resources. As explained in section 1, our solution is to ‘‘describe’’
the choices available for representing particular emotions, instead of ‘‘prescribing’’ a rigid grammar for emotive behaviors.
Identifying the meaning making resources in touch only partially solves the problem of representing touch behavior.
Another issue is the limited resources available in static images and their semiotic status. For example, duration and
velocity cannot be easily represented and strength can only be represented indirectly by the effect of the touch. Images
can only represent a ‘‘snapshot’’ of a touch behavior or its preparatory stage (called pre-touch). For example, in the
emoticons in online chatting, the kiss is signified by pouted mouth and the hug is signified by outstretched arms, both of
which are the preparatory stage of the actual behaviors. Our ability to recognize the pre-touch is based on real-life
experience as a member of a culture. From the cognitive perspective, the represented cues are sufficient to activate our
cognitive schema of that particular touch. For example, a person would normally duck after someone raises his/her palm
with the intention of slapping. According to Peircean semiotics, the preparatory stage of slapping is the sign, the actual
slapping is the object, and the ducking is the interpretant, as illustrated in Fig. 6 (cf. Kockelman, 2005). The first signifying
process involves part-whole relation and the second cause--effect relation (see Fig. 2). The employment of pre-touch in
comics is discussed in section 4.

3.3. Body orientation

Body orientation is one subcategory of proxemics which refers to the aspects of posture that concern the horizontal and
vertical orientations of the body. Body orientation is an important resource for the communication of emotion, as well as
attitude, identity and other dimensions of a social being (e.g. Hall, 1959). Mehrabian (1981) proposes two dimensions of
body orientation which communicate meaning: immediacy (e.g. social distance, affection) and relaxation. In the present
study, the meaning of body orientation is analyzed in terms of horizontal and vertical variables shown in Fig. 7.
The system for body orientation displayed in Fig. 7 is similar to Martinec’s (2001) coding system for facial expression
(Fig. 3) (left--right movement on the horizontal axis is not considered because normally it is not emotively motivated). The
dimensions are too simplistic to manage the complexity of facial expression, but are sufficient to describe body
orientations. Martinec’s (2001) classification of activity-based (active/reactive) and value-based (positive/negative)
emotions is adopted. The vertical orientation encodes value-based emotion of un/happiness and the horizontal orientation
encodes activity-based emotion of anger and fear. Upright body orientation often indicates happiness, confidence and

up
Vertical
down
Body orientation
forward
Horizontal
backward

Fig. 7. Emotion resources in body orientation.

2074 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

other positive emotions, while downward body orientation indicates sadness, guilt and other negative emotions. These
relations are so well established that they give rise to metaphorical expressions in language. For example, the metaphorical
expressions of ‘‘cheer up’’, ‘‘I am feeling down’’, and so on are based on our embodied experience of posture (Lakoff and
Johnson, 1980:15). However, unlike the case of vertical axis, the emotive meaning of body orientation on the horizontal axis
is not clear. While the backward body orientation normally indicates dislike, disagreement or other negative feelings
(Anderson, 2008:330), the forward body orientation may indicate either the negative emotion of anger or the positive emotion
of affection. Therefore, it is important to take into consideration both the context and the other emotion resources, as is
demonstrated in the analysis of the comics in section 4.

4. Data analysis: the representation of emotion in comics

The systems for facial expression, touch and body orientation are used to analyze the representation of emotion in
comics in this section. Comics are chosen because cartoonists have the freedom to design emotions using the available
resources in creative ways. As Forceville (2005:71) explains, ‘‘unlike realistic photographs and live-action films, which
more or less ‘naturally’ mirror real-life manifestations of emotions, comics and cartoons make use of stereotypical
exaggerations using a rudimentary ‘sign-system’ very much like a language’’. McCloud (1993) also points out that comics
have developed a specific language based on facial expressions and human gestures. Aside from the embodied
resources, comics also use more conventionalized, or symbolic resources, such as small stars to represent pain, sweat
drops to represent surprise or anxiety, and so on (Saraceni, 2003:26). In this study, we shall focus on the emotive meaning
realized through facial expression, touch and body orientation. As pointed out in section 1, we are not arguing that these
resources can cue emotions independently; rather, our aim is to systematically model how these three fundamental
semiotic resources are used in emotion representation. In the following discussion, emotions are identified first, based on
the multimodal expressions of the characters and the narrative context (cf. Forceville, 2005). Then analyses are presented
concerning how the resources of facial expression, touch and body orientation are selected and combined. Other
resources, such as metaphors and symbols, are also briefly pointed out in the ensuing discussion.
In this study, 66 images depicting clear expressions of emotions are selected mainly from two popular comic books in
America and Japan, Archie’s Pals and Gals (Gorelick, 2010) and Naruto (Kishimoto, 2000) respectively. As Archie
Comics Publications did not give copyright permission to reproduce the images, public domain American comics from
1940s are chosen as examples for illustrative purposes, although their differences with the contemporary comics are not
considered. Using these comics, we examine how emotive meanings are constructed and investigate potential cultural
differences between American and Japanese comics. The emotions depicted in the 66 images include the six basic
emotions proposed by Ekman and Friesen (1975), namely, anger, surprise, fear, happiness, sadness, and disgust
(72 emotions in all, including 6 cases of blends which are counted twice). The relative distribution of these emotions is
displayed in Table 2.
The 66 images are annotated using the system networks, including the details of what choices are made in each
image. As previously explained, the system nodes in Fig. 4 are numbered for convenience, the choices of touch are
annotated with their conventional names, and body orientation with the choice names in Fig. 7. For example, the image in
Fig. 9 is annotated as [(6) (7) (8) (12)] + [Pre-touch: punch] + [Body orientation: forward]. The complete annotation cannot
be reproduced here due to space constraints. Based on the detailed annotation, the construction of these emotions
through facial expressions, touch and body orientation is analyzed and the quantitative patterns are reported in this
section. The single instance of disgust is represented by [nose: wrinkled] (choice 10 in Fig. 4) and will not be further
elaborated. The remaining five basic emotions are analyzed in detail, with brief references to handbooks of comics (e.g.
McCloud, 2006; Blair, 1994).

4.1. Anger

In our corpus, anger is represented with the combination of facial expression, touch, body orientation, linguistic and
visual symbols, and so forth. In terms of facial expression, anger is mostly represented by the facial expression of [inner
brow: lowered] (choice 6 in Fig. 4) and [mouth: open: tensed] (choice 12), which appear in 12 and 14 of the 16 images

Table 2
The distribution of emotions in the corpus.

Surprise Happiness Anger Sadness Fear Disgust

American 12 9 8 0 5 1
Japanese 11 8 8 7 3 0
 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2075

Fig. 8. The expression of Anger (a) (right character) and Happiness (a) (left character).

Fig. 9. The expression of Anger (b).

respectively. In Fig. 8 (the right character), the inner brow is not lowered, but the wrinkles form a ‘‘\ /’’ shape and achieve
the same effect. The shape of the mouth is more complex. According to Ekman and Friesen (1975:87), there are two basic
types of mouth shape in anger: lip-pressed-against-lip and squared open mouth, as in Figs. 8 and 9 respectively. In our
data, most depictions of anger select the latter expression, contradicting Martinec’s (2001) proposal of anger as a ‘‘closed’’
emotion. The reason may be that anger usually occurs during confrontation and thus involves speech or shouting. As
Ekman and Friesen (1975:87) point out, the open-mouth anger occurs during speech, when the person may be shouting
or verbally expressing anger. Therefore, complementing Forceville’s (2005) interpretation of the open mouth as releasing
‘‘metaphorical anger’’, some may be literally releasing the words.
One difficulty of analyzing anger expressions is distinguishing the mouth shape from that of other emotions, such as
surprise and fear. Anger can be distinguished by the tension of mouth, but tension is also an abstract concept that defies
straightforward visual representation. Three features are identified in the representation of tension. First, for closed mouth,
the mouth is tightly shut, thus forming wrinkles in the cheek area, as is shown in Fig. 8. Second, for open mouth, the upper
lip is pushed up; the lower lip is drawn back and forms a horizontal line, or may even be dragged down to form an upward
pointing arc, as in Fig. 9. As a result of the tension in the lips, both upper teeth and lower teeth may be completely revealed.
Third, in more abstract representations, the mouth has a rectangular shape, which appears in Japanese representations
such as Fig. 10, but not in American comics.
The resources of body orientation and touch also play an important role in representing anger. In terms of body
orientation, 9 out of 16 images employ an easy-to-recognize orientation of [forward]. In terms of touch, 5 images use
clenched fist as a pre-touch to signify ‘‘punch’’, as in Fig. 9. The gesture in Fig. 9 is further reinforced by the dots around the
man’s fist. The urge to punch and the accompanying strength of the action accord with the experience of anger, in which
‘‘there is a feeling of power and an impulse to strike out, to attack the source of anger’’ (Izard, 1977:331). Based on the
2076 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

Fig. 10. The expression of Anger (c).

eyebrow—inner corner—lowered (6)

eyes—wide open (8)

closed—tight (13)

mouth—tensed upper lip—raised

Anger open (12) position—lowered

Expressions lower lip horizontal line
shape
down pointing arc
contact part—fist
vector—towards patient
pre-touch (punch)
velocity—high
strength—high

body orientation—forward

Fig. 11. Systemic options for representing Anger.

analysis of the comic book data, the choices available for representing anger are displayed in Fig. 11. As explained in
section 3.1, the round brackets map out the total choices available. That is, the system simultaneously involves different
areas of the face, touch and body orientation, but not all of them may be utilized in the representation of anger.
The system is largely consistent with the proposals in ‘‘how to draw’’ handbooks for comics creators. For example,
according to Blair (1994:24), a prototypical angry character has fisted hands, clamped teeth, V-shaped heavy brows and
so on. In comparison, the systemic description documents the choices available to comic creators in a more
comprehensive and explicit way.

4.2. Happiness

Happiness is perhaps the easiest to represent among the basic emotions, as its facial expression is ‘‘universal and
easily recognized by all people’’ (Izard, 1977:241). In our data, happiness is mostly represented as [mouth: corners up],
which appears in 15 out of the 17 images and [eyes: narrowed/closed], which appears in 14 of the 17 images. The mouth
may be open or closed, but the smile or laughter is easy to recognize, as is shown in Figs. 8 (left character) and 12.
[Cheeks: raised] may also be employed and is represented by wrinkles in the cheek area, as in both Figs. 8 and 12.
Because of raised cheeks, the eyes are narrowed or closed. In more abstract representations, the eyes are reduced to two
upward-pointing crescents, or even reduced to two dots, as in emoticons. The intensity of happiness is primarily judged by
the wideness of the mouth. For example, leaving other factors aside, the character in Fig. 12 is perceived to be happier
 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2077

Fig. 12. The expression of Happiness (b).

eyes—narrowed (9)
facial expression cheeks—raised (11)
open
Happiness mouth—corners up (16)
closed
Expressions
body orientation—up

Fig. 13. Systemic options for representing Happiness.

than the character in Fig. 8. The choices of facial expressions are consistent with the advice given in comic handbooks.
For example, McCloud (2006:93) suggests that ‘‘the smiling muscle pulls the corners of the mouth up and out,
compressing cheeks which, together with the squinter, produces the arched eyes of joy’’.
The body orientation [vertical: up] is used in two images in our data. The ‘‘up’’ body orientation typically involves raised
head, upright torso and lifted arms. The most salient choices are summarized in Fig. 13.

4.3. Sadness

Sadness is mainly represented in the brow and the mouth, in accordance with psychological studies of real life facial
expressions (Ekman and Friesen, 1975; Izard, 1977). The facial expression may include [eyebrow: inner corner: raised]
and [mouth: corners down] as evidenced by the corpus of comics where these choices are made in 3 and 6 images
respectively from a total of 7 images which depict sadness. Figs. 14 and 15 are the prototypical examples. The choices are
consistent with McCloud (2006:93) who points out that in sadness ‘‘corrugator/brow lifter struggle over squinted eyes,
while the lip stretcher, triangularis and pouting muscle produces the sideways ‘8’ shape of the crying mouth’’.
The body orientation of [vertical: down] is also an important signifier of sadness, which may involve lowered head and
torso. In lowered torso, the back bends forward and downward as if there is no strength to support the body. Touch is not
found in the representation of sadness, presumably because sadness is a passive emotion. As Darwin (1872:176)
explains: ‘‘sad persons no longer wish for action, but remain motionless and passive’’. The typical choice configuration for
sadness is displayed in Fig. 16.

4.4. Surprise

Surprise is mainly represented by facial expression in our corpus. The most prominent choices are [eyes: wide open]
(choice 8) and [mouth: open: relaxed] (choice 14), which appear in 21 and 15 images respectively in the total 23 images
depicting surprise. In the handbook of comics, McCloud (2006:93) states that ‘‘the brow lifter pulls the upper face straight
up in surprise over wide open lid-lifted eyes, while the mouth falls open’’. Figs. 17 and 18 are the prototypical examples.
The raised eyebrow may result in [forehead: wrinkled: horizontal], as in Fig. 17. According to Ekman and Friesen (1975),
surprise itself is neutral, but it can be blended with other emotions such as anger and happiness. There are four cases of
happy surprise and one case of angry surprise in our data. In Fig. 18, the happiness is represented by [cheeks: raised],
and the surprise is registered in [eyes: wide] and [mouth: relaxed: open].
2078 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

Fig. 14. The expression of Sadness (a).

Fig. 15. The expression of Sadness (b).

4.5. Fear

The expression of fear may be confused with that of surprise, although the experience of these emotions is significantly
different (Izard, 1977:364). Therefore, it is important to examine how the two emotions are distinguished in visual
representation. In the comic book data, fear is characterized by body orientation and facial expression. The use of touch is
scarce, presumably because the tendency to touch in fear is low (with the possible exception of pushing behavior).
In terms of body orientation, fear is characterized by the tendency to avoid or escape from a dangerous or harmful
situation, while surprise may involve a backward or forward movement, or no movement at all (Ekman and Friesen,
 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2079

corners raised (5)

eyebrows
facial expression drawn together (7)

Sadness mouth—corners down (17)

Expressions
body orientation—down

Fig. 16. Systemic options for representing Sadness.

Fig. 17. The expression of Surprise (a).

Fig. 18. The expression of Surprise (b).

1975:47). Therefore, the body orientation of [horizontal: backward] is a useful dimension to distinguish fear from surprise.
In our data, 6 out of 8 expressions of fear involve avoidance posture (backward orientation), as in Fig. 19. The danger may
also result in protection or defense, so fear may also be expressed by pushing the danger away, or by covering the face
with hands.
2080 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

Fig. 19. The expression of Fear (a).

In terms of facial expression, the most prominent choices are [eyes: wide open] and [mouth: tensed: open], both
appearing in 7 out of the 8 images depicting fear. The differences between fear and surprise may be registered in the
eyebrows and the mouth.2 They may both involve [forehead: wrinkled: horizontal], [eyebrows: whole brow: raised], [eyes:
wide open] and [mouth: open], but in fear, the eyebrows are sometimes drawn together and the mouth is tensed. The
wrinkles in forehead tend to be at the center as in Fig. 20, or there may be vertical wrinkles between the eyebrows. The
mouth is tensed as in anger, making the lower lip a down pointing arc, as in Fig. 20. However, fear differs from anger in that
the mouth is not drawn back so much and the teeth are not completely revealed. The facial and postural distinctions
between fear and surprise are shown in Fig. 21.
Along with body orientation and facial expression, fear and surprise are also distinguished by visual symbols. As the
facial coding accuracy of fear is the lowest among all basic emotions (see Table 3), other resources (e.g. sweat drops) are
frequently used to disambiguate the emotion.

4.6. General findings

In what follows, the quantitative features of our data are discussed in relation to psychological findings of coding
accuracy and then some observations are made with respect to cultural differences between American and Japanese
representation of emotion in comics.
The facial areas chosen for emotion representation correspond to psychological findings of the coding accuracy of
particular actions. According to Ekman et al. (1971:50), certain facial areas provide better predictions than others for
particular emotions, as illustrated in Table 3. For example, happiness is best recognized in the lower face, so if only one
facial area is chosen in representing happiness, it is more likely to be the lower face. This is generally the case for most
emotions, as is clear from Table 4, which shows the total percentage of specific choices from the five items and the

2
According to Ekman and Friesen (1975), fear also involves tension in lower eyelids. However, the tension is too delicate to identify even in real
human beings, and no case of tension in eyelids is found in our data.
 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2081

Fig. 20. The expression of Fear (b).

drawn together (5)

eyebrows
facial expression 0
mouth tensed (12)
0
backward
body orientation
0

Fig. 21. The distinction between Fear and Surprise in representation.

Table 3
The coding accuracy of facial areas (adapted from 1; the percentages refer to the accuracy rates when basic emotions are judged based on the
expressions in the three facial areas).

Brows-forehead Eyes Lower face Combination of the three areas

Anger 80% 50% 100% 100%

Happiness 70% 90% 100% 100%
Sadness 70% 90% 0% 90%
Surprise 70% 90% 90% 100%
Fear 29% 71% 29% 43%

respective choices in American and Japanese choices. The exact choices made are not indicated in Table 4 since these
have been discussed in detail in the data analysis above. We can see from the ‘‘total’’ rows that the frequency a facial area
is used in emotion representation is largely consistent with its coding accuracy of a certain emotion in Table 3. However,
sadness seems to be an exception. According to Table 3, sadness is not recognized based on lower face only, but in
abstract representations such as emoticons and some comics, [mouth: corners down] (choice 17 in Fig. 4) is the only
feature employed, as in the emoticon ‘‘ ’’. This discrepancy may be due to the fact that ‘‘mouth corners down’’, as well as
‘‘mouth corners up’’, has become conventionalized symbols for certain emotions (e.g. the ‘‘smiley’’ for happiness).
Table 4 also includes the choice percentages of (pre-) touch and body orientation. It is clear that the frequency of choices
made from these resources is significantly lower than that for facial expression, indicating their lower signifying power
compared with facial expression. Meanwhile, although body orientation is chosen with a relatively high frequency, it is
normally accompanied by facial expression or touch. The three resources of facial expression, touch and body orientation, as
well as language, visual symbols and so on, work together to guarantee the accurate representation of emotive meaning.
2082 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

Table 4
Choice percentages of facial areas, (pre)touch and body orientation in emotion representation (the percentages represent the instances of a
particular emotive behavior (e.g. eyes) used against the total instances of the emotion).

Brows-forehead Eyes Lower face (Pre)touch Body orientation

Anger Total (16) 75% 63% 88% 38% 56%

American (8) 75% 63% 88% 38% 50%
Japanese (8) 75% 63% 88% 38% 63%
Happiness Total (17) 0 71% 76% 24% 18%
American (9) 0 78% 78% 22% 25%
Japanese (8) 0 63% 75% 25% 13%
Sadness Total (6) 57% 43% 86% 0 57%
American (0) 0 0 0 0 0
Japanese (6) 86% 86% 43% 0 57%
Surprise Total (23) 43% 91% 70% 0 0
American (12) 58% 92% 58% 0 0
Japanese (11) 27% 91% 82% 0 0
Fear Total (8) 13% 88% 88% 0 75%
American (5) 20% 80% 80% 0 80%
Japanese (3) 0 100% 100% 0 67%

In terms of cultural differences, Table 4 shows no remarkable difference between the choices of visual representation
in the American and Japanese comics. Although an accurate statistical comparison is impossible for such a small corpus
(especially when sadness is not found in the American comics), the margin between the two is mostly less than 10%,
which means only one or two different choices for each emotion. We can tentatively conclude that the distinction between
American and Japanese representations of emotive behavior is mainly stylistic. That is, similar choices from the semiotic
systems are made, but these choices are represented in different ways (e.g. the shape of the mouth in Figs. 8 and 10).
Applying our model in Fig. 2, we can say the differences exist at the iconic stage, rather than the indexical stage. This is in
accordance with psychological findings that the basic emotions and their behavioral expressions are universal as
biologically determined products of human evolution (Darwin, 1872; Ekman and Friesen, 1975; Izard, 1977). Another
contributing factor to the cross-cultural similarity could be that the global genre of comic strips has developed a common
set of semiotic conventions due to the foundational influence of American comic strips of the 1930s.
The location of the difference at the iconic stage (i.e. stylistic) is a significant step towards the comparison of cross-
cultural differences. For example, using black--white images and very creative sketches, Naruto’s representation of eyes
and mouth are more ideographic than naturalistic, as in Fig. 10. This allows Mangakas (manga authors) to focus on the
essential emotive meanings and communicate them in a simple and effective way. For example, the ‘‘\ /’’ shaped eyes
exaggerate the lowered inner corners of eyebrows and represent intense anger vividly. However, although the two
datasets represent American and Japanese comics to some extent, some of the differences may arise out of the
idiosyncratic styles of the artists and thus should not lead to unsubstantiated claims about cultural differences.
Nevertheless, systematic annotation and analysis can offer an effective method for cross-cultural comparison and the
current study represents only a first step in this direction.
Although beyond the scope of the present paper, two points with respect to cultural differences need to be emphasized.
First, although the bodily expression of emotion is universal, there are culturally distinct ‘‘display rules’’ which are socially
acquired (Ekman and Friesen, 1975). Relating to display rules, we find that in the American comic book most expressions
of emotion are displayed in the presence of other characters while in the Japanese one, many emotions are displayed
when the characters are alone. Second, American and Japanese comics differ in many other respects (e.g. use of
language, background setting, framing, clothes and so forth), indicating the need for a multimodal analysis which takes
into account the complete range of meaning making resources (see, for example, Shinohara and Matsunaka, 2009 for the
analysis of pictorial runes in western and Japanese comics).

5. Conclusion

This study examines how embodied semiotic resources are used to represent emotion in visual images. We first
theorize the semiotic status of visually represented behaviors as partial iconic representations of real life emotive
behaviors, which are in turn indexes of emotion. Following this, the meaning making resources of facial expression, touch
and body orientation are formulated as paradigmatic systems from a social semiotic perspective. The choices of facial
 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084 2083

actions can encode basic emotions with a high degree of accuracy, and the dimensions of touch and body orientation can
encode the valence, activation and intensity of emotions. Finally, the systems are used to analyze the representation of
emotion in comics. It is demonstrated that anatomizing embodied emotion resources from the social semiotic perspective
is useful for describing and understanding how emotive meaning is created in visual art.
Significantly, the systemic choices are discussed in relation to both psychological findings and comic handbooks. First,
the systematic analysis shows that choices made in comics are largely consistent with the signifying power (coding
accuracy) of the facial areas. Second, a brief survey of comic handbooks shows that the system networks correlate with
the formulations derived by comic book experts. However, the systemic description of the choices is more comprehensive
and easier for comic creators to refer to. Therefore, the systemic frameworks for emotive behavior provide a useful meta-
language for designing emotions (and the teaching of emotion representation) in creative works like paintings, comics,
computer vision, and related domains. For further work, if more elaborate system networks are proposed to model the
systemic choices of a wider range of semiotic resources in more detail, a more or less accurate ‘‘language’’ for designing
emotion in comics and digital animation may be developed. Through empirical studies, the coding accuracy of different
systemic choices in comics can be tested, adopting the method of coding accuracy test in psychological studies (e.g.
Ekman and Friesen, 1976 for facial expression; Banse and Scherer, 1996 for vocal features). Forceville’s (2011)
discussion of the recognition accuracy of pictorial runes in comics represents a significant step in this direction.
In terms of cross-cultural analysis, the study not only confirms the universal nature of bodily expressions of basic
emotions, but also locates cultural differences at the iconic stage of representation. However, a more convincing account
of cultural differences can only be achieved by a multimodal analysis approach which is based on the annotation of a
larger corpus on the one hand, and the examination of the full range of semiotic resources and the situational context on
the other. As such, the approach developed in this paper is a first step towards systematically describing the realization of
emotion, and more broadly, the way we construct and understand the world. It is hoped that the study will help launch
further thinking about the visual (and multimodal) representation of emotion in both traditional and new media, from the
perspectives of social semiotic analysis, cognitive analysis, cross-cultural comparisons, empirical studies, and so forth.

Acknowledgements

The research for this article was supported by Interactive Digital Media Program Office (IDMPO) in Singapore under
the National Research Foundation (NRF) (Grant Number: NRF2007IDM-IDM002-066). We wish to thank Shueisha, Inc.
for permitting us to use the manga images and we wish to thank Dr. Mie Hiramoto for her kind help and great patience in
contacting Shueisha, Inc.

References

Anderson, Peter, 2008. Nonverbal Communication: Forms and Functions. Waveland Press, Inc., Long Grove.
Argyle, Michael, 1975. Bodily Communication. Methuen, London.
Banse, Rainer, Scherer, Klaus R., 1996. Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology 70 (3),
614--636.
Bartlett, Marian, Hager, Joseph, Ekman, Paul, Sejnowski, Terrence, 1999. Measuring facial expressions by computer image analysis.
Psychophysiology 36, 253--263.
Birdwhistell, Ray, 1970. Kinesics and Context: Essays on Body Motion Communication. University of Pennsylvania Press, New York.
Blair, Preston, 1994. Cartoon Animation. Walter Foster, Laguna Hills.
Burgoon, Judee, Buller, David, Woodall, Gill, 1996. Nonverbal Communication: The Unspoken Dialogue. McGraw-Hill, New York.
Darwin, Charles, 1872 (1965). The Expression of Emotions in Man and Animals. University of Chicago Press, Chicago.
Eerden, Bart, 2009. Anger in Asterix: the metaphorical representation of anger in comics and animated films. In: Forceville, Charles, Urios-Aparisi,
Eduardo (Eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin, pp. 243--264.
Ekman, Paul, Friesen, Wallace, 1975. Unmasking the Face: A Field Guild to Recognizing Emotions from Facial Clues. Prentice-Hall, Englewood
Cliffs, NJ.
Ekman, Paul, Friesen, Wallace, 1976. Measuring facial movement. Environmental Psychology and Nonverbal Behavior 1, 56--75.
Ekman, Paul, Friesen, Wallace, 1978. Facial Action Coding System. Consulting Psychologists Press, Palo Alto, CA.
Ekman, Paul, Friesen, Wallace, Tomkins, Silvan, 1971. Facial emotion scoring technique: a field study. Semiotica 3 (1), 37--58.
Forceville, Charles, 2005. Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics
37, 69--88.
Forceville, Charles, 2011. Pictorial runes in Tintin and the Picaros. Journal of Pragmatics 43, 875--890.
Forceville, Charles, Veale, Tony, Feyaerts, Kurt, 2010. Balloonics: the visuals of balloons in comics. In: Goggin, Joyce, Hassler-Forest, Dan
(Eds.), The Rise and Reason of Comics and Graphic Literature: Critical Essays on the Form. McFarland, Jefferson, NC, pp. 56--73.
Gorelick, Victor, 2010. Archie’s Pals and Gals, vol. 142. Archie Comic Publications, Inc., New York.
Hall, Edward, 1959. The Silent Language. Anchor/Doubleday, Garden City.
Halliday, Michael, 1978. Language as Social Semiotic. Arnold, London.
Halliday, Michael, Matthiessen, Christian, 2004. An Introduction to Functional Grammar, 3rd ed. Arnold, London.
2084 D. Feng, K.L. O’Halloran / Journal of Pragmatics 44 (2012) 2067--2084

Izard, Carroll, 1977. Human Emotions. Plenum Press, New York.

Kennedy, John, 1982. Metaphor in pictures. Perception 11, 589--605.
Kishimoto, Masashi, 2000. Naruto. Shueisha, Inc., Tokyo.
Kockelman, Paul, 2005. The semiotic stance. Semiotica 4 (1), 233--304.
Kövecses, Zoltán, 1986. Metaphors of Anger, Pride and Love. John Benjamins, Amsterdam/Philadelphia.
Kövecses, Zoltán, 2000. Metaphor and Emotion: Language, Culture, and Body in Human Feeling. Cambridge University Press, Cambridge.
Kress, Gunther, van Leeuwen, Theo, 2006. Reading Images: The Grammar of Visual Design, 2nd ed. Routledge, London.
Lakoff, George, Johnson, Mark, 1980. Metaphors We Live By. University of Chicago Press, Chicago.
Margritte, Rene, 1928--1929. The Treachery of Images. http://en.wikipedia.org/wiki/The_Treachery_of_Images.
Martinec, Radan, 2001. Interpersonal resources in action. Semiotica 135 (1/4), 117--145.
McCloud, Scott, 1993. Understanding Comics. Harper, New York.
McCloud, Scott, 2006. Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. Harper, New York.
Mehrabian, Albert, 1981. Nonverbal Communication. Aldine-Atherton, Chicago.
O’Halloran, Kay, 2005. Mathematical Discourse: Language, Symbolism and Visual Images. Continuum, London.
O’Toole, Michel, 2011. The Language of Displayed Art, 2nd ed. Routledge, London.
Ozawa, Tadashi, 2005. Let’s Draw Manga: Bodies and Emotions. Diamond Comic Distributors, Timonium.
Peirce, Charles, 1931--1958. Collected Papers. Harvard University Press, Cambridge.
Planalp, Sally, 1998. Communicating emotion in everyday life. In: Peter A., Anderson, Laura K., Guerrero (Eds.), Handbook of Communication
and Emotion. Academic Press, San Diego, pp. 29--48.
Richmond, Virginia, McCroskey, James, Hichson, Mark, 2008. Nonverbal Behavior in Interpersonal Relations. Pearson Education, Inc., Boston.
Russell, James, 1980. A circumplex model of affect. Journal of Personality and Social Psychology 39, 1161--1178.
Saraceni, Mario, 2003. The Language of Comics. Routledge, London.
Sebeok, Thomas, 2001. Signs: An Introduction to Semiotics. University of Toronto Press, Toronto.
Shinohara, Kazuko, Matsunaka, Yoshihiro, 2009. Pictorial metaphors of emotion in Japanese comics. In: Forceville, Charles, Urios-Aparisi,
Eduardo (Eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin, pp. 265--293.
Thibault, Paul, 2004. Brain, Mind, and the Signifying Body: An Ecosocial Semiotic Theory. Continuum, London.

Dezheng Feng completed his PhD in the Multimodal Analysis Lab, Interactive Digital Media Institute and the Department of English Language
and Literature at the National University of Singapore. He is currently a Visiting Research Scholar in the Department of English at the Hong Kong
Polytechnic University. His research interests include the critical analysis of multimodal discourse, social semiotic theory and cognitive linguistics.
He has recently published ‘‘Visual Space and Ideology’’ in the book Multimodal Studies: Exploring Issues and Domains by Routledge and
‘‘Intertextual Voices and Engagement in TV Advertisements’’ in the journal Visual Communication.

Kay L. O’Halloran is Director of the Multimodal Analysis Lab in the Interactive Digital Media Institute and Associate Professor in the Department of
English Language and Literature at the National University of Singapore. Her main research areas include a social semiotic approach to
multimodal discourse analysis with a particular interest in mathematics and scientific texts, and the development of interactive digital media
technologies for multimodal analysis of (multimedia) data. Kay L. O’Halloran is the Principal Investigator for several large projects in the
Multimodal Analysis Lab IDMI. Further information is available at http://multimodal-analysis-lab.org/.