CONTRASTIVE

ANALYSIS OF
MULTIMODAL HUMOUR
LN400: Fall 2022
Humour
◦ A basic dictionary definition of humour is “the quality of being amusing or comic;
the ability to amuse other people; or a state of the mind” (Oxford Online Dictionary).
◦ Humour is a socio-cognitive activity, detected in the various forms of interaction in
diverse social contexts. Humour has gained the attention of scholars of different
disciplines
◦ From a sociological perspective, humour has various interestingly conflicting social
functions; as a source of mirth and positive feelings, humour may function as an
affable means of in-group solidarity and cohesiveness, while at other times manifest
coercive antagonism. Humour is greatly purposeful in contexts where serious and
direct mode of communication bears the risk of being provocatively confrontational.
 
Incongruity Theory
◦ Within the cognitive framework of the Incongruity Theory, humour results of
sensory/cognitive perception of an incongruous situation violating our mental
configurations and expected models. In the Rhetoric (3, 2,), Aristotle claims that a
speaker can create humour by violating an expectation that is created for the audience.
James Beattie (1779) maintains that the cause of laughter is “two or more inconsistent,
unsuitable, or incongruous parts or circumstances, considered as united in one
complex object or assemblage, as acquiring a sort of mutual relation from the peculiar
manner in which the mind takes notice of them” (p. 320).
◦ For Schopenhauer (1818/1844 [1907]), humor arises when we suddenly notice the
incongruity between a concept and a perception that are supposed to be of the same
thing.
General Theory of Verbal Humour (GTVH)
◦ Based on the cognitive theories of incongruity, inconsistency, or contradiction between two planes of
thoughts, humour is examined as a linguistic activity in Attardo & Raskin’s (1991) General Theory of
Verbal Humour (GTVH) providing a linguistic framework of humour. A text is potentially humor-
carrying when it demonstrates the following dimensions or Knowledge Resources (KRs)
1. Situation: the original context; backdrop of the joke, i.e. contextual underpinnings of the joke
2. Narrative strategy: the overall textual genre, e. g., riddle, conversation, humorous short story, memes
3. Script oppositeness/Script overlap: refers to the process of combining scripts in a way that stretches
of text “are incompatible with one another. A “trigger”, the linguistic/non-linguistic element that links
the two scripts manifesting opposition or overlap.
4. Language: the actual textual artefact, whether spoken, written, mimed, gestured, expressed through
dress, etc., with all the choices at the relevant levels of linguistic analysis, e. g., syntax, phonology,
lexis, rhetoric…etc
5. Target: the usually stereotypical butt of the joke; the individuals, social groups or social practice
selected as subjects for ridicule and critique
Humour & Multimodality Combined
◦ In cartoons, memes, and comics, meaning and humor are produced either via two semiotic modes, the
verbal and the visual, or solely via the visual mode. Due to their condensed form and to the interaction
between language and image, caricatures and memes can communicate extremely relevant messages, with
or without words. They are often considered to be a direct and easy means of communicating highly
vindictive messages which can be accessed by different people.
◦ The readers’/viewers’ interpretations of the messages in cartoons and memes rely on their previous
cognitive and social context. The underlying bitter messages are processed and contrasted against their
background knowledge, beliefs and assumptions.
oMemes and caricatures are multimodal texts where multiple modes of communication contribute to an
audience's understanding of a composition. Everything from the placement of images to the organization
of the content to the method of delivery creates meaning. This is the result of a shift from isolated text
being relied on as the primary source of communication, to the image being utilized more frequently in
the digital age (replacing monomodality).
oMultimodality describes communication practices in terms of the textual, aural, linguistic, spatial, and
visual resources used to compose messages.
Kress and van Leeuwen’s
framework (2006)
o Visual elements are perceived as factors that express
people’s personalities and attitudes alongside verbal
communication.
o Kress and van Leeuwen (2006) highlight the
significance of placing pictures in compositions
(similar to language).
o Words and pictures are related, but pictures do not
have to rely entirely on words to be understood.
o The tools which Kress and van Leeuwen use to make
sense of the meaning of pictures are based on
Halliday’s (1985) metafunctions. This framework is
threefold:
Outline

The Representational
Structure

The Interactional
Structure

The Compositional
Structure
 Structures

Representational Interactional Compositional

Informational
Processes Contact
value

Participants Social distance Salience

Circumstances Attitude Framing

 1. THE
REPRESENTATIONAL
STRUCTURE

Processes
Participants
Circumstances
The Representational Structure
oImages can be still or moving. Kress and van Leeuwen (2006)
offer diverse tools to analyze these types of images.
oThe representational meaning is concerned with the relation
between entities and their cultural representation in the world.
oThis relation includes participants, processes, objects, and
circumstances.
oThis structure is categorized into narrative and conceptual
processes. The main difference between the two is the
presence or absence of a vector.
Narrative Processes
oThe narrative process takes place when participants are linked with each other
through a vector.
oIn this regard, participants are depicted in the process as doing an act.
oNarrative processes include events and processes.
oKress and van Leeuwen emphasize the importance of vectors for the narrative
process to come into existence, unlike conceptual structures that do not
require the presence of a vector.
oThe presence of vectors indicate that participants are doing something for
each other.
oThe vector that links between two persons can be achieved through eye
connection, a body part, or a tool.
 Narrative
process

Action Reaction
process process
First: Action process
oIn action processes, the Actor is a participant, which either
originates the vector or is the vector itself.
oThey are usually the most noticeable contributors to the
communication “through size, place in the composition, contrast
against background, color saturation or conspicuousness, sharpness
of focus, and through the ‘psychological salience’ which certain
participants) have for viewers”
oIn the case of having only one participant, the construction is then
called non-transactional. The process has no Goal because it is not
directed towards a certain entity.
oThe action process can be either transactional or non-transactional.
oTransactional process is further categorized into unidirectional or
bidirectional.
oThe process is unidirectional transactional when the vector is
made from a line joining two participants. These participants are
either Actors or Goals. Actors are the source of vectors; Goals
are the participants to which the vector is aimed.
oThe bidirectional transactional process takes place when two
vectors are linking participants, with each of the participants
pointing vectors at each other.
oAs for non-transactional action process, they have vectors
pointing from an Actor but do not have Goals.
 Action
process

Non-
Transactional
transactional

Bidirectional

Unidirectional
Second: Reaction process
o As for the second form of narrative processes, it is called the reaction
process, in which “the vector is formed by an eyeline, by the direction
of the glance of one or more of the represented participants” (p. 67).
o This process is further divided into two subcategories: transactional
reaction and non-transactional reaction.
o First, transactional reaction has perceived vectors; it is represented in
the eyeline that unites two participants.
o Second, non-transactional reaction takes place in the case of the
presence of a participant and a vector, but there is no receiving
participant or phenomenon that the vector is directed towards.
oWhen the vector is formed by an eyeline, by the direction of
the glance of one or more of the represented participants, the
process is reactional, and we will speak not of Actors, but of
Reacters, and not of Goals, but of Phenomena.
oThe Reacter, the participant who does the looking, must
necessarily be human, or a human-like animal – a creature
with visible eyes that have distinct pupils, and capable of
facial expression.
oThe Phenomenon may be formed either by another
participant, the participant at whom or which the Reacter is
looking, or by a whole visual proposition, for example, a
transactional structure.
 Reaction
process

non-
transactional transactional
Circumstances
o It is related to inconsequential participants that do not affect the action taking
place.
o The types of Circumstances are Locative Circumstances, Circumstances of
Means, and Circumstance of Accompaniment.
o The first one is Locative or circumstance of setting which is associated with
foregrounds and backgrounds.
o As for Means, when there is no vector between users and their tools.
o As for Accompaniment, it has no vector, but it has several participants who
share a relationship.
o These different types of circumstances affect the visual layout and the meaning
behind it.
The conceptual process
oThe conceptual process is different from the narrative
process in many aspects.
oIt does not require the presence of a vector and is more
general in nature.
oIt is an action-free process that depicts participants in a
static manner.
oConceptual representations have three separate
subcategories: Classificatory, Analytical, and Symbolical.
oFirst, the Classificatory process is related to taxonomy or
hierarchical classification. In other words, one participant
plays the role of the Superordinate while others occupy the
role of his Subordinates.
oSecond, the Analytical process revolves around a part-whole
structure. Two roles are to be fulfilled here: Carrier and
Attribute. In case of Carriers, participants take the role of
whole, while Attributes take the role of part.
oThird, the Symbolic process. It does not try to find
relationships between concrete objective images, but seeks to
establish what the images mean.
Conceptual Processes - Symbolic
Conceptual Processes

Classificatory Process Analytic Process

 2. THE
INTERACTIONAL
(INTERACTIVE,
INTERPERSONAL)
STRUCTURE

Contact
Social distance
Attitude
Contact
o “Contact is related to gaze; when represented participants look directly at the viewer’s eyes,
vectors formed by participants’ eyes connect the participants with the viewer”(p. 122).
o Contact has two divisions: demand and offer.
o In case of direct gaze towards the audience, familiarity and closeness emerge. “The participants
gaze demands something from the viewer, demands that the viewer enter into some kind of
imaginary relation with him or her” (p. 122).
o On the other hand, contact is further subcategorized to include Offer. When the participant does
not direct any gaze to the audience, so this results in lack of involvement between the two
parties. This means that the participants are depicting information and are being offered to the
audience
o Gaze can be controlled or sometimes it can be accidental, and this depends on how structured
the conversation is, the interactional arrangement of the gaze controls the conversation; “the
gaze of represented participants directly addresses the viewers and so establishes an imaginary
relation with them, while more schematic analytical pictures invite impersonal, detached
scrutiny” (p. 118). This imaginary relation or lack thereof affects the connection between the
two participants: the perceived and the audience.
Social Distance
o Social distance is related to the distances of shots in an image.
o The aim behind this distance is to allow the audience to differentiate between different
types of participants such as partners, companions, acquaintances, or complete strangers.
o Close shots demonstrate closeness, which is shown by displaying heads and shoulders of
participants.
o Medium shots include the knees of participants, which results in an increasing sort of
distance.
o As for long shots, they demonstrate public distance, and they show the full height of the
participant.
o This structure is categorized to intimate distance, close personal distance, far personal
distance, close social distance, far social distance, and public distance.
o It is related to the gap that people use in respect to other people’s and objects’ space.
Attitude
o Humans’ posture can be attributed to the way they perceive themselves, relationship
to others, and their attitude towards a person or a situation.
o This notion is related to the awareness that forms the image of participants.
o To decide on attitude, two pieces of information are needed; frontal plane and angle.
o Frontal planes of participants are either in parallel to those of the viewers’ or not.
o When the character faces the audience, the sense of involvement can be inferred.
The first case means connection, the latter means that participants and viewers are
detached and do not belong to the same world.
o As for angle, it is either high, low, or at eye level. High angles empower
participants, but low angles empower viewers because they look down on the
characters. However, eye level angles denote an equal distribution of power between
the two parties.
 3. THE
COMPOSITIONAL
STRUCTURE

Information value
Salience
Framing
The Compositional Structure

This structure is related to the “way in which

the representational and interactive elements
are made to relate to each other, the way they
are integrated into a meaningful whole” (p.
176).
Information Value
• Information value is related to the location of participants in a frame. The positioning of participants is
influential in terms of meaning making.
• A participant placed to the left is a Given, as they represent information that is not novel to the audience.
• On the other hand, participants that are placed to the right are New, as they offer new information.
• Top is Ideal, and bottom is Real.
• Also, the Center is more important than the Margins.
• “Ideal and Real and Centre and Margin are often the most significant compositional dimensions in three-
dimensional visual composition. Architecture provides perhaps the clearest example” (p. 256).
• Multidimensional entities increase further measurements to three-dimensional structure, and they have “front
and back, and the left and right side, an interior center and the exterior,” which enriches analysis (p. 256).
• The layout dominates the organization of people or furniture in a certain area. A certain way of organization
of people as they sit or that of furniture or any other props is governed by certain semiotic rules. Entities that
share a close space belong together, and if they are apart then they do not exist in the same environment or
space.
Salience
• Salience is a system that tackles the level of attractiveness of a given component in a frame.
• It is defined as “the degree to which an element draws attention to itself, due to its size, its place in
the foreground or its overlapping of other elements, its color, its tonal values, its sharpness or
definition, and other features” (Kress & van Leeuwen, 2006, p. 210).
• According to Kress and van Leeuwen, salience generates a ranking in a relationship between
people in a frame:
• Regardless of where they are placed, salience can create a hierarchy of importance among the
elements, selecting some as more important, more worthy of attention than others. The Given may
be more salient than the New, for instance, or the New more salient than the Given, or both may be
equally salient. And the same applies to Ideal and Real and to Centre and Margin.
• Salience is related to the visual importance of participants in an image. It is concerned with how
noticeable against a background a participant is, and it is also concerned with other items that are
positioned in the foreground. Salience also revolves around the level of eye catchiness of
participants.
Framing
• Framing is a term used to describe elements or clusters of elements
within an arrangement, whether they are disconnected from each other
or coupled together.
• According to Kress and van Leeuwen, framing takes many shapes or
degrees.
• Structures within a frame could be either intensely or faintly framed.
• The more intense the entity is depicted, the more separated it is in
terms of framing. On the other hand, “[t]he absence of framing
stresses group identity, its presence signifies individuality and
differentiation” (p. 42).
Practice
Practice