Professional Documents
Culture Documents
Visual Semiotics
M Danesi, Victoria College, University of Toronto, have great difficulty in carrying out such imaginary
Toronto, Canada rotations. People imagine faces and voices, locate
ß 2006 Elsevier Ltd. All rights reserved. imaginary places, scan game boards (like a checker
board) in their minds, arrange furniture in a room,
design a blueprint, and so on in similar, but signifi-
Introduction cantly differentiated ways (Kosslyn, 1983, 1994).
The study of visual signs and their representational Psychological studies have shown that images also
functions became a major branch of semiotics after vary along cultural lines. When asked to visualize a
the publication of two influential books: Rudolf triangle or a cat, people living in the same culture will
Arnheim’s Visual thinking (1969) and Jonathan come up with very similar mental pictures of the two
Berger’s Ways of seeing (1972). Visual signs are de- referents. These isomorphic images are known as
fined, simply, as signs that are constructed with a vis- ‘cultural prototypes’ (for the concept of ‘prototype’
ual signifier (form), i.e., with a signifier that can be in psychology, see Rosch, 1973, 1975, 1981, Rosch
seen, rather than heard, touched, tasted, or smelled. and Mervis, 1975, Taylor, 1995). The triangle that
Visual semiotics is a growing and robust area of people living in Western culture typically imagine is
inquiry, which includes the study of art forms, logos an equilateral one, because it is the type of triangle
in advertising, maps, diagrams, pictographs, alphabet that is perceived as the prototypical or basic form.
characters, and other visual artifacts and symbols. It is the one that is perceived to be exemplary or
Among the relevant works in this field are those by representative of all triangles. Obtuse-angled, right-
Barthes (1977), Sonesson (1987, 1994), Saint-Martin angled, and acute-angled triangles are perceived to be
(1991), and Sebeok and Umiker-Sebeok (1994). subtypes of the equilateral triangle. Similarly, the
A convenient classificatory framework of visual image of the cat that comes to the mind of subjects
signs can be found in Santaella-Braga (1988). living in Western culture is that of a household cat,
because it is the most typical in that culture.
Images are not generated only by visual perception.
Mental Imagery They can be elicited through other sensory and affec-
The semiotic study of visual signs overlaps consider- tive modalities. These modalities can be easily experi-
ably with the study of mental imagery within psychol- enced by imagining such things as: (1) the sound of
ogy. An image is defined simply as a mental picture of thunder, (2) the feel of wet grass, (3) the smell of fish,
objects, events, feelings, actions, situations, and ideas (4) the taste of toothpaste, (5) the sensation of being
(real or otherwise). Images allow us to recall events, uncomfortably cold, (6) the sensation of extreme
refer to them in various ways, and to generate hypo- happiness. The image that comes to mind in the case
thetical scenarios of situations or conditions that we of (1) has an auditory quality to it, (2) a tactile one,
may not even have actually experienced. Mental (3) an olfactory one, (4) a gustatory one, (5) a bodily
images are substitutes for real things, allowing people one, (6) an emotional one. Moreover, images are not
to recall, plan for, and predict things (Kosslyn, 1983, elicited solely by sensory modalities; they can be ab-
1994). But these are not monolithic. Individuals differ stract, fictitious, or narrative. As an example of an
widely in their abilities to construct images and use abstract image, think of love. The image that this
different aspects of imagery. For example, some peo- concept evokes is tied to some personal experience,
ple are better than others at moving objects around in such as the expression of a face in love. A fictitious
their heads. They can visualize, say, the letter N chan- image is something that can be made up in the imagi-
ging into a Z when rotated to a certain point. Others nation. An example of a fictitious image is a ‘winged
442 Visual Semiotics
Image Schemas with that idea; you’ll just have to turn back’; and
so on and so forth. Image schemas are, clearly, crucial
Ferdinand de Saussure (1857–1913), one of the mod- in abstract concept-formation, linking bodily and
ern-day founders of semiotics, used the word image in perceptual experiences.
two senses in his theory of the sign (Saussure, 1916:
16). He defined a signifier (such as a word) as a
‘sound image’ and its signified (meaning) as the ‘con-
Visual Representation
ceptual image’ that the word calls to mind. Recent Drawing pictures, making charts, sketching dia-
work on the conceptual imagery associated with ver- grams, and the like are such common things that we
bal forms has shown that Saussurean images are hardly ever realize what they involve. They are part of
‘schematic’ (Johnson, 1987; Lakoff, 1987; Lakoff an innate semiotic competence that allows people to
and Johnson, 1980, 1999). This has led to the devel- utilize a sensory modality, such as sight, to represent
opment of image schema theory within linguistics (see the world through signs.
Cognitive Linguistics). Image schemas are defined as Visual representation has been studied under vari-
unconscious mental outlines of recurrent shapes, ous rubrics both within semiotics and in other disci-
actions, dimensions, orientations, objects, etc. Al- plines. The focus in semiotics is on the use of
though they are unconscious, their ‘reality’ can be elemental signifiers (forms) such as points, lines, and
easily ascertained by prompting subjects in specific shapes (Dondis, 1986) that are combined in various
ways. For example, if someone were asked to explain ways to represent figures. Consider, for instance,
the expression ‘I’m feeling up today,’ it is unlikely that what can be accomplished with three straight lines.
he or she would describe an image involving an up- These can be joined up in specific configurations to
ward orientation. However, if the subject were asked represent by resemblance a triangle, the letter ‘H’, or
‘How far up do you feel?’ ‘What do you mean by up?’ a picnic table (among other things) (see Figure 1).
etc., then the subject would start visualizing the sche- The use of visual signifiers in this way is called,
ma in question and provide appropriate description more specifically, iconic representation (see Iconicity:
of the image schema. Theory). If arrowheads are adjoined to lines, they
According to Lakoff and Johnson (1980, 1999), can be used as signifiers to represent movement and
such schemas guide the formation of common con- direction (see Figure 2).
cepts. As an example, consider the image schema of This is an indexical use of signifiers for spatial and
an impediment. An impediment is something, such as orientational representation (see Indexicality: Theory).
a wall, a boulder, another person, etc., that blocks Shapes are visual signifiers designed to represent the
forward movement. We know from actual experience outline of something. For example, to show the shape
that we can go around an impediment, over it, under of the moon, we can draw a circle; to represent the
it, through it, or else remove it and continue on. On surface of a table we can draw a rectangle; and so
the other hand, the impediment could successfully on. Virtually anything that can be perceived visually
impede us, so that we would have to stop at the can be represented by a combination of points, lines,
impediment and turn back. All of these actions can and shapes: e.g., in drawings of scenery, a cloud is
be easily imagined in the mind. These are derivatives represented as a shape, a horizon as a line.
of familiarity with actual impediments. Now, as Other visual signifiers include value, color, and
Lakoff and Johnson have persuasively argued, the texture. Value refers to the darkness or lightness of a
very same image schema is used unconsciously to line or shape. It plays an important role in portraying
form abstract concepts. It is the psychological reason dark and light contrasts. Color conveys mood, feel-
why we say such things as: ‘We got through that ing, and atmosphere in a visual text such as a paint-
difficult time’; ‘Jim felt better after he got over his ing. This is why we speak of ‘warm,’ ‘soft,’ ‘cold,’ or
cold’; ‘You want to steer clear of financial debt’; ‘harsh’ colors. Color, as will be discussed below, has
‘With the bulk of the work out of the way, he was culture-specific meanings: e.g., in American culture
able to call it a day’; ‘The rain stopped us from yellow connotes cowardice, while in China it con-
enjoying our picnic’; ‘You cannot go any further notes royalty. Texture refers to the phenomenon
Visual Semiotics 443
that certain visual forms (lines, shapes, etc.) evoke winkey ;-) (meaning ‘I’m winking and grinning at
tactile and other sensory modalities. Wavy lines, for the joke here’), :-( (‘I’m sad about this’), :–7 (‘I’m
example, tend to produce a much more pleasant sen- speaking with tongue in cheek’), :D or :-D (big
sation in a viewer than do angular ones. By increasing smile; ‘I’m overjoyed’), and :-O (either a yawn of
the number of edges on angular lines, the unpleasant boredom or a mouth open in amazement). In an
sensation tends to increase proportionately. This is e-mail message or newsgroup article, a letter, word,
strong evidence that visual representation is inter- or phrase that is encased in angle brackets, indicating
modal, eliciting other sensory modalities in tandem. the attitude the writer takes toward what he or she
The term that is used to characterize this phenom- has written, is called an emotag. Often emotags have
enon is synesthesia. It is defined as the phenomenon opening and closing tags, similar to HTML tags, that
in which the stimulation of one sense elicits a percep- enclose a phrase or one or more sentences. For exam-
tion that ordinarily would be elicited had another ple: <joke>You didn’t think there would really be a
sense been stimulated, as when a loud noise registers joke here, did you?</joke>. Some emotags are used
as a blinding light, or vice versa. as single tags, such as <grin>.
Color Signifiers
Digital Visual Signs
As a signifier in visual representations color has be-
Today, elemental visual signifiers have found a new come a target of much study, within both semiotics
medium of representation – the digital one (Horn, and linguistics (e.g., Hatcher, 1974; Hilbert, 1987;
1998; Darley, 2000). Digital visual signs, such as the Davidoff, 1991; Hall, 1992; Hardin and Maffi,
icon that designates a tiny picture on a computer 1997; MacLaury, 1997; Tufte, 1997). At a perceptual
screen, are proliferating. Each icon represents a (concrete) level, we interpret colors as gradations of
command. The system of icons, pointer, and mouse hue on the light spectrum. Hue is the property that
is known as a graphical user interface (GUI), a system leads us to give a color its name – for example, red,
that provides a user-friendly way of interacting with a orange, yellow, green, blue, or violet. But the naming
computer. Users can usually tell by the icons how to process is hardly free of cultural factors. The actual
get the computer to do what they want. Without a color terms we use in English predispose us to see
GUI, the computer screen is black, and the only way ‘differential categories’ of hue. Experts estimate that
to tell the computer what to do is to type in com- we can distinguish perhaps as many as 10 million
mands. There is little doubt that GUIs contributed to hues. Obviously, then, our limited number of color
the rise of the personal computer in the mid-1980s, terms is far too inexact to describe accurately all the
starting in 1984 when the Apple Computer Company hues we are potentially capable of perceiving. The
introduced the Macintosh, the first personal computer restrictions imposed on color perception by color
to include a GUI. Because they make computers easy vocabularies are the reasons why people often have
to use, GUIs quickly became standard throughout the difficulties trying to describe or match a certain color.
computer industry. Today, most users encounter only In a real sense, color is what the color terms in a
GUI-based programs and never have to type in com- language say it is. Consider what happens by putting a
mands to control their computers. finger at any point on the spectrum. Only a negligible
Another common digital visual sign is the emoticon difference in the hues immediately adjacent to the
(literally an icon that conveys an emotion visually). finger at either side would be perceived by a viewer.
Emoticons are strings of text characters that, when Depending on where one puts it, however, perception
viewed sideways, form a face expressing a particular of the difference will vary. This is because we have
emotion. Common emoticons include the smiley :-) become accustomed to ‘seeing’ the spectrum in terms
or :) (meaning ‘I’m smiling at the joke here’), the of the color terms of the language we speak. But there
444 Visual Semiotics
Figure 4 A three-dimensional figure represented on a two- Perspective drawing is, in a phrase, the technique
dimensional surface. of creating an illusion of depth or length in two-
dimensional surface drawings. As a historical foot-
note, it should be noted that the craft of perspective
in fact, that sensory and emotional meanings attached drawing dates back to the Renaissance, after the Italian
to colors may even have been the source for the color artist Filippo Brunelleschi (1377–1446) developed the
terms themselves (Wescott, 1980). In Hittite, for in- technique in Western painting.
stance, words for colors initially designated plant and There are many and varied types of visual texts,
tree names such as poplar, elm, cherry, oak, etc.; in from paintings and drawings to maps, blueprints,
Hebrew, the name of the first man, Adam, meant ‘red’ comics, and the like, that have become targets of
and ‘alive,’ and still today, in many languages, red specialized areas of semiotics. Some of these, such
signifies ‘living’ and ‘beautiful.’ In all societies, colors as comic books, involve the use of both verbal
play a critical function in the realm of symbolism. The and visual techniques, producing very effective texts
Navajo of North America, for instance, allocate to (see Semiotics of Architecture; Cartography: Semiot-
colors a hierarchy of symbolic importance – blue is ics; Comics: Semiotic Approaches; Photography:
‘good’ and red ‘bad’; nation societies perceive great Semiotics; Pictography: Semiotic Approaches).
significance in the colors of flags and national Visual texts have an effect on perception, and can
emblems; and the list could go on and on. even mislead it. As a concrete example, consider the
classic illusion in which the line AB appears to be
longer than line CD (see Figure 6).
Visual Texts
In actual fact the lines are equal in length, but the
A text is anything that has been put together orientation of the arrowheads fools the eye into see-
with signs to represent something. Lines and shapes, ing AB as longer than CD. In some areas of the world,
for example, can also be combined to create three- however, psychologists have found that people see the
dimensional figures, which are visual texts that create lines as equal in length. The factor behind this illusion
an illusion of depth in the viewer. In the visual text of is thus cultural, not biological – visual perception is
Figure 4 there are 12 lines. The way they are put intrinsically intertwined with representational prac-
together, however, makes us perceive the text as a tices. Psychologists have found that people living in
three-dimensional box (see Figure 4). cultures that have never been exposed to illustrated
Although the figure has been drawn on a two- magazines are unable to recognize the photographs in
dimensional surface, we cannot help but perceive it the magazines as visual representations of human
as a three-dimensional box. The reason for this is beings (Deregowski, 1972; Dunning, 1991). They
that, as a visual text, it has been drawn with the tend to perceive them, rather, as smudges on the
technique of ‘perspective drawing.’ This deceives us, paper. Such erroneous interpretation of the photo-
in effect, to view objects as the human eye perceives graphs is not due to defects of intelligence or eyesight;
them. The flat surface of any drawn figure is known on the contrary, the individuals are clear-sighted
as the picture plane; the horizon line is called the and highly intelligent. Their primary assumptions
horizontal eye-level line that divides the scene in are different, because they have acquired a different
the distance; and the vanishing point is located on system of visual representation that blocks them from
the horizon line where parallel lines in the scene perceiving the photographs in the same manner that
appear to converge. This makes certain parts of the those accustomed to viewing magazines perceive
figure appear to be closer than others (see Figure 5). them.
446 Visual Semiotics
Rosch E (1975). ‘Cognitive reference points.’ Cognitive Sebeok T A & Umiker-Sebeok J (eds.) (1994). Advances in
Psychology 7, 532–547. visual semiotics. Berlin: Mouton de Gruyter.
Rosch E (1981). ‘Prototype classification and logical classi- Sonesson G (1989). Pictorial concepts: inquiries into the
fication: the two systems.’ In Scholnick E (ed.) New semiotic heritage and its relevance for the analysis of
trends in cognitive representation: challenges to Piaget’s the visual world. Lund: Lund University Press.
theory. Hillsdale, N.J.: Lawrence Erlbaum Associates. Sonesson G (1994). ‘Pictorial semiotics, Gestalt psychology,
73–86. and the ecology of perception.’ Semiotica 100, 267–
Rosch E & Mervis C (1975). ‘Family resemblances.’ Cogni- 331.
tive Psychology 7, 573–605. Taylor J R (1995). Linguistic categorization: prototypes in
Saint-Martin F (1990). Semiotics of visual language. linguistic theory. Oxford: Oxford University Press.
Bloomington: Indiana University Press. Tufte E R (1997). Visual explanations: images and quanti-
Santaella-Braga M L (1988). ‘For a classification of visual ties, evidence and narrative. Cheshire: Graphics Press.
signs.’ Semiotica 70, 59–78. Wescott R (1980). Sound and sense. Lake Bluff, IL: Jupiter
Saussure F de (1916). Cours de linguistique générale. Paris: Press.
Payot.
Learning Burden
Word Form
Knowing a word well involves knowing several
aspects covering the form, meaning, and use of the Words that are difficult to pronounce are usually
word. The difficulty these aspects have for any par- difficult to learn (Rodgers, 1969; Ellis and Beaton,
ticular word depends on how closely the aspects re- 1993). Words that are easy to pronounce can more
late to knowledge the learners already have. This easily be held in working memory and thus have a
knowledge can come from the learners’ first language greater chance of entering long-term memory. Thus,
or other languages they know, and it can come from the words introduced early in an English course
the knowledge they already have of the second lan- should be easy for learners to pronounce. Substantial
guage. For example, if the learners’ first language uses listening practice and a small amount of guided
the same written script as the second language, then pronunciation practice can make it easier for words
this will make the learning burden of the written form with unfamiliar sounds to be learned.
of the second language much lighter. If the written Although English spelling is notorious for its irre-
form of the second language is very regular with one gularities, there are numerous patterns and rules that
letter always representing the same sound and vice can help learning. First language research on reading
versa, then the learning burden of the written form has shown that developing phonemic awareness (the
will be even lighter. Similarly, if the two languages idea that words are made up of separable sounds) can
share a lot of cognate vocabulary, some of the have very positive effects on learning to read English
learning burden will be lighter. Spanish learners (Ehri et al., 2001). Similarly, giving some attention to
learning the academic vocabulary of English find phonics (sound-spelling relationships) can also help
that these words have a light learning burden because with learning to read and write (Stahl et al., 1998).
most have very similar forms and meanings in A very large proportion of English words come from
English, for example acquisition and evaluation. French, Latin, or Greek and thus make use of pre-
Finnish learners of English are faced with a more fixes, suffixes, and stems that can occur in many
difficult task. different words. Learning can be helped if the most
Teachers can try to lighten the learning burden of a common affixes and their meanings are learned, and
word by explicitly relating it to known items that are often a simple explanation of a word’s etymology
similar in the first or second language, by showing the (e.g., the word rank has the same stem as the word
patterns or rules that the word fits into, and by pre- arrange) can help learning. Irregular affixation can
teaching items and features that will make the new increase the learning burden. Laufer’s (1997) study of