Journal of Vi$ual Literacy. Spring 2002 Volume 22. Number 1.

19-28

The Symbiotics of Semiotics and Visual Communication

Sandra E. Moriarty, Ph.D. Boulder University of Colorado

Boulder, Colorado, U.S.A. Abstract This article reviews visual semiotics, a philosophical approach that seeks to interpret messages in terms of their signs and patterns of symbolism. Semiotic analysis is particularly relevantfor messages that rely on visual communication cues. The theory is that Peircian semiotics, rather than Saussurian semiology, can provide a central organizing perspective for analyzing how the "standsfor" function in sign systems produces meaning in nonverbal communication situations.

obert Craig's challenge to communication scholars to develop a consensus on communication theory as a field (Craig, 1999), can similarly be seen as a challenge to visual communication scholars. He makes the argument that there is no unifying theory of communication, nor should there be, but instead, there is a set of assumptions that can unify the diverse traditions of communication theory. Likewise, visual communication has emerged from a diverse set of traditions as, in his words, an "interdisciplinary clearinghouse." The search for a "field" of visual communication also parallels the work of Umberto Eco who has conducted a similar effort to define the "field" of semiotics. In contrast to Craig's premise, Eco's discussion of semiotics begins with a notion about the importance of having some sense of a central theory. He explains, "One cannot do theoretical research without having the courage to put forward a theory; and therefore, an elementary model as

20

a guide for subsequent discourse " (1979, p. 7-8) Likewise, in visual communication, it would be useful to have some organizing theory. However, like semiotics, the field of visual communication is broad and varied (Moriarty and Kenney, 1995; Moriarty, 1995; Moriarty 1997); and, similar to Craig's review of communication theory, it is probably too broad and varied to have any central or unifying theory. Eco takes his search a step further in distinguishing between a field (a field of studies, that is, a repertoire of interests that is not as yet completely unified) and a discipline (an area with its own method and a precise objective). In his analysis of semiotics, Eco says that if semiotics is a field, "then it should be possible to define semiotics inductively by extrapolating from the field of studies a series of constant tendencies and therefore a unified model." On the other hand, "If semiotics is a discipline, then the researcher ought to propose a semiotic model deductively which would serve as a parameter on which to base the inclusion or exclusion of the various studies included in the field of semiotics." He then suggeststhat we must keep in mind the semiotic field as it appears today, in all its many and varied forms and in all its disorder. The challenge he sees is to propose a simplified research model, "in order to suggest a unified method of approach to phenomena which apparently are very different from each other, and as yet irreducible." Likewise, Craig says that, "although communication theory is not yet a coherent field, I believe it can and should become one." He explains that "A field will emerge to the extent that we increasingly engage as a communication theorists with socially important goals, questions, and controversies that cut across the various disciplinary traditions.. .." Both Craig and Eco concur that communicationand semiotics are founded on sets of theories that need organizing. Likewise, visual communication, with its links to aesthetics and literacy (education), among others, is equally confused. Although it is too much to presume that a central, unifying theory might be developed for visual communication, it might be within the limits of feasibility to articulate a set of central, organizing theories - the theories that intersect at the point where visual processing departs from verbal processing. It is the purpose of this paper to explicate a theory of visual semiotics as one possible theory that might be included in such a set.

Semiotics: A Theory of Signs

Semiotics is the study of signs and signals, sign systems, and sign processes. Jakobson defines semiotics as "the exchange of any messages whatever and the system of signs that underlie them." (Sebeok, 1991, p. 60) Fiske adds the notion of generation of meaning to this definition (1990, p. 42). In other words, messagesare made of signs and conveyed through sign systems called codes;

Journal o/Visual Literacy, Volume 22, Number 1

---

21

meaning is derived only to the degree that the receiver of the message understands the code. Researchers in semiotics come from varied areas, such as communication, linguistics, anthropology, and marketing, as well as the natural sciences, where sign systems are studied in such areas as cellular biology and zoology. The focus of our concern is on the communication aspects of a sign, and particularly the communication of nonverbal signs. What is a Sign? In semiotic theory, a sign is anything that stands for something else; a sign stands for an object or concept (Hopes, 1991, p. 141; Eco, 1986, p. 15). The Swiss linguist Ferdinand de Saussure, who is known as the father of continental semiology, expressed this relationship as between a sound or image---called a signifier-and the concept for which it standscalled the signified (Fiske, 1990), p. 43-44). Because of his linguistic background, Saussure saw the relationship between these two as arbitrary, in other words, the link between the sign and what it stands for is understood by convention. This arbitrariness is true in most spoken and written language, however, that may not be so for other types of signs, such as visuals that provide cues to stimulate recognition through resemblance, or mimesis, a point well argued by Paul Messaris in his book Visual Literacy (1994). C.S. Peirce, who is characterized as the founder of American semiotics, defines signs as something that stands to somebody for something in some respect or capacity. That's a tripartite relationship that includes Saussure's signifier (the sign) and signified (the object or concept) as well as the Peirce's notion of an interpretant (the "stands for" element or process). By interpretant he means the idea contained in the concept as it is decoded, or a subsequent thought to which the sign gives rise (Hoopes, 1991, p. 34). Because Peirce approaches semiotics as a cognitive philosopher, he is more interested in meaning interpretation and modes of cognition, and does not limit his analysis to only arbitrary codes. This is clear in his analysis of three types of signs: iconic, indexical, and symbolic. Iconic are mimetic representations, that is, they look like what they represent. With an indexical sign, there is a physical connection - evidence between the signified and the signifier. For example, a footprint means someone has walked by and smoke means there is a fire. An indexical sign cues something or is a clue or evidence of something. Only in the third category, the symbolic sign, (a flag, school colors, an abstract logo or trademark) is the meaning arbitrary. In other words, Saussure's approach is more useful for analyzing languagebased sign systems. Nonverbal systems might be better analyzed using Peirce's approach because, besides the arbitrary, it allows for other systems of signifying, such as mimesis and evidence (clues and cues), which are fundamental to visuals. Peirce's semiotics also focuses on the concept of interpretation, which is key to making sense of visuals, a point I will elaborate on later in this paper.

Moriarty - The Symbiotics of Semiotics and Visual Communication

22

What are the Key Issues of the Semiotic Tradition?

The biggest problem faced by visual communication scholars is sorting out the factors and processes that are intrinsic to visual processing and separating them from language-based processing. The Problem of Language Dominance One of the problems in organizing the field of visual communication (if it can be called a field) is the dominance of verbal language. The language metaphor, that is, presuming that visual communication operates in the same way as verballlingualllexicalcommunication, has led to a subordinateposition for visual communication studies because visual communication doesn't operate in the same way as language. Hence scholars like Barthes (1971) decry "the paradox of a message without a code" and the lack of a metalanguage that corresponds to the signifieds. The problem is that scholars attempt to find parallels to language in the way visuals construct meaning. In her book, VisualIntelligence, Ann Marie Barry (1997) describes how visuals are processed leading us to the conclusion that the language metaphor may be inappropriate because the two systems are radically different in their basic operations. Privileging language as a primary system through which other systems are expressed has made it difficult to develop the concepts and, yes, even the language to describe how visual communication works (Moriarty, 1994). As Thomas Sebeok, founder and director of the Center for Language and Semiotic Studies at Indiana University observed in a review of the contributions of various philosophers to semiotics, Bacon, for one, "did not commit the vulgar error of identifying language with communication." (1991, p. 71) It is the thesis of this essay that visual communication is equally as important as verbal language and that it deserves recognition as an area of study on its own. But this brings us back to the problem stated earlier: that there is no recognized discipline-like linguistics and speech communication (rhetoric) in the verbal language area-that focuses on visual communication. There is no field there; no central theories and, therefore, little support from the academy and little scholarly firepower.

Symbolic Meaning and Propositions

The irony is that Messaris, in his book VisualPersuasion (1997), limited the symbolic function primarily to language and, thus, unfortunately, restricts the ability of visuals to function symbolically. He says: " .. .as far as semantic features are concerned, it is the indexical and iconic properties of visual images that most clearly set them apart from language...It is true that some kinds of visual representations (e.g. technical diagrams or maps) are arguably based at least in part
Journal of Visual Literacy, Volume 22, Number 1

--

23

on arbitrary conventions, and in that sense, they can be said to entail the type of semantic relationship that Peirce labeled symbolic. However, symbolic signs are, if anything, even more characteristic of language and other major modes." (p. x) One would think that art historians who have spent centuries uncovering the symbolic meanings in Renaissance paintings would challenge such a statement. However, there are challenges to this view from his own school. Larry Gross and Sol Worth,both scholars in the tradition of the Pennsylvania school of visual communication, write of two interpretive strategies for visuals, the symbolic and the natural (Gross, 1981, 1985).Inhis article on visual narratives, Gross explains, "symbolic events are those we assume were intended to communicate something to us [author's italics]. Further, we assume that these events are articulated by their 'author' in accordance with a shared system of rules of implication and inference." I believe, however,that Messaris's point about symbolic meaning reflects something else, and that is his doubt about the ability of visuals to carry propositional meaning. This point was made earlier in the work of his mentor, Sol Worth, who proposed that "pictures can't say aint" in an article by that title (1982). Messaris states that position very clearly later in the introduction to his book when he says, "what visual communication lacks most crucially is a so-called propositional syntax." (p. x) So the question about the communication of symbolic meaning comes down to implication and inference and how they relate to propositions. A proposition is a statement that can be proven or disproven using logical means such as an argument. An inference is the forming of a conclusion from premises, also an exercise in logic. It should be noted that inferences, and, yes, even propositions are not just language-based concepts. Gretchen Barbatsis, for example, has been working to articulate the propositional meaning of form in video and film and is developing a body of work to demonstrate how propositions and inferences can be "stated" visually. And Ann Marie Barry in her book VisualIntelligence, (1997, p. 333) explains, "how the simplejuxtaposition of one image next to another can affect our thinking and feeling in radical ways, and how in rapid succession images can ultimately form convincing arguments which, when received and believed, can change the values, attitudes and lifestyles of critical masses, and even topple governments." An example of a propositional visual is an illustration that shows a bee approaching a photo of flowers. Even in black and white, the meaning of the proposition is still clear: the photo (in this case a photocopy) is so realistic that it can fool a bee.
Moriarty - The Symbiotics of Semiotics and Visual Communication

24

Which Concepts or Metaphors have been Most Productive?

Inapplying semiotics to visual communication,one concept is particularly important, and that is the process of interpretation. The "stands for" element in Peirce's definition of a sign refers to the way meaning is created through encoding by the source and decoding by the receiver (reader, in semiotic analysis). That is the model as it is phrased in communication theory, however, visual communication involves other types of meaning beyond those in the intended/perceived schema. I pointed out in the review of the definitions of signs, that the definitions are based on different notions of how the process of signification works. To summarize: Arbitrary signification: works by convention Mimetic signification: works by iconic representation Evidential signification: works by cues and clues Signaling: works by recognition (hard wired or convention) Peirce's most important contribution is in the area of interpretation, which rests on the notion of "a dynamic view of signification as a process" (Deely, 1990, p. 23). Since he was more centrally concerned with thinking, rather than with communicating, his primary contribution to cognition-and only incidentally to communication-was his proposition that "all thinking is the inferential interpretation of signs" (Hoopes, 1991, p. 11). In other words, a thought is a product of the signification process, an idea that provides the link between cognition and communication. He explained that the meaningmaking process-finding the signified-is an infinite process of interpretation using all of the signification techniques. Furthermore, that "to interpret" means to define a relationship in the something-stands-for-something-else construct (Eco, 1986, p. 2, 44). So we have at the base ofPeircian semiotics a thinking process based on inference that results in interpretation. This process of interpretation involves the consideration of the meaning of a sign in terms of all possible signification possibilities - a sign can be a signal, or an iconic, indexical, or symbolic sign, all at the same time. Interpretation is done by puzzling out the inferences from these multiple levels of signification. And this type of search for meaning is infinite, in Peirce's view, because every signifier can be translated into other signifiers and interpretants through an endless process of inference chaining. The complexity of this type of inferential interpretation is illustrated in an ad for a Jaguar car that uses the car shot against a background of a high school building to stand for achievement. In order to make sense of an ad using a visual like this, you have to recognize the car and the fact that it symboizes accomplishment, and the high school building and the fact that it is cueing a class reunion. Furthermore, through this complex process of symbolization you understand that the ad is also intended to engage all the emotional and aspirational context associated with a reunion situation. This
Journal of Visual Literacy, Volume 22, Number 1

----

25

is an example of how the inference chaining process works in visual communication. The reason Peirce's concept of interpretant is so important to us in visual communication is that visual meaning is more open for interpretation than is verbal. Contrary to conventional wisdom which suggests that if you can see, you can understand (i.e. the meaning is transparent), the interpretation of many visual messages may be more complex and more demanding of the decoder because of the inferential dimension on which visual interpretation rests. Visualinterpretation,however,involvesmore than simple inference. From semiotics we know that what is missing is sometimes as important as what is there. Eco (1979) suggested that a viewer goes through a process of "synthetic inference" which involves both denotative (realism, representation) and connotative (associations, attitudes, emotions) processes. Association, in particular, unlocks this chaining process. In this complex inferential process where information is being actively synthesized, an involved audienceextends and fills in meaning, as well as decodes the meaning (Fry, 1983).

Which Epistemologies are Most Useful?

Peircian semiotics is a theory of knowing, rather than a theory of languaging. This philosophical grounding leads to a theory of visual communication based on how we come to know things, rather than how we transmit knowledge. It can best be explained through the understanding of a concept Peirce referred to as abduction. In trying to better understand the complex inferential process used in interpreting visuals, scholars have tried to articulate the logic behind this process using Peirce's theory of abduction (Moriarty, 1996; Buchler, 1955; Peirce, 1931-35; Hoopes, 1991). In contrast to inductive (reasoning to) and deductive (reasoning from) logic, abduction is an inferential process that fashions conjectures based on "clues" that are available or conditions that are known. As Neiva says in his discussion of semiotics, rather than mimesis or conventional meanings, "the logic of signs is the logic of possibilities" (1999). In order to accumulate clues, the abductive process begins with observation, the bits and bytes of perception. It's similar to the way a doctor accumulates symptoms until he or she arrives at a diagnosis. Peirce described the formationof an abductivehypothesisas an "act of insight," the idea coming "like a flash"-the proverbial lightbulb. In a more formal statement,abductive reasoning assembles the observations and attributes a variety of characteristics or conditions to a subject (the conjecture process) until a match is made and a conclusion can be stated. Another metaphor for abductive thinking is the semiotician as a detective, which Eco and Sebeok (1983) presented in their book, The Sign afThree.
Moriarty - The Symbiotics of Semiotics and Visual Communication

26

Beyond the linear forms of deductive and inductive logic, abductive reasoning more closely resembles massive parallel processing, one that is not at all like language processing. The information is gathered, assimilated all at once, and matched to previous patterns of knowing, which is the way visual recognition works. This opens up the question of whether, in response to Messaris, visual propositions are not impossible, but rather different, from language-based propositions, based as they are on intuition, rather than induction or deduction. Which Research Questions are Most Relevant? Because of its ability to analyze visual meaning-making as a different process than verbal meaning making, the visual semiotics approach is hypothesized to be a useful tool to free visual communication from the limitations of the language metaphor. Hence, additional research and theory development continues to be needed in isolating those visual factors and processes that truly are distinct from verbal language processing. In particular, the issues that remain surround the role and function of visual symbolism and the way propositions and inferences actually work in visual interpretation. Messaris (1994), in his earlier work on visual literacy, identified a need for an understanding of a different set of interpretive skills that are more intuitive than conventional. Scholars in visual communication continue to move toward that goal. To that end, it also would be useful to better understand the differences in the meaning-making processes of the four types of signification as they are embedded in visual processing. As referred to earlier, these are hypothesized to include the following: Arbitrary signification: works by convention Mimetic signification: works by iconic representation Evidential signification: works by cues and clues Signaling: works by recognition (hard wired or convention) Conclusion How is meaning produced, conveyed, and interpreted in messages that are primarily visual? This question is particularly relevant when the message is one that relies almost exclusively on visual communication cues. The reason there has been little work in this area is largely because of a lack of a structured approach, or theory, to use in analyzing such processes. If we agree there is a need for an organizing theory and if we agree that there is a field of visual communication developing that is deserving of such a theory, then this paper proposes that semiotics provides a useful theoretical foundation to apply to visual communication because it helps unlock the

Mt\\~l~iitiMM Vi~"M it\t~~r~tMilU\.

Journal o/Visual Literacy, Volume 22, Number 1

--

27

This analysis relates semiotics theory to visual communication in an effort to better understand how visual communication works. Table 1 summarizes the key issues raised in this paper.

Table 1 The Symbiotics of Semiotics and Visual Communication

Semiotics A theory of signs (visual, as well as verbal) and how they are interpreted The problem of language dominance; the challenge of symbolization and propositions Interpretation and the role of the interpretant The philosophy of knowing-abduction inference-rother than tmnsmitting and

Key issues Most productive concept Most useful epistemologies Research questions that remain

How interpretation-and abduction-work; The differences in the four types of signification

References Barbatsis, G. (1998, June 24-28). Analyzing meaning in form: Domestic comedy's angle of refraction. Paper presented at the Visual Communication Conference, Winter Park, CO. Barry, A. M. (1997). Visual intelligence. Albany: State University of New York Press. Buchler, J. (Ed.). (1955). Philosophical writings of Peirce. NewYork: Dover. Craig, R.T. (May, 1999). Communication Theory as a Field, Communication Theory 9:2, pp. 119-161. Deely, J. (1990). Basics of semiotics. Bloomington: Indiana University Press. Eco, U. (1979). A theory of semiotics. Bloomington: Indiana University Press. Eco, U. (1986). Semiotics and the philosophy of language. Bloomington: Indiana University Press. Eco, U. & Sebeok,T.A. (Eds.) (1983). The sign of three.Bloomington: Indiana University Press. Fiske, J. (1990). Introduction to communication, (2nd. ed.). London: Routledge. Fry, D. L. & Fry, V.H. (1983). A semiotic model for the study of mass communication. In M. McLaughlin (Ed.), Communication Yearbook 9 (443-462). Beverly Hills: Sage.

Moriarty

- The Symbiotics of Semiotics and Visual Communication

28

Gross, L. (1985). Life vs. art: The interpretation of visual narratives. _Studies in Visual Communication, 4, 2-11. Gross, L. (1981). Symbolic strategies. In L. Gross (Ed.), Sol Worth:Studying visual communication (pp. 134-147). Philadelphia: University of Pennsylvania. Hoopes, J. (1991). Peirce on signs. Chapel Hill: The Unviersity of North Carolina Press. McKibbon, B. (1993). The age of missing information. New York: Dutton. Messaris, P. (1997). Visual persuasion: The role of images in advertising. Thousand Oaks, CA: Sage. Messaris, P. (1994). Visual literacy: Image, mind & reality. Boulder, CO: Westview Press. Moriarty, S. (1997). A conceptual map of visual communication. Journal of Visual Literacy, 17, 9-24. Moriarty, S. (1996). Abduction: A theory of visual interpretation. Communication Theory, 6, 167-187. Moriarty, S. (1995). Visual communication theory: A searchfor roots. Paper presented at the Visual Communication conference, Flagstaff, AZ. Moriarty, S. & Kenney, K. (1995). Visual communication: A taxonomy and bibliography. Journal of Visual Literacy, 15, 7-156. Moriarty, S. (1994). Visual communication as a primary system. Journal of Visual Literacy, 14, 11-21. Neiva, E. (1999). Redefining the image: Mimesis, convention, and semiotics. Communicatoin Theory 9,75-91. Peirce, C. S. (1931-1935). Collected papers, Vols. 1-6. (Harshorne & Weiss, Eds.). Cambridge, MA: Harvard University Press. Sebeok, T.A. (1991). A sign isjust a sign. Bloomington: Indiana University Press. Worth, S. (1982). Pictures can't say 'ain't.' In S. Thomas (Ed.), Film/culture: Explorations of cinema in its social context (pp. 97-109). Metuchen, NJ: Scarecrow Press.

Journal o/Visual Literacy, Volume 22, Number 1