Designs for Multimodality in Literacy Studies:

Explorations in Analysis

Marjorie Siegel
Teachers College, Columbia University

Carolyn P. Panofsky
Rhode Island College

Literacy studies have taken a semiotic turn. We read the recent torrent of publications on
multimodality in literacy studies as a sign of dissatisfaction with verbocentric theories of ‘text’
and ‘literacy,’ and a desire to acknowledge the permeable boundaries among sign systems in the
contemporary semiotic landscape. This emerging field of scholarship has not settled on a single
definition, theory, or set of analytic tools, even if some approaches have nearly become synonymous
with ‘multimodality.’ The unsettled status of the field appears to be a productive moment of
experimentation, invention, and problem-posing as researchers design analytic approaches that draw
on a range of theoretical frameworks relevant to their research interests, purposes, and questions
(e.g., Albers, 2008; Harste, Leland, Grant, Chung, & Enyeart, 2007; Hull & Nelson, 2005; Ranker,
2008; Rogers, Winters, & LaMonde, in press; Taylor, 2006; Wohlwend, 2009).
The most influential among these designs for multimodality is the social semiotic theory
outlined by Kress and his colleagues (Jewitt, 2006; Kress, 1997, 1998, 2000a, 2003; Kress & Jewitt,
2003; Kress, Jewitt, Bourne, Franks, Hardcastle, Jones, & Reid, 2005; Kress & van Leeuwen, 1996).
This theory provides a metalanguage that calls attention to the multiple modes of representation
available and their affordances, as well as the way a sign-maker’s interests shape the choice of mode,
and, consequently, the production of meanings. Without this metalanguage, modes might be (and
often are) taken as “natural” and the sign-maker’s interests ignored. Yet, we believe this approach
is, by itself, not enough to understand multimodality, and propose that analyzing multimodality
requires a hybrid approach—a blend or “mash-up” of theories. Although a researcher might elect to
focus primarily on the way an individual’s interests and choice of modes shapes meaning-making,
to do so in the absence of historical, cultural, and political theories of literacy curriculum, teaching,
and learning is to limit what a multimodal lens can offer educators. In what follows, we trace
the history of attention to multimodality in literacy studies as a prelude to our own experiment
with blending different theoretic lenses to read a classroom literacy event from an early study
of transmediation (Siegel, 1984). We conclude by posing some of the questions that challenge
researchers as they pursue the analysis of multimodality.

MATTERS OF HISTORY AND DEFINITION

In their look at trends in literacy education, Comber and colleagues (Comber, Green, Lingard,
& Luke, 1998) see a “tradition of the new” (p. 20), in that new theories and approaches have
successively displaced each other in recent decades. So it is not surprising that multimodality has
been constructed as “the new” by the literature on multimodality in literacy studies. Pointing to a
constellation of technological, economic, and social changes, this literature traces the inception of
multimodality to 1996 (Jewitt, 2008, p. 246), the publication date for the New London Group’s

99
100 National Reading Conference Yearbook, 58
(1996) manifesto on multiliteracies and Kress and van Leeuwen’s (1996) grammar of visual design.
Design was a central concept to both projects and was recruited to signify the process as well as the
product of designing meanings. The elements of design were conceptualized as modes of meaning
and each mode (Linguistic, Audio, Visual, Gestural, Spatial) had a metalanguage that “describes and
explains patterns of meaning…. Multimodal Design, however, is of a different order to the others
as it represents the patterns of interconnection among the other modes” (New London Group,
1996, p. 82). These publications catapulted semiotics to prominence in educational discourse, in
part because they adopted a more accessible metalanguage than semiotic theory (note that ‘sign’
is both a morpheme of and synonym for ‘design’), but also because the time was ripe for moving
beyond language.
We would argue that multimodality has a much longer history in literacy studies, and trace
its appearance on the literacy scene at a time when conventional theories of literacy and literacy
learning were being uprooted by semiotic and sociocultural theories (see, for example, Dyson,
1982, 1983, 1989, 1993, 2003; Harste, Woodward, & Burke, 1984; John-Steiner, 1985, 1995;
Lemke, 1993; Rowe, 1986, 1998; Siegel, 1984, 1995; Smagorinsky & O’Donnell-Allen, 1998;
Suhor, 1984). A social semiotic theory of multimodality took up many of the concepts introduced
at earlier points in time. These include philosopher Suzanne Langer’s (1942) idea of “discursive”
(sequential) and “presentational” (simultaneous) forms of meaning, developmental psychologists
Werner and Kaplan’s concept of “symbol-weaving” (cited in Dyson, 1982), semiotician Umberto
Eco’s (1976) Peircean-inflected synthesis of semiotic theory, and language educator Charles Suhor’s
(1984) notion of “transmediation,” which, together with “symbol-weaving,” outlined the key move
in what has since come to be called multimodal theory—the juxtaposition and transformation
across sign systems.
Definitions of multimodality are grounded in a semiotic perspective on meaning-making
which assumes that meanings are made through multiple modes of representation (not just
language), and these modes can be combined “in quite remarkably dynamic relationships” (New
London Group, 1996, p. 78). Kress and van Leeuwen (1996) defined multimodality near the end
of their visual grammar, including it as part of the “meaning of composition” (p. 183). They wrote:
… the question arises whether the products of the various codes should be
analyzed separately or in an integrated way; whether the meanings of the whole
should be treated as the sum of the meanings of the parts, or whether the parts
should be looked upon as interacting with and affecting one another. It is the
latter we will pursue…. (p. 183).
Some scholars reserve the term ‘multimodality’ to mean the simultaneous display of more than
one mode in a single text or event. For example, Duncum (2004) defines ‘multimodality’ as the
“interaction of two or more discrete sign systems” (p. 253), and distinguishes this from pedagogical
practices in which one mode is used to “inspire expression in another” (p. 254).
However, there is growing agreement that “all meaning-making is Multimodal” (New London
Group, 1996, p. 79). Reflecting on their studies of young children’s initial encounters with
print, Harste, Woodward, and Burke (1984) argued that even a single modality (e.g., linguistic
signification) is a “multimodal cue complex” (p. 207). They explain:
Cue systems from alternative communication systems are embedded in and
Designs for Multimodality 101
make up this linguistic cue complex. … In written language this is done through
formatting, type size and thickness, packeting, layouts, charts, graphs, and
pictures, all of which in transaction with print are signifying structures with
potential to use in sign meaning. (pp. 207-208)
Kress (2000a) came to this same conclusion in the course of arguing for a semiotic theory of
meaning-making:
Importantly, the question arises whether modes such as ‘written language’ or
‘spoken language’ can, in any case, be regarded as ‘monomodal’: in fact, my view
is that they cannot. This means we have to rethink ‘language’ as a multimodal
phenomenon. (p. 184)
If texts and events—even those that appear to be monomodal—are produced through the
juxtapositioning of multiple sign systems, then understanding “what people make of the space
between discrete communicative systems” (Duncum, 2004, p. 257) is central to multimodal
analysis. We hypothesize that understanding what people make of the space between multiple
modes requires more than a theory of multimodality since what people make of the space between
modes depends on the activities, identities, and power at a particular moment in time.
EXPLORATIONS IN ANALYSIS
There is no ready-made tool-kit for analyzing multimodality in literacy studies, but researchers
have turned to a range of theories in search of analytic guidance. These include Peircean semiotic
theory, approaches grounded in Halliday’s functional-systemic theory of language, and sociocultural
analyses that reference both Vygotsky and Bakhtin’s influential ideas. Each approach has inspired
distinctive lines of inquiry that, we argue, could be productively blended.
This first line of inquiry (Clyde, 1994; Harste et al., 1984; Leland & Harste, 1994; Rowe,
1986, 1998; Siegel, 1984, 1995; Suhor, 1984; Whitin, 1996, 2005) tapped Peirce’s triadic
explanation of semiosis to examine the process of sign-making (semiosis) involved in acts of reading,
writing, and transmediation. Literacy scholars (Dyson, 1993, 2003; John-Steiner, 1985, 1995; John-
Steiner, Panofsky, & Smith, 1994; Panofsky, 1999; Smagorinsky & O’Donnell-Allen, 1998) also
turned to sociocultural theories for analytic tools—especially Vygotsky’s cultural-historical theory
of mediated cognition and Bahktin’s theory of dialogism and texts—to examine literacy learning
and participation. Kress and van Leeuwen (1996) introduced a third approach, known as social
semiotics, by applying Halliday’s three meta-functions of language (ideational, interpersonal, and
textual) to images drawn from contemporary Western culture. Their efforts produced a grammar of
visual design that shows how “depicted people, places, and things combine in visual ‘statements’ of
greater or lesser complexity and extension” (p. 1). Finally, Kress and his colleagues (Jewitt & Kress,
2003; Kress, 2003) have continued to develop social semiotic theory to examine the multimodal
character of literacy curriculum, teaching, and learning. Recent empirical work (e.g., Jewitt, 2006;
Kress et al., 2005; Pahl & Rowsell, 2006; Stein, 2008) has increasingly incorporated theoretical
perspectives and tools to complement the insights of social semiotic analyses of multimodality in
literacy studies.
102 National Reading Conference Yearbook, 58 Designs for Multimodality 103

MULTIPLE READINGS OF “SKETCHING RADIATION:” AN ANALYTIC Figure 1. Alan’s Drawing of Killer Whales
MASHUP

To illustrate these approaches, we will revisit an example taken from an earlier ethnographic
study conducted by one of us that examined the ways in which fourth-graders used drawing to
transact with a text they had read (Siegel, 1984, 1995). The study was conducted over a nine-month
period from October 1981-June 1982, and included participant observation and action research.
Our example comes from the action research component in which students were introduced to
the “sketch to stretch” strategy and invited to draw their transactions (Rosenblatt, 1978) with texts
and explain their “sketches” to their group members one hour per week for 12 weeks. Throughout
the action research phase of the study, ongoing participant-observation made it possible to
contextualize the children’s participation in sketching. The data analysis combined semiotic analysis
of transmediation (using one sign system to mediate another) with ethnographic analysis of the
social world of the classroom and the sketching lessons, and was framed by a reflexive narrative of
fieldwork that documented the researcher as a sign in the field (Herzfeld, 1981) and the children’s
interpretations of the sketching lessons. These analyses generated two working hypotheses: (a)
children’s sketches were acts of invention in which they made codes from available codes, and; (b)
children’s sketches and processes of sketching were mediated by children’s definition of the situation
as “play,” their friendships, their interests, and their definitions of themselves as artists. In what
follows, we introduce a key event from this study and read it through multiple analytic lenses,
starting with Peirce’s triadic theory of sign-making (semiosis), followed by multimodal theory, and,
finally, sociocultural theory.

MAKING SENSE OF “RADIATION:” INVENTING A SIGN

Peirce’s theory of sign-making highlights the generative and mediated nature of semiosis (Eco,
1976). For Peirce, a sign does not simply stand for an object, but tells something about the meaning
of that relationship and this requires a third component, which he called an “interpretant.” Thus,
a sign becomes meaningful when it produces another sign (the “interpretant”) that mediates the
“representamen” (sign-vehicle) and the “object” (concept). An intriguing example of this process,
recorded in the field, involved a group of boys who included “radiation” in their sketches. When
the boys shared these sketches, the group of 12 children would break into gales of laughter, making
the re-establishment of order nearly impossible. And so “radiation” became a kind of puzzle: it
seemed to have no other purpose for the boys than to entertain themselves and the group. A
Peircean reading, however, shows that over time, these boys invented a sign—radiation—in which
an interpretant (punishment) mediated the concept of radiation and its visual representation as
jagged lines.
The first time “radiation” appeared was in Lesson 8, when the students had been asked to revise
the sketch they had originally produced after reading a short non-fiction text on killer whales. Each
session ended with a sharing time, and in this lesson students were asked to explain how this sketch
of Killer Whales (Simon, 1978) differed from their original drawing. When it was Alan’s turn to
explain he said, “Well, this sketch you get radiation” (see Figure 1). At this point, a Peircean analysis
of the triadic relation among representamen, object, and interpretant was impeded by the absence
of connections. There was no way to interpret what “radiation” meant.
Two weeks later “radiation” was again introduced into the sharing session, first by Curt, then
by Jack. When Curt finished talking about his sketch of the story, It’s Not Fair (Supraner, 1976),
he added, “And they’re getting radiation,” at which point Alan asked, “How come they’re getting
radiation?” Sam admonished Alan, saying, “Don’t ask dumb questions—you always do that,” but
Curt replied anyway, “Cause they’re beeping up there.” In this sketch a relationship between the
object (concept of radiation) and the representamen (lines drawn to look like a bolt of lightning) has
been established, but the interpretant—the sign that arose from this relationship—is as yet unclear.
It was clarified to some extent by Jack in his explanation of Oliver Button is a Sissy (dePaola, 1979).
Jack described his drawing, and added, “And then I did this little thing about one of the goons are
picking on Oliver and he’s [the goon is] getting radiation by some electricity and he doesn’t like it.”
The connection between the notion of radiation and the representamen in the sketch now seemed
to be that radiation is the punishment one receives for being a bully. At this point, then, the semiotic
triad of object, representamen, and interpretant has been invented.
Two weeks later, Jack made the connection between “radiation” and “punishment” explicit
when he’d finished explaining his sketch for the Aesop fable, “The Dog and His Bone” (Kent,
1972). When asked what he thought the author was trying to teach him, he responded, “On the
back is a picture of the dog getting radiation. That’s the evil gargoyle giving him radiation.” Jack
104 National Reading Conference Yearbook, 58
then offered a title for the sketch: “The title of it is if you’re greedy something bad might happen.”
Alan picked up on this idea when his turn came. When asked for a title he said, “Well see—some
things don’t pick. When a sign says ‘don’t,’ you might end up in trouble. And this guy was picking
the grapes.” Sam continued, “And he got radiation.” Alan repeated, “And he got radiation” as he
stood up and jerked his body to show what it looked like to get zapped with radiation. In sum,
“radiation” eventually came to signify punishment for some wrongdoing, whether it be picking on
someone, being greedy, or disobeying a posted rule.
This sequence of events points not only to the process of sign-making but the role of
friendship in creating a sign. Alan, Jack, Curt, and Sam were good friends and obviously enjoyed
inserting radiation into the lesson. What might have been interpreted as “mere” fooling around
and disrupting the lesson was, importantly, an occasion for meaning-making. But the meaning the
boys generated was not limited to that which was explicitly expressed. Another meaning was being
signified, namely, the fact of their mutual participation in the event. “Radiation” was not only a
way to signify the consequences of breaking social codes, but a way to signify that they were doing
something together. Given the unusual structure and organization of this activity, these boys had
developed a definition of the situation as a recess, or “play.” Inventing a sign to signify they were
doing something together was a way to call attention to their friendship and their pleasure in the
activity. One could hypothesize that sketching was an opportunity for them to publicly negotiate
this interpretation of the situation and eventually arrive at a consensus of what was happening. This
is what Herzfeld (1981), after Jakobson, calls a poetic analysis. Such an analysis reveals that the sign
points to one’s participation in an event and permits the sign-user to make a comment about him/
herself. From this perspective, one could argue that “radiation” was a way for Alan, Curt, Sam, and
Jack to make a collective comment on their intersubjectivity.
A Multimodal Reading of Alan’s “Killer Whales” Drawing
Our interest, now, is to go beyond what Peirce would call the “openness” of the sign, to
explore what Kress’ theory of multimodality makes visible. Kress (2003) proposes that each mode
offers particular affordances of meaning, and their selection and arrangement as texts is shaped
by the sign-maker’s interests. Kress (2003) writes that any text (by which he means any instance
of communication) has a site of appearance: text that appears in a book is ordered by the “logic
of writing,” even when there are images; in contrast, he argues that text that appears on screen
is ordered by “the logic of the image” (p. 48). Thus, on the screen, the logic of images arranged
and interrelated in space and their simultaneous display dominates over the logic of time and the
narrative representation of elements in sequence. The logic of the image with its rhetoric of display
clearly characterizes Alan’s “Killer Whales” drawing. The drawing is ordered spatially by division
into five sections, each displaying a different part of the textual meanings. Although the drawing
is dominated by the logic of the image, it is a multimodal text that also uses written language
in several ways, including these: 1) word as label (“whale,” “Enemies”); 2) word as proposition
(“Lives in water,” “it is a mamle [sic]”); 3) word as indexical marker (“Danger” positioned inside a
conventionally stylized arrow); 4) word as symbolic image (“Food” with the double oo illustrated
to look like eyes peering down as if functioning like a thought bubble for the shark, though there
is no bubble-shape around the word).
Designs for Multimodality 105
Kress (2003) identifies genre as expressing “the social relations of participants in social events –
who is involved, with what purposes, what roles, what power, in what environments” (p. 47). This
drawing seems to be designed like the kind of text-and-image displays found in children’s illustrated
information books (such as the Usborne series, e.g., Hindley, Wingage, Cartwright, & Goffe, 1998)
that use “comic-book-like” layouts, with a page divided into many sub-sections (also the kind of
layout sometimes seen in a museum or library display case). But instead of signifying narrative
movement, the separate boxes or spaces show taxonomic or other non-narrative arrangement of
information. Here, there is the boxed field at far left center that apparently shows the various fish
eaten by killer whales that Alan and others refer to in the transcript; interestingly, none of the fish
referred to in the transcript are mentioned in the “Killer Whales” text, including “the” shark, which
Alan’s definite article refers to as “given” rather than “new” information (Chafe, 1970). In either
case (comic format or display case), layout is used to distribute “pieces” of information in different
boxes or spaces, creating a text held together by virtue of layout and juxtaposition (rather than verbal
cohesion) that Kress would see as characteristic of the display logic of the image (and/or screen). In
some parts, there is a more clearly symbolic representation: for example, the stick-figure placed in a
box and crossed out next to a second stick-figure that appears to smile and gesture as if celebrating
triumph. The ‘X’ through the boxed stick-figure is apparently what Alan refers to when he says,
“Well, this sketch… you get radiation.”
So, in this drawing, we find multiple modes and genres mixed together in a way that clearly
elicited an enthusiastic and creative response from its creator. Going one step further, we may also
discern what Kress (2003) refers to as multiple discourses in the drawing, where discourse refers to
“what is at issue, what is being talked about” (p. 47). Material referred to earlier as the labels and
propositional information are the academic content that the child understood as needing to be
included, that is, school discourse. At the same time, transforming ‘Food’ into a symbolic image,
bringing in the word ‘Danger’ and enveloping it in an indexical arrow, and providing the stick-
figures as a kind of narrative gloss on ‘Enemies’ seems to bring a peer discourse into the drawing,
enlivening the academic discourse with an implied narrative of conflicting forces. In this semiotic
work, the child seems to be using representational resources to accomplish work with peers at least
as much as with the teacher. In this way, we might also see the drawing as a mixed genre expressing
social relations with peers in terms of purposes they understood and with different purposes for
the academic context. Notably, it was this very atypical school context in which the peer-oriented
genre was expressed.
A Sociocultural Reading of Alan’s “Killer Whales” Drawing: Meanings Beyond a Multimodal Analysis
Whereas for Kress, primary focus is on text and dimensions of text, such as genre and discourse,
for sociocultural theory, the primary focus is on culturally mediated social practices and how they
are enacted through historical time. Focused, then, more on the actors who make the texts, than on
the texts that are acted upon, a sociocultural analysis of the killer whale drawing provokes questions
about the origin of radiation as a signifier for Alan and his friends. Origin is key for Vygotsky (1978)
when he argues that, “behavior can be understood only as the history of behavior” (p. 65); that is, to
fully understand the boys’ invention of “radiation” as a sign, one would need to see the full context
in which “radiation” emerged as a newly created meaning, shared and used by the boys, and this
context might even have to extend to prior activities that served as groundwork for those meanings.
106 National Reading Conference Yearbook, 58
Although Kress (2003) seems to involve such social activity when he defines genre as “the
social relations of participants in social events—who is involved, with what purposes, what roles,
what power, in what environments” (p. 47), he gives no suggestion of the way those relations
work through time in a dynamic and generative (in the sense of onto-genetic) way. Although Kress
claims a view of meaning as creative, rather than as “use,” one finds little, if any, mention of a social
process of creating meanings. But, for sociocultural theory, the historical, dynamic, ontogenesis
and sociogenesis of meaning is key in an instance such as this because the children are creating
something new and unique, perhaps through transformation of some other sign, a new sign,
mediated through their own collaborative activity, shared knowledge, and joint actions. That is,
sociocultural theory takes us further in exploring the creativity of semiosis.
Sociocultural theory foregrounds what human beings do as organized in activities that
are practiced by social groups. Working in such a framework, Bakhtin (1986) identified “social
languages” of specific groups and “speech genres” associated with specific activities of groups.
From this perspective, “getting radiation” is part of this peer group’s social language, the way they
share meanings. In addition, as part of a speech genre that accompanies their play activity, “getting
radiation” includes forms of graphic representation as well as embodied performance. In addition,
the social group and its activity are, in cultural and historical terms, notably gendered: only boys
seem to participate in the discourse about “radiation,” and the activity depicted in most of the
drawings and highlighted in the “radiation” events foregrounds physical aggression and physical
violence, popular culture themes common in children’s cartoons and comics and traditionally
appealing to young male viewers.
In extending the sociocultural concept of activity to reading and writing, Smagorinsky and
O’Donnell-Allen (1998) write that “activity is predicated on ‘the notion of internalization [that]
is concerned with the ontogenesis of the ability to carry out socially formulated, goal-directed actions
with the help of mediating devices’ ” (Wertsch, 1981, p. 32, as cited in Smagorinsky & O’Donnell-
Allen, 1998, p. 201, emphasis in original). In such a view, context is key and includes such elements
as institutional values, rules, and goals, as well as counter-values and goals that may be held in the
peer group in an institutional setting such as school. In addition, any text that is created must also
be understood as highly contextualized, deriving from other texts known to both the author and
his/her audience, and thus presupposing other texts and their sources, and playing with and “off of ”
the multiple voices of those texts. In this sense, any text is an inter-text, a text built through dialogic
interaction with others and with others’ texts, whether literally present or not at the time of the text’s
creation. As its etymological origin suggests, a text is a weaving, through time, of meanings. For the
weaver, the meaning-maker, the text realizes a continuity of experience through time (as well as a
transformation), even if that continuity is not evident to chance readers—such as we are, reading
Alan’s “killer whales” multimodal text years after it was composed.
Despite our late arrival on the scene of Alan’s textual dialogue with his peers, we can still detect
traces of his identity and sense of agency in the creation of this text: he describes it as differing
from an earlier drawing about the same article as, “Well, this time you get radiation.” We know
that radiation has nothing to do with the article on killer whales, but we also know that “radiation”
signifies something important to his friends. So in this newer version, Alan has found a way to
take part in the discourse of “radiation,” the social language of their group, and while his peers
Designs for Multimodality 107
will contest other claims he makes about the drawing—such as the accuracy of his claims about
types of fish—no one contests his assertion of the relevance of “radiation” in this drawing. In the
two weeks or so that elapsed between Alan’s first killer whale drawing and the one that we see,
something transpired in the peer group to make Alan’s inclusion of “radiation” not only possible but
also desirable and its inclusion seems to position him well in the group. Including “radiation” in his
drawing seems to be an act on which he can proclaim his agency and enhance his identity within the
peer group. It is through such social actions that sociocultural theory “reads” this multimodal text.
As for its multimodality, specifically, sociocultural theory has long recognized the varied forms
of mediational means in human communication. As John-Steiner (1985, 1995) has shown, the
variation in kinds of meanings to be represented have, historically, required the creation of multiple
modes for representational tools; some meanings simply cannot be communicated or represented
in words. For Alan, representing his ideas about killer whales requires the use of words, images,
icons, and other signs, assembled in a layout that communicates relations between meanings and
images in the drawing. The kind of text that Alan creates carries an intertextual resonance with the
kind of displays seen in some kinds of non-fiction texts for children, such as the Usborne books on
many topics. Thus, it’s likely the child has not created this particular kind of text, but has drawn
on one he—and probably his friends—are familiar with. This kind of text is another way the
drawing communicates, partaking of the meaning potentials (Halliday, 1978) or affordances of that
particular genre, one that likely figures in the out-of-school reading activity of some members of
this group. At the same time, Alan’s text incorporates the social language of school as well as out-of-
school activity. As noted in the earlier analysis, there is both the school discourse (or social language)
in the declarative science knowledge, as well as the peer discourse of images, icons, and “radiation.”
And as noted earlier, the text works both with and against the discursive norms of schooling,
displaying appropriate information, while at the same time incorporating material that transgresses
the boundary of school—and the teacher’s—knowledge. In this way, Alan’s drawing may enhance
his social position—at least for now; one might infer that Alan experiences an ongoing challenge to
his position. In a later event (previously mentioned), Alan will ask Curt about the radiation depicted
in Curt’s sketch for It’s Not Fair, and Sam will admonish him, saying, “Don’t ask dumb questions—
you always do that!” The peer group is ever in flux, yet some members are more privileged than
others—and their social language and speech genres are integral to their shared world and activities.
CONCLUDING QUESTIONS
In our reading of this literacy event, we have sought to suggest a “mash-up” of some of the
theoretical tools available to literacy researchers for analyzing multimodality. But given the fast pace
of research in this emerging field, we are left with ever more questions. We conclude, therefore,
with some of the methodological and conceptual questions researchers will need to consider as they
design their own analyses of multimodality. Many of our questions arose as we attempted to make
sense of what we felt were tensions within the literature spawned by Kress’ social semiotic theory of
multimodality, and although we are critical of some aspects of this work, we do so in an effort to
explore the methodological implications for multimodal research.
108 National Reading Conference Yearbook, 58
What is the unit of analysis? A text? A literacy event? A unit of study? An individual child or youth? A
classroom? An interaction?
Not surprisingly, answering this question depends on the purpose of the research and the
theoretical lens on literacy and multimodality the researcher brings to the work, but it is important
to point out some of the issues at play in this list of analytic units. First, there is the question of what
counts as a text. The field of literacy studies has long embraced a semiotic perspective on text such
that anything that can be read or interpreted can be taken as a text, whether it is a surface (page or
screen), a gesture, a conversation, an interaction, and so on. This is significant since Kress’ theory
has inspired researchers to focus on pages or screens, although when studying classrooms, Kress and
his colleagues do expand to include gestures, movement, and talk. In addition to surfaces (Kress,
2003; Kress & van Leeuwen, 1996) and classrooms (Kress et al., 2005), researchers have identified
other units of analysis, including literacy events (Pahl & Rowsell, 2006; Whitin, 2005) and literacy
performances (Kontovourki, 2008), reflecting the particular literacy theories and research questions
that shape their work. The question of what it means to study individuals must also be theorized.
From a sociocultural perspective, the concept of individual is problematic since mind, human
creations, and meanings are understood as social: for Vygotsky, the unit of analysis was always
social, never individual (1978); for Bakhtin, all utterances are filled with “dialogic overtones” of
others’ speech, for “our thought itself—philosophical, scientific, and artistic—is born and shaped
in the process of interaction and struggle with others’ thought, and this cannot but be reflected in
the forms that verbally express our thought as well” (1986, p. 92); similarly, Gee (1992) writes that,
"mind extends beyond the skin to mutually form and be formed by an ever-changing physical and
social world" (p. xvi), and that "meaning is not in the head at all, but in the world" (p. 1).
What are the methodological implications of selecting particular theories to frame data analysis?
Here we want to raise two methodological issues that arise from Kress’ social semiotic theory:
the meaning of “social” and the positionality of the analyst. Social semiotics was first proposed by
M.A.K. Halliday (1978) to redress what he regarded as the limitations of Sausurre’s structuralist
semiotic theory, which treated language as social conventions but not as social practices. Kress later
drew upon Halliday’s insight that language must be interpreted “within a sociocultural context in
which the culture itself is interpreted in semiotic terms” (Halliday, 1978, p. 2) to propose a social
semiotic theory. Social semiotics thus seeks to explain how signs participate in social life as part of
specific meaning-making practices with all that implies about power and the social order. Kress’
transformation of Halliday’s ideational and interpersonal metafunctions into the analytic categories
of discourse and genre suggest his interest in attending to issues of power and social order. And
yet, as we suggest in the section that follows, the sociocultural and sociopolitical meanings of
multimodality often seem absent, or muted, raising the question of what is included and excluded
in social analyses of multimodality. Even in a recent collection seeking to forge connections between
New Literacy Studies and multimodality (Pahl & Rowsell, 2006), the attention to the social and
discursive takes a back seat to the multimodal ensemble. A related question arises from the emphasis
placed on the “interest” of the sign-maker in designing meanings. Kress argues that this “interest”
is culturally formed and historically produced, but the sign-maker’s perspective on her/his own
design is not always investigated, resulting in a privileging of the researcher’s reading of the text.
Designs for Multimodality 109
Social semiotics was forged from the rejection of the autonomous code, so it seems imperative to
investigate the social interests and social positioning of both the sign-maker and the sign-reader.
What is the role of critique in understanding multimodality?
We find it troubling that Kress (2000b) has assigned critique to the dustbin, an artifact of a
point in time “of relative social stability [when] critique has the function of introducing a dynamic
into the system” (p. 160). He continues:
In the new theory of representation… [a]ll these circumstances call for a new
goal in textual (and perhaps other) practice: not of critique but of Design.…
While critique looks at the present through the means of past production, Design
shapes the future through deliberate deployment of representational resources in
the designer’s interest. Design is the essential textual principle and pedagogic/pol-
itical goal for periods characterized by intense and far-reaching change. (p. 160)
This theory/politics of textual production not only seems surprisingly utopian but vests a
great deal of power in the designer’s interest. Even though we agree in the generative power of
multimodality, as demonstrated by the early studies of transmediation (e.g., Siegel, 1984; Harste et
al., 1984; Suhor, 1984), this view of design seems a retreat to individualism and a canceling of the
complex meanings of “the social” that have characterized contemporary scholarship on literacy as
a discursive practice. Who can “count” as a designer? The boys who invented a sign for radiation
were clearly designers and their act of semiosis did indeed reshape the social world of the sketching
lessons. But it also served to reinscribe the gender order so its potential to reshape the future seems
suspect.
Instead, we would argue that a critical perspective on multimodality in literacy studies is more
important than ever. The U.S. is very far from being a socially just society. Race, class, gender,
citizenship status, and sexual identity continue to serve as interlocking systems of oppression,
and with the rapid spread of neoliberal discourses into educational discourses and practices, there
is little reason to believe that critique has passed its “use-by” date. Our central thesis—i.e., the
analysis of data in studies of multimodality and literacy requires a blend of theories—calls to mind
Hillary Janks’s (2000) argument that various orientations to critical literacy (domination, access,
diversity, and design) are interdependent and cannot be theorized or practiced independently of
one another if critical literacy education is to achieve its aim. Her argument is that design—i.e.,
teaching students to select and combine multiple semiotic resources to design meanings—may end
up reproducing the status quo, remain on the margins, or privilege dominant forms if it remains
separate from issues of domination, access, and diversity.
If studies of multimodality are to contribute to the reimagining and redesign of literacy
education in schools, it needs to be located within contemporary discourses about literacies,
diversities, and power so as to offer a full account of meaning-making. Just as multimodality
involves the generative juxtapositioning of different sign systems that “contradict and balance one
another,” we think it may be similiarly fruitful to approach the analysis of multimodality with a
hybrid of different theories and analytic tools (including others we have not used here, such as
Scollon, 2001). A social semiotic theory of multimodality is a valuable resource, but it does not
stand alone.
110 National Reading Conference Yearbook, 58
REFERENCES
Albers, P. (2008). Theorizing visual representation in children’s literature. Journal of Literacy Research, 40(2),
163-200.
Bakhtin, M. M. (1986). Speech genres and other late essays. Austin, TX: University of Texas Press.
Chafe, W. (1970). Meaning and the structure of language. Chicago, IL: University of Chicago Press.
Clyde, J. A. (1994). Lessons from Douglas: Expanding our visions of what it means to “know.” Language Arts,
71(1), pp. 22-34.
Comber, B., Green, B., Lingard, B., & Luke, A. (1998). Literacy debates and public education: A question of
‘crisis’? In A. Reid (Ed.), Going public: Education policy and public education in Australia (pp. 18-34).
Adelaide: Australian Curriculum Studies Association.
dePaola, T. (1979). Oliver Button is a sissy. Orlando, FL: Harcourt, Brace, and Co.
Duncum, P. (2004). Visual culture isn’t just visual: Multiliteracy, multimodality, and meaning. Studies in Art
Education, 45(3), 252-264.
Dyson, A. H. (1982). The emergence of visible language: Interrelationships between drawing and early writing.
Visible Language, 6, 360–381.
Dyson, A. H. (1983). The role of oral language in early writing processes. Research in the Teaching of English,
17, 1–30.
Dyson, A. H. (1989). Multiple worlds of child writers: Friends learning to write. New York, NY: Teachers College
Press.
Dyson, A. H. (1993). Social worlds of children learning to write in an urban primary school. New York, NY:
Teachers College Press.
Dyson, A. H. (2003). The brothers and sisters learn to write: Popular literacies in childhood and school culture. New
York, NY: Teachers College Press.
Eco, U. (1976). A theory of semiotics. Bloomington, IN: Indiana University Press.
Gee, J. P. (1992). The social mind: Language, ideology, and social practice. Westport, CT: Bergin and Garvey.
Halliday, M. A. K. (1978). Language as a social semiotic. Baltimore, MD: University Park Press.
Harste, J., Leland, C., Grant, S., Chung, M-H., & Enyeart, J. A. (2007). Analyzing art in language arts research.
In D. W. Rowe, R. Jiménez, D. Compton, D. Dickinson, Y. Kim, K. Leander, & V. Risko (Eds.),
56th Yearbook of the National Reading Conference (pp. 254-265). Oak Creek, WI: National Reading
Conference, Inc.
Harste, J., Woodward, V., & Burke, C. (1984). Language stories and literacy lessons. Portsmouth, NH:
Heinemann.
Herzfeld, M. (1981, December). Looking both ways: The ethnographer in the text. Paper presented at the
Annual Meeting of the American Anthropological Association, Los Angeles, CA.
Hindley, J., Wingage, P., Cartwright, S., & Goffe, T. (1998). Usborne time traveler. E.D.C. Publishing.
Hull, G., & Nelson, M. (2005). Locating the semiotic power of multimodality. Written Communication, 22(2),
224-261.
Janks, H. (2000). Domination, access, diversity and design: A synthesis for critical literacy education.
Educational Review, 52(2), 175-186.
Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. London and New York: Routledge.
Jewitt, C. (2008). Multimodality and literacy in school classrooms. Review of Research in Education, 32, 241-267.
Jewitt, C., & Kress, G. (Eds.) (2003). Multimodal literacy. New York, NY: Peter Lang.
John-Steiner, V. (1985). Notebooks of the mind. Albuquerque, NM: University of New Mexico Press.
John-Steiner, V. (1995). Cognitive pluralism: A sociocultural approach. Mind, Culture, and Activity, 1(2), 2-11.
John-Steiner, V., Panofsky, C. P., & Smith, L. (Eds.). (1994). Sociocultural approaches to language and literacy: An
interactionist perspective. New York, NY: Cambridge University Press.
Kent, J. (1972). Jack Kent’s fables of Aesop. New York, NY: Parent’s Magazine Press.
Kontovourki, S. (2008). Becoming literate in assessment-saturated classrooms: Elementary students’ literacy
performances at the intersection of literacy curricula and testing. Unpublished doctoral dissertation. New
York, NY: Teachers College, Columbia University.
Kress, G. (1997). Before writing: Rethinking paths into literacy. London: Routledge.
Kress, G. (1998). Visual and verbal modes of representation in electronically mediated communication: The
potentials of new forms of text. In I. Snyder (Ed.), Page to screen: Taking literacy into the electronic era (pp.
53-79). New York, NY: Routledge.
Designs for Multimodality 111
Kress, G. (2000a). Multimodality. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the
design of social futures (pp. 182-202). London and New York: Routledge.
Kress, G. (2000b). Design and transformation. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy
learning and the design of social futures (pp. 153-161). London and New York: Routledge.
Kress, G. (2003). Literacy in the new media age. London: Routledge.
Kress, G., & Jewitt, C. (2003). Introduction. In C. Jewitt & G. Kress. (Eds.) Multimodal literacy (pp. 1-18).
New York, NY: Peter Lang.
Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K., & Reid, E. (2005). English in urban
classrooms: A multimodal perspective on teaching and learning. London and New York: RoutledgeFalmer.
Kress, G., & van Leeuwen, T. (1996). Reading images: A grammar of visual design. London: Routledge.
Langer, S. (1942). Philosophy in a new key. Cambridge, UK: Harvard University Press.
Leland, C., & Harste, J. (1994). Multiple ways of knowing: Curriculum in a new key. Language Arts, 71(5),
337-345.
Lemke, J. (1993, December). Multiplying meanings: Literacy in a multimedia world. Paper presented at the
Annual Meeting of the National Reading Conference, Charleston, South Carolina.
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational
Review, 66(1), 60-92.
Pahl, K., & Rowsell, J. (Eds.) (2006). Travel notes from the New Literacy Studies: Instances of practice. Clevedon,
UK: Multilingual Matters.
Panofsky, C. P. (1999). Getting to the heart of the matter: Literacy as value commitments. American Reading
Forum, (19), 13-38.
Ranker, J. (2008). Composing across multiple media: A case study of digital video production in a fifth grade
classroom. Written Communication, 25, 196-234.
Rogers, T., Winters, K., & LaMonde, A. M. (in press). The cultural sites of youth digital video production as
a new literacy practices: A play of genre, positioning and critique. Pedagogies: An International Journal.
Rosenblatt, L. (1978). The reader, the text, the poem: The transactional theory of the literary work. Carbondale, IL:
Southern Illinois University Press.
Rowe, D. W. (1986). Literacy in the child's world: Young children's explorations of alternate communication
systems. PhD dissertation. Bloomington, IN: Indiana University. Retrieved May 11, 2009, from
Dissertations & Theses: Full Text database. (Publication No. AAT 8707819).
Rowe, D. W. (1998). The literate potentials of book-related dramatic play. Reading Research Quarterly, 33(1),
10-35.
Scollon, R. (2001). Mediated discourse: The nexus of practice. London & New York: Routledge.
Siegel, M. (1984). Reading as signification. Ed.D. dissertation. Bloomington, IN: Indiana University. Retrieved
May 11, 2009, from Dissertations & Theses: Full Text database. (Publication No. AAT 8429377).
Siegel, M. (1995). More than words: The generative potential of transmediation for learning. Canadian Journal
of Education, 20(4), 455-475.
Simon, S. (1978). Killer whales. New York, NY: J. B. Lippincott Co.
Smagorinsky, P., & O’Donnell-Allen, C. (1998). Reading as mediated and mediating action: Composing
meaning for literature through multimedia interpretive texts. Reading Research Quarterly, 33(2), 190-226.
Stein, P. (2008). Multimodal pedagogies in diverse classrooms. London and New York: Routledge.
Suhor, C. (1984). Towards a semiotics-based curriculum. Journal of Curriculum Studies, 16(3), 247-257.
Supraner, R. (1976). It’s not fair. New York, NY: Frederick Warne.
Taylor, I. D. (2006). Dangerous literacies: Contextualizing the interplay of critical literacies and identities of
African American youth. Unpublished doctoral dissertation. New York, NY: Teachers College, Columbia
University.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (M. Cole, V. John-Steiner,
S. Scribner, & E. Souberman, Eds.). Cambridge, MA: Harvard University Press.
Whitin, P. (1996). Exploring visual response to literature. Research in the Teaching of English, 30(1), 114-140.
Whitin, P. (2005). The interplay of text, talk, and visual representation in expanding literary interpretation.
Research in the Teaching of English, 39(4), 363-397.
Wohlwend, K. (2009). Damsels in discourse: Girls consuming and producing identity texts through Disney
princess play. Reading Research Quarterly, 44(1), 57-83.