Available online at www.sciencedirect.

com

Journal of Pragmatics 41 (2009) 1459–1463 www.elsevier.com/locate/pragma

Book review
New Directions in the Analysis of Multimodal Discourse Terry D. Royce and Wendy L. Bowcher (Eds.), Mahwah, NJ: Lawrence Erlbaum, 2007. ISBN 0-8058-5106-2 (cloth). 403 pp. A rapidly waxing flow of ‘‘multimodality’’ floods publishers’ catalogues, journals, and conferences. This is understandable, since discourses that draw on more than just the good old language modality present themselves in ever more emphatic ways, due to developments such as the growing multifunctionality of cell phones, the rise of computer games, and the huge popularity of Youtube and Internet-surfing in general. In this monomodal review I discuss a book that promises innovative perspectives on multimodality, Royce and Bowcher’s edited volume New Directions in the Analysis of Multimodal Discourse. The coherence between the twelve chapters in the book resides mainly in the adherence to the Systemic Functional Linguistics (SFL) approach rooted in Hallidayan grammar, while Kress and Van Leeuwen’s (1996/2006) pioneering application of this approach to the visual realm is cited in every chapter. I should say straightaway that I have no first-hand knowledge of Halliday’s work. This has the drawback that I cannot verify how true to the letter or spirit the authors are to their primary sources of inspiration, and the advantage that I can assess with relative freshness what promising tools SFL has to offer for analyzing multimodality. Let me further add that my own primary affiliation is to Cognitive Linguistics and Relevance Theory models, as developed by such scholars ¨ vecses, Turner, Fauconnier, and Sperber and Wilson. However, I consider myself as Lakoff, Johnson, Gibbs, Ko a critical and eclectic follower of these paradigms. The tree is known by its fruits, and I will happily make use of any theoretical insight that can help understand multimodality, irrespective of its source. I will first briefly, and with no claim to exhaustiveness, discuss each chapter separately, and end by charting and evaluating shared tendencies. The 60 page-long first chapter by Matthiessen is devoted to ‘‘The multimodal page: A Systemic Functional exploration.’’ After some preliminary remarks, the author proposes first to ‘‘examine multimodality within language’’ (p. 11), because allegedly ‘‘the ideational resources of language provide us with a range of strategies for modeling [. . .] all modes and media of expression’’ (p. 21). Then I more or less lose track. I cannot help feeling that the many technical terms Matthiessen introduces come as a substitute for explanation, rather than as its prerequisite. The complex panels and figures (figure 1.14 has more than 80 boxes!) are not particularly encouraging either. And while the author makes some interesting observations on ‘‘articulation’’ and ‘‘prosody’’ as distinguishable modalities in spoken language, and on the rhetorical patterns in verbal texts, he has very little to say about the visual realm, let alone about interrelations between word and image. Royce usefully begins his chapter ‘‘Intersemiotic complementarity: A framework for multimodal discourse analysis’’ by briefly outlining the three metafunctions Halliday distinguishes in language: the ideational (pertaining to ‘‘meaning’’ as it arises out of the relation between text and world), the interpersonal (pertaining to the relation between communicator and addressee), and the textual (pertaining to the internal coherence of a discourse and the way it links up to extratextual context). Royce argues that these metafunctions are applicable to non-verbal communication as well and uses an Economist article, accompanied by a cartoon and a diagram, to demonstrate that ‘‘repetition,’’ ‘‘antonymy,’’ ‘‘hyponymy,’’ ‘‘meronymy,’’ and ‘‘collocation’’ are concepts that define relations both within and straddling the verbal and visual modalities on the ideational level. He comes up with similar modalityindependent concepts for the interpersonal and textual levels. Moreover, he introduces the notion of ‘‘visual message elements’’ (VMEs), ‘‘visual features which carry semantic properties [. . .] potentially realized by a variety of visual
0378-2166/$ – see front matter # 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.pragma.2009.01.007

which is part of a larger ‘‘Genre and multimodality’’ project. accessible online.’’ and ‘‘orbital’’ versus ‘‘serial’’ navigation paths. a ‘‘grapheme’’ is the visual expression of the verbal ‘‘letter. ChF] does not mean that readers of those textbooks interpret them in the same way’’ (p. closure). is made communicatively significant. but persuasively argue that this modality – space – needs to be taken into account in all communication that involves physical motion. e. English. the author also discusses photographs and headings. of a newspaper article. 278). Among the concepts he develops and discusses are the ‘‘Associating Element’’ (e. while others produced texts that explicitly reflected causal relationships’’ (p.’’ Her analysis. and visual taxis. and production/consumption-related dimensions that need minimally to be taken into account for an adequate description of how multimodal documents communicate. distance. and advances proposals for the labelling and description of pertinent variables in multimodal discourse – indispensable for enabling generalizations and formulating testable hypotheses.e. Bateman. bridging indigenous and ‘‘white’’ perspectives on history and culture. emphasizes the importance of corpus research. What I particularly like in this research. she tries to show how information in the verbal and visual modalities are connected. and Japanese readers to respond in their own language to a diagram representing the ‘‘water cycle. Mohan et al. and further cooperation is invited via the MCA. scientific textbooks in different languages.. He relies on McCloud’s (1993: 70–72) six types of frame-transition. Her tentative claim that the direction in which the rugby players in the photographs look correlates with where in the verbal text ‘‘goodness’’ and ‘‘badness’’ are discussed is suggestive. and the ‘‘visual linking device’’ (VLD). which compares traditional and electronic versions of newspapers. A case study exemplifies the approach. 70). The hypothesis is confirmed – but what has been proven? Surely there is something circular . specifically visual taxonomy. Baldry reports about the benefits and prospects of the Multimodal Corpus Authoring (MCA) system he helped develop. The authors focus on a single modality only. the latter enabling a viewer to understand how the action in one panel relates to that in a previous panel. visual.1460 Book review techniques at the disposal of the visual designers’’ (p.. is dauntingly detailed. but it is a pity that the author tends to be rather verbose. by the repetition of (part of) a character or scene. even though these types are sometimes difficult to distinguish. and their location on the page.. He proposes a typology of graphic dimensions of verbal elements (e. spatial orientation.. Bowcher draws on her co-editor’s model for ‘‘A multimodal analysis of good guys and bad guys in Rugby League Week. the central idea being that one can only discover interesting patterns after identifying and quantifying salient variables. directionality or orientation. In a fine chapter. The authors then hypothesize that ‘‘causal line texts exemplify the features of scientific discourse more than do time line texts’’ (ibid. too.’’ A sample of the texts was subsequently analyzed in more detail.g. Thibault’s chapter focuses on the visual dimensions of language. systematic data analysis. Among the ‘‘tools’’ they deploy in their analyses are the ‘‘given-new’’ dichotomy. revealing that ‘‘some participants produced texts which had a sequential line of meaning. and applies them to comics. But even if correct many more case-studies will have to be conducted to assess whether such ‘‘gaze-direction’’ mechanisms are in any sense generalizable. and visual features. ‘‘bound’’ versus ‘‘unbound’’ spaces. claiming that ‘‘the mere fact that similar graphics occur in textbooks [i. he discusses the issue of segmenting a text.g. and Henschel convincingly expound how the impact of content shifts when it is presented in different media. prosodic. Lim takes his cue from fellow contributor Martin’s earlier work to develop visual counterparts of verbal organization mechanisms. continuity of direction. This is insightful. He goes on to illustrate how the levels of content and graphic expression combined create meaning in a Duracell ad. He. similarity. elegantly underscores this point. visual configuration. 277). and that allow for comparison between documents belonging to the same genre. Importantly. asked Chinese..’’ and ‘‘logogram’’ corresponds to ‘‘word’’). Although mainly focusing on verbal aspects. visual reference. and to use an overdose of jargon to explain sensible but uncomplicated ideas. The in-depth analysis of a single case-study in the light of clearly outlined theoretical concepts provides a useful starting point for applications by other scholars to other (types of) texts.). They present lists of verbal. The case study. Delin. Among other things. a cornflakes package connotes ‘‘breakfast’’ and a tree connotes ‘‘outdoor scene’’). is that it combines commonsense theoretical concepts with honest. Martin and Stenglin describe how spatial organization and layout in a ‘‘post-colonial’’ museum in New Zealand. taking into account linguistic. drawing on Gestalt theory concepts (figure-ground.g. Thibault ends his chapter by showing how visually deviant language (as in comics and cartoons) can resonate with visuals in a way that enhances a sense of rhythm that is difficult to capture in words alone.

First. direction. but not others. Shrek. and provides fruitful angles for analyzing audiovisual modalities (see Johnson. ‘‘Japanese semiotic vernaculars in ESP multiliteracies projects’’ is Ferreira’s report of a classroom task performed by Japanese students in two consecutive years to produce a multimodal tourist brochure aimed at an English-speaking audience. It makes sense to add its opposite. and Chinese subjects. given a clear theoretical framework. spatial dimensions are crucial. typical of old-style semiotics. line. this is not always feasible. this classic type of remediation research assumes some urgency because the target audience consists of children. both in the language and visuals used. At the very least. Moreover. By definition. . Even comparing two things using the same measuring tool is already infinitely more useful for theorizing than an application to a single case-study. and dimension/motion. would pass their Physics test . this work outlines an important roadmap for the humanities’ curriculum of the near future. at least not to the degree voiced in this book. Let me signal and comment on some themes and patterns in the book. 4. Most chapters insist that teaching multimodal communicative competence is a necessary component in (secondary) education. Here. 2. shape. This confidence.Book review 1461 going on here: the only conclusion we can draw is that subjects who produced causal texts (‘‘Y happens because of X’’) understood the water cycle better than those who produced timeline texts (‘‘Y happens after X’’). and genres (see Forceville. 2006a). Many chapters subscribe to the idea that the visual modality constitutes a ‘‘system’’ comparable to that exemplified in the grammar and vocabulary of the verbal modality. and effects in the real world). He notes. I also see serious problems: 1.) has overlap with the importance attributed to ‘‘vectors’’ in the SFL framework. Comparing more than two items (dozens. The notion of a visual ‘‘grammar’’ (Kress and Van Leeuwen. Unsworth investigates how stories are affected by their ‘‘translation’’ into a different medium. which particularly in Shrek reveal that much of the prickly unconventionality of the original has been smoothed out in the big-screen version. texture. 1996/2006) needs to be used with extreme caution. A bonus of a corpus approach is that. 3. whose usefulness is demonstrated by sample analyses but which are flexible enough to be applicable to new texts. some English. comparative work can be divided over a group of students who are to earn credits for course work. He unveils a number of salient differences. He ends the chapter by presenting a number of classroom tasks that can aid multimodal communicative competence. tone. ease of production and processing. Focusing on multimodal discourses also sharpens pupils’ awareness of the relative strengths and weaknesses of different modalities. among other things. the positive and promising developments I discern: 1. Ergo. Having to teach multimodality forces researchers and educators to develop clear-cut tools and models. The term betrays the extent to which work on . For the analysis of graphic elements he draws on a list by Donis Dondis which consists of dot. but given that qualitative research of multimodal texts can be very time-consuming. My view is that this pair is very helpful in the assessment of the opportunities and impossibilities pertaining to media. detecting patterns requires comparing things. Royce’s programmatic ‘‘Multimodal communicative competence in second language contexts’’ would have made a good opening chapter. and The Little Prince and their CD-ROM/filmic counterparts – in terms of fostering a critical ‘‘pedagogic practice’’ (p. hundreds) is better. of course. is not warranted. modalities. though sometimes small ones. and that language and cultural background had no effect on this. The laudable use of corpora. surfaces in several chapters. appropriateness to context. color. ‘‘constraints’’ (that what you can not do with a thing or concept). 1987). In the visual modality. Specifically the ‘‘source-path-goal schema’’ (discussed by Mohan et al. . Unsworth emphatically presents the comparative multimodal analyses – of Stellaluna. given that space involves movement it is not surprising that several chapters mention ‘‘embodiment’’ – a central concept in the Cognitive Linguistics approach. Japanese. An often-quoted term in the volume is Gibson’s (1979: chapter 8) notion of ‘‘affordance’’ (roughly: that what you can do with a thing or concept). 333). that Japanese right-left writing and the possibilities for vertical layout of text can have (sometimes: unintended) effects on the visuals in the brochure. scale/ proportion. He takes his inspiration from the ‘‘New London Group’’ to promote the teaching of multimodal competences by extending Dell Hymes’ four criteria for effective communication (grammatical/formal correctness.

forthcoming. We thus run the risk of being blind to the specificities of these other modalities. London. McCloud. Poland. Out of the Gutter. Jefferson NC.1462 Book review multimodality is undertaken by scholars trained as linguists. McGraw-Hill. 2005b.). Mark. Berlin. Urios-Aparisi. Chicago. HarperCollins. Kurt. D. Charles. Berlin. Boston. Leech. Scott. R. Michael H.. Forceville. James J. 3.. Longman. 1987. Leech. Short. Jeulink. Journal of Pragmatics 37. while McCloud’s work (1993.. Thompson. Paper Presented at the International Cognitive ´ kow. 2. 1996/2006. I am worried that they are simply considered as facts that can be unproblematically used to ‘‘prove’’ things. New York. while many of the concepts proposed by Kress and Van Leeuwen (1996/2006). 2006b). 2008. In: Kristiansen. As to the short shrift paid to the visual modality: to some extent. Linguistics Conference. Multimodal Metaphor. Non-verbal and multimodal metaphor in a cognitivist framework: agendas for research. Charles. London. G. Language and Literature 8. ´n ˜ ez. Geoffrey N. Charles. 163–178. (Eds. 1999).. 241–261. Van Leeuwen. Leech and Short. 1981. Cognitive Linguistics: Current Applications and Future Perspectives. Moreover. Johnson. E. 1966.. 2007. 1969. pp. Understanding Comics. Charles. The HUMAN VICTIM IS ANIMAL metaphor in horror films. 1999. 1979. Routledge.). C. 2006. 2006) holds promise for comics). 1981). Charles. and associated with the journal Language and Literature. 423–444. Scott. space. (Eds. For my own recent attempts to practise what I preach. (forthcoming). moving images. Van der Keuken.b.’’ pioneered by Leech and Short (Leech. 2006b. English in Advertising: A Linguistic Study of Advertising in Great Britain. 2006a.. Kress. Eggertsson and Forceville (in press). 69–88. 1993. 1966. Reading Images: The Grammar of Visual Design. London. sound. McCloud. Forceville. pp. Gunnar The (Eds. The source-path-goal schema in animation film. compared to the verbal modality. Marloes. 1969. . see Forceville (2005a. The Body in the Mind: The Bodily Basis of Meaning. Hassler-Forest. Apart from the fact that ‘‘modality’’ itself is not adequately defined. Dirven. Gibson. such as given-new. As to the verbal modality: analysis in this vein continues (uncited) work in the venerable tradition of ‘‘stylistics. this book focuses predominantly on (the interrelations between) two modes only: the verbal and the visual. The Ecological Approach to Visual Perception. In: Forceville. music. Even the visual modality gets little attention. by definition. 247–278. this is inevitable. are exciting and potentially fruitful... Charles. Achard. F. David. and Forceville et al. Paradox. Geoffrey N. Film Art: An Introduction. References Bordwell. Charles. A Linguistic Guide to English Poetry. Forceville and Jeulink (2007). and gestures. Hard work is needed to develop this non-linguistic knowledge. ´ odor. place. and vectors. 2005a. Making Comics: Storytelling Secrets of Comics.. 8th ed. are absent or downplayed in verbal communication. for instance. in press. Tony. the latter because the language model may blind researchers to affordances characteristic of other modalities and media that. Ruiz de Mendoza Iba Gruyter. Gunther. London. Longman..). and of dangerously narrowing the concept of multimodality. Addressing an audience: time. Educating the eye? Kress & Van Leeuwen’s Reading Images: the grammar of visual design. Forceville. Martin and Stenglin are absolutely right to point out that the two-dimensional page ‘‘affords’’ representing language and (static) pictures at the expense of. Forceville. University of Chicago Press. New York. Longman. Leech. Feyaerts. Jagiellonian University Kra Forceville. J. basic knowledge of how they function on their own is indispensable. McFarland. Boston. Theo. 15–20 July. Manga and Graphic Novels. Forceville. Geoffrey N. Forceville. centre-margin. But in order to discuss how different modalities interact in multimodal discourse. and entails the risk that claims about the need to acquire visual competence are either exaggerated or misdirected – the former because understanding pictures comes to humans a lot more naturally than understanding language. Forceville. 379–402. Balloonics: the visuals of balloons in comics. Mouton de Gruyter. Kristin. Mouton de M. since there are few tried-andtested tools for the analysis of contemporary visual culture (film theory is an exception. Houghton Mifflin. Visual representations of the Idealized Cognitive Model of anger in the Asterix album La Zizanie. whereas in reality these concepts are at present no more than hypotheses requiring further critical evaluation as well as empirical testing (see Forceville. see Bordwell and Thompson (2008). The source-path-goal schema in the autobiographical journey documentary: McElwee. otherwise what is bound to happen – a problem surfacing in many of the chapters of the current book – is that the visual modality is seen as always merely supportive of the verbal. Humor 18. Eggertsson. Imagination and Reason. The New Review of Film and Television Studies 4. Cole. In: Goggins. and genre in Peter Van Straaten’s calendar cartoons. Style in Fiction: A Linguistic Introduction to English Fictional Prose. Veale. Charles.

medewerker. In 1996. University of Amsterdam Turfdraagsterpad 9.j.uva. Since 1999 he has been employed in the Department of Media Studies of the Universiteit van Amsterdam. is forthcoming.nl. he published Pictorial Metaphor in Advertising (Routledge).j. Ch.nl/c.forceville/ . where he subsequently taught in the Departments of English. URL: http://home. the volume Multimodal Metaphor (Mouton de Gruyter). cartoons). His teaching and research interests focus on the structure and rhetoric of multimodal discourse in various media and genres (documentary. Between 1987 and 2007 Forceville wrote some 200 literary reviews for the Dutch newspaper Trouw. Forceville* Department of Media Studies. co-edited with Eduardo Urios-Aparisi. 1012 XT Amsterdam. fax: +31 20 5254599.J. comparative literature. comics. E-mail address: c. and Word & Image. where he is now associate professor and directs the Department’s Research Master programme.forceville@uva.: +31 20 5254596.Book review 1463 Charles Forceville studied English language and literature at the Vrije Universiteit. animation. The Netherlands *Tel. Amsterdam.

Sign up to vote on this title
UsefulNot useful