Multimodality A social semiotic approach to contemporary communication Gunther KressMultimodality ‘Gunther Kress’s Multimodality is the definitive statement of a social semiotic approach to communication in all its modes and their admixtures from the man who Virtually invented the modern form of this field’ James Paul Gee, Arizona State University, USA “This book represents a paradigm shift beyond linguistics and literacy studies as we have known them. Yet, for all its path-breaking intellectual innovation and conceptual profuncity, it contains a d'sarminaly lear view af meaning-making, one in which human agency is at the centre.’ Mary Kalantzis, University of llinois at Urbana-Champaign, USA The twenty-first century Is awash with evermore mixed and remixed images, writing, layout, sound, gesture, speech and 3D objects. Multimadality looks beyond language and ‘examines these multiple modes of communication and meaning-making ‘Multimodality: A Social Seriotic Approach to Contemporary Communication represents a long:awaited and much anticipated adcition tothe study of multimadality from the scholar who pioneered and continues to play a decisive role in shaping the field. Written in an accessible manner and illustrated with a wealth of photos and illustrations to clearly demonstrate the points made, Multimodalty deliberately sets out to locate communication Inthe everyday, covering topics and issues not usually discussed in books of this kind, from traffic signs to mobile phones. In this book, Gunther Kress presents a contemporary, distinctive and widely applicable ‘approach to communication. He provides the framework necessary for understanding the attempt to bring all modes of meaning-making together under one unified theoretical roof. This exploration of an increasingly vital area of language and communication studies will be of interest to advanced undergraduate and postgraduate students in the fields of English language and applied linguistics, media and communication studies and education, Gunther Kress is Professor of Semiotics and Education at the Institute of Education, versity of London. His numerous titles include Reading Images (co-author with Thee W) Leouwen, 2nd edn, 2006), Literacy in the New Media Age (2003), Early Spelling (1999), Before Writing (1997) and Learning to Write (1993); all published by Routledge.Multimodality A social semiotic approach to contemporary communication Gunther Kress 4) Routledge RoteFirst published 2010 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN, Simultaneously published in the USA and Canada by Routledge 270 Madison Ave, New York, NY 10016 Routledge san imprint ofthe Taylor & Francis Group, an informa business ©2010 unter Kress ‘Typeset in Bell Gothic by Keystroke, Tettenhall, Wolverhampton Printed and bound in Great Britain by Td International Ltd, Padstow, Cornwall All rights reserved. No part ofthis book may be reprinted or reproduced ‘or utilized in any form or by any electronic, mechanical, or other means, now known or hereafter ievented, including photocepying and recording, or in any information storage oF retrieval system, without permission Jin writing from the publishers. British Library Cataloguing in Pubication Data A catalogue record for this book is available fram the British Library Libcary of Congress Cataloging-in-Publication Data Kross, Gunther &. Multimodalty : a social semiotic approach to contemporary communication” Gunther Kress. pcm, Includes bibliographical references and index 1. Semioties. 2. Communication. I. Title, P99.K64 2009 302.2-¢22 ISBN10: 0-415-32060-7 (hbk) ISBNLO: 0-415-32061-5 (pbid) ISBN10: 0-203-97003-9 (ebk) ISBN 13: 978-0-415-32060-3 (hbk? ISBN 13: 978-0-415-32061-0 (pbk) ISBN 13: 978-.0-203-97003-4 (bk) Meinen Eltern Elly Johanna Charlotte Kress, 1912-1988 Johann Michael Kress, 1911-1987 Zwei Leben, gegen eine grausame ZeitContents Chapter 1 fi CiApter J List of ilustrations Preface Acknowledgements Where meaning is the issue Multimodalty: simple, really 1 From semiotic system to semiotic resource 5 Cultural difference and communication: the ‘reach’ of the theory and the ‘reach’ of modes & The politics of naming 12 A satellite view of language 15 ‘The social environment of contemporary communication ‘An ethical approach to communication 18 Assumptions 19 Environments for communication: social frames and communicational possibilities 19 Power, authority and authorship 21 Social and theoretical consequences: ruling metaphors of participation, design, and production 22 Personal choices: existential insecurity or agency through particivation and connection 23 Communication and meaning: fluidity, provisionality, instability 23 A prospective theory of communication: rhetoric, design, production 26 From language and grarnmrar to semiotic resources 27 Mobitity and portability 28 A word on ‘pace’ 29 The nex! for apt metaphors 30 13Contents > Giliel Part! 746 _/Shapter 3_/communication: shaping the domain of meaning 32 C Chapter 4 Labo ¢ Chapter 5 Chapter 6 Chapter 7 Communication as semiotic work: a sketch of a theory 32 ‘Reading’ and the reader’s design of meaning 37 Provisianality in communication: rhetoric and design, newly configured 43 Environments of communication: ahistorical view 46 Refashioning social and semiotic domains: rhetoric and design 49 A | of pyinpsatiy Ls Yptlrdlong! 54 From 4 linguistic to a *ultimdAal social-sefnidtic theory of 4 ~ meaning ané communication 54 Linguistics pragmasiewaga a speak semiotic aporaach V to represetatio ss A Horses for courses: at theories seful Frarflg“60 The motivated sian 62 The everyday, the banal and the motivated sion 65 Interest and the partiality of reoresentation 70 Mimesis, signs and embodied experience 7 y ” é ‘Materiality and affordance: the social making of mode 79 The ‘reach’ of modes 83 What is a mode? 84 Is layout a mode? 88 ‘Mode, meaning, text: ‘fixing’ and ‘framing’ 93 Mode as technology of transcription 96 Mode Meaning as resource: ‘naming’ in a multimodal social-semiotic theory 103 Naming aptly 103 New frames, new names. 105 Making signs: resources, processes and agency 107 Processes and effects: making and remaking meaning 120 Design and arrangements: making meat 19 material 132 Design in contemporary conditions of text-making 132 Design: an essential (refocusing 133 Wiat is design? A homely example 135 Desiga in social-semiotic environments 1 Changes in design: a brief look at recent history 14) Chapter 8 Chapter 10 Contents Arrangements: making meanings material 145 What else is framed? 153 Discourse: ontological and epistemological frarning 157 Multimodal orchestrations and ensembles of meaning 159 The world arranged by me; the world arranged for me 159 The world arranged by me, the world arranged for me: ‘orchestrating ensembles, staging of movement, motion, “pace” 162 Aesthetics, style and ethics in multimadal ensembles 169 Applying the theory: learning and evaluatio knowledge identity and Learning and identity in a communicational frame 174 Reading as design 175 Semiosis, meaning and learning 178 Recognition, metrics and principles of assessment 182 ‘The social semiotics of convergent mobile devices: new forms Cf composition and the transformation of habitus (Elizabetta Adami and Gunther Kress) The social frame for the semiotic analysis 184 Tne affordances of Smartphones: a social-semiotic account 185 Affordances of the hardware 186 “Shape’: designers" intentions and social implications 187, The representational affordances 187 The functionalities 189 Mobile Web access and usability: changes in social habitus 190 Irnplications 193 Gains and losses: some open questions 195 References Index a7? 1e4 198 208Illustrations Colour plates (between pages 78 and 79) Colour plate 1 The operating theatre Colour plate 2 CBBC homepage 2005 Colour plate 3. Salt and pepper, SAS: ‘health’ Colour plate 4 Salt and pepper, Delta Airlines: ‘social practices’ Colour plate 5 Salt and pepper, Lufthansa: “racition’ Colour plate 6 Salt and pepper, Austrian Airlines: “light and cheeky’ Colour plate 7 Bottle: as ‘toiletry item’ Colour plate 8 Bottle: as ‘ug’ Colour plate 9 Lil Snail 1 Colour plate 10. Lit Snail 2 Colour plate 11 Hedgehog, Em Colour plate 12 “My Birthday’: affect Colour plate 13 Lego car Colour plate 14 Outside the room: ‘Armenian wedding’ Colour plate 15. Inside the room: ‘Armenian wedding’ Figures 1.1 Morrisons supermarket 1.2 Waitrose supermarket 1.3 Temporary parking arrangements, Salzburg, UEFA 2008 1.4 Cell with nucleus 3.1 Saussure‘s diagram of communication 4.28 Map of a museum exhibition ‘London before London’: ‘the prehistoric cam’ 4.2 Map of a museum exhibition ‘London before London’: ‘Heathrow! 3.3 1935 Science: digestion 4.4 2002 Science: digestion 4 “This isa car’ 4.2 Football: “Arsenal vs. Chelsea? 4.9 Child's drawing of the visit to the British Museumli Ilustrations Ba 52a 5.2b 5.26 53 54 55 56 57 58 59 baa o.1b 62 63 64 65 b.ba 6.6b 7 72 73 74 aa a2 a3 a4 85 gaa 9ab 10.1a 10.18 10.2 2002 Science: digestion The digestive system: columns reversed The digestive system: rearranged Blood-circulation: concept map Alphabetic writing Character-based writing Western timeline The toner of my memories Modular layout: ‘dependence’ independence’ Bento box Senate House, University of London Entrance hall, Senate House Passport photo Posed shot EU passport stamps, ‘entered’ —»> and ‘departed’ < Giorgia Michael: classification 1 chorne) Michael: classification 2 (school) Potato peelers: new and traditional Poetry Archive website: General homepage Poetry Archive website: Teachers’ homepage Poetry Archive website: Children's homepage Blood circulation: circle Blood circulation: arrows and loop What English is for: English as popular culture What English is for: English as National Curriculum CBBC homepage 2005 Frogspawn 1 Frogspawn 2 ‘Smartphone closed Smartphone open Aesthetics of interactivity 85 90 a OL 95 8 98 100 100 101 101 cert n2 5 15, 8 us 126 127 135 140 140 ai 163 163 167 167 17 180 180 186 186 189 Preface This book has been a long time in the writing. I tell myself that maybe it will be the better for that. All this time a lot of work has been going on under the label of ‘iwitimodality’. While I have not documented any of that here, the traces of it are evident everywhere in the book. Work in multimodality comes from quite different disciplinary) perspectives to that taken here and one part of the task at present is to see how to make use of the connections. Carey Jewitt’s Handbook of Multimodal ‘Analysis (2009) gives a very good sense of the range and diversity of that work; and. of the interconnections. My aim here is to acknowledge the many people who, over a considerable time, hhave helped me in al sorts of ways in shaping the ideas: the smallest reconition of a (great gift of conversation, of challenge and argument, of a need to explain and defend ‘and to add; in many places, over many years. And, most important of all maybe, to have a sense of interest, support, engagement from and with like minds. I have been lucky in colleagues and friends whese ideas 1 admire and whose judgements I trust; ho took ideas about which I felt not a bit confident as ideas just like others, with a simple °Yes, 507’; or, more often, getting straight into a discussion, just carrying ‘on with the conversation, For many years | have wanted to mention Bob Hodge in that respect. We became olleagues at the University of East Anglia in the early 1970s. We taught courses Jointly; with him it felt safe being ignorant. Ideas that | had never had the courage to ay out aloud even to myself became the ordinary stuff of our talk, of disagreement ‘and agreement, of discussion. That gave me the confidence to say things; to try to publish ther even. In that long conversation, with David Aers and with Tony Trew too, we developed what has remained for me most important about meaning. | co-taught and co-wrote with the three of them (Aers, Hodge, Kress, 1982; Fowler, Hodge, Kress, Trew, 1979; Hodge and Kress, 1979). Roger Fowler, by then already tablished author, was generously supportive in our efforts to get this work published. Out of teaching a joint course on Philosophy and Linguistics over several years, Tony Trew and I wrote 2 book which we called The Chomskian Caunter- revolution in Linguistics; a book about Chomsky wresting Linguistics back from the jocially oriented path set out by his teacher Zellig Harris; turning transformations Irom the sociallyextually/discursively motivated operations they were in Harris's, writing and theory, into psychological operations. It was not a notion to catch the fnvood of the mid-1970s; and so the book was never published,v Preface Ten years on, in Sydney, I met Mary Kalantzis and Bill Cope: enormously generous personally; as profoundly serious in their intellectual positions as In their political views; utterly remarkable as activists. There too, Jim Martin and I joined forces to ‘support each other's thinking around genre, shelving theoretical differences in favour of comradeship, support and the shared political-pedagogical aims of equitable access inschools. And in Sydney Theo van Leeuwen and I started on a project trying to figure ‘out something akin to a ‘grammar’ of the visual. That project lasted a good ten years; with great enjoyment and benefit for me. Moving to London in 1991 meant moving to the —for me — new ‘site’ of Education Three research projects conducted jointly with Jon Ogborn, then Professor of Science Education, over the next nine years, here at the Institute, supplied a wonderful proving ‘ground for developing social semiotics; as well as an intensive part-time education in matters scientific with a wonderful teacher. In the last of these, ‘The rhetorics of the science classroom, Jon decisively ‘scaffolded’ ane of my more stumbling serniotic, attempts by throwing the word ‘affordance’ into the conversation. Carey Jewitt had come as the Research Officer on that project; she stayed on for the next one, “Enlish in urban classrooms’; and we have worked together since: enjoyable, illuminating, challenging, now as much as on that first project. She has been a source of inspiration and inestimable support. In London | met Brian Street. We decided early on that complementarity might be more productive than contestation; and maybe more ‘enjoyable; and so it has proved for me and I hope for him. ‘Ata conference in Toulouse, in 1990, I had met Ben Bachmair. Since then we have talked and worked on different projects around media and media education, talked and wrote about culture and communication. For me, his friendship has been truly delightful and an insightful means of reconnecting with things German; while hi (predominantly German) sociologicaUhistorical ‘take’ on education has been eye- ‘pening in thinking about issues of pedagogy. Christoph Wull’s more anthropological interest in ritual, in performance and his work on mimesis in particular, pushed me to try to became clearer about my own interests in meaning and learning, especially in relation to "the body’. Ingrid Gogolin’s coinage of the term ‘mono-lingual habitus! ~ in her work with immigrants in Germany ~ made me see Bourdieu’s concept con- cretely. In New London, as it turned out, | met Jim Gee. With his usual nonchalance, hhe made clear for me the connections of pedagogy, curriculum and “ast capitalism’, ‘while shooting some baskets with Phillip, the youngest member of our group, in the parking lot at the back of the hotel; for me, with the same ‘white light’ effect of Jon Ocborn’s mention of ‘affordance’. Staffan Selander had invited me to be a Visiting Professor in his Didaktik Design ‘group at the Laerarhoegskolan in Stockholm, between 2004 and 2006. His distinct sociological approach ~ broad, generously encompassing, precise and with a strangely real sense of the geographical proximity of Russian psychology ~ has been a constant stimulation and a challenge to staying alert. The time in Stockholn) gave me a chance to be still; to think and talk in an atmosphere of a taken for yranted intellectual an personal generosity. Among the scholars in that group, | Lenetited greatly trom Preface ‘conversations with Eva Insulander and Fredrik Lindstrand and their work. [also came lo know Roger Saljé, whose ways of thinking ~ with so much affinity to a social semiotic perspective and yet so productively distinct — have proved greatly enriching for me In London I met Denise Newfield and Pippa Stein. Both had been working at the University of the Witwatersrand during the years of apartheid and at the same time they had worked with teachers and children in schools in Soweto. Through their ‘experiences and in their reflection on that experience they had much more to teach me In their doctoral research than I could offer them in return. Their hopes for those students and teachers and thelr unbounded optimism for them in a future South AAirica showed me how entirely linked inspired academic work and political action are. Pinpa Stein's death since, is for me, as for so many people in mary places, a vast loss. ‘Some two years ago Roger Kneebone approached me with a view to doing some Joint research on the operating theatre seen as a pedagogic site. That has opened a ‘whole nev world of challenge, as well as truly enjoyable conversation. My colleagues at the Institute of Education continue to inspire and support me, with coffee and conversation. { want to thank Di Mavers for her enthusiasm, energy and sharp insights into meaning-making. Jeff Bezemer brought his ethnographic take land ways of working, a great addition to the social-semiotic stew. Anton Franks has heen an always insightful companion, Caroline Pelletier, Hara Sidiropoulou and Sophia Diamantepoulou have given me much material for thought from thelr research ‘and their own reflections; they have — I hope wittingly ~ been willing to listen and respond to my ideas. Jan Blommaert brought his galvanizing energy and ideas, so close yet with an importantly different take, uring his much too brief period at the Institute of Education, Norbert Pachler has been an enormously patient guide to some Of the pedagogical and wider social affordances of contemporary technologies; and in that, has helped me get some real sense of its profound implications. have learned with all the doctoral researchers with whom I have worked. It is Invidious to single out individuals; yet some need to be mentioned. Paul Mercer's exploration of forms of representation and social meanings in TV programmes ins for me an exemplary detailed study in semiosis, as well as an important re- jssment of the category of realism. Lesley Lancaster's meticulously detailed study ff semiosis of a two-year-old; Shirley Palframan’s study of semiosis in the work of early teenage students in classrooms, all have far-reaching implications theoretic~ ally and practically for learning and assessment and pedagogic practices generally. Inv the work that she calls ‘re-genreing’, Fiona English has opened new horizons on teaching and learning and representation. Elisabetta Adami came for a year and with hhor [ was able to think again about issues long debated in Linquistics and seemingly led ~ Grice’s conversational maxims for instance, which get pretty well stood on their head in the triple context of multimadal representation, ‘digital media’ and a social-semiotic theory of communication. Sean McGovern’s work on the semiotics of Japanese culture has far-reaching implications; and it has been very insightful for me. J wish to thank him for allowing me to use some of his data; Ihave borrowed from himwi Preface the notion of ‘modular composition’ and want to make acknowledgement of that right here, Marie-Agnes Beetschen and more latterly, Rebecca Elliott, have given me an ‘opportunity, over a considerable time, to learn, in a very full sense, about the issues of representation in the practical world — in a commercial context with its pressures and demands. That has been an invaluable source of insight and of confirmation too, for me. I wish to thank them for their wisdom, insight and support. Margit Boeck gave me comments that were incisive and proved crucial at just the right moment; what coherence and order the book may exhibit is due to her clear advice. [want to thank Lilly Ramp, who kindly took some photas of a sign by the side of a ‘ousy road in Salzburg. Rachel Kress gave me permission to use some of her images and drawings (colour plates 9, 10, 14 and 15), as have Michael Kress and Emily ress, and for that | wish to thank them. Eve Bearne kindly allowed me to use some of her data, which has been very insightful for me; it appears as Figure 4.3. The book draws on three research projects, funded by the UK's Economic and Social Research Council: ‘Gains and Losses: Changes in Representation, Knowledge and Pedagogy in Learning Resources’ (2007-2009); “English in Urban Classrooms’ (2001-200); and "The Rhetorics of the Science Classrooms’ (1998-2000). It aso draws on a project funded by the Swedish National Science Foundation, conducted in Stockholm and London "The Museum, the Exhibition and the Visitors’, directed by Professor Staffan Selander of Stockholm University and co-directed in London by myself. The deep benefits of such Support will, | hope, be evident in the pages of the book. Anyone who engages in this sort of work knows about the people who are essential yet 100 often remain unsung. No academic can work well without the help of those who smooth the paths of administration and who create a climate conducive to feeling ‘supported’. In that respect | would like to mention Sarah Gelcich and Manjit Benning for their help; differently, and each essential. It remains to thank the more than patient editors: Louisa Semlyen, over many years, and latterly Sam Vale Noya, whose job was not made one whit easier by my slowness; and without whose patience and increasingly firm pushing this book might still not be done. GRK London, October 2009 Acknowledgements ‘The publishers would like to acknowledge the copyright holders for permission to reprint the following materi 1935 Science: digestion from General Science, Part I1T, Fairbrother, F., Nightingale, E, and Wyeth, Ful. (G. Bell & Sons, 1935). 2002 Science: digestion from Salters GCSE Science: Year 11 Student Book (Heinemann Educational Publishers, 2002). Austrian Airlines for the reproduction of their salt and pepper packaging. The Poetry Archive for Screenshots from their website at www.poetryarchive.org, British Broadcasting Corporation for the screenshot from their 2005 website at worm bbe.co.uk/ebe. Every effort has been made to contact copyright holders. If any have been inadver- lently overlooked the publishers will be pleased to make the necessary arrangements ‘at the first opportunity1 Where meaning is the issue Multimodality: simple, really On my way to work the bus gets held up before a large intersection, even quite early In the morning. Sitting on the top deck, my eye is drawn to a sign, high up on the wall opposite; it shows how to get into the car park of a supermarket. It is nat a com- plicated sian by any means, nothing unusual about it really. But I have puzzled about It: how does it work? Above all, how does it work here? It is about 150 metres before this complicated intersection. Drivers have to keep their eye on the traffic; there’s fo time for leisurely perusal. OF course, my academic interest in the sign les in its Joint use of image and writing. And so, one morning, when the bus is held up in just the right spot, I take a photo on my mobile phone, as one does (Figure 1.1). If writing alone had been used, would this sign wark? I don’t think it could: there 4s to0 little time ta take it in. A litte later in the day, if shoppers tried to read the sign, the intersection would clog up. With writing alone, the message would, quite simply, be 100 complex. Using three modes in the one sign ~ writing and image and colour as ‘well ~ has real benefits. Each mode does a specific thing: image shows what takes too Jong to read, and writing names what would be difficult to show. Colour is used 10 highlight specitic aspects of the overall message. Without that division of semiotic labour, the sign, quite simply, would not work, Writing names and image shows, while colour frames and highlights; each to maximum effect and benefit. If writing by itself would not work, could the sign work with image alone? Well, Just possibly, maybe. Writing and image and colour lend themselves to doing different Kinds of semiotic work; each has its distinct potentials for meaning ~ and, in this case, linge may just have the edge over writing. And that, in a nutshell ~ and, in a way, as simple as that ~ is the argument for taking ‘multimodality’ as the normal state of human communication. Except that, just across the road, on the other side, there is another supermarket. It too has a sign, on its side, just as high uo; it shows its customers how to get into itscar park. Figure 1.2 isa photo of this other sign. The sign is different: not different Ii the modes used but in how the modes are used. Colour is different, tines are differently drawn; the sign has a distinctly different aesthetic. Multimodality can tell Us Whiat modes are used; it cannot tell us about this difference in sty; it has no means {to toll us what that difference might ean. What is the difference in colour about or UMuttimedatity 0058-9 Leys 1] LJ L. Carpark OFF HERTSLET ROAD a) oe) | Figure 1.1 Morrisons supermarket the difference in the drawing style? What identity is each of the two signs projecting? What are the supermarkets ‘saying’ about thernselves? What are they telling their ‘customers about themselves? Are these differences accidental, arbitrary? Would the style of one sign serve equally well for the other supermarket? To answer questions of that kind we need a theory that deals with meaning in all its appearances, in all social occasions and in all cultural sites. That theory is Social Semiotics Of course, there are many theories of what communication ought to be and how it might work. And then there is power. So just because something might nat really work writing by itself, in this ease —it does not mean that it wor't be used. There ar2 many n more important than ‘will it work?’. There is tradition, for instance. Writing has traditionally been used to do certain communicational things ~ regulations id Instructions being just two. Then there is officaldom. Bureaucracy assures that as long as something has been announced in writing it has been communicated and the rest will look after itself; or else it can he left to the law, where the excuse ‘but nobody could have read it in that time" doesa’t count. Here (Figure 1.3) isan example ofthat latter approach, It comes from ane of the cities whicly played host to games in ne European Soccer Championships of 2008. The sign ann temporary ehanay Where meaning isthe issue Fiqure 1.2 Waitrose supermarket {o parking regulations for the duration of the championships. A rough translation might be: Dear drivers! During UEFA 2008 the official times for bus lanes will be changed as follows. From 7 June to 29 June 2008, the normally applicable times for the bus lanes 1e Griessgasse, Rudolfskai, Imbergerstrasse, Giselakal and Schwarzstrasse will be extended until 2 a.m. of the following day. On days when games are being played, that is, 10.06, 14.06 and 18.06 2008, the ‘no-stopping rule in the bus lane in Griessgasse will come into force earlier, from 10am The instructions are not unfriendly ia tone; though they are complex, official and, above all, impossible to read in the brief time before the lights ahead turn again. But then, there are times ~ perhaps many times ~ when communication isn't really the Issue, and power is. That isa crucial point to bear in mind in thinking, theorizing and Writing about meaning, communication and soctal mattMultimodatity tober ge Figure 2.3 Temporary varkiny arraiyements, Salzburg, UEFA 2008 Where meaning is the issue ‘Simple paints often have profound consequences: and so it is here: consequences for learning, for knowing and shaping information and knowledge, for attending to and communicating about the world and our place in it. Developing ways of thinking about This a once simple_and complex phenomenon’ ~ that Ts, setting OUCa-sacial- semiotic theory of multimadality —is what th's book is about. From semiotic system to semiotic resource If that is the issve, then one ought to ask ‘What has produced the explosive interest In the issue of multimodality over the last decade or so?’ and, most obviously, ‘Why fnow?’. Image has been a part of human cultures longer than script ~ though the difference Between the two is not at al clear-cut. Image has been the subject of much interest, academic or otherwise, aver millennia. Gesture is a presence in all cultures, fen if in quite different ways, As ‘sign language” it has been elaborated and articu (ated intora-fanly Tunetioning representational resource. Whether as TeSTUre OF as signing it has been much stuaied (British Deaf Association, 1992). Academic ‘disciplines have their interest in particular modes: Psychology in gesture; AFt History in image, as has Mathematics, if differentiy so; music is studied in conservatories {te world over. One difference is that whereas before these were the subject of interest t areas of academic work, now there is an attempt to bring all means 1 making meaning together under one theoretical roof, as part of a single field in a unified account, a unifying theory. ‘A further reason, quite simply and yet most powerful of all, this: the world of ‘onynunication has changed and is changing still; and the reasons for that lie in a vast \web of intertwined social, economic, cultural and technological changes, ‘Any attempt at a satisfactory answer to the questions “Why?! and ‘Why now?’ has to uo in the direction of the Factors, everywmere connected, which have been and still fare sweeping the world. One shorthand term wich points to a collection of these is globalization’ ~ which, for me, refers not only to financial globalization but to Conditions which make it possible for characteristics of one place to be present {ani active in another — whether economic or cultural or technological. The Forces ‘of neoconservative ideology have sponsored and amplified these conditions, though they have been only partially causal. The factors at issue ~ let us say, forms of Management, or ideologies around schooling — always impact in one locality and there they enicounter locally present factors. There isa struggle of local forms and traditions With the features from outside, in which both are transformed, transformed in ways dependent on arrangements and dispositions of power in a locality. The effects of this vastly diverse and complex phenomenon have led in very many places to the corrosion, fraying, dissolution, desteuction and abandonment of older relations, forms, structures, ‘givens’. Globalization is not ane ‘thing’ it is differ ly constituted i different places, as are its effects and impacts, interacting with thw vastly varied cultural, social, economic and political conditions af any ane specific locality, Yet the deep effects are constant yind recognizable everywhere, They have 56 ‘Muttimodality brought a move from a relative stability of the social world over maybe the last two centuries (as in Western Europe) to an often radical instability over the last three decades or so. Stemming fromm that ~ and generated by it ~ are far-reaching changes in the domain of meaning: in representation and in semiotic production’; in dissemi nation and distribution of messages and meanings; in mediation and communication Al have changed profoundly. “The semiotic effects are recognizable in many domains and at various levels: at the level of media and the dissemination of messages — most markedly inthe shift feom the baok and the page to the screen; at the level of semiotic production in the shift {rom the older technologies of print to caital, electronic means; and, in repre- sentation inte shift fom the dominance ofthe mode of writing to the move of image, as wall as others. The effects are felt everywhere, in theory no less than in the practicalities of day-to-day living, Academic interest in the characteristics ofthis new communicational world, the world of the screen and of multimodality, has been relatively belated, stumbling after the horse which had left the stable some while ago. Belated or not, there isa need to catch up and get back inthe saddle. The effects of globalization are clear. From (relative) permanence and stability there hasbeen aimarked Sh to orovisionality ang instability. Curentiy fashionable metaphors are always revealing in that respect. For instance, while lam writing book, the metaphar of mobility has areat curreney, as in °mobile technologies’ or “mabe learning’. Like others, I se such metaphors; they are attempts to name and capture something ofthe essence ef the alterations, transformations, remakings of social arrangements and practices. Desig, isa term Ihave been usig since the early 1990s; for mei indicates a shift away both from the earlier social and semiotic goal of competence as from the somewhat later critiave Design accords recognition to the work of individuals in their social lives and builds that into the theory. in my use of the term, designs about a theory of communication and meaning, based — atleast potentially —on equitabe participation inthe shaping of the social and semiotic world. Design, by contrast with competence, foregrounds a move away from anchoring communication in convention as social regulation Design fecuses on an individual's realization of ther interest in their world. The move away from critique has, for me, aaierent focus and motivation: away from engaging with the past actions of ethers and their effects. Competence leaves arrangements unchallenged. Critique is oriented backward and towards superior power, concerned with the present effects of the past actions of others. Design is prospective, looking forward, Design focuses on my interests now in relation tothe likely Futur effects of my actions. The understanding which inheres in campetence was essential to carry cut evtique, jut a the understanding developed threuigh critique ic ecental in the practices of desian. Desian draws on both these, carves their insights forward anc deepens them, focused in a socia-semiati theory af muiltimodaiity. ‘As anther example, take the shift fom the ferry rammar tothe use ofthe tern semiotic resources. It indicates a shift from meanings travkitionally attached to Was fixed and highly constrained regularity. Occasionally I use te te Where meaning isthe issue 7 \garar’ to contest its meaning with my sens of ‘relative regularity ofa semiotic ester’ Tous the boc T eer to dro atierion tte trscrionsand ths Bi ecives ct ste mecaghorat ne arene #0, etree selene ante possible effects on what I want to achieve, While it may Sern pedantic t dos, I fpen! time here ad there drawing out such differences perspectives and effects In eof metaphors rn oats have oor Wc ‘lois ue ta understand acount Sor se word f cmmunisation asi aw. Nor therefore do we Rave an adequate set of tes tries ura: ue'ree to-do. a thr merce we tend to use he lorms that we have inherited from the former era of relative stability, in particular {rom theories and descriptions of language, where, over a long time, a complex set of ols has been developed. Bu using tools that had served well te Ye horse-drawn tarrages Becomes a prob in mending contemporary cars. Many ofthe older terms Beta epnces wii nat 2 coder aru see fo" Pen puri a BI rwrtineer comma aod obs calla n traducing ene nee te Many trequenly ised terms need careful re-examination; represetaion' one of those, The points: adequate theoretical tools ae needed to deal lr with the presant peste polkical ar itu esiatlon fed teresa Conon ft Fae tics ot rececary rea! vesnures il bey vetrn Iria» iers we haves ten kn to ery nme In Wester oles ‘thdugh there is not much agreement as to what it might mean. There is a certain feel- i= held with diferent depress GT conviction Tn different domains ~ that grammar fabout rules, conventians. certainty: phenomena that are fixed, settled. When business leaders, politicians, media pundits, bemoan that the young leave school and are Boheme thy cone e-n grapes) serena, ey lok hack some ‘thirty oF forty years and seem to discern, through a sepia-tinged haze, that in their day the teaching of grammar ha produced just such iealconitions. They rot thatthe young do not use the rules which they are certain that they had followed way back then; and which they think they use still, in their present practice. Adherence to linguistic convention is equated with adherence to soclal convention and consequently With social ‘stability’ Alas, the present is deeply unstable; no amount of nostalgia {ay Ghange that. The need is for a Tabour force that can meet the semiotic demands ‘ot conditions now. But in this, ‘grammar’ in its older sense of ‘a stable system of rules’ {s,m obstacte to necessary action. Representational and communicational practices are constantly altered, modified, _y~ f(s all of culture, in line with and as an effect of social changes. That inevitably Jakes the gramunar of gramunar bonks a veeaed ofthe past sacial practices of parti lar groups, in speech as in writing: useful maybe then, but neither used nor useful for those who wish it were atherwise. If | use the term Ygrammar’ it How, not eve Igaves me with the need to contest its implied meanings of ‘fixed rules’, ‘stable onvention’, and so on, oF to use a term which is free of such histories of meaning. At fhe moment 1 generally choose the latter youte, using the term ‘resource’, as in8 Multimodality ‘resources for representation’. Resources are constantly remade; never wifull arbitrarily, anarchically but precisely, inline al What (need, in FeSponse ‘to some ‘demand, some ‘prompt’ now — whether in conversation, in-waiting i silent engage- fent with some framed aspect of the world, or in inner debate Semitic resources are socially made and therefore carry the discernible regula ties of social occasions, events and hence a certain stability; they are never fixed, let alone rigidly fixed. No degree of power can act against the socially transformative force of interaction. This isnot the point (nor the purpose of this book) to talk about the benefits or disbeneits of stability, or the need for stable ‘ules in order to attempt ~ even if never to guarantee — relative security of communication. of Cultural difference and communication: thé ‘reach’ of the thi and thé ‘reach’ of modes Most rSadeF will take it as given that a society, its cultures and the representation ‘of their meanings, form a tightly integrated whole, at a certain level of generality at least. If that is so, then differences between societies and cultures means differences in-eppresentation and meaning. That is close to-@ commenplace, We know that lan- ‘quages differ and that those differences are entirely linked with differences of histories and cultures. What isthe case for ‘language’ isso for all representation ~ for modes as for discourses as for genres and in all communication: patently so with music, with image, with clothing, food, and so on. Chinese opera isnot the opera of Mozart cr Puccini. For me, as someone from “the West’, it takes much facused semiotic Gworbto understand its meanings, its aesthetics; roughly the same amount of work, imagine, hat itakes for someone wh has grown up with Chinese classical musie to make sence of ‘Western’ music The more pronounced resources of represent erences the greater are the the practices of thelr use. This means that in izing and writing about commurigatian can tale wth somé-canTdence about Tres that | know reasanably well Where my knowledge becomes veGue or ‘imore General, | can only talk vaguely and generally. T have a Gearee of confidence, ar ‘inwardness’, with several anglophone cultures, and different degrees of ‘inwardness’ with some ofthe cultures of mainland Europe. I have a glancing acquaintance wit sore cultures in Asia and Latin America, where I Fave acen a casual visto. I make ne strong claims about those cultures and none about those of Africa: it would be foolish fr me to do sc. ‘So what isthe reach’ ~ the applicability ~ ofthe theory I put forward here? Hove eel is it, how far and where and in what ways dies or an t app? Is # confines to Western Europe alone? What claims to understanding or insight can I make for this theory outside ‘the West, broadly? I hav, ary case, a problem with the notion of ‘universals’~ sueh as Yuniversls of language, oF “niversals of communication, $0 am nol tempted much in that direction. The universe ofcullures and of cultural Ailvereace on ouF small planet i to vast for such generalzan, Where meaning isthe issue There tend o be contradictory views on how to deal with this. As far as “anguage’ 's concerned, we know, on the one hand, that languages differ’ the way they name the word ~ in ‘words asin syntactic and grammatical forms; we know that lexical fields are close mappings of social practices. To give a banal example, Enalish does ot have the word“Weltschmerz’ and German does not have the word ‘literacy’. Each ‘ny strugate to find ways of bringing the other's meanings inte their culture; in the {ose ofthese two examples, with lite success so far. Culture is to complex to toer- ‘te difcut transplants readily On the other hand, we sor of assume that “anguage fuquls lanauage’s that is, if thare is @ novel in Russian it ought to be possible to \tanslate it into English. We expect significant overlap, even if nat a one-to-one ‘correspondence. In many cases the incommensurability matters. In a debate, Germans fo talk about “Wissen” Cknowing that’ and "Koennen’ Ckrowving how’): when the Wiseussion moves to translating ‘knowledge’ from English to German (or Swedish) itis not stratghtforward to know which of these to choose. Or, another example ii Enalsh there are the two words ‘teaching’ and ‘learning’, verbs with an implied Aivectionality of authority, in which teachers’ have ‘knowledge’ and ‘learners’ have (0r used to be assumed to have) the duty to acquire this knowledge. Vast theoretical A) practical edifices and industries can and have been erected on such distne- flons. Yet in many languages the same lexical root is used for this social domain i German (as in Swedish for instance, “leen’ and ‘ernen’ are morshological infec Nions ‘alternations’ ~ ofthe one lexemic stem ~ ‘learn’ in English, That makes it likely that German and Swedish societies might develop quite ctferent theories around ‘Viication’ to English society; and articulate them in their languages and their instititions. In each cas it isthe ‘accident of exis” ~ not of course an accent but the ‘pression of a history of non-accidental social differences ~ which leads to such Onotonical witd goose chases Two theoretical approaches have characterized this debate. One is focused on ‘ule; the other is focused onthe hurman brain. Warn Chorsky's universalism isthe Wot known recent exemplar ofthe latter, through his posited link between the organ- leaton ofthe bra ain mina ering and that of Tnavane— assupti Aout the Yinateness of linguistic competence (Chomsky, T9BET This asserts that ‘the organization (the ‘nature’) of the human brain is shared by all humans; the ‘deep’ Srganization of language derives from ‘dee, ‘innate’ mental organization, so that Irrespective of superficial differences, at a basic level all human languages share the same organization. From this, Chomsky derives a noble political project (to which J subscribe), namely that this profound and essential equality of humans ought to have lis reflection in a polities of real equality. From this I infer that the reach of .Chomskian theory in respect to my icsue here is general; Ty HAM Hyman; and the social and cultural really does not enter. My view on that is that there are some highly general semiotic principles, which are fommon to all human (as well as to most mammalian and some other a ‘eonymunication. Consequently, these are present and evident in all human societies and hele cultures. The most significant of theses that humans make signs in which form °10 s co— y ‘Multimodatity G6 meaning siard in a ‘motivated! relation. These signs are made with very many ferent means, in very many different moses. They are the expression of the interest of socially formed ndvguas who, wih ese sans, realize gh UIWATGexpresHon _{o= thelr meanings, using culturally available serniatic resources, which have been shaped by the practices of member: their cultures. Instead of a shared innate linguistic competence | assume shared secial, semiotic, communicational principles and dispositions ~ which includes the linguistic as one instance. These ‘principles and dispositions are articulated in communities in the ceaseless processes ‘of social (interaction. Hence the principles and dispositions take particular form, as ‘the result of the specific social concerns of a community. Given this stance, I assume that ‘translations’ across modes within a culture are both possible and hugely difficult; from image te Speech ~ the evocation’ rather than & &escription — of a painting in a conversation about av exhibition; or from image to writing; or from oral poetry to poetry in written form. I assume that translations across cultures, whether in the same mode (from writing to writing ~ from Russian ‘novel to that novel in English; from gesture to gesture ~ from the ‘French’ shoulder shrug_of indifference to an English version) or across different modes are also possible, though always achieved with enormously difficult selection; at a considerable level of generality; and inevitably with significant changes in meaning. “Stated ike tis, it implies that in its most general features, in is outline and general direction, the theory applies to all cultures. In the specificities of cultures there are ‘offen vast differences inthe articulation of these principles, which lead to difficulty in the transfer of the theory from culture to culture. Readers need to make their assessment whether, where, in what ways or to what extent my comments, based on ‘observations in some cultures and speculations based on these, are possible to ‘translate’ from one to anther of the cultures with which they themselves have “inwarcness'. This features as a topic in both Chapters 5 and 6. » To give an example of principles shared by all cultures: | have mentioned the three ‘most important: (1) that sic ‘motivated conjunctions of form and meaning; that conjunction is based on (2) the interest of He sign-maker; using (3) culturally -aeaabte resources. These are principles of sgn-making: There are Wen, aka differen Tevel, resources for making signs. Here too there are commonalities of @ genera kind. As one instance, for meaning-making to be possible, human cultures need and do provide means for franiing aspects of the Word te which an individual needs or wishes tOattend. A culture will therefore arovide Ws distinct semiatic resources for frais (complexes of) signs: what sorts of things are framed, how they are framed, what kinds of frames there are, and so on, and these will vary from culture to culture Expressing this arly ne meal wilt Fag These are instances of general principles; and of means ard processes of meanicg ‘making in any culture and in any mode. They provide a starting point in the analysis of meaning in any one culture. I do not call these “universals', though ¥ regard them as shared by all human cultures, as well as by many other species, These shared principles are based on experiences coramon to hivimans in social groups in their Where meaning isthe issue 11 ‘ngagerent with a world both vastly different at one level and yet presenting common hllenges Iba take ts Tine of argument we can acknowledge commratites at a~ vory general Tevaland yet be able to Focus or the specifieties of cultural difference. The theory can then provide a shared frame at its most generat Tevel and respond to <> Specie needs atthe level of ary one culture and ts That represents one theoretical tae on te issue hare is then the social yd cultural aspect of that notion, namely that of the reach’ of modes. Here ‘question is: ‘What areas are “covered” by a mode in a Specific culture?’ iffer Wily, "Bo all modes cover the same terrain?. In a multimodal aporoach@ is modes father than say,Tanguages’ = which are compared. Medes are the result oF asoclal— ai Historia saning of aera chaser by a socety for representation: there is broason to assume that the mode of gesture in CultureT covers the sare “area’ or 4. Wi same concerns, oF 1 used for the same purgoses and meani of pestore-ww-CuTture-2-— quite apart from the Clexieal and synta ‘mentioned just above, For instance, gesture may be used to deal with meanings around ‘attitude in two (ullures; yet what is vaddresse2", made Tito an area of common interest within the | personal stance to some event oF utterance. English culture does nat Fav ‘a equivalent to the French quick, limp-wristed Shaking of the drooping hand — usec yy women more than men — to convey ‘disapproving astonishment’. Quite to the trary, is likely that aspects of the cultural domain cavered by a mede in one iy ot covered by tat mode in anther culture. An area sah a ples ply. be dealt with extensive in-one-culture and by Afoxemics, by gesture by ficial expression in another eather Se ae ee Fhwaning, there is no reason to assume that the ‘modal division of labour’ will be the: ‘sane across societies. That is a radically different position to one which held when— the assumption was that Yanguage’ did all significant cultural semiotic work. howe differences have been addressed in the study of intercultural communication, ‘Yor here to0 an approach focusing on modes might be highly useful. An area in which his is real issue is that oF translation. Until recently that has focused on language ‘lone. In the subtitling of films for instance this becornes significant. But what about @lvanslation of a movement, an action, a gesture that is entirely understood in one soelety and either entirely misunderstood or not understood in any way in another? We simply can no longer assume that the each of modes is the same across different societies and their cultures. Modes occupy different ‘terrains’ from one society to jolly. We have to begin lodking at the Feld of meaning as a whole and see how” ‘noaning is handled modably across the range of modes in different societies.Multimodality The politics of naming Somewhat related to this is the difficulty of ‘naming’, It has two distinct, maybe unrelated, aspects: one is political, the other theoretical. The politcal aspect is how to name the political/social/eultural world about which Tam writing It isthe normal, ‘unremarkable, ordinary world, for me. As an insider, it characteristics are dificult for me to see; however much I attempt not to do so, I use my ‘insiderness’ as a yard: stick in judging other parts of the world around me. Should I use names such as the First World’, ‘the developed world’, “the post-industrialized world’, “the West" in talking about the world in which I have lived and worked; and about ‘the East’, 'the ‘Third! oF Fourth’ world in talking about parts ofthe world unfamiliar to me, different socially, economically, with profound differences in history and culture? Such ques tions are usefully discussed in the writings of Immanvel Wallerstein (2001). Apart from the difficulty of drawing meaningful boundaries for any of these constantly rruitiplying and overlapping terms, each brings with itspecific meanings, more often than not potentially harmful in their application, meanings which I would prefer not to import into my text From a semiotic perspective, all cultures, all semiotic ‘worlds’ are rich, if differ- ‘ently so. Each of these domains ~ the ‘First World’ say, or ‘Western Europe’, the nglophone world ~ is highly diverse within itself, culturally and seriaticaly, Seme parts of ‘the First World? (whatever that might be) are deeply diferent to ‘the West; and from each other. As ane small semiotic example, some societies in ‘the First World make use of character-based scripts while others use alphabetic ones. This one difference alone brings deep differences of ontology, of cultural disposition, cof dispositions towards representation, to media systems; it has effects on knowledge and identity. = = ‘Absolutely related too is te issue of globalization’. I take it that this term names something real ~ for instance, the fact that in very many places around the world, economic, political, social and cultural values and practices are subject to forces which come from ‘outside’ the domain regarded as the immediately framed “local anc have telling effects within i. Within that frame these ‘external’ factors interact with the meanings, values and practices of the ‘local site" differently therefore in diffrent places. Semiotically speaking, ths leads to constant change, transformation, ‘blending’. Constant blending is by no mears a new phenomenon, even though the: now fading) fashionability of the term ‘hybridity’ ad seemed to suggest otherwise Blending is as old as humankind itself, as archaeologists demonstrate even in parts of the globe still remote naw to the ‘first world’, Or as we can see everywhere around Us, even now, at any time. On a recent train journey in Germany ~ from the northern Rhineland to Munich ~ three couples in thei late forties or early fifties boarded the train in Frankfurt, about 9.30 in the morning. I had caught the train in Essen, about two hours tothe north. Listening to the group's German dialect, I could not tll f they ‘were from a Hessian dialect area, or from a Franconia oF a Swabian one (my overt dialect is middle Franconian). They were in high spirit rly indeed ~ inviting Were meaning isthe issue ‘ve to join in a morning coffee-cum-picnic which they had unpacked and spread on {wo of the tables, with garlands across the windows; and so I asked where they were rom. The answer was Germsheim: a town just north of Heidelberg, a region which Just about straddles all three dialect areas. Clearly, people in this area — as in very ‘many places (Rampton, 1995; Sebba, 1993) ~ are engaged in constantly assimilating, {ransforming features of the dialects around them and integrating them into a fonstant'y changing yet constantly coherent resource. What is new is the intensity of this phenomenon and the effect in its present manifestation, its ‘pace’, aided by current means of transport of economic, social and cultural commodities. Electronic forms oF communication can now make aspects “Lary Specific ‘where’ into features encountered everywhere, with an unspoken and Luegent requirement for it to be made sense of ‘there’. What is not new however, is the set of nares, the lexicon, which we have to name {his differently paced social worid. That is a problem, which makes the issue of theory 3 ail of naming within theory urgent. There is a need for new names. Eerdinand de ‘Aoussure suggested, atthe very beginning ofthe twentieth century, that Linguisties — Wi its theory and terminology — as the then’ most advanced study of aay semiotic ‘Gilem, es” for the semiotic feasts to ‘ine in the nov century. This seemed a reasonable ope, given that that expectation Was expressed in the high era of abstraction in the Social Sciences: Linguistics an Jomiotis included. In certain areas of the study of language in particular, concepts ‘elvas Slanguage, “angue', ‘parole’, ‘stood in’ for the tendency towards achieving ‘grasp of the particular via very high degrees of abstraction. By entire contrast, the the making of signs row, in this environment for this occasion. fn its focus on the Wilerial it also focuses on the bodilyness of those who make and remake signs in fonstant semiotic (inter action, ‘a. move away from high abstraction to. Wi snecific, the material: from the mentalist to the bod For that, itis essential to develop apt labels for a theory of representation and (ommunication for the whole domain of multimodal meaning as well as for the level ‘of specitic modes. The theory and the labels will need to attend to the materiality, th specificity and the histories of (social) work of any one mode. The specific asaects lf mode will need to be assimilated ta superordinate terms which capture what is Aerviotically general to all modes in, that society and to modes even across cultures; ven though what is general is always articulated distinctively in a specific mode. Itdevotea chapter to this issue of categories ofthe theory and of naming, Two terms (igure prominently in what 1 have written so far and do throughout the book: society id cultuce. the distinction between them is not easy fo make, hare Ts wetaclear line {i draw, Nor is there a generally accepted serse in which the two terms are used (Williams, 1985); for many writers the two are Gear synonymous, The distinction Uhl [ make has two features: sociely emphasizes human action in social groups (that Ji, groups organized in a recognizable fashion around goals, purposes, organizations, shared practices, values and meanings+” Social groups and actions are always S14 Multimodatity ies by (differences in) power. Cults asizes the effects, the products social actions and social work, whether physical and material or abstract and, for By powex(difference), “want of a better term, conceptuz re the social is marked by powe(d 2 Thecalra is marked by valu evaluation sll te effect oF Ca power. Culture, in_my_use, in_of socially made values; tools; meanings; knowledge; resources ofall kinds; society isthe field of human (inter/action in aroups, always; / oC work: ices of the use and ef Tuse the term work, wishing to stra sacial orientation and aims. Work involves 2 worker, tools and that wich is worked on, Work praduces change, in the worker, in the tools and in that which is worked on. Being socal and purposive, change produces and embodies meanings. [n_working,-the worker has experienced_and learned ||) something; the worker has changed. The tools have (been) changed by their use; anc what has been worked on has changed, Each ofthese chances has produced meaning, C) sew meaning The effect of these changes is to produce cultural resources. For me, ‘therefore, culture-ts-the name for the resources which have been made-produced, remade, ‘Yransformed’, as the result of social work. Everything that is socially made land remade becomes part F CUNTUraI Fescurces, imbued with the meanings of the work of those who have made and remade the resources. Cultural resources, being meaningful, are semiotic resources, It isthe social’ which generates ‘the cultural and, in that, ‘the semiotic’. Where meaning isthe issue A satellite view of language ping outside the long tradition of seeing “language” as a full means of making 19, seeing it instead as one means among others, one can gain a ‘satellite view" ff language (Kress et al,, 1996). That metaphor recalls ‘our’ fist views-of the Earth \Wvough photog-aphs from a satelite ~ that is, from outside the Earth, beyond Its ‘mosphere. That view gave ‘us’ on Earth a starilinaly different perspective; for Instance showing with frightening clarity the boundedness, the limits of our planet: his ls what we have, there is no more. From that time on, for many of us, it became {/oubling to pour old oil down the kitchen sink, because we could see, actually, that | wasn't simply going to disappear, that it would emerge somewhere else on this Hiownded Earth, with unknowable consequences. The satelite view showed us what We had knovn and had been able to ignore, in a way: that aur planet, our Earth, was (we small part of a much bigger whole. ‘A multimodal social-semiotic theory has a somewhat similar effect. It shows the Init st= al ne ere oo EIT ich. ewe Eg — a idee we id nown about the destiny of the oil Language’ Tsv"ta big enoug [etupraete Tor all the semiotic Se et se we uid pour Thto Tt. But a ‘wall foaiball or snooker, tennis or goTf, I can see that the conceptual world of the (outball, snooker oF tennis player as of the golfer Isa Tleld where semiotic wark js Constant enzagenent in (social) work as defined above, with socially made cultural- Midst docidedly not done via speech — lt alone writing = the occasional shouted ‘here’, semiotic resources and their constant remark fay actions, shapes my own Taner ‘ol or “Tessa"on an Enalish football pitch notwithstanding. With snooker, the e, transformed by ine, This describes the effects of communication-and'of ning: attention to an aspect of the social world; engagement with it in trans: formative action; which constantly remakes my inner resources; and in that, changes ‘my potential for future action in and on the world. Differently seen, it describes my construction of knowledge for myself — always in social environments; with culturally available resources imbued with the meanings of those who have shaped and reshapes! them in their social environments; responding to the needs of their times. In bot! of these processes — a differently focused perspective on very much the same phe nomenon ~ I am constantly remaking myself and refashioning myself in terms of my subjectivity (Seen from a social and political perspective) and fashioning ani constantly refashioning my identity ~ the process viewed from the perspective of my biography (Endress et a., 2005); a biography which shares much and yet is never Identical with the biography of any other member of my or any other social group. ‘Throughout the book, two other terms turn up frequently: ‘sign’ and ‘concept While the book does nat deal with ‘concept, the term names muuch the same territory as does ‘sign’ — which, together with made, is atthe centre of attention. The term si belongs to Semiotics ~ the conjunction, imvsible and indivisible, of form and meaning wile concept belongs to PsychoToay-and, deriving from there, to Pedagoay, wher it names something alled 'knawledge’ in an entity called ‘nnd | plible precision oF planning of geometrical strategies for specific shots and for the ‘ilonvied shape of the arrangement of the balls four moves ahead, is a damain outside [am tempted to say ‘beyond’ language’. That is before we get to the practices of ‘uallwmaticians (to which snooker or football players of course belong). The reach ‘al spoech oF writing simply does not extend there. The semiotic/conceptual work doné WW these fields, as in countless other human social tasks, pleasures and professions, ‘elles on modes remote from speech and writing, There are domains beyond the reach oF language, where it fs insufficient, where semiotic-conceptual work has to be and is ‘Hone by means of other modes. {W this as in other ways, a social-semiotic approach to multimodal work makes a ilforence, The task is to establish, with as much precision as we might, what Tese iilfererices are, in specific cases and circumstances. What new kinds of questions anorye and are made possible; how do persistent, older questions get recast, in ways jpotelbly that lead (o more plausible answers; and who might benefit in what ways from {iw different answers, In my area of work, the insights and benefits will accrue in all Ue areas of (heorias off learning, meaning, evaluation, assessment, subjectivity and entity JW this itis important to be aware of the twin focus of the social-semiotic theory and A multimodality, The issue of ‘access for instance, can benefit from the insight that Huinans ray have different orientations to modes and ensembles of modes. ~ maybe Wil specific preferences for the tempora-er the spatial, for image or speech, for the 1516 ] Multimodality fap the communicational encounter. This demands _rhetorical approach to communication, The hitherto sufficient (sos of competent performance and possibly oF critique cannot meet the nen demands. Inwardness with ‘grooved convention had, in the past, been sufficient for -ompetent communication By contras) fhe rheiprlas maker of a message now makes ‘an assessment of all asgectef the com: fonal situation; of her or his interest ‘BE The characteristics of the audience; the semiotic requirements of the issue ot Stake and the resources available for making an apt representation; together wit © Tstablishing the best means for its dissemination, ~ Tn a model of communication for full and equitable participation in the new com ‘municational worid, the rhetor’s interests need to be fully acknowledged, The rhetor in turn is aware of the resources needed to give material shape to these interests the world; she/he understands the audience and its characteristics; and understand what the matter to be communicated demands. This is the basis for desfgns to sha these representations ready for their production and dissemination. A rhetoric: approach draws on the resources both of competence and af critique and utilizes the in the process of design. Given the presence of modal choice in ropresentation ii a multimodal world of communication and a social world where choice is demande and the instability of the environment of communication, a rhetorical essential. Design meets the interests of the rhetor (most usually the same person) in {ull awareness of the communicational potentials of the resources which are available The socia environment of contemporary communication 27 lerprowse drawn In the environment and needed for the implementation of the rhetor’s interests. DeSz ives shape to the interests of both rhetor and designer in thelr world. Production isthe implementation of design with the resources available in the world ich the communication takes place. In production, meaning is made material and bbocomes subject to review, comment, engagement and transformation. Production happens both in making of messages and in their remaking Ta participant's engage Went with and transformation of a message. Production has semiotic (form-as- fontent), conceptual (content-as-concepts) and affective (semiosis-as-expressive, Always reflecting interest and personal investment") features; all three always at th sine time. ‘A rhetorical approach is based on the agency of maker and remaker of messages. |\ has direct implications for knowledge production. Knowledge is made and given shave in representation, according to the potentials of modal affordances; the process ‘of representation is identica! to the shaping of knowledge. Makers of representations V We shapers of knowledge Mi this conception, knowledge production is entirely part of social-semiotic Wocesses and organization of participation. That is, knowledge is always produced, falhor than acquired. Acquisition isa relatively non-agentive conception of the relation 6f individual, learning and knowledge. The new, participative sites of aopearance and tes of dissemination of messages and knowledge ~ YouTube, FaceBook, but also ‘hin logitimated sites such as those created by graffito artists ~ are an integral part Wf the new media landscape. Sites are associated with specific characteristics of iMtvibutions of power and agency in communication. Sites of appearance and sites Wf dissemination are a part of the process of knowledge production; they come with {jinciples of evaluation as part of the logonomic rules of ‘engagement//reading’/ ‘yaluation in communication (Hodge and Kress, 1988). Theories of communication ave to describe this world and attempt to articulate and theorize its practices both in Wins of continuities and changes. Given the changing environments for meaning and knowledge-making — multimodal seprowentation, participative production and sites of appearance’ — knowledge-making {uke new routes in content, form and social engagement, These routes shape what i aged with, processes of engagement and as a result sociaVindividual dispositions. Tite is a need for careful considerations of designs for meaning and knowledge- Making: the shaping of routes and enviconments of meaning-making and production Ji Knowledae and, in this, the shaping of ‘inner’ semiotic resources. The sites, the ‘yencosses, the desians all shape ‘concepts’ and, in that, they shape what dispositions Hegre habituated as subjectivities and as identity. From language and grammar to semiotic resources Ji. the monomadalyy conceived work, in other words, in a world regarded as operating Willone-kind-of resource In a sprcific domain, reflection on the potentials of that lage! warraill that there was; and “language” was Fasdurce could not arise8 Multimodality regarded as a means fully capable of dealing with all human (rational) meaning. In a rmultimodally conceived communicational world, two questions arise: one is about the ‘aptness’ of the means for representation; the other is about the complexes of mades designed for achieving complex representational and communicational requirements and tasks. Instances of commonly used modes are speach: still image; ‘moving image: writin sic; 3D modols; action; colour. Each offers specific botentials and is therefore in principle particularly suited for specific representational’ severe oneves Ipcomuniaton Gone gads) Const ats, in_ modal eg cen ere Dau he et ta Ce ee A j Design isthe process whereby the meanings of a designer (a teacher, a public speaker q but also, much more humbly and in a sense more significantly, garticioants in everyday interactions) become messages. torical) analyses, on aims salons ofa rica al hey are the implarented ooo te STRAT Fe Peg ewe at ten imsenened rows He eSenTeT Design resis on the possibility of choice ~ “this could have been chosen rather thar that. That permits the description of style as the effect of a series of choices mae ir the design of a message. Choice is always circumscri wer in different ways, foreta soci, clr fower ans se ee ice: Siyesare sect 7 alae Ese ead To a social ranking expressed ae ester iudgeiments, Hence socialy, aestetics can be seen as the politics oT SIV. Gestion around eth, such as: "Who benes?-OF Wh Sffers damage Or Fur", How ar social environments likely to be changed by this?” lead to judgements of social benefit. In this approach, ethics is the politics of value and (evaluation, It is not fanciful to see a progression from social fragmentation and the dislodging of individuals from social bonds as forming part ofa sequence of aclons Gahere G an on and the GisvougTng ofan indefdval from socal v nds, forms, struct ism — individuation — social ‘social consequences and costs of that dynarifc are visible mow in many ways and i many places. In a periad when social pathologies are increasing — aided by th: dominance of the market and its values ~ this is a major consideration, Mobility and portability Currently, mobility is a hot topic. In my view, mobility is in part both an expression and an effect of larger-scale social moves towards instability and provisionality ‘Mabitity tends to be discussed in terms of the rently available tech Tologies: physic ‘mobile as hand-held ot various Kinds; and lending communizatenal molly thous fast_increasing rapge of Features of the (former) mobile phone. This is the subject of discussion in Chaptor 10. The social environment of contemporary communicatian 29 The Interest in mobility is a response to aspects of current conditions. Jncertain domains ~ institutional education, for instance ~ mobility is seen as a paniacea to a Pile of pcblems. Two stand out effectivenes and motivation OL uinan resources’ (ie, teachers) promises tosohearange Pesto cat acai! | (oblems of The-School (conceived in traditional terms); ag methation ie sams to Iylrrpatr aay from the alienation from schoo! felt by many = predaminantly male” | i physicality of the school, its confining time structures and its general lack ff attractiveness to many students and to provide instead the physical, social and | ‘iotlonal mobility as freedom ofthe individual. Both come together in promising to inal the attractions of the market thr unded access and individual choice fi oxpressed in concepts such as ‘the personalized curriculum’. ‘What is suggested Tor tre school can serve as a metaphor for wider social domains Ii \seisential to subject tis metaphor and the claims of mobility to searching scrutiny. | are profound questions about what social effects ‘mobility’ might have: Who or imobile2, What is portable? And in any case, in what ways are portability Wily a social, evitural, pedagosic, affective, Cognitive and generally hurnan ‘vod? What conditions for sociality does mobility propose anc what kinds of socality ‘wl hese be? Is i the case that society ether can or ought to ‘develop’, change, at {iw sane pace as technological devices or in the directions indicated inthe facilities fhe ‘new’ technologies? A world of constant accessibility and availability is also a 1 constant surseilanse and in that word. what opportunity i tere for doin Tor being idle’, fr reflection? What vision Tor society is entalled? The person- Tel curriculum is at the same time the loss of one major force Tor social cohesion, {Wit of knowledge, information, values shared by all members of a group. Are total Hiumwetlvity and accessibility a social good or are they markers on the road to (ivnatarrsotr pao? Arete a ter asta fatal Fagrerttion Aljhlipiegration or are they covert means for dealing with the absence of refabTe Saal bonds? ‘The Combined effects of these factors of communication suggest the need for aids’ as an absolute requiremert: in relation to text-making, to reading, iveornment and discrimination, to reflection on metaphors and their effects. us i they suggest the need for communication to-be founded on principles which A word on ‘pace’ Hivloglea’, Individual, social, economic, political and technological developments 90h) have their own, distinet ‘pace’. The current fashion is for valuing ‘speed, the AuAHVPtION being that slowness is not just boring, Dut wasteTUT DI Fit Tie Tat ynesTTon th coVMMMERTTON as Teather soctal practices TATqM MAE TOR30 Multimodality Nat tumae cr vs Wa bum cede greet ssn onditions?’, ‘Under what conditions is slowness of pace essential?’. The pace oF tech- eal cane cannot possaly be mitrored by sll ietltions, ver though tha ‘Seems intended in calls to accommodate to every innovation as a means of furthering ~gitncies he ceasles restrucurrgsaForganizalions ar ope sush splat “thereqarement of individuals constant to adapt The more urgent question Isto reflect what the relation between technalosica instutloal, social and human pace shouldbe. Society cannot hope te mimic the pace ier in evetytchnolagieal nevetionrotsoul Hater ada, ta bly cil ica, urine a Pras ie ake pede The need for apt metaphors Wis essential iE o K herent in the theory being constructed am manana around Taming’, which T develop at various points in the book. yf “fll sassaemetactrs Alsons or aay ely rade, So maghors=i aizabvays ney made bl eee ecco FRITS EROMECGE aval. Slane ar rowedge are tools dealing wih prob ithe Sgemaker’s Tfe-world ol Sis ame caus anh ‘purposes, so metaphors too are ride Tor specific audiences and purposes, That make= it @ssentTal to Be aware of what and who the audrences are and vit the purpases are The fact that signs-as-metaphors and metaphors-as-knowiedge are tools makes i essential to exercise the highest care in the fashioning of tools. A sigoetpnor rade fr aay ainsi ts pos fie, aga ishing understanding. As one example, take the metaphor of ‘multimedia’. It seems: a SEE well Tor quick SEE WAITTOF queK communication. But what is or are multimedia? What does th: signifier multimedia refer to? What does it name? What does the metaphor project AAs far as I can see the metaphor of multimedia has much the same relation to thy Present communicational landscape as the metaphor of horseless carriage has t the age of the car. It is a naming of new givens in terms of old frames, thoroughly lodged in a previous environment where the media of print, of sound, of photography, and so on were all discrete. Each of these dealt with specific modes: radia with soun’ as music and speech and soundtrack; print with writing and image. Film was, in that context, already a border category as it brought together a large variety of modes with their then still distinct traditions of production. with present givens; to So-to.use Lhe term sauitimedia is to confuse past practice confuse the cultural teclnologies of dissemination —raMio, newspapet,-TW,etc.— with ase the culture chr Oloples ot dissemination “rad £; ntoprcbetern Iie communicaenal ensemble might be signTicant for Ts eanerirainee, =) V7 Jn Fameg —> fe ae conta and ently atte to otaMUaTy fica ces ‘uleitial promats tis his interest as trainee surgeon that turns any one of these — or ©? Jif nto a pronpETarRim- TCs his decision. Once turned ito a promot, his eet fares De rae a el ny asec ears fran tat res sofa . a se ae hans ta epee Vora Te aaa UTS Sonia 1. Wor changes tings: the too, the worker and that fl srian ae so ae orSSUTII-GEOTa ness tobe a least equally which is worked on, Semiotic work is no exceptions It is work it the dormall AMlontive to prompts of acl ical athe than a pcos king, Freq the saa ua canes proceed yaa seve work rear Mean maiein MMW" ihe ae oie te he Gad ot a eT com ris for SMe ertion Westy toheTie rar rater han ie eceean ir et retances Assume that we take the situation in Colour plate 1 as @ normal condition of com ff Soi ‘munication. Colour plate 1 shows an operating theatre; an operation is in its ear! iatonniles ta ant in this situation applies to the other participants; Woroutly, depending on their position, role, perspective Ti the complex ensemble — Communication as semiotic work: a sketch of a theory stage. A ‘scrub nurse’ is in the foreground. Behind her, to the right, is the ‘lead surgeon’; opposite him is a ‘trainee-surgeon’ ~ a qualified medical doctor training Sent ot thelr postion. Crucially, communication’ to become a surgeon. Behind them, separated by a screen, is the anaesthetist; fa Siperoting theatre ig raltimad: a gaze, a touch, e'spoken comment, a gesture, back on the right stands an Operating Theatre Technician, Representatives of fou Sh otin see ee ee distinct professions are present; each with specific traditions and practices, way: Piju pvt alma acts apron ToWEpTion of communication showin Colour plate 2 is a very different one. of talking and doing things. Their tasks are closely interrelated and integrated. The yp iy thing, eve mode fs not an issue. It may be that language as speech is assumed occasion i first and foremost a clinical situation; an instance of (communication in) 44 Jy-th modo of communication; or, more likely, the focus Is on a message as professional practice: a patient is here to be made better. It is also a pedagogi “edntont’ quite absteactly and generally, rather than on the material and semiotic form situation, an environment of (teaching and) learning: a trainee surgeon is here to yf iw inemage become a fully trained surgeon. Wf the Saussurean schema, yo interlocutors ave linked in a dyadic structure, One Communication is multimedal: by speech at times, as spoken comment, as instruc lig a message: the diagram andthe theory both suggost that itoriginatostrom / sore ea re Sas aL Teaching out for 31 WIN literlocutor’> NERC; There Its shaped into speech, seemingly; it is uttered; ion rac les communication js. response a rome a azz j Qght produce a spoken comment; that Teads to an action; [poking at the screen by and is met ey sn-antcument being passed, Communication has happened 1 space aaa acused on some aspect of the communication; 3 \, Fas ten thao a mess = that message asa prom {QUIET SF Timself. That prompt has been inter ‘a new Inward Framing —> Interpretation is ceaseless; it involves all tv participants here, at all Wines, though aifferently in each case.” “THETAFGer social event here can be framed in at least two ways ~ from a clinic and from a pedagogic perspective. IF we frame the event pedagoaically, as one 0! teaching and learning, the senior surgeon and the trainee move into focus. Questions flyupe 4.2. Saussure’s diagram of commnamreation om)34 Multimodatity the other participant receives this (spoken) message; and in that interlocutor’s ‘head It becomes the basis oF a respanse “Versions of this model had been active in twentieth-century conceptions communication; the sll haunt conceptions ef communication = er If nthe ser negation of many adaptations and alterations ofthat model. Glaus the domina conception of communication in the latter part of the ce on the Sende: eS Receiver scheina of Shannon and Weaver (1948), a model derive: fom lere the origi ive cause of communication lie with the sender, who ‘encodes’ a message in a code shared by sender and receiver, ser along a channel, to be ‘decoded’ by the receiver. This version recei in critique in Roland Barthes! (1968) article ‘The death of the author’, which insisted o ‘fiz dominant role of the reader in communicgbign. ——— “Tid Lemme te cosas ah 0) is s0cral interaction and i ound meaning, oriented to the processes! ‘igingad orale meaning teeugh eatin a ogee Seresetaan Sgnmakers a ther agincy a8 sol actor are pI TET dard with them the social envronrentsn wich they make sian. Ce ish or making meaning on modes and ther aforances Te ‘air condion and nears for RSanPaT AG meaTing= The mela and Use ace PHtheory Or communication MEETS TO WEA Wi te semiotic work done in relation to 21 thes ard withthe meanings which est. Questions ofthe Ked “Who des wht for whom are etal ys acs. racist rere, hs teonsfarmation of ther sell from the porspecive meaning. tn that proces the social” ar wiles and forme, 2 OO Dractces~is constantly articulated in (materia) sero frm: the Sot Callovated e-rgistered with semoticfcultaral resources. The process of (usualy Iminute)'ecaliration’ ensues that gaps between social structures and processes the one hand and semiotic accounts of them on the otter donot open to ready too far; and thatthe semiotic resources remain capable of representing St phenomena, This ceaseless recalibration acadues a serse of Tab asenss pf noticeable gaps hes sense to members of the: Poa That things are as they ‘nave always been ~AY_the-same Time, social interaction via semiotic means produces more thev ‘documentations, records, more than ralTications’ so to spea, oT Socal harge: ‘produces Tew meanings. Even the most ordinary social encounter is never entire predic, new in some way, however slcht, so that the ‘accomnis dations’ produced in any encounter are alwavs rew in some way. They project social | possibilities and potentials which clffer, even if slightly, {rom what there had bee before the encounter. As a consequence, the semiotic wark of interaction is alwg \ socially productive, projecting and proposing possibilities of social and semiotic forms, entities and oraces ch reorient, refocus, and ‘90 beyond’, by extending » el GI dr oom eit Communication: shaping the domain of meaning {onmunication can only be understoog it as an always complex interaction whodded in contradictory, contested] fragmentarysocial environments: whether liveen groups or between individuals, coming together front sacTaT Tocatfons which aways distine. in some PESpEETS Tr the interaction, Te SOCTaT divergences) rovide the generative dynamic of com- on. Inthe process cifferences are reshapadfransTormed Th temporary social HTotic accommedations. These in turn are the semiotically and socially {hctive force of communication: they project what is socially problema into a Hille ace’ and produce temporary recordings of the social and the semiotic state Witfars; in transforming it, they shape it diferent Iiother words, the huely-burly of social ife isthe generative force whch constantly ‘He Yehapes a society’s semiotic resources and in doing so documents and ratifies new Aielal givens. Communication and the resources made in that process, have the ‘rictristics that they have because both bear the imprint of their social environ ‘inte. The central point for the theory is: she soctal ee motor for communicational ijalic chance: for the constant remaking oF éuTtural/semfat resources; and forthe Minis tence ist that ‘the social has priority’ is to say, above all, that the forms, the snl thé contents of communication are social MOT, they are socially ve mms hed, that communicatiow TH social environments, arrangements Wil practices; that communication is itself a form of social action, of social work; and {et communication is always a response by one participant to a prompt by other AHtM pants in social events, Some element of ‘the social’ prompts me into shaping a Ngil-complen as a message — which is my response to a priar prompt. Individuals act HW Aoinmunication, prompted by and in a social enviranment, with social-cultural Foyourees. Hinicipants are embedded in networks of social relations with athers who make Hieanlrgs by making signs. Signs and sign-complexes — messages — have ‘shape’, ‘Wiuelure and content, representing the interest of the sign-makers. Some of the signs/ ‘Heyes ore taken up by participants in an interaction as prompts. As I have ‘Wiyested, the two central assumptions are that communication Is the response to a WeHP ad that communication happens only when there is interpretation’ (see Kress Yl Pais Leeuwen, 2001), Communication depends an the transformative/interpretative Lb av want iti an interaction with a message made by another ~ in Medici cs teroett ons is tninesesar oF eormenieson ber angen Diner commutation Biearuerce for is treory of communiation and We practice that oven | essage with the intention to be a prompt, by another participant, A message, ranges the attention of a participant in the interaction Aliention derives in the first instance from the participant's interest — where inter fis the momentary Nocusing’ of a social history, a sense of who Lam in thi $4418! iKuation now, as we as a clear sense of the social enviorenent in which the 35%6 0 a ‘Multimodatity prompt occurred, ‘Refuse negation both of a bath ofa soc aula. Tterpretation is central in communication and so, therefore, i the interpreter. A intefretalion ea rcporce to aptlur marae Ths careers, the shape of tat ccemngE CORRES Hr TT on wilt heir pettin happens, At HEE EET tnercTore always mb oF enor oF he "Ground as We prompt framed b the interpreter, with the resources brought by the interpreter. An interpretation the result of a series of teansformations in which aspects of the prompt and aspee of the resources brought by the Tnferpreter are shaped, jointly, into a pew senior: eit. — ~The theory needs to be able to serve as the basis for description and analysis c{ all instances of communication. In suggesting that the environment of the operatin: theatre could be seen as ‘normal’, [ am aware that reading the pages of a book cr engaging with a website are different kinds of activities, in some sense. However, or theory should, nevertheless, be able to deal with all instances of communication. “TETATGHT be Useful to say — simplifying massIvely — that theories of communication ve veered, broadly, between dyadic, interactive models, such as Saussure's, either more sociologically or more psycholocically oriented; and dyadic, unidirectionat models such as the mode! of Shannon and Weaver (1948). Both imply social rela ns: Unidirectional models tend be more authoritatively oriented and interactiv models less so. The power relations implicitly coded in ‘authoritative models’ requiv: thes “ to recover — to decode — the meaning encoded by the authoritatig Both the models mentioned earlier were conceived of as monomadal ~ one depen! ing on the mode of speech, the other on a modeticode) based on the materic! affordances of electricity, miodulated into ‘code’ by relevant technologies. Or, it might be better to say that the issue of made did not arise: it wasn’t ‘present’. That has remained 0 right into the present. As mentioned, both models have been challenge! in different ways — for instance by the Gournalistically based) model of ‘uses and ratification’ (Blumlee and Katz, 1974; Palmgreen et al, 1985; Dervin et al., 198) /—n which users and their needs are in focus; and fundamentally by the explicitly sem \ otic theory of Roland Barthes in his ‘The death of the author’. From that perspectiv my sketch here is an attempt to give a social-semictic articulation to the critiq offered by Barthes in 1968. To restate: in the sketch put forward here, three assumptions are fundamental ‘ommunication happens as a re ympt: communication has happened wh Tras been an Interpretation: communicatian is alvays multimodal Betas Interpretation is cortral, so therefore isthe interpreter; without interpretation there {ig_conmunicati t vompt_whith Coristitute the ground on 2 yes Staae on Sean ike Thay comorantor arate wistin staes. by the interest of MTINTTal maker of the sign-camples. the thetor ——ET—e = Communication: shaping the domain of meaning 37 ion of disseminating the sion-complen as to be taken ii-o1amn. In_stage two, the iferest and attention of an interpreter is in focus it Joni to selection of whats criteria Tor the Tterpreter Tr the Tita messace and to Nag often ape oT eat re The seTectod aspects of the nilal message as a prompt mh is, sub Seen, nerrete Ts mea made Th hat nereretavion can become Te Basis i aker terete sequence ofa prior promo and is Infergretatin») new sia | Mle Cased on We makers inferest and a sree of the auciences characteristics) einlenced as prompt =» te itera sent Ins does not make the semiotic work of the rhetor and the audience’s/interpreter’s mt Tot the same kind of work, with the sa Vaud chara hesame results. After all, ‘setting out the around’ ‘ory different in all kinds of ways to ‘making Selections’ from that ground and Jiderpreting aspects of it. The rhetor and the interpreter both perform semiotic wark; Wl Is liferent work with eiferent eects, Oden structures, in genres, is ol 1 gion- complex. com, a | odie ‘Heading’ and the reader’s design of meaning Jip model of communication rebalances-power and attention, with equal emphasis WW the fiteroreter of a message-promot and the initial maker of the message, the Mawr tose afr of eon eT Ot Hwatie, in the ‘reading’ of a quite traditional text; in the engagement with a website; 1 iting an exhibition; or in other environments. To show this, I will discuss two witint examples: one, Colour plate 2, is a website — the homepage of Children’s BBC; {Nie other, Figures 3.2a and 3.2b, comes from a ‘visitor study’ in a museum (Selander, 80008) Insulander, 2008; Diamontopoulou and Keess, forthcoming). Heading is communication. In mary approaches — especially in pedagooic ay f ents — ‘reading’ has been thought of quite analogously to the Saussurean mode| Figure 3.1, and maybe even more so to the $ -> Mi» R schema. ‘Older’ pages ‘wilting embodied notions of autharity and authorship quite compatible with those ugilaley the author assembled and organized knowledge on behalf of readers and Myplayed that as a Gwe ordered) text on pages of print. Readers engaged with that Yulas knowledge in the-oedorset-ont-b the author: an order OF TS, OP Warts On The of arrangements of words as syntactic elements; of genres; oftexts-as-qenres, Hipage; of chopters. That had been the embodiment of a unidirectional, herarchical, Hib roltionship in which power rested with he author38 ‘Multimadality Contemporary pages, whether of information books such as those produce’ for instance, by Dorling & Kindersley or of websites such as the Children’s BB« homepage, Colour plate 2, are designed on the bass of a quite diferent social relation of author, reader and meaning-making. Above all the texV/age is shaped generical with the assumption that engagement takes place on terms of the (child) reader’. interest: an assumption based on 2 very different social relation to that of th “traditional text and its genres. Unlike the traditional page, designed with a given arderfarrangoment for the reader engagement, this site—a ‘homepage’, which has visitors’ rather than readers ~is give an ordering by the readers” interests through thelr (ordering-as-Jcesian, The readers’ Interests determine how they engage with this page. In doing so they design not so mu a coherence for this siteMpage’ as an ordering which represents their interest. T sequence in which the elements are ‘read’ and ordered by them accords to choices whic! veflect their interests. In effect, the readers’ interests, reflected in the manner of thei fengagement, provides for them the design for this page: readers redesign the pe (see Kress, 2003). This notion of ‘reading’ accords with the theory of communication ‘outlined: material is presented; and readersiviewers shape their ordering of that material. Formerly, an author, asthe initial maker of the text/ message, had proviced that ordering; now that is work shaped by the reader's interest. Clearly, this chome)page fs designed. In its design it embodies and assumes the necessity ofthe reader's semiotic work. To an older reader the page may well give an impression of incoherence; for its intended reacer it provides what this reader expec to:do (atleast some of) their own semiotic work of design. With this site the question of modes is unavoidable, Ata first look itis not at all evident that writing and lexically conveyed content do provide the main path of engagement for the reader/vsitor; or whether image does; or colour; nor whether placement of the main semiatic elemen —for instance, in terms of a left-right and top-bottom order of the traditional paye determines an order of engagementreading for the readerNisitor. In the page, 2 i is encountered on the internet, the cartoon characters along the band at the top of the page move. Movement as mode provides a further point of choice; again there may be generational factors at work in shaping preferences for engagement. It is 2l:0 important to observe that a formerly profound divide between the ‘programme isn and the programme itself has become blurred. Whereas differences of genre hil usually been done within one mode, by a specific selection and ordering of semio' ‘entities ~ syntactic, lexical and textual ~ here the generic differentiation is done iy means of modes: the mode of moving image makes the programme isting itself into performance. In some ways this is akin to a crossing of the boundaries of social semiotic ‘domaine’ ‘work’ versus 'pleasute!, for instance. But the generic lifermce marked modally is more profound. In considering the homepage, the question of ‘yeading’ and generation became central one. If it is the case that members of a. ‘younger’ generation (quite likely by now to be separated by two generations from generations more ‘traditionally oriented) read according to a design of the reader even though on the Communication: shaping the domain of meaning ‘yeni of someone else’s message ~ it is highly unlikely that they will not carry this Jiwhitus into their engagement with any semiotic domain or entity in their world. These ‘lyorgent practices and expectations about reading by readers of one generation Fnwelirs) ~ and another (students) is, I assume, ane of the major problems about “Feaiing’ in schools. It might be objected that reading for pleasure is one thing and What readino/engagement which takes place in institutional sites — of work, of Jfesion, of school ~ is another. For one thing, the issue of power is overtiy and ‘{istently present in institutional sites. Some readers will respond in alignment with {iver vw others not; with generation playing a major role. Once develope, a habitus ‘Wf Feading is likely to be applied in all instances of communication, though the ‘Willigent factors of particular environments and the specific characteristics of ‘HeiiiOr will, as always, play their role. My fast example here ~ Figures 3.2a and 3.2b ~ comes from a research-project on ‘Viilor studies in museums (funded by the Swedish National Science Foundation). It WH) fonducted in Stockholm and in London (at the Museum of National Antiquities ‘W Mockhoim; and at the Museum of London, in London). In Stockholm the abject of WE Feweareh was an exhibition on Swedish prehistory; in London two exhibitions were ‘We liited ~ ‘London before London’ and ‘Roman London’. The research project aimed. WW ive Insioht into how visitors make meanings for themselves of these exhibitions; il 0 a lesser extent, what sense they did make. Unlice schools, museums tend not WH Phireike power over their visitors in their engagement with an exhibition; even HON they might wish to be able to so. That makes conditions of learning different Wilpon vay. Ji Heth Stockholm and London, visitors were invited to participate as ‘couples’: as WAipnront and grandchild; as friends; as married couples; and so on. Those who Hee the invitation were given a camera to take photos of objects or displays which rr, ‘Their interest. They were videoed as they made their way through the exhibition. {He Gonclusion of their visit they were asked to ‘draw a map’ representing their WHEE Of the exhibition; they were also interviewed briefly. Each of these four ‘takes’ ‘4.1 means of obtaining material that could serve as data to gain insiht into “Hf eragement and serve as evidence of ‘learning’, Two maps, from the London yuh drawn by a member of each pair, Figure 3.2a and Figure 3.2b, are shown titow Mi Wahibition is designed; its designer(s) have specific aims: to show objects, TW, reconstructions; to tell stories of the prehistory of a nation, and, in that, to ‘WHieve social, cultural and maybe political purposes. While these aims tend not to be ely sated, in interviews with curators or the curatorial teams it was clear Hl Ulweke exhibitions were the result of much discussion, framed by policies of the pate, (ommuinicationally and semiotically speaking, an exhibition is a complex sign, ‘Wialined 10 function as a message. It is meant as a prompt for the visitors’ engage- WHEN. Pedagopically speaking, ay exhibition presents a curriculum for the visitor/ Jeni (0 that context, the 1 Way here are indications of the visitor's interest 3940 Muttimodelity Figure 3.24 ‘Map of a museum exhibition ‘Londo before Londen’: “the prehistoric camp’ Figure 3.2b ‘Map of a museu exhibition ‘Londo before London’ ‘Heathrow’ which shaped their attention; which in turn framed particular aspects of the overall (design-)message. Semiotically and communicationally speaking, the maps arc response to this prompt. Pedagogically zpeaking, the maps are signs of learn. ‘The ‘maps’ are of interest from any perspective. They do not, of course, provid full account either of what the exhibition offers, nor of all of the meaning made by either of the map-makers ~ one an 18-year-old wornan, one an 11-year-old boy ~n their visit, The photographs they took offered other material for data, as did the king video and the final interviews, Neverthelo tra the maps provide one teas on Communication: shaping the domain of meaning Meir engagement and transformationtransduction of aspects of the exhibition in each ‘90 They show a clear difference in interest; a consequent cifference in attention and Janing; nd distinctly different interpretations ofthe same large overall message. The W exhibition was turned into distinctly different prompts by the interests of each of We two map-makers. Most immediately they show a starkly different sense of what a ‘map’ is, what it ‘hit mean, what it does or can be, based on different resources brought; inthis case ‘uikaplions of what might count 2s a map, and of what is to be mapped. In neither ie Is the notion of the map a conventional spatial one; the exhibition had been ‘Aiyajwed as.a large ‘room: with relatively indistinct sections. Some questions posed, W Show map-makers, seeminaly were: "What was the central topic to engage my slleniion?’, ‘What objects cid 1 find most interesting?’ The maps are a record of Wwplures of the exhibition, not of the layout ofthe exhibition; a record of the objects ‘il tableaux that seemed most salient and interesting. In the case of Figure 3.26, Wie notion of map is ‘conceptual/atfective: one question posed, seemingly, was: * ‘Whit for me, was (the most) significant cbject/entity of this exhibition, along with ‘Hie others and how shall I arrange them spatially to give a sense of ther relative ‘iypifiance?’. For the first_map-maker, that which incited her attention was {faved along vith some of the entities which prompted that; for the second map- Heke, the map vas set out as a visual display, a spatially ordered arrangement, in ‘Wii lle most significant object has the mast salient position. Fran the same displayimessage the two visitors had each fashioned their own 1 Jesign and a distinct interpretation. A number of questions jostle for attention ‘Wwe interest has been dominant: the curator's or the visitor's?’ “Has the SiyAlor suecveded more in one case anc less so in the other?", ‘What interest and what Heehuries of the visitors are evident in these mags, ‘Has one of the two visitor ‘ples failed in their ‘reading’/engagement, or have both2’. These are questions ‘Heal aad urgent for a curator; they provide one motivation for engaging in ‘visitor Wille allure’ or ‘success’ are probably not concepts the curator uses in relation Wi tle visitors; though ‘effective communication’ might be. They are central and Will uestions within this theory of communication, irrespective of the site: Wier In. museum; a school; a site for public information; a site of professional aller wich as surgery; or in an ordinary conversation. What conclusions can be drawn from these examples for this theory of commun- ‘Allon? Taking the two maps as.an instance, one response would be to say that Fewpo ings made from the exhibition ~ are unpredictable, individualistic’, “Wihieeivo’, anarchic even. Or we might say that the most significant criterion isthe Valles Interest. Both ace unsatisfactory accounts: tho First amounts to an admission Wal hore is Ho theory that ean account for differences in engagement and reading Wie second ionores the role of the maker of the initial sign-complex/message in AHowwina the message/promyt. That, alter all, has ‘set the around” Fromm which the Seeminaly ‘anarchic’ or Ssubjectve" responses are produced, a fact which is demon- eat bots maps s of the exhibition, such as the model ) a2 ea ‘Muttimodality airplane at a neolithic camp. While the response to this complex message/promp! may not be predictable, it is not anarchic. 1f we stay with the perspective of th interpreterfrecipient, there seems ta be a sequence where the recipient's existing iereseersta bora vequence where:the:ren|plentis cxlsting interest shapes attention, which produces ‘engagement leading to selection of elements from the message, leading to a AW cm is He iran ofthese eleven, ch fads their re a new ner sgn f ! nf Men Aso ‘The interpreter’s interest produces attention; attention shapes the form of the engagement this leads to sles bring made; te selections are frame there the subsauet trarsfomationard transition ofthe elements nthe ame; ar, that, the inwardly made) sions produced. The sequence reshapes (aspects) of th nial message, the around, laa prompt interest isthe motive tore: isthe bas {or atientian tothe “qroundconsitued bythe extn, fr engagement with al ‘ground’; it shapes selection, transformation and transduction; and interest become evident nthe new sg, the a. The maps are subjective inte ee of being bast onthe maker's ners: yeti exiting the prints of the makers” Inerest Clear relation tothe grourlprompt, the maps ate principled. They are not anarei This Ureory es welier te sipnifcanceof the semiotic work ofthe make of he tial mesons, the reer, nor rat of te Interpreter: the work of cesion ht fashions the ‘grou’ on which interpretation ake place is one essential element the t-par structure of he proces communication. Forte aragst as rh as cverydey communication, some aspects ofthe dein of te Inia ressape rust Trobe Sent camel nf ak ese Oe eas Wore inerprlaion, Th semiotic work of transformation and rasan whieh under interpretation is te ste of the production ofthe new’. It Is work whch fad semiotic enlies whichare always new, nmovative, rete nat because ofthe pei ef the patcponts in te tteractin but because ofthe very characteristic forms these nracions, nuh one cancption of he worl the ermal express interes of one parteipant~ ime bythe diferent terest fe ieterioctor. 1 he Alek out ofthat clferncey wih pow and ws paying ena necicrl new conception is arrived at. --Fealled that Yan accommodation’ to indicate | both its provisionality and i ence cuof different interests, power and affec| Practically, the curator INirto ask what consequences follow fram this concept of communicatian far her or him or for the museum in strategies of design. There ve specific questions: ‘What path had the curator constructed for the visitors?, “Why «iN! how was that path followed or not followed?’. The curator might well want (0 understand the principles underlying the differences in the path taken: in attention The concept of interest provides the overarching reason. As i 1s in interpretation, Communication: shaping the domain of meaning 43 Jnppened, one of the map-makers was an 18-year-old student from Germany on a self- ‘uHyanized 'study-trip’; her interest is expressed in the detail of attention to the objects, ‘Wi lior map; the other map-maker was an 11-year-old boy, a highly reluctant visitor. Hu in citferent ways—each time precisely and specifically ~ that is the case with each el tie maps a Ue, oem seta tee ‘Frovisionality in communicé ‘and design, newly fontigured qe like any semiotic entity — a website, the operating theatre, a book, a i's drawing ~ is the result of design. Design is shaped by the prior analysis provided Ihe rhetor through questions such as: \Wllatis the environment of communication2 Telations of power ace at issue?’ Who are the participants in communica oo i val these, the rh in-ifferent ways, with ciferent ions of purpose, power and affect ~ the lkaly Yesponses-ot-aTmagined re Tree-dinerstonal objects more Salant, Tore “Slane”, are le than writin caption o than longer written accountfexplanations?”, "Are ed scenes mg 3 than tse dimensional tba? What fect es [7 have in creating root and affect? “1s the distance at whiphrvitors are able With objects —for instance, whether they are sented en object, © er?”. The es a glass panel or a rail — a significant Wok ot affect “Addressed in all aspects of the éxhibition: affect modulates jie ota “Agfest oblate te seen Filed, I ae Cc (oransoclra sivana totem oa REairensional) image: though elements are drawn in to the ‘mag’ from other Mat oF ‘map’. The big skull at the top left of the map is located at the very beginning Wile exhibition. Selection by the map-maker has changed a large room with many “wher image where few objects are drawn into an entirely new coherent ee oT Heripby-transtorcation ~ changes in ordertnerand ConTTgUTatIONS oF efements within the change from meaning expressed in one mode to44 Multimodality Curators might see themselves first and foremost as communicators and their response to these maps might be shaped by 2 wish for better, more ‘effective’ co munication. Just Uist cneatng theatre, te museum more anc more becom aateatih is Soubly famed 3s a etveatie Governments See museums as environments for social education. In both framings — a site of entertainment or of learning — curators are bound to be interested characteristics oftheir audience which have an effect on both. In the present unstable social and communicational environment, toe category ol abl ishing the characteristics of communicatici as 10 be treated — conditions of communication: ‘What are my purposes, "What do I wish ‘communicate?’, ‘What are the characteristies of my audience?', ‘What are the best sources to do this, ven the characteristics of my audience?”, What relations 0 fower abtain between myself and my audience?’ ‘What resources for communication ‘are available”, “How are these resources best arranged to represent what is tole communicated?’ and the larger framing question: ‘Am I attempting to educate, entertain, or both?’. These are essential questions forthe rhetor, in any environme:i at al times. Inthe past, convention had provided ‘routines’ which obviated questior now they are starkly present and need to be addressed, newly each time. Rhetoric needs now to be seen in the two-phase structure of communication, witheut which it cannot deal with contemporary social givens. Communication is joint ay ttcipracal wack The sign-complex which is sketched by the rhetor on the basis 0’ preceding analysis is elaborated in detail by the designer and is then given mate” al form by a producer. The sign-complex is presented to the audience as a (comolex) ‘message. Ideally, members of the audience shape the message as a prompt and ear! transforms that prompt into the new inner sign inthe light of interest and the semio\i: (and wider social, cultural, aesthetic and ethical) resources which each brings ence To cr as eee ng ene Pe attend to and endaae with and interpret Ba prompt for he 1 Crnmurtcaton rest on both phase ital work ofthe rhetor and The subsea. engagement and interpretative work of the auience, seen as interpreters. To cal part o agement is to use a relatively ‘neutral’ naming; it @ESErt« ‘fended by she-tetor= It doas assume the attention to the rhetor’s message. Whi! aitention and engagement Tead to section and the reraniiaa ab ubal has been sete] ae Communication: shaping the domain of meaning 45 Ln leersny Crane off an, Food Comme SL Lich Jeg ‘ahr, whether of the initial maker of the sign-message oF the interpreter of the Wael. Or we might use the label chetor for the maker of the message that initiates ‘Heeauence and becomes the around for the interpretation; and use the label interpret WW the sign-maker who interprets the prompt. That brings a distinction between ) ‘Willing the ground’ and ‘shaping and interpreting the prompt’, both seen as the ‘Hub iny oF signs. | adopt that naming ~rhetor and interpreter ~to mark the distinction Weipeen setting the ground and interpreting the prompt. Each of the two has a distinct Wid! function and effect. How each of them is carried out is a separate matter, Ives sway, decisively, from a conception of successful communication measured TW ietins oF the ‘closeness’ of the interpreter’s sign to the sign of the rhetor. Social present twice inthis framework: theough the interest of the rhetor, FENG Sonse-OF the social characterstiesG fn Wo the rhetor; and through the soctal location and interests of interpreters. Ine distinct vantage points oftheir social stories and present socTaT post ons, Hill bing their sense of the demands of the immediate social environment, ‘ieMentational and communicational requirements of the phenomenon ~ the ever 1 thject = to be communicated and interpreted. Each brings their culturalsemiotia\ ‘Peerevs and values. iis conception of communication gives appropriate recognition to the semiotic rh of curator and visitor; to teacher and student; to writer and reader; to the person ‘WH oloads a video to a social site and to those who respond to that video. It provides 4 Wines (rare of rhetorical and communicational work, modified according to the use. , writing was the dominant made, carrying all the information Mijught tore essentiat or central to the curriculum; image had the function of Mlustyatina’ In 2002, writing is one of two rades with seBriing equa semTate st TAA no longer has the function of ‘illustrating’. Image shows those curricular Wailala yuich-ava.test_— most aptly — represented in image. Writing deals with Aestion ws well, though not the physiology of digestion as before, but the bio. shieaniitry of digestion, a748. Multimodality Anil Seiten» Sorin Mana 18. Dent at stn 9 Taare tte ee ee sat sp iag ted dace ee ot SRS Shay bee pen Figure 3.3 1935 Science: digestion In 1935, writing was seen as a sufficient means for carrying the meaning ‘hl had to be carried; image was very much a supplement (Barthes, 1966). By 2007) writing and image are treated by the designers of the textbook 2s offering spec i affordances; and these are utilized in relation to present ‘curricular need” and \ly assumed characteristics — as likes, dislikes, preferences — of the audience. The dominant view, that writing carried all that was crucial and significant, me: that no real attention could be paid to what image contributed. Figure 3.3 shows {hi Image did in fact provide information not present in writing; but that was not in foc ‘The dominance of the mode of writing provided categories both for representa’ and categories for "recognition’: what was not highlighted in the lexis, the syntax 0 the grammar of ‘language’ did not emerge into visibility. So the fact that the ina ‘made specific what the diameter of the cesophagus is compared to the size of the chi cavity, was not visible’ as a significant contribution by the image. Communication: shaping the domain of meaning Piyire 4.4 2002 Science: digestion Jashioning social and semiotic domains: ign eerste: and communication are distinct social practices. Representation on my faterest in my engagement with the world and on my wish to aive ‘erin realization to my meanings about that world. Communication focuses on my ‘ened to make that representation available to others, in my interaction with ‘The dual frame of rhetoric and design permits both: rhetoric as the polities Aonimunication and design as the translation of rhetorical intent into semiotic yiwortation. Rhetoric is oriented to the social and political dimensions of n;_ design is orient ~The retor has a political purpose: to bring about an alignment between her or his cn With its ideological position and the position of the audience with their Jogieal position. The designer has a semiotic purpose: to shape the message, using Wie available representational resources, for the best possible alignment between 1 Wirposes of the rhetor and the semiotic resources of the audience: mediating the Feauves of what is to be communicated with the resources and characteristies of / Hw mitionce. Benin as esigne hase sino intaess wile thle tasks ifr. The thetor environment for She oF he need pication-as, 49 \ /| Multimodality Communication: shaping the domain of meaning S1 ssuch-that the audience wll engage with it and, ideally, assent ‘0 swoment:Chird, ny interest is shaped by mn experiences overtime Te That UNS tsk Te cevnr asses wha nic apreeiaenal” | oT arts: sn er eure fod roy meres OT resources are valle, wif aul uacerstandig ofthe. celars needs and ain Init whet SARC TO RTENG OTST social ervrcrnent RTE ARETE STE {uch a way that the rhetor’s intevests, needs and requirements, are met and make th on to this phenomenon or object. The question tn representation Is What are the best possible match with the interests of the audience, in ynment where thy A the a Tor siving matertal form, material realization to my meanings’. FESOUFES Tar doing so are usually Inadeouate. Thetis 2 @rfoipeack. Ac seca (al we representation envrarments Change, eo Ue designs oF Te message Ed TS Change. Tha se moter | EQpmmreRTOayby contra, to ut the meanings to vbich Lam aving mater hich dies snag charge ine wth social change STG Cs text) into a THEPRTTOT wEers in my enviroment: to make * ox @verviay cammunicatiop the two tasks and roles come together in ori ingnings known To my a5 Tmierest has shifted fom person, so that rei? Is also WaT fe aptess oF ataral Tor ta reaizt Ay means egos pone Tar andere are-ParCe pars Th he That: ime with oi process of communications the socral relations with the audience = STEEL ESSE TE TOT TETTTETTaMI_ oF mode with the phenoMenor Hisehanger What Is rip relaTOM Te TrGse with won’ I am communicating?”, be represented and communicated. And designing, too, a communicational enviror oy members of my more Immediate soctal group Or are they more distant”, ‘ent congental to The audTence’s interest and for that which is to be communicates for instance, designing ‘everyday realism’ for a younger audience in school or au 2 greater degree of abstraction for a scientific one; or with aesthetic considerati uppermost for particular audiences. Always too, designing matters of affect, whether lighting in a room at home, in an exhibition space, or for a film; designing the messarx q also for the apt medium of dissemination and designing the spaces of display within Holl representation and communication are social processes, but differently so. that. Fjrwentation is focused on me, shaped by my social histories, by my present social 7 The exhibition at the Museum of London, “London before London’, shows amon) JiWi, iy my focus to give material form through socially available resources to many other objects, dioramas, tableaux, etc. — neolithic tools found in the Thame iw element in the enviconment.. Communication is focused on sacial Cinter-Jaction ‘They are cisplayed in glass vitrines, lit in a bright bluish light: much as they might ov | J go¢lal relation of me with others, as my action with or for someone else in a in an art gallery or in a jeweller’s shop even. In the exhibition on Swedish prehistory | elf social environment, with specific relations of power. Interest remains central, at the Museum of National Antiquities in Stockholm, the lighting employed for tiv | Wal Wy focus, its direction and attention shifts: from ‘me and my focus on aptly rooms dealing with the equivalent period Is ‘low’, with greenish and reddish/brow | ¥ajjaisenting some entity or phenomenon to my satisfaction’, to ‘me in inter-action tones. The one exhibition seems to want us to see neolithic artefacts as objects of | Wil) others in my social environment and my focus on success in engaging and beauty; and to think about the people who made them in that ight’. The lighting of | jMawiadiny others’. Representation is oriented to self; communication is oriented to the aher exibition encourages us to se the periods diy wsbe fr us, ort UMM, Representation takes place ina social environment; communication con. age’ tous, even though it also displays artefacts of great delicacy and beauty in srl! | iis social environment. SiansCas Trays Shap , alass cases integrated into the tableaux of the rooms. Wwleyeats: by my interest in aptly realizing my meaning and my interest in aptly ‘of rhetoric and design are neither exceptional nor rare. They are part of | uiyeying it to an Other. the everyday, mundane, banal, unremarkable business oF Conmmcafion as rich Wwlununication is a quintessentially social activity. It may be that as humans we ‘aLTimes part of ‘heightened’ occasions of interaction Servant OF Tetoric | gapulefined by our need to communicate as much as by our abilities to do 50. Certainly, ~"or to put it differently: the political and social interests of the PHetor are the | Willwn@ is an effect, a result, of communication and not possible without it; in turn, o the semiotic arrangements of the desire” | wjiyiinication is framed and shaped by culture and changes culture in the process ‘The perspective of fepresentatiy, asks: ‘What is it that TWish To say, write, g&tae; | WE wyjiwnnication. Germanic languages have words Such as: mittefen in German “exoress”,at this poif?™"What is my interest at this point in giving material sare | Kill wily Yellen = to share) —Yo have and share something in common with ya, and form to my meaning?’. Representation is focused on myself and_my inter We Hake you and me the same in respect to some knowledge or feeling’; Mitteflung in _Cegnnounicatioy is focused on iny interest in tS relation to others, With representation | WePFAN ‘sharing sornething about myself with you in respect to this message’; and there is(irst) gomething to whis alization, making some [NMAMAlende in Swedish ~ letting you participate in a part of me’, as crude translations. rnaganing fangible in he take" on wliat I wish to represent ari Filey te s0 basic and common a trait for higher-order mammals as to qualify as out of my interest interest directs my attention to somelhinu Ihab now engages me, a yitlonal ofthe species, human as well as other‘Moltimodality Communication: shaping the domain of meaning 53 Being social, the conditions for representation and communication chance wit changing social conditions; at the same time, representation and communicatio Hiclng (what are now non-existent internalizations of) boundaries of ‘public’ and constantly change social conditions, though each differently so. The social condition ts the notion of Your space’ has broken dawn with the breakdown of a sense of set the ground, they lay out the arena, so to speak, for representation and commun’ 4 4uiic’, The British state responds by criminalizing such actions (through the ever Cation. Communication constantly (reconstructs this social ground, the soc! MMMM frequent issuing by courts of “ASBOs" — Art Socal Behaviour Orders). An ces ee a earch, Choire Nia issued with’ an ASBO is constrained not to engage in certain actions, thatitatways potentially changes distributions of power. Potentially at least it mak: ing for instance; or to stay away from certain spaces. In other European societies ‘Toratmestin Folically- proleratc. Representation constanly_comakes, th Yall be treated asa sotal misdemeanour and dealt within that fashion From resources for making meanings and, in t@ remade resources, shapes those who ‘We could attempt to characterize social conditions, construct social taxonomies remake them. That_is the eff resentation 1 The COTBaM ser" 7 ‘{juulate their likely social and political effects, and from that assess conditions. effects on power and identity, communication and TepreSE Hore to be kept clean. In the UK, public disapproval no longer works as a means jwinication Iiyportant question is whether there isa need to distinguish between presentation ‘presentation. Is every sign that we make a re-presentation in some way of (a) ign(s)? Is every text a remaking of prior texts? At one level the answer has ‘yes. The model of communication that I am proposing here assumes that nication begins as the response to a prompt. There is the possibility of seeing hig either as constantly transformative, as a constant remaking —the position SF nopt; or we assume that sign-making, at times, proceeds from a fresh start’. ‘Hille one case we acknowiedge that our utterances work on the ‘ground’ that has ach vifferenty. mde interest, active in the world, Tt constantly changes the resoUrCes and the potentials For-configuring how the world Ts construed. Tn. CHangTNg’ the TESUUPCES or SEM THE WOTTE, Te CHANGES WT at any One moment to be seiticd knowledge. That makes representation ontologically and epistemological prose" atic. Communication reshapes, (reconstructs the social environment; it change ‘the potentials Tor acon, OF agency, OF Mose wto-are-pa re process ol | faajepablished by the form and the content ofthe prompt we have received. By doing ‘ssmaunication. That makes communication politically p gis ane Sa We we ourselves as always connected to, integrated into, the (prior) actions of ‘omalexes are shaped by both and hence they are always problematic for knowledr and poner. /-__Theeffects of communication ar both moe overt and more key toe subi policing than is the case with representation. *Policng’ may range from an unnctic adherence to conventions which have become ‘naturalized’, ‘made mine 0 a strovs \ enforcing af them, so that I hesitate to incur the penalties of contravention; to outro prohibition on engaging with oF producing specific messages (censorship), wt penalties for transgression. It follows that members of a community adhere to or challenge conventions for quite diferent reasons. ‘Narmai times’ might te those when conventions are kro are ‘second nature, by nd laroe regarded as essential and generally adhered to, « | amatter of course plo vert porremg Ts needed )Where they are not adherd lo, penalties may take the form of disapproval in ess normal times, penalties take sev" , Teumiwhere power & used overty to force acherence, A banal example might |x “ittering’. ACBottom, “littering” is a matter of the boundaries of public’ and privat of ‘purity’ and ‘dirt’; while the private space is kept clean, the individual fh no responsibilty about the public space (Dauglac, VORA). Fuery act of ciscardiny 9 sweet wrapper by dropping it on the ground s in that sense an act of marking ad communicating boundaries of public and private. There are, even now, socteties 10 | Europe where, by and large, these boundaries work such that itera’ fs not significant issve. Iti held in check by public disapproval and even more by interna ized boudaries of ‘public’ and ‘private’ *p a Your collective space’ an | liv the other case we assert that we right set the ground newly, in our own 1, in our own right; using, of course, existing cultural resources of all kinds, ting them to set our agenda newly. The concept of representation places us in ‘Heaweless chain of semiosis, which we reshape in our signs and texts, but reshape iv the limitations set by that chain. The concept of presentation suggests that atleast at times ~ able to step out of that flow and may wish to start with W agenda, ip a point of ideological, social and personal choice: to see ourselves immersed ‘ceaveless flow of sernfosis ~ both inner and social ~ and yet wish to frame an ment with the world and its presentation in such a way that our agenda, our #1), uF interests newly set the ground for an ensuing chain. Here ideological shapes the theoretical frame. =A social-semiotic theory of multimadality 35 Wuking of signs are made in social interactia the semiotic 4 Asocial-semiotic theory Uhre the form ofthe sian i on Fi ; ietiag hl to eld tna ese te fe of multimodality [ii te nurs oe caer fhe nearing It) Figure 4.1 below, a circle is used ~ by a three-year-old boy — as the signifier to Bes tre crewing, several crcles-er-wheel serve oo apt siglo express We signified ‘car’. Figure 4. 2 is a drawing by-a six-year-old boy of two footballers ‘squaring up’, to From lingutstie toa multimodal sacta:semiotic theary Wy the versclar Be positon ofthe vo frs an gt ster terres = of meaning and communication Hynified of ‘confrontation’ or “opposition. The drawing is &sfan-complex made with We vival node Toregroundes and carrying major fu Like al signs and Multimodality names both a field of work and a domain to be theorized. Anyoxi| jp complexes\ is a metaphor, newly madéTin €Social-semiiotioXeke on represen- \working inuttimodally needs to be clear what theoretical frame they are using; «| Yaion and cornmonicationgall signs are metaptogs, always newly mage. Init ‘realist! ‘ake that position expint. SgsTar-semiotic Theory is interested aT i ation this connects with aspects oF the eurrently popular theory of metaphor oT forms. Meaning arises in social environments and ih social interactions. That mish wd Johnson (1982; also Laka TTS Tir soctal basis However, the social Tito UR Sourze, TE OFT TIEPATOT OF WEARTAG. TE theory he lols departs from the cogntivist approach of Lakoff and Johrson. “the sociat is generative of meaning, of semiotic processes and forms, hence the theory pb ond. Is a social-semiatic one. dahon. The core unit of semiotics is the sign, a fusion of form and meaning. Signs ex's ral nego Ui odcerecd ence oto taeee net Gy rani oa sgnaorp, aes tr heel cen Tei) ‘he active making of signs in social (interactions — SITS APE TAGE reer than 0 V “i fice. on sien ane rae War sla ose ie OF severe Temare wed — ‘inalshessociasemitc theory from other forms of senToTRs. rrsoetseri | CN coun of ening, aids, wide social Wve socaly shaped, cates | social environments, using socially made, culturally available resources, are agen and generative in sign-making and communication Figure 4.2. “This is a car’ There are several, elativaly distinc. al Sena rivet ] the writings & aViewed from a relatively abst | s \eFGh they end to cfer in relation to one issue: whether they base themselves on Y —chinguistiyor the Semitic perspective of Halliday’s theory. Wie there 15 2Oree? i BUEN Th broad soettne ane THaeh detail among provorens of either of the \v possibilities, the differently placed emphases do have significant effects. The thei] a , lyf used ee acops the seme perspective of Halliday theory, Rater tan ae to construct a compromise view, from here on in | use the label ‘Social Semitic refer tothe aporoach set out in Hodae and Kress, 1988; Kress 1993, 19973, 200) 7 Kress and van Leeuwen, 1996/2006; Kress and van Leeuwen, 2001; and in respects, van Leeuwen, 2005). It rest on several fundamental assumptions: sigs. ; aluays newly aden soll latracln, sons are olvat at aca fla Oe PA | (of meaning and form: the motivated relation of a form and a meaning is based of ai Figure 4.2 Football npis2s Out ofthe intorest of makers oF signs; the forrm/igniiers which are used ie iy Arsenal vs, Chelsea’6 Muitimodatity A social-semiotic theory of multimodality 57 jorrelates with these social features’ or ‘the occurrence of this form points this social environment’). Correlations show links(but provide no inguistics, pragmatics and a social-semiotic approach to representation Before elaborating this approach, I briefly want to ‘locate’ it among a few disciplinary relatives to provide sore orientation Given that thinking about meaning has in ray Nays been shaped by a linguistic understanding of Tanguage, it might help clear the | decks ~ soto speak ~ of some longstanding notions around meaning and commu cation to sketch some differences in focus between a linguist, a pragmatic and » social semiotic “ake on representation and communication. The orip of the Tinguistic take’ on communication rests an the long history of th study of language _in European intellectual traditions. Like so much of ‘European soci iba ass ron tLe Tn this caSe fradittons going back io ‘the grammarians of Sanskrit (e.9. the granmariap Panini, between 500 anc iC. STEMS We aramid y valaoe Gitano Green to Rome ia ab scholarship to medieval Europe ~ and much transformed along the way. In tie Sanskrit grammatical tradition, concern for the ‘purty’ of transmission of sacred texts fed toa nee fs angus a & necessary guaantor for that security of eta in ian In many banal but social sia Ways sare Tie that alftude remains, adapted to contemporary givens, Now, reliance on tie stability of languageis-seen as essential in relation to significant public tex! canonical or ot Jnty apout the re i it Sil a the guarantor of rationality and knowledge, of all that was and is seen as qui eSentiatly hartan, Nor is the notion of purity far from current public” as witness the outrage caused by changes to ‘conventions’; of which panics about th moral and-social effects of ‘texting’ are a current instance. If language guaranteed meaning and rationality, then theories of language cov also be relied on! to provide tools to explain what needed to be understood abou central issues in representation; as indeed about much else of value in culture aiv! society. In that way language and understandings about language have long sinc ‘come to provide ‘naturalized’ access to theories of meaning and communication. To sketch some differences of linguistic, pragmatic and semiotic approaches, | will use an example from my first years of teaching, in the early 1970s, at a ‘new University in the UK. It was not uncommon ~ things have become ‘tighter’ since — (or students to ask for ‘an extension’ to complete some work. The approach was usualy something like: fovin of this oft repeated request puzzled me. It seemed that power and power- nee were the key to understanding what was going on and would lead to ining variations in language use. My question at that time was: how can 1 ii) th social meaning of ths simple yet complex request "I wanted to ask, could suiextonsion far my essay?" | had other, more specific questions: ‘How is it tha Tnnwe (the morpherne Ged), whether in ‘wanted oF in (can + ed) = could) can sign of (relations of power?’. A grammatically/syntactically founded approacityan readily provide a description of clauses and their relations in ace; oF of the "sequence of tenses’. It can indicate what elements il rules of ‘grammar’ are being used: the use of the past-tense morphemes; Wee, The categories of 2 syntacticlgrammatical accowit, €-9-chawse, subject sentence, or adjective, tense-marker, noun, can readily be used to descri "Ws of that utterance Wher This approach is used, the focus is on tl H/o OF form and of formal relations. ontrast, erate Cy scinliaaulé) approach asks: ‘Under what con- Wh, When anc =, are these forms used and what tend to be likely effects of That provides descriptions of the environments of use and ofthe conitions N such forms tend to appear. It describes the socialinguistic roles of Nts) their status and relations of power; Figen see etc sinh as face, repair, politeness. To some een Te Tocus ison correlation: iste {orn Correlates with that Social context. The orientation is to use, to ‘ul environments of use and 4a same extant to-offe joach asks: ‘Whose interest and agency is at work here in the iilig?', ‘What meaning ts being made here?', *Hovr is meaning being [With what resources, in what social environment?” and ‘What are the mean- Mite of the resources that have been used?’, ‘How can past tense be an lof of power’, “How is it that a form that signals distance in time can signal Hi islonco, a “distance” produced by difference in social power?’. There is an Wun 10 the intorests of the sign-maker Cdistancing’, making her/himself 10 the environment in which meaning is made (the relations of power which AN the sonctiors which exist in not acknowledging power); to meaning and to jl/eultural resources wich are available for the realization/materialization tivated sign, bol the social-semiotic aspects. It does not deal with the “Iwanted to ask, could I have an extension for my essay?” Ae that time I was teaching linguistics ~ undoraraduate courses in (transformation!) syntax, semantics, sociolinguistics, psycholinguistis, stylists. I was trying to unc stand regularit was not comfortable’ with the ‘correlation Aha mode One response might be: ‘wel, the only mode here is speech, approaches la William Labov (1966, 19727 then dominant in sociolinguistic yulstie account is sufficient, nothing else to say.” A Jinguistic account, owe To me they did not offer a sufficient explanatTon of what it should be possible to know Mga tla it worwnyac 4 what Linguistics recognizes as siniticant about) iis about other ‘and say about such utterances (e.g, Yin this environment, this linguistic oF phone wy ciftax pikes ees58 Multimodality sneallin ree stele the moo peach itaeentd-atven Core eton of Spee ras 7 EAT us wheter few spoken ath 8 rising ora talon intonation at Tr “BEE am pcan oth souls eter eter ofthese yes i saa smile, acheeky grin, a serious or emt er supenstat? ilTirasesier ok at rf “gorse eer? de rsh cve ra i TE TOUTTOF ai they stay wedged ir TFE WAT-ope door? | wanseroton of ies Gut anyother canola aspects of mann 4 ie Calegory-of the mode of speech-forces attention toll signs 1): ? ‘snake lanolin” ard the cotgord ‘which that supplies. The semiatic category of mode attends to the potentias | making rearing of sound-as-sveech differently to the linguistic category of spe««l The other point here is that if a linguistic account of this interaction is see suiicient, necessarily it cannot give attention to other modes witch are in play: ca facial ex tial p9sitioning. ~ Each of the signs at Eh positioning inthe haf-open dor rmuthas the shape and the extent of the intonation contours. The ensemble of sic a a whole makes meaning: the distancing past tense coheres with the positioniny the half-open door; as does the tentativeness ofthe rsing intonation, The indivisis si semble can of course be used to contrast, to contradict, to ci tension: the distancing oF the past tense can Be ‘BeFSOM wali © iahein tothe office; or with the marked fal in intonation which turns the gran) matical form of interrogative into th ategory of statement. \ its shape, is ‘ready’ inrors’ Ors sak — the ‘meaiiny Tiere, a (Grammatical) indicator oF Temporal ds. ‘Whe past leMSETHOTpTIeMe -ed) is used as the 19 social distance, distance of power. It is possible to call that an Ganic relations ein time ‘is ke ther ference between the tree approaches les in what I cll he eo eeepc fe prac, Aah i description of naulstie forts and tel relations. As soclotngTtes, Tis ese” AE area at the absitton of conditions an coretan. redma spp ‘Recs gallon in meaning inked wo variation BeoreeT=y "Necessarily, the linguistic approach has nothing to say about other modes in makin meaning; though te principles of pragmatics do rot exclude an eres In extend its reach to other modes (Morris, 1970). In some sense both linguistics and pray in terms such as Yextra-tinguistic f matics recognize the presence of other modes A social-semiotic theory of multimodality “WM inauistic’, ‘non-verbal’ or in different kinds of acknowledgement to features of iNet’. That, however, is a recognition of the phenomenon in the same moment as ‘stant dismissal; a backhanded theoretical compliment: | notice you and you're THE Higniticant enough for me to bother. 1} course, as a generalization this is too sweeping. There are and have been clear PH wlons (iictmsiev, 1961; Halliday, 1978). Yet in a ‘disciplinary common-sense’ If such a thing exists ~ it is not too far off the mark. Often this position is ‘S/eiypanied by an assumption that, in any case, images are organized like language, HWilnoss terms such as ‘visual literacy’, ‘visual language’. End of story. Tike the example in Figure 4.2. In a social-semiotic multimadal account of JN, all signs in all modes ace meaningful mntation: to modes, mez Tigements. It can, for instance, ig.about meaning relations and Ur Tistartation in image through the | avrangemeadof visual elements; it can elucidate a syntax of this visual ilalion ~ the meaning-potential of spatial orientation of the players standing Joce, the use of colour as an ideationat-resuree; to identi sams; of that is, Te use of distance as a mean ; avon sect tyealized Wiplonce at which the players stand); their Eis epreson dour Tally to fective size and the prominence of the stua® or their boots. Should we want TMF We dichotomizing terms, we could say that affect and cognition are equally HNultaneously present; though much more significantly, social-semiotic theory sin Uslorhaleng thet dicho, via ths xorple france. Hi Is a multimodal text; the modes in use are ing, image, number, colour (and pression), Social Semiotics is abe to say something about the function of each Modes in this multimodal text; about the relation ofthese modesto each ather; ht the main entities in this text, Hi sunnmarize: linguistics provides a descrotion of forms, of their accurrence and of Jelllons between them. Pragmatics ~ and many forms of sociolinguistics —tells us sa! circumstances, about participants and the environments of use and likely {6 Socal semiotics and the multimodal dimension of the theory, tell us about Hai) ageocy; about meaning\-making); about processes Of sian-making in social HNonts; about the resources for making meaning and their respective potentials illers i the making of signs-as-metaphors, about the meaning potentials of WAenotic for. The theory can describe and analyse al signs in all modes as AAA Mlle Interrotation in any one toxt. = Wr Felavion to the example of the spoken request, Sal Semiotics tap say: power Ht retative social postion it can be expressed ab a distance MAAN Ti T0 sia al distance of powor. The degre of Wenn etrar terre rilated by @ manced and fayered use of apt pipiens Hor brtance °t had wanted to ask I wondered, could | possibly ask if 5930 Multimodality A social-semiotic theory of multimodality 61 you'd mind, if...) to make ever more complex/wvanced signs of relations of power ‘hilting the frame only slightly, the choice of a theory from Distancs is less power ys ideologically founded theorieSTs Tiself Bar herp tra) speaker both to acknowledge the distance in power bety the ald Hitloning is unavotdable; positfoning isthe result of chotce Fram-amiong a range of Fessee and ¢ means ~ being distant in time {rather than in space) — of temgor,) iilties; that choice is Socially meaningful ~ itis ideological. That is the-case with distancing from the request; in Tam telling you something about a pas! han with other forms of semiotics. It is nat to lapse into wish, about something that 1 had wanted to do then, whereas actually, now, Ta Pea io longer in What positon.’ That is, the request can be expressed and STWTTANEG) ‘Semrdented in this (weakened) form; it is modalized.\Being modalizeg (Halliday / 1967, 1964; Kress, 1976), Hodge and Kress, 1979\ TORS TT Fecipient tly ‘opoortunity to adopt o al possible readings. fn finguTsUIcs This s usualy treated as an instance of @fybiauly. In the sketch oF communication developed Chapter 3, the utterance/ressaJE provides a ground which can readily be framed @ prompt in either direction, according to the interpreter’s design. In the case of Figure 4.2, Social Semiotics can say which mode is foregrouncd which mode carries major informational weight (functional foad); which made |x what function in the overall textual entity (writing as caption, image as major ai! number as minor carrier of meaning). In the mode of image it can describe tiv arrangement of the entities of the image and their meanings ~ of ‘opposition’, of ‘confrontation’, of ‘challenge’, and so on. sod to describe multimodally constituted texts. The descriptive, analytical and logical apparatus of the prior theory is brought along and leads, necessarily, toa Woscription of the domain to which it is mow applied but for which it had not been oped. [The choice of a) theory carries, however implicitly, potent messages about what ake to be conceptions of culture, the relations between ‘representation’ and culture te fundamental characteristics of culture itself. A full theory of meaning needs a Hotion of culture and of meaning. |modal Social Semiotics th Ueniess 5 PTET TTS a re media of communication, From arsectve of eins ry el ih aan al mesa the relations bet ss. Here there ar€ question such as: ‘What Kinds of actual eke are common to all mades and to al te tyes of relations Horses for course: pt theories, useful framing Theories, of course, are just that, theories: multiple, competing, aften internally contradictory. Theories specify their domain, more or less explicitly and precisely they provide categories to describe and analyse the phenomena which they constrict Language, more maybe than other social phenomena, suggests itself ‘naturally’ 2% 9 ‘metaphor of the social or the political ~ or of their negation; and linguistic theor i act as potent metafoy ‘Among linguists, / a trenchant opponent of political oppression it the twentieth century, Is TE UMewWho has been mast insistent on separating languay) Tar aTfordances, Ws Wstories of social shaping ard the val origins/provenance of elements of that mode. As an instance, the mode of hn English has the semiotic category of intonation. Even though the modes Soundtrack oF music also use the material of sound — with its affordance of pitch from social and politic = er iS his political writing has becor lution ~ these modes do not have the category of intonation; nor of course do modes; UTTveTEliy’ the common characteristics and capacities of human brains every and at all times. This (seemingly) asocial theory isa potent metaphor of his poi position. A similar case can be made for the linguistic theory of Michael Halliday though i ite dire +s assum out the Generative pow! the social” explain thea jal organization, human action in scl ‘environments and the meaning potential of linguistic forms and processes. ll ‘rnetical terms, the three perspectives are difficult to keep apart; yet, for certain. Hiptive and analytical purposes it is useful and at times necessary to do so. 7 ‘Most Amainstrea fiLseMullimedal Soca! Semiotics deals with IE miwaning and ppear-as Bn integrated whol, a sigr>y\s Signs are always newly made according n ngustic theories separate meaning and form — syntax be analVlic edge was used in Critical Tes Ukress and Hodge, 1979; Fowler ct vi. | (AWW interests of sigr-makers in specific social enviranments, there is neither need 1979; Hodge and Kress, 1993) ag the tool for social critique: the social agent Ye place for a theory of use, that is, for Pragmatics. Use is, simply, a normal part capacity to exercise ‘choice’ from a complex system of /inguistic meanimaypotential iy Wh he making of every sign. A theory of use is reckundant in-an-appreaety- which has: Me COMEXT OT a SPAT sOTTAT evra Auman te sign maka ts centre the sian, + all, is made in and for theMultimodality Conditions ofits use, Signs are made in a specific environment according to the sign ‘ihaker’s need at the moment of sign-making, shaped by the interest of the make ‘of the sign in that environment. The environments and circumstances of ‘use’ are. therefore, always an absolutly integral part of (the making of) the sign: they are a the centre ofthe concerns ofthe theory. The signs made are as precise as it i pessib to make them to realize the sign-maker's me re are other theoretical consequences, ILsgns are precise, that leaves th Geegy e catnatnt ont itis reveled as an-alempE TD patch up problem at the core of a problematic theory; a theory, that is, which had no plausible Account of meaning-mating, Siar arguments apply to enterprises suchas Smet: and Stylistics. Cleary attests important to focus on meaning; and ste is important component of a social-semiotic multimodal theory. Yet the need or justi, cation for (sub-Iaisciplines such as stylists, semantics has gone: style isan outeor of Simrescive-chaiceso jn-the-agking OF Sig's ar slgn-complenes i i ‘Similarly, Yhe,goncer oF Sociolingatsticg vanish when all occasions of sigr-making are embedded in and A social-semiotic theory of multimodality " fielice classified signs according to the, {rind which they {0 that which they represent in the/world: an iconic sign ‘resembles! what it “Wifesents ~ the circle as weet; an indexfcal sign ‘points to’ (as in ‘detxis’) an object, Nt ~ Peirce’s example was that of snoke as indexing fire; 2 symbolic sian stands conventionally agreed relation between a form and the WH Crossed cresce bols for a humanitarian organization (Colapietro |, 1966). In his theory, the function of the sign does not — except in the case of wile signs ~ stand in an arbitrary relation to the world, but is shaped in and by Hite rotations. 2d}, was on relations of the ‘ouler4yorld to-aa in the sign., relation dued-getween a phenomenon in the o Din that inner mental world the sianifiee cFasTan, the sigi is expressed in an external, outwardly ‘oF otherwise materially TaamibTe form.,in Saussure’s sign, signifiarand Wve linked in a relation which is arbitrary and bound byxEomvention> Shaped by social environments ‘The motivated sign The sign is the central concept of semiotics. In the sign, meaning and form are fuse! in one entity. In 2 Social Semiotic theory, signs are made—not used by a sign-maker who brings meaning into an apt conjunction with a form, a selection/choice shapes! by the sign-maker’s interest. In the process 0 dealing with the social world ‘The two main strands of Semiotics which have dominated in ‘the West” are based on ‘the work of one or both of two major figures: Charles Sanders Peirce (1857-1913), an American philosopher; and Ferdinand de Saussure, a Swiss linguist (1839-1914), both teaching and writing int 1e nineteenth century ard into the very sort part ott treneh Coe Patan mod) offers three crucial emphases: 9 classification of signs based_on the TeTatToM-of signs to “the world’; the process of iosis; and the category of the ing ts = In Peirce’s account of signs (musty the process of semiosis ~ ‘recipient«/ inierpreters of a sion make thelr sense of the sign they ‘receive’: they form interpretant of that sign for themselves. The faterpretant is the meaning of that si for the recipient. ‘Readers’ are agentive and transformative in their semiotic engase ‘ment with signs. Once formed, the interoretant becomes (asthe new objec in Peirc terms) the point ef departure for a new sign, leading to an ongoing, constan'y transformative chain of semiosis. In this process the 'veader'Pvecipient’ as interpreter 4s the maker of new signs and, in that, is the remaker of the semiotic material of the culture. It is a process of a ceaseless remaking of meaning, of interpretants newly with a prior sin, ] formed in the transformative engageme . irariness- and convention pcint to social pawez, though in distinctly Invent ways: @rarine ae dcation ofa socal power which is suficin {0 tie any Torm to-aty meaning; and égrventioD the effect of social nower i a socTal TOrEe which acts to keep signs stable, a stabilizing force for the Monty whe 5-10 i. In this account, the actions of individuals cannot ‘signs or the relation oF the sian to the system of signs (Hodge and Kress, ean approach assumes change based on the actions of individuals i :[ulination of thacinterpretantyyavich Bee ‘Object of the next stant The iuiyian accolar rites oor TaTvranAT action as a possible means Tor chang (yaleiils stable held a pracechy-the Torce of Collective social paw alized Jotin( social account of meaning based on the significance of the agency of Ula, is entireTy-at odds with a conception of an arbitrar il, established and held in place by conventipn. ‘tsanple used by Saussure to make his point was a lexical one, the French wo Meee re ee ifier/sound shape (tri:] is linked with that same signified, as Is Lbauml in German? milation between sound shape and meaning is an arbitrary one. That seems Hible enough, J. castle say, ther we could use any small object, a button maybe, to replace/ | This seems to confirm both the arbitrary relation between form and meaning (thé Aeetion of Ue chess piece) and the strength of convention, agreed on by both players {We the duration of the game. 63 >) Wiligenstein (1935) extended the scope of this reasoning, stressing the force Ot SHeeilon by introducing the example of a (chess) game. Assume, he suggested, that ‘we were setting Up a game of chess, we discovered that a piace was missing, a. yy |64 Multimodatity A social-semiotic theory of multimodality tease ck ates tty ta we woud wea Dk ben sad for the black piece even though the notion of arbitrariness suggests that the choice of | awhite button would do just as well. At this point we have made two decisions that avr not arbitrary: the decision to use a button to stand for a chess piece and the choice vf at bate. te omer een tecnico ur anae auton forte acest ay sn nema end eso Src sre tance or toe ietane ne mf paca | Svan: yh prnr rl bv aged on se sew crs arias oderewoten erhearatona hesene Had wei oe Bathe ole vaso daca tad atts of eres we wale erg ton St ner Dice egress seo tor agin fe ape Iie I tthe Hck en ewe pres were mtn path e wel ardor sabe ero bats De me mu are a Sct lay rnd te ming we ew let a versa It seems clear that ny rate the principle of arbitrariness won't wor What seems lke wirat convent are socially motivated, aistneey a Choosing “bTacK" in the rst example meant select ‘Colour’ as the criterial aspect of the object to be represented; choosing ‘size’ in Ii second case meant taking "value’C-as-size) as criterial. Tobe effective, whether in game of chess or more so in the social game of language — as Wittgenstein put i both selections need the joint agreement of the players. But the decision also needs \0 ‘work effectively for playing the game: using white buttons for black pieces and smal buttons foe-mare valuable pieces would work against the necessary “transparency i: We can push his example further. If the black pawn was missing and we had son Inlues the symbotie sin, Theoretically, to allow for itl bs of arbitrariness” or there, allowing maybe forthe edd erbitrary sign in some environments s Bod istake, The effect so totally unckermine the power and uschaess of fy ofthe motivated sign ~ wheter Inthe direction ofthe pssbltyof refutation 'the theory itself or in the use of the notion as a heuristic device to hypothesize about Hig, Assuring the positon that al tgs are metivated conunctions of form ireatina fin orale ta eign of he Red Cros Te REO CORT beg 9 oer) the colours of the Swiss flag) might pethave been yafrelated to Swiss neutrality, aie ofthe sive’ oF being an ‘urirwole, reutral party ‘an Sparta tation of assistance’ et to menor the nately ofthe fener afte ization. Nor Is it implausible aiven the religous mearing of res, that in some lies the Red Crescent was chosen asthe apt signifier or ther. tn that attr case ‘Tp the ‘cross’ as religious symbol rather than as reminder of the Swiss flag which fred criteria. T have not checked, but it would be revealing to find what versions of I raed Greseet tore ar,and wh Wrench coe thingie of rath: i ightforwardly in ordinary everyday interaction or in forms of research. If the ype’ of the signifier aptly suggests the ‘shape’ of the signified (e.g. the circle and Giotvatjod each ooint to soci ‘Mee, it allows an analyst ~ whether in everyday interaction or in research ~ to icin Se maga = ay jwothiesize about the features which the maker of the sign regarded as criterial about and tramspare Tations of form and meaning in the sign. The apt Fea ‘abject which she or he represented. Positing that relation between ‘sign’ and ‘oF material form and cultural meaning isan expression OF the Sonsmaker’s interest i (s crucial; it opens the possibility of a path to understanding what in the g ‘ms meme ey eee ee 7 wmenon or object to be represented was treated as criterial by the maker ofthe rzation” of Their meaning and tha, in-uuc, is needed crucially in communica Al the moment of representation. That can lead to an understanding ofthe sign- ers position in their world at the moment of the mal H hypothesis is of fundamental importance in all commur ‘Historically as critique or as design, w of the sign, Such tion — whether framed ‘The everyday, the banal and the motivated sign “Diyneure's Aarstaken assumption th of signifier and sianified is an Vultee vasa all tory, a product ane realization of he social conditions Wis ting. Here ate three objects. Kis arbitrariness takes no account citer. of Social Senate, i | want > be wners08 by preference [Use the resorcisthat ose Taig the signs WHET reed to make. [[Tan-nol fariiar wih Urose resource Take sia i TU Tarn avery saapene te ean TORSO 655 Multimodality the patent facts of the histories (of change) of semiotic resources (see in this respec raymond WilTams’s (1985) Key ‘ior of the facts oF contemporary sign-makine prasiee nary soe See ts ona soles or SURES SETA ia characterises and OS Ah sper se anmete CRE Genies agency To those who make meaning in making Sigs: Twisting To buttPess one ‘ilar of he cca ary tre FeO RETNE PONTO pero ecOTECT Oe Tord Source o at poner valve agree Trova We at v MOTWTaUATs acting as Sigi-MaKers Out of socially shaped interes with socially made resources in sock is. To deal with te Tast SE Tlrst: observation of semiosis at any level show: constant change rather than rigid conformity as repetition or ‘copying’ — whatever the modes which are involved. Every example in this book provides an instance of this, ‘The second objection rests an a confusion on Saussure's part: the signifier of “ree ness! is not a sequence of sounds, that is, not Lazbr], but an existing lexical-item-as signifier, ‘tree’, used in its potential for becoming a new sign. The meaning-potenta Of the signifier ‘tree’ is the sum of all the instances in which I have encountered tiv sign tree as a signifier: that enables me to make a prediction about its aptness a a signifier for the new sign that J want to make now. The signified TREE and the sic nifier ‘tree’ are elements at the same level and of the same kind: not as in Saussure's assumption where one is a semantic entity and the other is a phonetic one, one a entity of meaning and the other an entity of sound. ‘Assume | have bought a pot plant, a ficus. | know nothing about the plant’s natural habitat or the dimensions that it reaches in its natural habitat (in Australia it is 0 rainforest tree which grows to huge expanse and height): my plant is about 50 ¢m i height. Should I call what I have bought 2 ‘bush’ or a ‘shrub or maybe somethings else? Or is it best described as a ‘tree’? At the point of searching for a name, th signifier that 1 am looking for is not a string of sounds but a abel’, a name, that [have encountered before, sufficiently often to know that its meaning-potential ~ 0 woody, perennial plant, of significant size with a strong, usually single stem — best captures what this thing that I need to name is. My question is not ‘will [shrub] sou! best for this plant, or maybe Cbush?” Rather, my question is: ‘Where does this pia “belong’”? *Does it belong with flowers, bushes, shrubs, trees?’ and: ‘What lexical entity will serve as an apt signifier for this plant in terms of its present characteristi or in terms of its future grovith?’. Someone from Norway or from the Black Fores! used to trees as towering pines, might, on a first visit to the National Parks nea Sydney, (Kuring-aai Chase to the north; the Royal National Park to the south) be unwilling to dignify the twisted, gnarled, wonderfully shaped vegetation on the sans stone plateau with the name ‘tree’. The ‘settlers’ who encountered the Mallee (re of the South Australian inland used the ward ‘scrub’ to name the expanses of these hauntingly beautiful, low-growing eucalypts. That gave legitimacy to uprooting thousands of square miles of that forest and sow the land to wheat ~ at east until the A social-semiotic theory of multimodality in topsoil had blown away. The use of the signifier ‘ree’ fs not automatic; itis Klally motivated and individually enacted as apt for me, on that occasion. ‘My problem with naming the pot plant is not about the sound that would seem right this plant, but about the lexical and social ‘place’ where it would fit. Had it been, former, then, yes, bub or sub, slib even, might all have served equally asa name; though questions Gf euphony aight have been a consideration, itself “Wojarmatier of an arbitrarily made dSTS79%. In other words, insian-making there js Mircloapyiacen sented ane soni: both are fom the same ler. Soa pes dato Wal instances of sion-n makina, Tor” 25s, 1988). jal, the everyday and unreTTarCaDTe El ‘@xamples here are chosen with that in mind. Colour plate 3 shaws two Tmages 0 ts of salt and pepper, the kind you used to get (and still do, in some cases) on iy ray, travelling in economy class on an aiine. These things have a practical use: snake the meal on your tray more palatable, if you like salt or pepper. They ar also Wand they interest me as signs —of salt oF pepper. On a different airline the sait pepper sachets are diferent (Colour plate 4); and ona third and Fourth they are wnt yet again (Colour plates 5 and 6). This variety requires an explanation; and thmonstrates to my satisfaction that the sign-makers have chosen ~ lferently fch case —what isto be taken as criterial about the object to be represented: how Jopresent ‘salt’ and ‘pepper’. There is a vast range — infinite maybe - of possibilities Aepresenting ‘salt-ness’ and ‘pepper-ress’. Each of the choices provides insight the sign-maker’s intrest. In Colour plate 3, the colours on the sachets ~ a kind fange red for the iodized salt and dark brownist-orey for the pepper ~ point to A\est and motivation: the colour of iodine andthe colour of pepper ‘As with the chess pieces, in each case a criterial feature of the thing to be repre ted chosen, In Colour plate 3, colour is chosen to distinguish the condiments. For passenger dealing with lunch oF dinoer on their tiny tray — with the light not ail 1 Q000 in the middle seat of the central row ~ the most important thing is easy Heognition. Most users might agree that the greyish-brown is an apt choice as a lier for pepper and would not work well as an apt signifier for salt. And wile it ibe objected that salt is rarely orangey-red, the colour does work both in inguishing it from the pepper and in signifying that this salt is ‘iodized’. with of me, I am unlikely to go wrong in my selection of salt or pepper, even ht not manage to read the word “iodized! on the sachet. A feature ofthe et toe represented is selected as criteria and becomes an apt signifier in these iti circumstances. Di nother avtine (Colour plate 4), a different criterion ie selected asa signifier for “Hie samme object ~ salt and pepper. There are two words, as labels; the message is Wejwated’ in a way; maybe because, asin the other case, words may nat be reliable ya! under the circumstances. Here colour is not criteria for distinguishing salt Mil perwer Instead, there isa visual reference to a culturally specific practice of 1 arn using salt and pepper in do nts — in salt and esti¢ oF public envir he best site to anchor thean 67 mer us +2 Multimodality pepper shakers. We can ask whether colour is still important: there are bound to be Colours that would not be apt as signifiers for either salt or pepper. If so, it woule indicate that colour is significant here too, though implicitly. This too is a motivated sign, even though the signifier chosen is from an entirely different domain to that 0 colour, namely a reference to social use. That signifier too is apt for the signified “The tivo signs show a different ‘angle’. In Peirce’s schema, the Delta sachets mish be regarded as an indexical sign, pointing to an earlier sign, an actual shaker and its use. in Multimodal Social Semiatis this is seen as a matter of provenance, a referenc: booth to a cultural practice and to the object ‘salt and pepper shaker’. The question posed and answered here is “Where does this signifier come from?’ (Kress and vai) Leeuwen, 2001). The signs of the sachets with iodized salt belong to the Peircear category of iconic sign ~ of “looking like or being lke’. Its the sigr-maker’s decision as to what is to be taken as criterial ~ with any sign. The different ‘angle’ indicate ferent positioning: we are in the realm, however mildly, of ideology. There may be other criteria. The deep navy blue colour of the salt and pepper sachets in Colour plate 4 loaks to me — | am not a frequent user of the airline ~ like the ‘corporate blue’ of the airline's livery and logo. In other words, the sachet of salt and pepper may be being used to signify a ‘corporate meaning’, ‘Delta-ness’ —a fee good factor to sprinkle on and digest with your food. Provenance ~ the questio’ "Where does this come from?’ puts us inthe difficult waters of difference of cuttur practices: in some cultures you get salt from the shaker with the one hole and in other from the one with many holes; and vice-versa with pepper. That does not change th principle of motivation: it points to different social histories, valuations an! consequent differences in the use of these condiments in the cullnary and gastronori histories of different cultures. In Colour plate 5, the sign refers to specific social practices: the uses of sait an pepper in the social histories of cooking — the practice of grinding the ‘raw’ materia ina marble mortar; a cultural reference leaning on tradition, on a specific aesthetics and evoking notions of authenticity ‘The relation of signifier to signified is not an arbitrary one in any of these example Tt Is doubtful whether any one of these signs will become generally accepted 0! signifiers for salt or pepper. The airlines’ intentions, if anything, are to keep the distinctive; and maybe to allow themselves to engage in alittle mild playfulness wi) their ‘guests’. Nevertheless, the theory ought to be able to tell us under what cov ditions this might or might not happen. What it does show is that the relation ol ‘gnifier and signified in the sign has a social and therefore ‘political’ and ideologic component. A ‘playful relation’ with someone nowadays often called a ‘custom fr a ‘client’ i to make a sacial and idenlogical paint about that relation and about a wider ideological frame. In my last of these examples, Colour plate 6, ‘ideology’ in the sense of the suggestion or projection of a particular ‘world’ seems to me quite clearly present. Hevs the colour-scheme works together with the faint 's p ic y’ printed acrass the bottow fof one side of the two joined sachets ~ spicy meting that adds fun to lie A social-semiotic theory of multimodality {s no longer just about salt and pepper, but about a characteristic which might je not just an individual but a community, about a whole attitude to fife, about ‘Austrian and Viennese. Here discourse ~in the sense of a positon in and to life, oping of a way of being in the world, of ways of being and knowing ~ is most ly evident, Equally with genre, as an expression of the social re Ui and its ‘guests’ and the ‘address’ of the reader/viewer. ‘uci point is the unnoticed, near invisible social and ideological effects of signs of the everyday, the sions of ordinary life, of the unremarkable and banal, in Wiscourse and genre and with them ideofoay are potently at work — nearly ibly ~ as or more effective than in heightened, clearly visible and therefore ble instances Koncept ofthe motivated signin no way places restrictions on sign-makers; the Jp ns open or as restricted as the sign-maker's intrest, which shapes the sian an 1 which isan effect and a realization ofthe histories in social environments of ‘ignemaker. The examples show this clearly: one speaks of health (the iodized J) another of social practices; the third of culinary traditions, of the aesthetics ing and of kinds of authenticity; and one speaks of lifestyle, of pleasure and All Awse are present, differently weighted, differently valued. The social, its ies, knowledges, its forms of social relations, its discourses and genres, are here, {in these unremarkable, everyday, banal objects-as-igrs. the examples show the social embeddedness of sions and sign-making practices: ies ofthe value and valuation of pepper inthe histories of European cooking; of {uses of salt; of their preparation in households; of the recovery of tradition; Just being “cool. At an Italian restaurant near my place of work — one of a ul ‘chain’ ~ there are small, shallow, white saucer/plates on the tables, filled arse flaky sea-salt. They invite me to imagine myself as an Italian peasant, up salt with my fingers, crushing it and sprinkling it on my bread after dunking Ae saucer with olive oll. 1 am reminded of the one-piece heavy glass salt and I containers of my childhood, with two shallow ‘bowls’ on either side, in the othe table for everyone to take their pinch of salt or pepper, or ‘dip’ their cr bread. These objects show that signifies and signs carry, in their make-up, hice of long histories of practices. The meanings of these practices are present ‘ignifiers asa potential for meaning and are carried forward’ in constantly lornved fashion into new signs, remade in the light of the resources that (re)- #9 sions bring with them. In signs, sian-makers mediate their own social } their present social position, their sense of their social environment in the ‘of communication; and this becomes tangible in the reshaping of the cultural 196 used in representation and communication. The makers of signs ‘stamp! ML social conditions into the signs they make and make these signs into the Jor of social histories. CS70 Vv Muitimedality A social-semiotic theory of multimodality vinaking however remains constant. Partiality of interest shapes the signified at oment of the making of the sign. AT Te vary next moment the sign-maker’s Interest and the partiality of representation The next example shows the processes at the core of sign-making with great clar'ty A three-year-old, sitting on hs father’s lap (mine, as it happened), draws a series o! Circles, seven to be exact. At the end he says: ‘This isa car' (see Figure 4.1, above) How is, or could this be ‘a car’? While drawing, he had said ‘Here's a wheel Here’s another wheel... That's a funny wheel... This isa car.’ For him the criter feature of car was its ‘wheel-ness'; it had (many) wheels. Two steps are involved! ‘the making of this sign. At the first step, ‘wheels’, the signified, are represen by circles, as apt signifiers. At the second step, the signified ‘car’ Is represented by \w ‘apt signifier of “(arrangement of) seven circles’. To represent wheels by circles rests as with the replacement ofthe chess pieces (‘a black button is like a black piece’) — 0 the principle of analogy: ihe moment of the making of the sign, representation is always partial; yet it Always Mull’, ‘complete’. It is partial in relation to the abject or phenomenon full in relation to the sign-maker's interest at the moment of making ln. That isthe case with this ‘car’ as much as withthe representation of any car ¥ avhertisement. Interest produces attention. Attention frames the world to be quslales THETEST A selects'what is to be represented as the Lbasis of the relation of analogy. The outcome of that prOCESSTS a metaphor. Step one : aphors, always TERY made, resting on, maigrlallzing and Giplaying analogy —> ‘circles are rourd; whee's are round; circles are lke wheels; Stays TenIT made, resting on, matealzing and Spaying Step twor jnlation =the mearing that 1 wish to realize, to make material ~is not analogy —» a car has many wheels; many wheels are like a car’, “inne race ator tach errant or Seheeanwa Gar. Thetorpleesioy car rade ferent probate waldng two slg cool, a dub proses of analy, eli in two metaphors rl are le hea ar ny hess ea car's anally a oe meta (the complex ua si ea car Tse how or why wheels could be the citer feature for cr, we have oa thepoint of view, eral phyilonely, pachoogally, eaturally sent the tree eared. Trwe lage en lack ate faye aa 82 YW Ga it Oneal Sis uPeh, epee sk TERRE AC coreegeanalel wart Eovelude thet this sign'maker’s poston inthe worl, Iteraly,phystealy, But = psreialy,atfecively, ight welled im ose ‘ear In tht way. Hs tet a ‘out of his (physical, affective, cultural, social) posit in the world at that monien!, ny THE Dro ore as Aierey Tose RpTeEaton ToS On Ty Mt) communication focuses on the assumed interest ofthe TIDE OFTE ig, ‘ign. That leads to the demand for transparency in communication. fildr an example, wellenwn to mast of us We are In-a plane, jst before Hof, The cabin staff are going through the safety instructions, and are about to ie ereroency ets. Sxtcen ote rane afte plan, alg he fpf the cockpit, that with her bark to the passengers, an atenant Bors o ve spokevanncurcement: "vo exis ate ony ave el ore rah” alses cr ie arm ard gestures nthe drection of the lefhond ext she rales Berman geaures the rection of ie eah-tand ex er gestures are fis the raising of arm ard gesturing with er hand reap sinters let arm Hl exit, right arm to right exit. Spoken and gestural signs are in agreement. Both Berar what sto be represmcd, as announcer cots wi “bey ‘foar, one left, one right’, the attendant has turned around and now faces the yigers. On ‘one left” she raises her right arm and gestures with her right hand ‘the exit which, for the passengers, is on the left-hand side; on ‘one to the right’ as tr et ar ae ester with er fe han to the exit othe right of Mibsanges She docs the sare to "end four es over te wings tw Kf oo Ih anata Ue wave othe hands asi lapping’ ings, Now Mba contrat orc hr rte tee aod conor tr are area ew Tat TaTerest at the moment of sign-making arises out of the position in the world; it shapes attention; that frames a part of the world and acts 0 a principle for the selection of apt signitiers. Clearly, the chilo’s interest I: tyere is more to a car than wheels, even for # three-year-old. Theoretically we have a choice: we can treat this as an instance f TE Tepresentation and dismiss it; or we ean take it as a contral feature al representation in all instances. That is the route | have taken: all representation spoken announcement: they are no longer rapresentatianally apt: ‘ore left” is always partial, What the sign-maker takes as criteria! determines what she or lv | lalied with her right arm and hand and Yone right’ with her left arm and hand. In wl repre bo tha ety. An dl cho ae mare shaped by history ‘worth, in this environment the demands of communication, the overriding need, ‘experiences in various sacial envirSimments; and adults have greater awareness of uni) WW adopt the passengers’ viewpoint, which makes her (complex) signs internally lure, The princi olf itfudictory. The interests of the pwssengers have overridden her interests; or, ‘access to the resources for representation available in ti ¢ n +72 Multimodeality A social-semiotic theory of multimodality 73 better, she has factored the interests of her audience into the complex sign that shiv hhas made. The signs are communicationaily apt. The greater demands of the com ‘municational partner override the representational needs of the sign-maker for th aptness of realization of her meanings. 00, class, gender, profession, lifestyle, have their specific and distinct semiotic Ee, differently arranged and valued, That reinforces the point I have just mad iy the need to shape my signin relation to the person or group for whom I have i the sign and on the effect which that is Vkely to have. As societies and as ‘This may seem too obviously necessary to be useful or persuasive as an examp lions of communication become ever more diverse, that is an increasingly lent factor, an area of work which Intercultural Communication has dealt with for the theoretical point here: yet in many situations contradiction is exactly whal hhappens; often it is the normal condition oF sign-making. The sign 1s Contraqiexon, ite some time (Scollon and Scollon, 2000). from Ter perspectNe as much as WONT the Perspective of (the luckily few) literal e-be expressed relationally, as position and direction, charac minded passengers. It is also the case that most passengers would not notice ti | Power is constantly contested, with v Instance ih Subversive genres such as allegory, joke, aware of the degrees of superior contradiction. Most passengers (like mo3t of us at most Times) assume that the wor'd werted to limit such expression (Bakhtin, 1986). Even i Tistances or periods of sign-making is ordered by our interests, or at least on this occasion; so no I pawer difference ~ or maybe particularly then ~ such challenges exist. contradiction is noted. On this occasion it is the passengers who count; and so their interest prevail ‘urrent fundamental challenges to and shifts in social power in nearly all lis of Western’ sites have had and are having semiotic effects. Equally, there od Peoreicaly i means tat af Sin-making has 1 be Founded on a careful assess " of the social environment and the relations of power 1 ‘eavironment. That musi ieee te pons ferro Fe Uegone oration of sign potentials of semiotic power as means of challenge to social power. Power and making is that others’ sign-making is ordered to fit the interests oF the powerful jets were the core of Critical Linguistics (Kress and Hodge, 1979; Fowler rer ste erties AMAL Ir raSparery FAO Te GreTion of opal 1979; Wodge ard KresS-T98B7 Hodge ar Kress, 1993), parts of which ater into the larger and theoretically more diffuse project of Critical Discours asses as ‘politeness’ (in anglophone cultures) is evidence of 1! skewing effects of power in communication, The more powerful the maker of the sion, tess, 1984/89; Kress, 1991; Fairclough, 1989). It remains one of the ‘Tecriore she or he can ignore te requirements cf ransparency = hati, attention io I inerests in communication, That includes (Foucault's notion of power from ye te cw ena a ls, THE UTS Tv to dhe Se (Foucault, 1982) or power exerisodimarshaled(seemingh) barony \ vfork that makes up for the neglect of the privilege¢———__—— this affects the neteBor of postion; n each istance-thase who exert power nf up, even if only fr a moment of ra istace” oF ‘All signs a ‘these tno perspectives and interests: mine in relation toy representation and interests; and yours in relation to commun"eaffon and to the nev! wailing action, Refusal to acknowledge the demands of power is for factoring Tm your Titerest and the requirements of power. While power introduc Jit.as refusal to act is to exercise power. “cpacity into the world of signs, it does not disturb the principles of the motivate! ‘we all familiar with examples Ir Tanguage. In their landmark article ‘The Sign. A common assumption in debates around communication is that ‘good communi (9 of power and solidarity’, Brown and Gilman (1966) drew attention to a cation’ is “transparent communication’, that Is, communication where there are 110 ‘nl semiotic change in seventeenth-century England, when the second person ‘obstacles to an understanding between the person or institution who shapes tly; pronoun you (the vous of French, the Sie of German) changed from being message and the individual or group who are its intended audience. The idea ol foun of power and became the pronoun of address for everyone. Such social *aptness’ works in that direction: the maker of the sign attemy i the material ‘Winlotic change produces a far-reaching unsettling of social arrangements; sTgnifier which is best fitted to ‘realize’ the (meaning of the) signified, However, (nt lv te expression of power and of power itself but it does not — and did not in raker’s Tnterest cannot ever be known To the reciprent. But ifthe recipient can safely ‘(hi ~ abolish it. New forms for expressing power needed to be and were found. assume that the relation of aptness between form and meaning will hold, then sl I | can address my boss as you and by his first name (students in higher or he can form a useful hypothesis about the signified and the interest of the maker of on in the UK tend to address their teachers by their first names) but I do not the sign an the basis of the ‘shape’ of the signifier. The recipient may not know abo W ny boss with an imperative ~ e.g. “Shut that window, Bil’; nor, usually, would any limitations in the availability of signifier-recources ta the maker of 1 tole te ‘An example is my knowledge of French: anyone reading something written by 10 1 Ie case For speech isthe case for all modes. Speech provides means fo in French would be helped by knowing about the limitations of my resources in tlt TsocTal other In interaction, but so do gesture, gaze, proximity and } language. TH nat Uifcult to test out How, with each ol these modes can Indicate A usual situation is eke Lo power oF my amelioration of power in cornplex signs. Averting my 4a: groups? — co HAW talking here about ‘my’ social group ~ these signifiers and Signs are highly creed 77 winication acrass boundar whe by virtue of factors such74 Multimodaity A social-semiotic theory of multimodality 75 ‘When 1 got to the museum it looked bigger than I thought. When I went in I took “lf ny coat and went into the men’s toilet and after [ ran upstairs I went into the ‘ft, Then I went to see the mummies and all those stuff. Then we went to our ‘lakroom so we can get our coat and then we went to Waterloo station on a tube WM! train to Clapham Junction and walked back to school and went home very ‘nny and I told my mum, sister and my brother. The end. sensitive to social difference) has a diferent effect to fixing someone with my and heldina it, Where modes are Involved which do not afford the ‘address’ of ao: ier but tic nevis Inet veneesept social rai, stn eX of foros and cf emooratian cans produced ina solomag te ete pre {high to tow; low to hah canbe varied Infinitely anced gradations, In wate the ening hows It de possi to make reasonable Inferences about the sate oft “British economy based onthe poston from which Ue an oT ENG bul ‘TS Shown. Distante-tar be mantpotaret ira shor ih Him or in a photo; as it ca tHephysieal avangements of enture na place of wark xin seating errengemen ata dinner table In all ese, power relator can be represanted to indete det ence or amelioration. Soft focus or its opposite in photographic image; choice of w: from a formal or a familiar register; the direct expression of a request or indirect yo the choice of medium — not text-message for those to whom T defer but a more Fors o tien recount and the drawing are both recollections of the visit, set down on the 19 day. The ‘story’ of the visit conforms both to the genre of recount and to the ie) logic of speech ~ a mode familiar and congenial to a six-year-old — here Kn its transcription into ‘writing’: a chronologically arranged sequence of ‘and events. The depiction of the day in the drawing is entirely diferent. It too Ws to the semiotic logic of its mode: the simultaneous presence of entities, 1 spatial relations within a framed space. The logic of speech seems to suggest, tion: ‘What were the salient events and actions and in what order did they I, The logic of image seems to suggest a different question: "What were the i eniitos in the visual recollection of the day and in what relation to each other stand?” leotard nay ara ae eating oa od ate snag poner ARE pos sition ih revtably ei assh nears prove SoS UN Ts environment point to The power of a participant, then her or his use of si of the diminution of power — in ‘politeness’ for instance ~ can be read as such anc ni as a sign of lesser power. This complex relation of signs (and ensembles of s yi ~~" of often quite different kind) Tg encapsulated In Te Totion of the logontomic syn) J (Hodge and Kress, 1988), itselt a complex of signs Wat GWeS readers DEE reading, ‘navigating’, of ‘placing” te Interrelations and valuations of signs in sa } complexes: means for reading comPadiction, tension, oppositfon and apparent “Mid Tacticlty, and so on. Th that context, contradiction oF One STG WET others W 1 SIgr-complex Ts in no way dysfunctional but supplies essential infarmation for 3 accurate reading of the sacial/communicative environment, which includes the nay of the sign. Makers of signs, no matter their age, live in a world shaped by the histories of th work of their societies; the results of that work are available to them as the resow of their culture, Inevitably, what has been and what is around” and available, |i shaped and does shape the interest and the attention of the maker of the sign. Ivo year-olds can pick up pens and make marks on paper because pen and paper <6 there. They can (still — these materials may be passing) shape objects with playdov yh make shapes with scissors and paper. Culturally provided material resources ‘we their effect in shaping and directing, inthe ‘channelling’ of interest A request by a primary school teacher to write a story about a visit toa muse suggests specific possibilities to the child writer, different to those of a request 10 ray an image of that same visit. The world projected in the written recount of the visit. ‘the world projected in the visual account (see Figure 4.3) are likely to be profount, different. Here is the written ‘story’ Piquer 4.4. Chile's drawing of the visit to the British Museum Hei thanks to Eve Bearne)76 Multimodality A social-semiotic theory of multimodality 77 ough the modes of image, writing, colour, but also in actual or imagined ‘inner’ Aiesis through touch and feel, scent and smell, in action — imagined or real. The feel the plastic container; its texture; the shape of the ‘bottle’; the action of pouring or ruse suggested by the shape; its imagined and actual fit into the hand; the scent the lid is undone; all engage more of our body in their materiality than sparser jllons of representation’ might usually suggest. Jn the engagement with any sign, the materiality of modes ~ where sign and mods stood broadly ~ Tera res ATT Signs, whether that I make in my actions, or remake in my inner transformative and/or trans- embodied, for maker and remaker alike. In 3 Way The question about the image seems to be answered as I was really impressed ly the facade ofthe museum, with those huge columns; and by the marmies which Ts: in the museum, The two mummies were a bit different from each other, but they Ww: equally interesting, and I drew them in front ofthe museum in which {saw them Uf this, in its brief Form, a reasonable account, we might infer that the tw moc at image and speech there as written transcription) organize representation di ently, that each poses specific Kinds of questions’. Putting it ore stronaly, we mish say that the request ta make a representation of the day's experiance in two mos Itads fo sign-complexes which order tre recaTectTon of The Way Tine with tv affordances Cand the logics) of Te Two TOTES WE MTG GOA ep FATT FEAT “whether a habituation to representing in one made or the other could come to orga iz and shape our engagement wit the world prior even to any request from outsi: 0 nee to make a representation aterestYs decisive because it Forms the basis of the choices of what I take criteria, the availability of modes, their materiality and Tet artorganees_IT entas ARE TTS TT ‘a can be or what fs more Hikely to becors real When fre ois oe ems rush vn than whe hear thro the auditory apparatus). The mades and the kinds oF ‘ade of ede in a society bring with them certain orientations, a certain ake” 0 OAR By and large, Tat Take” becornes Trwksole” (audible, intangible) — Tn ie arse of explicit awareness to those who do take” it. Cultural an social habit, to modes, genres and oractices shares hour we represen tn time That habituatio SFgage with the world which we then represent, OTH Ue broad Tange oF r Te individual's decision to make Choi To_use fhese modes rather Wan frase TVs environment Tor these reasons processes named as affect ard cognition coincide absolutely a5 ONF lly effect. In this way too, identity is embodied and becomes more than a merely pmenon, an ‘attitude’, or perform, 7 that children diic many of the physical attributes, practices, habits — facial expressions, gestures, ios of walking, other forms of bodligexsy- of thelr parents or of others close to 1) At another fever TOTTWITCH TOrCES us to reconsider, profoundly, the Miration of categories such as mind and body, of cognition and affect. From an analytic perspective of a social-semiotic theory of sign- and meaning Jing, the following might be said. [he sign which the sign-maker has made gives us of the worid that Isat issue; am indication of their expe TAT Tees WME WOT, Signs are shaped by That and give usa sense of the ia, the principles, the interest, which ted to that representation. In doing that five us an insight into the subjectivity ofthe sign-maker. Taking this stance forces me to add another element to my example of the chess aps, There 1 attempted to show the motivated relation between form and meaning. Ht iy alscussion there Lt ot constr the- materiality of the form ofthe sign- ful it simply: my characterization of playing chess was—as T were aisembodTed, ed mentaistic account. Chess can be (and at times is) played like that, as ital chesd’; it is then regarded 45 a particularly difficult achievement. In most. of playing, the pieces are there in fully three-cimensional extension and for Mimesis, signs and embodied experience nthe reception of a sign the materiality of modes interacts with the physicloy) of bodies. (When t-seeargesture i understand itn large part in an action of ‘silent Sar mineds (Wal, 2005¥-Leame to unerstand te meaning say the eT th Sqeay OT RTNOTUVERER or pace ENT DOA aT EARS REMC Ua gre MONET Nae Trae tee mganing. There are, as well, the meanings suggested by signs in Uhe accompany ood reason: they have a material wali they have an aesthetic effect; they erlronment, Stans made in gesture are culturally shaped as are all signs in any 0 tellectual activity, At that level it comes to matter y shares thep ly iF it happens on foo miany occasions ~ that the button-pawn lies flat on the J ary sians we encounter ae in tiree-dimesional form; or they are in time; ae BBB Ts awkword to pick up, and takes away even slightly from my pleasure in signs we make, for instance, via the senses of smell oF tou. Heve too the conc «l Ml Mlayng the game. My pleasure lessens, Something i different, missing, The material mimesis is crucial. We engage vith the objects represented in the images in Colout | MMs pee — carved or cast —has a tangibility and an aesthetic which hasan effect om plates 7 and & ~ when we encounter their i a supermarket oF at home ~ not only fe AMTarialy, a a body in The wor miboaied a rraerTatity that18 Muitimadality ‘adds to my pleasure of playing the game. I might much prefer to play with pieces wi a certain dimensionality, weight, texture, aesthetic and which for reason of tho qualities give me pleasure. That sensory, affective and aesthetic dimension is tooo ignored and treated as ancillary. I reality, tts tsSOTUBIy part of semiosTs Boor vr Colour plate 2 CBBC homepage 2005 (Pages 37Colour plate 3 Salt and pepper, SAS: ‘health’ (Page 67)Colour plate 11 Hedgehog, Em Colour plate 12 ‘My Birthday’: affect (Page 109) Colour plate 13 Lego car (Page 120) Colour plate 14 Outside the roam: ‘Armenian wedding (Page 149) (Page 109) lour plate 15 Inside the room: “Armenian wedding Page 14! 5 Mode Moteriality and affordance: the social making of mode fis ssi shaped and cutraly svn sri resource for making meaning Inaye, writing, layout, rusk, gesture, speech, moving Tage, soundérack and 3D Dbjgcts are examples of modes used in representation and communication. Phenomena iW objects which are the product of social work have meaning in their cultural ronments: furniture, clothing, food, ‘have’ meaning, due to their social making, The purposes b¢+thetrmmaking and the regularity of their use Tn social life. As their_ Bary srtion sot ta of ereetton and conmnniaton theres a avesion lier They should be considered as mode=— ever thougtr We RTIOW Tat Mey can Be W#RTo make meaning and to communTeate.— The introduction ofthe concepts of mde and multimodality produces a challenge to Vitherto settled notions of language. Ate al, if all modes are used to make meaning It powes the question whether some ofthese meanings are merely a kind of duplication Hf wwanings already made in, say, speech or writing ~ maybe for relatively marginal Feasons such as ‘illustration’ or for aesthetic reasons such as ‘ornamentation’ — or Wether they are “full” meanings, always quite distinct from other modes. If the latter Ih the case, then *language’ has to be seen in a new light: no longer as central and Homhinant, fully capable of expressing all meanings, but as one means amang others fir makina meaning, each of them specific. That amounts to a profound reorientation, Wis the route taken in social semiotic approaches to multimodal representation. Different modes offer different potentials for making meaning. These differing pileitials have a fundamental effect on the choice(s) of mode in specific instances of Fiimurication, Writing Gn English, as Tm mary atfer Tanguages) has words, clauses, PONENCES, Organized through grammar and syntax. It has graphic resources such as Hoi size, bolding, spacing, frames, colour. To frame its units, it has syntactic, textual Wil social-semiotic resources (e.0, sentence, paragraph, textual block, genre). In Witing, the trames use graphic resources such as punctuation marks, visual means ‘ und paragraphs and increasingly, ‘blocks’ of Writing, often in diferent colours, on surfaces such as pages or screens or others. rent cultures. Crucially, cultures may use Aforont script systems. That makes it problemiatc to speak of writing as such; instead iced 1a say. iting in his culture ot Lat, with an alphabetic or a character-based wernt, What Was spice hetwren words or Those resources have specific forms in dif jlies inthis respect to the mode of writing applies to all modes, +30. Multimodality The mode of sp and grammar with writing, Th ? materia SUD speech however, sound, is enticey diferent from the graphics! aT wetting Sous received via the physiology of hearing; the araphie SUF oF writs CT ae ysology oF sight. Sound offers reources such as (variation in) reray —TOMIMESS OF SOTTESS— AHTEh cam be Used to produce alternations oF stress! “ecunsresec elements, oF rhythm and accent, wMich produce the rhythmic orga: “eal of spech andthe acentuatln oT WOE PIED A PEN Varo” TWAT i" ‘OF We Trequency of oscillation of the vocal” chords) prow re TanGUaG ‘ever a Marder or Too) and lneanattor Ty arguages Such ae ERTS Sew vowel quality, lenath and ues Sega ees one-or Te ng of sw. Suits, English for instance, the contours of lntoration are used ta make and frame Titration AigisTTRSEMATETIaT entities are used to mark out given and new VaTormation ineiniotic entities, information units. (Halliday, 1967). If say It was last Saturday he came! with high intonation on ‘last’ and falling away (i.e. contrasting with ‘rol Saturday two weeks ago"), the informational meaning is different to “it was ls Saturday he came’ (ie. contrasting with ‘not last Sunday’, These units, held together in larger intonational frames, produce chunkings ol rmeaning similar in some vays to the (written) paragraph and yet also quite ditt in their potentials for meaning, Sound happens in time and that allows the voi'v to ‘sustain’ a sound, to ‘stretch’ it as a resource for meafing — as in the lengthen) ES Ee TT a ape come Fer SRM, FEES Writing uses graphic means — bolding, size, spacing — (0 ) elev selec eters wilh ary only sue rene SuTree re ! SHUT SBBECT. BOTEAT and Toudness” differ materially; yet insome cultures ty are Tnfegrated Tito and ‘belong’ to one more general semiotic category of intens! Bolding in writing and “loudness in speech are both signifiers of intensity and are 0 ‘means of realizing meanings (signifieds) of ‘emphasis’. Ina social-semiatic approach to mode, equal ernphasis is placed on the afforda of the material ‘stuf’ of the mode (sound, movement, light and tracings on surfac etc.) and on the work done in social life with that material over often very low) periods. The distinct material of sound (in the case of speech) and of graphic stuf (i the case of writing) is constantly shaped and reshaped in everyday social lives, in \\w ‘most banal as in the most extraordinary circumstances. Banality and exceptionally ales ensure that ale sala if is captre in that shaping, Social —and there rk. with the same! material, sound can ead very different mod: of course, to soundtrack as 77 Im Janquages; to so-called whistle languages. These rest on different use of the affor ances of Uie same material fealures: variations ia_eneray_(laudness/sofin Teresi itch (igh ptivow pic, ising and fli oie “changes in dura) TEMES Tong vowelsehor VOWETST pauses and silence, Social work performed ceaselessly by members of social groups with ty affordances of the material, together produce semiotic resources. That is, mod the product jointly of the potentials inherent in the material and of a culture Hip tts underiying-sesniotierfogic™ The Togies’ of space and tire are profoundly fet and offer dane - - rhe modes. esture or movly i space. In gesture uence in time through mevement oF arms and ‘Mode lection from the bundle of aspects of these potentials and the shaning over time by jvembers of a society of the features selected. Hence the resources of modes, say e oF Speeci, are both similar to and different from each other within one society: weds and demands of its members, yet different in that the materiality of different Modes offers different resources and potentials for social shaping. That difference ‘MW modal affordance also leads to differential use and hence to a further cause for wons: the different requirements of different societies and their members and the Terman) differences in the-custural-use-of say, vocal. Wh as English, Frency, ocalic quality, and so on, Be rareeetevateian a nan ma ts Wariner Conmoraltes. Speech havpes tne. In speech one sound, one rd, one clause follows another, so ei isane Tundamental organ- ina orrchie raTaeIgearng in tis ode. THs shore across ‘are Simultaneously present; the arrangement of TT That space in relation to each other is one fundamentally organizing ent elements Th that (usually Wamed) space igo The Togic OP SEqUENTE Th Time is provided by the succession Of frames of images, ptt otet pwn ormOREDTBEE Hersey {W alphabetically written languages, wrlting SOTMeWaT OF border category in Breer 1s satially islaved yeti lane on speech ins lose scans fo ged i ig pn sce a ners Bs ipl systems which use the Tine Lan which anne olsplayes THis spatial display H writing and of its elements on the line, its ‘linearity’, gives to the impression Hat it works like an image. However, the elements of an image can (usually be ‘reac! Wan order shaped by the interest of the ‘viewer’, while the reading of writing Is overnied by the ordering of syntax ard the directionality left to right or right to left ‘the lie, Writing & not, dominantly and finally, organized by the logic of space; fs readers we are bound both by the orderings of syntax and the directionality of the Hine. Culturally, both speect and writing share ~ even though insignificantly dtterent al snsequent different shaping. As a semiotic resource, image in one culture is i identical to image in another: aly related cultures and ‘languages! or of pitch 2 #2 i Muttimodality ways ~ the organizing features of syrtax and the resource of lexis. Superimposed 0» that are the stint different material features of sound and graphic “tut, al socially shapes Looked atin this way, there are significant ferences between speech and itn ‘al times maybe more significant than Te stmitarities—This Makes it surprising thet Peony a soTa-senotc perspective We use ofthe oe label obscures Westnet as modes wit related yet importantly distinc aFfordences. “The resources of the move of image differ from these of ether speech or writin Image does not “have’ words, nor sounds organized as phonology, nor the syntax and grammar of speech or writing, nor ary oftheir enitiesnits. There fs no poi searching forsylables, morahemes, words, sentences clauses or anyother languas based category in image. While spech i based onthe logic of tie, (stil) ima is based on the logic of space. It uses the affordarces ofthe surface of a (frames) spacer whether page or camvas, apiece of wall or the hack or front ofa T-shirt. in mage, meaning Is made by the postioning of elements in that space; but aso by Size, colour, ine and shape Image does nt ‘nave’ words; fuses “depictions. Wor Gan be ‘spoken’ or "write, images are splayed’ Image uses depition’, cons 0 various kinds ~ circles, squares, triangles for instance. Meaning relations are esta! lished by the spatial arrangement of entities in a framed space and the kinds of relation between te depicted entities Given the distinct afordances of different modes they can be use to do spec semiotic work. The uses of mode constantly reshapes arGant2s along te nsf he Socal Trements of those who make meanings; that ensures that mode i constantly changed tn the direction of social practices and requirements. Mod Shang vats cl hag, We ci et do et ati ner elaborated inthe enities of ammode, Ara COnSaqUENE, Te FOTeKials inherent materiality are never Tony Uses to become affordances of a mode in a particular cultore; nor ae all the affordancs which are available used for similar purpo across diferent cultures. My mention above of pitch may serve as the example. tove-languages pitch i used — among other things - fr lexical purposes: differen in itch withthe same syllabic Cor mult-sylabi) form protuces diferent words. In English pitchmoverent is used for arammatical purposes: for forming question ov statements Pitch is used for lexical effet in English too, but to a much smal extent: ry saying "es" to mean “Are you realy sure? or Maybe’ ora whole range ster meanings. Societies and ther cultures select ‘waterials’~ sound, clay, movement (of pats) of the body, surfaces, woad, stone which eer sel or ecessary for meaning 0" Inthat eure te be done. Selections irom the potentials for making meaning wich these materia offer, are constantly made in the socal shaping of maces. 1 communities of humais wh are speechimpaited the afordances ofthe body ~ th positioning end the movement of limbs of facial expression ~ are developed ito Mode lable, only a narrow selection of these affordances is used, in gesture, Different “sities have selected and continue to select diferenty, shaping different cutural/ iviotic resources of mode. The focus on materiality marks two decisive moves: one is the move away from shag language’, the Tguste system’, grammar” and a rove towards he sneciticity of a mode and patertals as Geveloped in social uses. The othe Wir makes Tope cation With the bodiliness mans: nat only in the physiology of sound af hearing, oF Sight and see also nthe fact that humans make "0% af taste and tasting, Bi an [itirough all these means are the tact Twat all These are Inked ane Take MEANS gether Br ITY OFFerS Me pOSSIIMy OF Seer MEATS modied — as in our bodies: a means of getting beyond separations of those other stractions, mind and body, of affect and cognition, ‘The ‘reach’ of modes Biroons> rage withthe world through these socially made and eutturally spec Greer and Grey Gipson ways that arse ut of Ter Uterets. Tis gives rise to the ‘Well enough understood problems of (spoken or written) translation: certain cultural pains may be well supoled wih syrtactic and lexical resources in oe cuture but Bry suplied a arother, or cse adorn ay BeeTrcy ia as Set prod 4 parte varia oF WepIHler” compared tothe world that might ‘We hamed or depicted, The semiotic ‘reach’ of modes — what is ‘Covered™ by Uitte or speech or wrltng or pestare=Teatways specie ana paral Wa clare Nah aifereny spear paral This partiality Ut raming-or-depetion-of labelling, is a feature of all modes and all cultures: there are always gaps. Aveas in the Bip sclal attention are wll suplied wit sent resoures; others lax su'or ott ai Becitics have modal preferences: this mode is used for these purposes, that other ‘Wide for those other purposes. Over long periods, ‘Western’ societies have preferred Writing to image for most areas af formal public communication. Hence there is a UWileential erst" of semiotic entities, of naming and offers of relations even Within one society. If a mode is preferred for a specific social domain it will become ‘Wore specialized in that respect. Not all cultural domains either can be or need to be ‘Wieribed and elaborated equally in each mode, The material affordances of modes By io: the en” of a gesture nity variable expres fc ture fords infty gragabe signifies. Gestures areimperanent once made, pete trace roman ot them Gesere ning WareTaes Be Sow Ra OS THREATS War the reach’ of modes varies from culture to culture. What may be se by speech in one carte may DE Tanaker by gesture in another; what Way be-weil a3 U € V J done fi Tagen one culture may be better done WT 3 Torms Tr another, and so on. We anol assume that teanslation from one niode to that (same) mode acrass cultares— ’