See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/332991173

Social Semiotics

Article · May 2019

DOI: 10.1002/9781118978238.ieml0226

CITATIONS READS

4 2,132

2 authors, including:

Clarice Gualberto
Federal University of Minas Gerais
39 PUBLICATIONS   52 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Multimodalidade e Semiótica Social View project

panmemic View project

All content following this page was uploaded by Clarice Gualberto on 02 July 2019.

The user has requested enhancement of the downloaded file.

 Social Semiotics
Chapter for The International Encyclopedia of Media Literacy

Authors:

Dr Clarice Gualberto, Universidade Federal de Minas Gerais (Belo Horizonte, Brazil) clagualberto@gmail.com
Dr Gunther Kress, University College London (London, UK) g.kress@ucl.ac.uk

Word Count: 4,111 [Includes Main Text, SEE ALSO: (cross-references), References, and Further Readings. Total word
count does not include title, abstract, contributor bios, or keywords.]

Abstract

This entry provides a brief overview of social semiotics. It begins with a brief historical account, presents some major
texts and mentions some research which has contributed to the development of the theory. Key concepts are introduced,
such as mode, interest, sign, design, semiotic work, text and meaning making. The issues of literacy and education are
discussed under a social semiotic perspective, showing how these fields are related to each other. An example from
teaching-learning practice is used to exemplify some of the concepts.

Keywords

social semiotics; design; mode; interest; meaning; learning.

This is the author accepted version, to be cited as:

Gualberto, C. & Kress, G. (forthcoming, 2018) ‘Social Semiotics’, chapter in the International Encyclopedia of

Media Literacy, edited by Renee Hobbs and Paul Mihailidis, NY: Wiley-Blackwell.

Social semiotics: A brief historical overview

Social semiotics is a social theory about meaning and meaning-making in (inter-) action; it examines the varieties of ways
texts can be made. The word “semiotics” derives from the Greek “semeion” – “sign” – the smallest unit of meaning as a
combination of form and meaning. In and through signs made in different modes, meaning becomes evident – visible,
tangible, audible: available for semiotic work and sensory engagement.

Considering language as a system of signs, Michael Halliday (1978) in “Language as Social Semiotic: The Social
Interpretation of Language and Meaning”, argues that language must be interpreted “within a sociocultural context” (p.2).
Strongly influenced by the semiotic thinking of the linguist Halliday’s work, as well as by the theories of Marx and
Freud, in “Language as Ideology” (1979) Robert Hodge and Gunther Kress provided their “account of language as a
social phenomenon” by developing tools for the analysis of the interrelation of power and language (Hodge & Kress,
1988, vii).

This perspective gave rise to “Critical Linguistics”, later providing a major input into the development of “Critical
Discourse Analysis”. Hodge and Kress realized that language was only a part of a much wider range of cultural resources
for making meaning. That assumption became one of the main premises of “Social Semiotics” (Hodge & Kress, 1988).
This book, followed by others, has been the basis of ongoing research, using and developing the theory in a variety of
domains (see e.g. Kress, 1997, 2003, 2010; Bezemer & Kress, 2016; and, in a distinct vein, van Leeuwen, 2005). A
significant part of current work in Multimodality derives from its insights.

Another example of a study derived from social semiotics is the work of Gunther Kress and Theo van Leeuwen
(1996, 2006) in “Reading Images: The Grammar of Visual Design”. There, the authors present a framework to analyze
images with categories inspired by Halliday’s Functional Grammar (1985). It is important to highlight that Multimodality
is not a theory; it is an outcome of the Grammar of Visual Design. Both are perspectives / frameworks which emerged
from social semiotics.

Social semiotics: key concepts

In social semiotics, texts are seen as multimodal complexes: they are constituted by more than one mode. Writing,
speech, gesture, music, dance, layout, are examples of modes, the material resources for making meaning evident. Modes
are the material resources for the production and materialization of signs. The distinctness of modes is the effect, jointly,
of the (usually) different materiality of modes and of the distinctive social shaping of each mode, over often long periods
of time, in constant social (inter-) action. This ceaseless shaping has produced and continues to produce resources whose
regularities are recognizable and understood by the members of a social group. Such ceaseless shaping constitutes the
history as well as the present of the development of modes.

Under a social semiotic perspective, meaning is the outcome of a response to the prompt of a social agent (the
rhetor) whose meanings are given semiotic articulation by a designer, and made material by a producer. Most frequently
these three roles / positions are merged in one person. The rhetor / designer / producer orchestrates modes and other non-
material semiotic resources to produce texts-as-prompts, motivated by the interest of the rhetor, in a given moment and
environment. When producing a text, the rhetor / designer carefully assesses the environment: “what is the world, for and
in which I am to design my prompt?” “what do I want to achieve?”; “what are the salient characteristics the audience?”;
and the resources: “what semiotic resources are available to me, and how will they serve my aims?”.

The last question leads to the notion of affordances. Semiotic modes and resources are apt for different purposes.
For instance, let us consider the popular game “Draw, speak, mime – Charades”, one player has to describe a given word
by either drawing, speaking or miming. The other participants have to guess the word. Marcela (a five-year-old girl)
thought that “tree” was easier to be drawn than described with spoken words or mimed. In a simple situation like this, the
child chose the mode of drawing, arguing that the members of her group would guess it faster than with her speech or
gestures. According to her interest, she believed that the mode of drawing was most apt to achieve her goal. That is, she
considered the potentials and limitations of the modes available to her (speech, drawn image and mime); Marcela
considered the modes’ affordances.

This assumption leads to a central premise of Social Semiotic theory: the prompt is a sign-complex. In a sign the
conjunction of meaning and form, of signifier and signified, is never arbitrary. It is motivated by the interest of the sign-
maker, who chooses a (material) signifier as an apt means for expressing the meanings of the signified. Signs are
motivated by the interest of their maker, bearing in mind what are seen as salient factors of the environment. The specific
combination of signifier / form and signified / meanings is due to the interest of the maker of the sign. As every occasion
of sign-making differs from every other, social semiotics asserts that signs are always newly made. In every sign-making
action, the already designed world is constantly re-designed.

In interaction and communication, signs and meaning-making happen twice. Signs are newly made by the initial
maker and by the person who engages with the sign, who, in their engagement and transformation / interpretation,
deriving from her or his interest and drawing on their semiotic resources, designs a new sign. In that process, modes are
constantly modified, and hence are the outcome jointly of the material affordances of modes and of the transformative
semiotic work of both the initial rhetor / designer and of the re-designer.

In social semiotics, a text is seen as a composition of interconnected signs: a sign-complex designed with signs in
different modes. The text (as-sign-complex) is produced to be cohesive and coherent internally, and to be coherent with
its environment. It gains its completeness from the occasions of its social use and making. A text is the designer’s apt
representation of her or his attempt to meet the rhetor’s interest.

Literacy through a Social Semiotic lens

In the perspective of social semiotics, it is essential to approach pedagogic issues such as ‘literacy’ in a much expanded
frame. Describing media literacy, in the “International Encyclopedia of Communication”, Livingstone & van der Graaf
(2010) state that it is important “to develop a subtle and detailed account of how people understand, create, trust, and
critically evaluate information and communication contents delivered on new platforms, and disseminated and regulated
in ways often unfamiliar or complex ways” (p.360).

Such a demand, with its comprehensive and expansive sense of “pedagogy”, fits entirely with the potentials and
the ethics of a social semiotic approach, in which every school’s curriculum has to be focused on preparing young people
for the demands of the new world of communication. Beyond the School, “the Media” in all their forms, are also
regarded as pedagogic institutions acting in the widest public domain. In that frame, every instance of sign-making is an
occasion of learning. Hence in this approach learning – in whatever social environment – is present everywhere,
constantly and ceaselessly. Learning has to be considered as an utterly usual, unavoidable phenomenon, of all everyday
and unremarkable or heightened and remarkable social action and interaction.

This has implications for the field of education, well beyond the walls of a school or other institution. In social
semiotics, (success in) communication is defined by interpretation, that is, as the process of engagement by a person with
a prompt, in which that person as interpreter selects from the prompt that to which he or she gives attention, regarding it
as salient, and interprets that in the light of her or his interests. All occasions of learning and teaching are regarded as
instances of communication; conversely every instance of social interaction is seen as an occasion of learning. In any
such instance it is the interpretative action of the learner, whether as student or as the man, woman or child in the street,
which is crucial, and which guarantees both the fact ofcommunication, its success, and its content. Each occasion is an
instance of the ubiquity of learning.

Hence, in social semiotics, the processes of learning and teaching are instances of communication; success in
communication is defined and guaranteed by interpretation. The outcome of communication is meaning-making, which is
taken as semiotic work both by the initial maker / designer of a prompt and by the re-designer of that prompt in making a
new sign (Kress, 2003).

The actions and interactions of social agents using semiotic tools (modes and other semiotic resources), constitute
semiotic work; it leads to the (re-) shaping of the material and theoretical semiotic resources in a community. Learning is
semiotic work, as the (work and the) outcome of the transformative engagement by an individual with an aspect of the
world which is the focus of her or his attention, with principles brought by her or him to that engagement.

This approach challenges main-stream assumptions about what is generally recognized or considered as learning,
whether in the hitherto narrower conception of learning in specialized institutions with formal curricula, or of learning in
every aspect of social action and interaction, with or without the intervention of the media. In school, for example, means
of making meaning and of learning therefore, are recognized according to strict canonical guidelines. The social category
of correctness dominates, in the form of different metrics of assessment established by institutional power. In a social
semiotic perspective by contrast, what is central are the principles of interpretation made evident in the learners’
interpretation. That demands profound changes regarding traditional assumptions about what constitutes knowing and
learning, and beyond that, agency, power and knowledge.

In a Social Semiotic approach to learning, canonicity is acknowledged as a social category, but it is not the
criterial issue in deciding whether there has been learning, nor as the basis of principles of (e)valuation: every sign is a
sign of learning. A social semiotic approach to meaning considers all means for making meaning; the task is to bring
them into visibility, giving them recognition. In doing so, such a perspective demands making what is “currently
unnoticeable noticeable, what is inaudible audible and what is invisible visible” (Bezemer & Kress, 2016, p.5). It is an
approach to meaning which goes well beyond mainstream notions of meaning. At the same time, it resists extending the
term ‘literacy’ to all means of making meaning, regarding that as a move which risks sacrificing descriptive and
analytical precision.

Learning is semiotic work. Work involves a worker, tools and something worked on. Work produces change; and
change is meaningful. Every use of semiotic tools – literacy included – involves the semiotic work of social agents; and
that produces change in meaning. Every time someone designs a message, assembling semiotic resources, she or he is
involved in semiotic work which produces a semiotic change, with social effects. The emphasis on work in social
semiotics, is at the same time an emphasis on the social agents who do the work.

There are two major processes of semiotic work and therefore of effects for socialchange: transformation and
transduction. In the former, the semiotic work changes existing arrangements of elements; it occurs within one mode. In
the second, “change occurs across modes, in a shift of semiotic material from one mode to another” (Bezemer & Kress,
2016, p.53). Modal affordances – the potentials a mode offers for representing the world – are not neutral with respect to
epistemology and ontology. Each mode imposes its epistemological and ontological commitment. Changes of
arrangements within one mode have epistemological but not usually ontological effects; changes across modes have both
epistemological and ontological effects. The effects of the two types of change are fundamentally different.

Applying the framework

Figure 1 reproduces part of an assignment, in which Brazilian students (14 - 15 years old) were told to take pictures with
their cellphones during their visit to the city of Mariana (one of the oldest cities in Brazil). The photos were intended as a
“visual report” of what they had learned on the tour and how they felt about it. Later, in the classroom, each student
presented a couple of PowerPoint slides to the class, with some photos; each talked about what these photos meant while
showing each photo.

[Figure 1a and 1b near here – Figure 1: Student’s visual report of Mariana (Brazil), 2011. Reproduced with
authorization of the teacher and the student]. Figures 1a and 1b, taken together with what the student, Paula, said in
presenting these to the class, allows us to exemplify several concepts so far discussed. There is, first of all, the notion of
transduction. Each of the two photos – Fig. 1 a and 1 b – and the spoken comments made while showing them to the
class, is a reflection of a part of what Paula saw and heard, felt and thought, while walking in the city, listening to the tour
guide. During the walk, Paula – as the learner who was engaging with the city environment – took photos and reflected
on what she saw and linked it with what she heard. In presenting her experiences later to the class, she distilled her
experiences largely in two modes – speech and image – as a complex multimodal sign, as a multimodal complex. Her
semiotic work was that of entexting her experience, using the modes of speech and of (still) image. These allowed her to
design a sign complex, as a text which produces some of the meanings she made about the visit.

In presenting her slides using the platform of PowerPoint, Paula used the mode of speech to talk about aspects of
the Roman Catholic Church that she found remarkable. She used the mode of writing to add some captions to the photos
– now integrated with the images on the PowerPoint – she had taken, and she framed her presentation of the visit with her
speech.

In other words, her presentation was an instance of a multimodal complex, consisting of the modes of image, of
writing (as captions), and of speech for framing and elucidating, in an interaction of the platform of PowerPoint and her
face-to-face spoken presentation. Given the circumstances of the visit and of the presentation to the class – it was not an
occasion for formal research – we do not have a precise account of her speech. Hence what we are presenting here is a
report of what she said, as a kind of vignette.

With Figure 1a, Paula wanted to convey by showing, the dominating power of the institution of the Roman
Catholic Church. During the visit, the tour guide had explained that all churches in Mariana were intentionally built on
the highest places of the city, in order to suggest that they were “above all things and everyone”, and were closer to
heaven, to God.

Paula had wanted to represent her sense of herself as viewer as being below the church. She said that she
“literally” felt like that, because, in order to get the angle of the church in the photo, she actually had to lie down on the
road. ‘Vertical angle’ is a semiotic resource of the mode of image; here it was used as the signifier to express her
interests: a feeling of the overwhelming, crushing power of this institution.

The image of Figure 1b was taken inside a church. The wooden barrier shown in the photo had, in the past,
functioned to separate men and women. Only men were allowed to stand close to the priest, closer to God. Paula wanted
to emphasize this aspect through her photo. She explained that she intended to represent the barrier as having no end – to
achieve such an effect, she used the semiotic resource of horizontal angle of “engagement” (see Kress & van Leeuwen,
2006). In Paula’s opinion – stated in her presentation – such barriers between men and women still exists. Now, the
barrier is not concrete or made of wood; it is realized, now, through abstract and implicit ways. This reveals one of her
interests related to her design of the multimodal complex in the mode of speech and of (photographic) image: using the
semiotic affordances of horizontal and vertical angle.

The effects – and uses – of vertical angle are well known to western viewers of images. Its affordances, effects
and meaning are and remain entirely different to a spoken or written transduction. The horizontal angle, showing the
barrier extending into the infinity of the vanishing point, with no end in sight, exceeds, in its effects, the use of words,
spoken or written.

We can observe a significant aspect about affordance. Had Paula made her text using the mode of writing alone,
such impressions, meanings, and visual aspects would have to have been expressed using the resources of writing. The
effect would certainly have been different. She could have used words to describe – to mediate and to abstract – what she
thought of or felt about the visit in Mariana. The images however, provide a distinctly different potential and take;
accompanied by her talk – the mode of speech – provides a remarkable different effect in the multimodal complex. With
writing alone, the student would not have had the same resources available to design the text. Readers / viewers would
see it differently, making their interpretations, their various meanings with the written prompt produced by Paula. Of
course, viewers / listeners will make their interpretations of the multimodal complex of speech, image and writing: but
they will be different.

Here, on the page, the affordance of image and speech, combined with the mode of writing, together provide a
deeply different meaning. Placing the mode of image as the focal, central feature, orients the person who engages with
the complex in a distinct way. The example shows how each mode’s affordances differ, and how therhetor / designer can
and does shape them aptly for her or his specific purposes, according to their interest. Each mode has specific
affordances; and in multimodal complexes modes can be used to complement or supplement or strengthen the meaning
overall.

In designing the PowerPoint presentation, Paula performed a semiotic and a social change. It was accomplished
through an intra-modal process – through transformation. Both slides and photos are images; so all changes she made to
transform photos into PowerPoint slides were realized within the same mode – image, though now with the addition of
the mode of layout. When the photos became a presentation to the class, they gained a different mean of visualization – a
larger screen. Both the use of PowerPoint and the use of the large screen were also changes of platform, which brought
with them a different set of social relations, and with that, different genres. The photos gained another audience – her
classmates and the teacher (instead of herself). The photos also gained a new appearance, a new framing in PowerPoint.
All these changes indicate that a new text was created, with different signs and hence different meanings.

In effect, the photo, casually though carefully and deliberately made, probably with the camera of the mobile
phone, and as such in the genre of memento, has now taken on the function of a meta-form: a means of reflection on a
larger social issue. The point here is that meta-forms are possible in each mode; and we need to begin to develop an
awareness of that potential in each mode, rather than falling back on each such occasion on the notion of meta-language
as the appropriate resource for reflection.

What is the range and domain of application of social semiotics?

social semiotics is a theory about communication. When someone is engaged in an act of communication, he/she acts as a
rhetor first, then as a designer, who, as producer is orchestrating semiotic resources to make meaning. Meaning is made in
communication as interaction, which is the foundation and basis of social life. Its essential effect is producing mutual,
reciprocal meaning-making in on-going semiosis. Meaning is made evident in many material ways, depending on the
demands and needs of particular societies and the availabilities of means for making meaning. That is why social
semiotics is ‘plural’, as an approach which deals with all means of making meaning in all their variety.

To return to the notion of literacy in the briefest way. One foundational assumption of social semiotics is work,
semiotic work, with semiotic resources, involving semiotic workers. All work, any work, produces change; all change is
meaningful. The kinds of semiotic work, and the kinds of tools and resources for making meaning suggested here – in the
briefest fashion – are many and they are hugely diverse. They range from the many modes – each distinct in affordance,
epistemologically and ontologically – to the sites in which semiotic work is done – social sites, material sites,
technological sites. These sites become platforms, in their social uses, through social work. That work is meaningful in its
distinct ways, given the potentials of the platforms. The meanings produced through semiotic work involving modes –
representation, and the work involving the shapingplatforms – dissemination, are of an entirely different kind, even
though they are everywhere connected.

To apply the term literacy to such diverse kinds of work is to forego precision, and to avoid producing essential
tools. In a world in which the means and sites for making meaning are proliferating at a hitherto unknown pace, the need
is for tools which offer precision. Social Semiotics aims to deal with that large environment of means and sites, of
resources and tools, as well as agency and social environments, in the terms of one coherent integrated social – semiotic
domain, with a promise of precision about tools and resources.

A social semiotic perspective requires that the issues of meaning-making be continually discussed, with
appropriate attention being given to the potentials – and limitations – of each mode, speech and writing included. Falling
back each time into the use of the “universal semiotic solvent” of language levels out essential differences in an
understanding of the affordances of all modes. Literacy, in its specific reference to writing and its capacities, will need to
continue being a focus of attention, because as the forms of multimodally constituted texts are changing and new modes
and ways of making meanings emerge, writing will change in ways that need to be fully understood.

As texts change, conceptions – imagined and real – of literacy have to change as will curricula and teaching
practices. Learners have to be encouraged to demonstrate their signs of learning in different modes, while being given a
full insight into those means of making meaning which are essential in their social groups. It will be essential to continue
to reflect carefully on the affordances of traditional forms of writing, still used in sites of power – economic, cultural,
political, and social. In those forms of writing are deeply valuable resources to retain. This requires a constant
(re)thinking by educators, theoreticians and intellectuals, so that the means for recognition of learning and knowledge can
remain useful in the near future and can be usefully expanded in step with social change.

SEE ALSO:
ieml0031

ieml0102

ieml0171

ieml0220

ieml0116

ieml0100

References

Bezemer, J., & Kress, G. (2016). Multimodality, Learning and Communication: A

Social Semiotic Frame.London, United Kingdom: Routledge.

Halliday, M.A.K. (1978). Language as Social Semiotic. London, United Kingdom:

Edward Arnold.

Halliday, M.A.K. (1985). An introduction to Functional Grammar. Baltimore,

MD: Edward Arnold.

Hodge, R., & Kress, G. (1979). Language as Ideology. London, United Kingdom:

Routledge & Kegan Paul.

Hodge, R., & Kress, G. (1988). Social Semiotics. Cambridge, United Kingdom:

Polity Press.

Kress, G. (1997). Before Writing: Rethinking the Paths to Literacy. London,

United Kingdom: Routledge.

Kress, G. (2003). Literacy in the New Media Age. New York, NY: Psychology

Press.

Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary

Communication. London, United Kingdom: Routledge.

Kress, G., & van Leeuwen, T. (2006). Reading Images: The Grammar of Visual

Design. London: Routledge.

Livingstone, S., & van der Graaf, S. (2010). Media Literacy. In: Donsbach, W.
(Ed.) The International Encyclopedia of Communication. Malden, MA: Wiley-Blackwell & International
Communication Association (ICA).

van Leeuwen, T. (2005). Introducing Social Semiotics. London,United Kingdom:

Routledge.

Further Readings
Jewitt, C., Bezemer, J., & O’Halloran, K. (2016). Introducing multimodality.London, United Kingdom: Routledge.

Kress, G. (2014). Design: The Rhetorical Work of Shaping the Semiotic World. In Archer, A. &

Newfield, D. (Eds.). Multimodal Approaches to Research and Pedagogy: Recognition,

Resources and Access (pp. 131-152). New York, NY: Routledge.

Kress, G., & van Leeuwen, T. (2001). Multimodal Discourse: The modes and media of

contemporary communication. London, United Kingdom: Oxford University Press.

Brief Authors Biography

Clarice Gualberto is a Postdoctoral Fellow at the Federal University of Minas Gerais (UFMG, Brazil), in the department
of Linguistics. Her research is focused on social semiotics and it aims to disseminate this theory in Brazil, mainly through
papers and books written in Portuguese. As a PhD student at UFMG, she analyzed Brazilian school textbooks and her
research was funded by the Brazilian Government. As part of her project, she studied at UCL/IoE, as a Special Research
Student.

Gunther Kress is Professor of Semiotics and Education at the UCL – Institute of Education (IoE). His research is in
communication and meaning-making in contemporary environments. His broad aims are to continue developing a social
semiotic theory of multimodal communication; and, in that, to develop a theory in which communication, learning,
identity are entirely interconnected. He has led and contributed to a wide range of research on multimodal interaction and
environments, including the gains and losses of changes in representation, knowledge and pedagogy, the English and
Science classroom, and more recently the surgical operating theatre
Figure 1 – Student’s visual report of Mariana (Brazil), 2011. Reproduced with
authorization of the teacher and the student.

View publication stats