Knowledge-Based Systems 14 (2001) 419±424

www.elsevier.com/locate/knosys

What Semiotics can and cannot do for HCI

Peter Bùgh Andersen*
Department of Computer Science, Center for Human Machine Interaction, University of Aalborg, Fredrik Bajers Vej 7E, 9220 Aalborg, Denmark

Abstract
Semiotics is `the mathematics of the humanities' in the sense that it provides an abstract language covering a diversity of special sign-
usage (language, pictures, movies, theatre, etc.). In this capacity, Semiotics is helpful for bringing insights from older media to the task of
interface design, and for de®ning the special characteristics of the computer medium. However, Semiotics is not limited to interface design
but may also contribute to the proper design of program texts and yield predictions about the interaction between computer systems and their
context of use. q 2001 Elsevier Science B.V. All rights reserved.
Keywords: Computer-based signs; Aesthetics; Context of use

1. The mathematics of the humanities
In my experience, Semiotics can be useful for the HCI-
®eld, but the purely analytic character of traditional Semio-
tics has to be supplemented by a constructive one. In addi-
tion, the semiotic community has to acquire a solid
understanding of the technical possibilities and limitations
of computer systems in order to become creative in this
domain. Being able to explain why there are problems
with Apple's trashcan is simply not good enough.
Given these requirements, Semiotics may be helpful in
creating interactive systems that better convey their
intended messages. The reason is that Semiotics is an
abstraction of individual disciplines such as linguistics, art
theory, drama theory and ®lm theory. Therefore it can serve
as a common language for transferring insights from one
domain to another in a systematic way. This is useful in
designing computer interfaces, since computers are inher-
ently multimedia where codes from these diverse ®elds meet
and amalgamate in practice.
Thus, it is in its capacity of `the mathematics of the
humanities' Ð an abstract language where insights from
different disciplines can be represented and compared Ð
that Semiotics is most useful. Semiotics offers a set of
general concepts and analytical procedures that can be
applied to many ®elds, but I do not think that Semiotics as
a discipline has much substantial and practical insight to
offer. Exaggerated claims to that effect have led to frustra-
tion in the past, and unfounded presumptions may be a
* Tel.: 145-9635-8924.
0950-7051/01/$ - see front matter q 2001 Elsevier Science B.V. All rights reserved.
PII: S 0950-705 1(01)00134-4
major reason why Semiotics has never caught on in the
HCI community.
I shall elaborate this basic position in the next sections:
Semiotics may be helpful in
1. Making HCI more coherent.
2. Exploiting insights from older media.
3. De®ning the characteristic properties of the computer
medium.
4. Situating HCI-systems in a broader context.
My motivation for emphasizing these four points is
brie¯y: (1) HCI involves interaction between two very
different entities, machines and humans [1]. They are placed
on each side of a strong faculty border, which makes it
nearly impossible to describe their interaction within a
unique coherent framework. (2) Computers are new
media, and the aesthetics of new media normally evolves
out of older media by borrowing and restructuring older
techniques. (3) However, new media normally have their
own unique artistic effects, and it takes time to discover
what they are. (4) Like any other sign-system, the meanings
created by means of computer systems interact with the
other semiotic systems surrounding them.
2. Making HCI more coherent
Interface design has been dominated by computer scien-
tists and psychologists. The problem is Ð to put it bluntly
Ð that Computer Science has no concepts for the human
mind, whereas Psychology lacks concepts for algorithms
and data structures!
420 P.B. Andersen / Knowledge-Based Systems 14 (2001) 419±424
There is therefore a need for a theory in which both
domains can ®nd a place.
The core of Semiotics is the sign that integrates a physical
(the signi®er) and a psychic side (the signi®ed). Therefore,
Semiotics can talk about representations (the algorithms and
data structures as signi®ers) as well as about the user's
interpretation of these representations (user interpretations
and domain concepts as the signi®ed). But it does so with a
particular focus, namely the sign. Thus, only those parts of
the computational processes that in¯uence interpretation,
and only those parts of the interpretation that are in¯uenced
by the computation, can be analyzed by semiotic methods.
Because of its inherent bridge between physical, psychic
and social phenomena, we will expect many topics from
traditional textbooks in interface design to ®nd their place
in semiotic theory. Here are just a few, taken from a couple
of in¯uential books [2,3]:
² The important difference between the system model and
the user's model simply signi®es two interpretations of
the same sign-complex produced by two groups that
access different parts of it (designer and user). The system
image is the collection of signs interpreted by the user.
The direct manipulation style can be de®ned by the domi-
nation of iconic signs based on similarity. The technical
notion of a view (e.g. in database design) corresponds
neatly to the pragmatic notion of perspective, the way
organizational or geographical position determines the
selection and structuring of interpretation. The fact that
users normally interpret systems differently than
designers do is predicted from the fact that interpretation
of signs always happens inside a larger semiotic system.
If two semiotic systems are different, the interpretation of
the same sign will differ too [4, Part 3].
² Semiotics sometimes enforces a reinterpretation of tech-
nical issues. From a semiotic perspective, many
processes, which Computer Science sees as data storage
and retrieval are really communicative processes. For
example, a database is not merely storage of information
but also a channel of communication [5,6]. This means
that it must conform to the normal principles of commu-
nication by providing a context for interpretation. For
example, a patient database in a hospital [7] ought to
indicate the sender of the information (knowing who is
the doctor in charge is important for interpretation). It
should also indicate an email or phone connection that
will enable doctors or nurses to engage more easily in the
kinds of negotiation which is so frequent in communica-
tion.
3. Exploiting insights from older media
Interface design should be informed by the knowledge
and skills collected within older media, such as pictures,
sound, movies, literature and theatre. Some of the older
insights are still valid in the new medium, others must be
restructured, and still others turn out to be invalid.
I shall take movies as an example [8]. Movies are similar
to computer interfaces in that they exploit both the spatial
(mise en scene) and temporal (montage, editing) dimensions
of the screen ([9,10]). The ®lmmaking tradition has devel-
oped very sophisticated techniques for montage Ð the
temporal succession of shots Ð involving rhythm, lighting,
use of colors, the nature of change from one shot to another
(the 1808 rule), and patterns of shot types (point of view,
match on action, etc.). We would like to learn from some of
these editing techniques in order to avoid becoming `lost in
hyperspace' in hypertexts. Filmmaking concepts, such as
the scene and the sequence, translate easily into computer
interfaces, but the basic ®lm concept, the shot, does not. A
shot is a continuous sequence of celluloid, but computer
interfaces are not made of celluloid but of sequences of
pixel changes. Therefore we need to abstract the notion of
shots so that we can ®nd the closest analog in interfaces and
thereby become able to transfer concepts hinging on the shot
in a systematic fashion. A possible abstraction is the smal-
lest sign whose signi®er is continuous. In hypertexts this
corresponds to a node (a page in WWW), and, having
realized this, we can substitute `hypertext nodes' for
`shot' in ®lm theory and see if useful guidelines result.
But semiotic research needs not to be con®ned to the user
interface. The fact is that all parts of the computer system
are made up of signs that are interpreted by stakeholders. In
particular, the program code is a sign that is created and
interpreted by (teams of) programmers. Therefore, Semio-
tics has an interest in programming methodology too [4,11±
13]. Although it probably has little to say about traditional
computer science issues, such as complexity, correctness,
reliability and ef®ciency, I think it has important contribu-
tions to make with respect to organizing the program text so
as to make it understandable.
4. De®ning the characteristic properties of the computer
medium
However, computer interfaces are not movies. The basic
point of view here is that computer systems add interaction
as a new sign-feature. We do not move the mouse and click
its buttons for the sake of moving and clicking itself, but
because moving and clicking mean something special in the
program we are using. They are signi®ers that can mean I
am drawing, copying, pasting, sorting, saving, etc.
In opposition to passive signs, such as pictures and
movies, interaction also exploits the ability of one interface
object to in¯uence another and it uses the movement of the
user's hand to create meaning. Thus, we should not see the
interface as consisting of passive objects, such as icons and
animations, onto which action is added. Rather, we should
see all interface objects as consisting of four features that
 P.B. Andersen / Knowledge-Based Systems 14 (2001) 419±424
must work in concert to produce meanings: the individual
sign possesses permanent and transient features like a
movie does, but it also harnesses action and handling
features [4, Part 2].
Although drama and ®lm have accumulated a lot of
insight on the art of producing interpretable sequences of
actions, we cannot transfer it wholesale to computers
because computers are neither theatres nor movies. We
have to realize that computer-based signs have their own
characteristics that are different from texts, pictures or
movies, and therefore the old concepts must be restructured.
Let us take the shot as an example again. A major differ-
ence between shots and hypertext nodes is that the ®lm-
maker knows exactly which shots follow which shots,
whereas in hypertexts several nodes can follow a particular
node. This means that nodes must be more self-contained
and combinable than shots must. Film theory does not tell us
how to accomplish this, but Linguistics might help out. The
smallest independent unit in language is the word, and, like
nodes, words must be able to combine with many other
words and still be able to produce meaning.
5. Situating HCI-systems in a broader context
Under this heading, I subsume two aspects that are often
neglected by HCI textbooks but are predicted to be impor-
tant by Semiotics: (1) Humans are compulsive interpreters
and (2) Humans are compulsive talkers!
(1) We cannot help making sense of the signs we perceive
(try to avoid reading the subtitles in movies!). This means
that users do not only interpret the interface they are
supposed to use but also make guesses as to what goes on
`behind' the interface, cf. [14,15]. From this point of view,
the strict separation of a user-friendly interface from the
ugly technical processes behind the stage is not user-
friendly at all [16]! It only hinders users in trying to under-
stand what `really goes on' (which they will do anyway) and
when the friendly interface eventually breaks down, it
leaves them in a helpless position. This motivates a semiotic
interest in end-user programming [17] and tailorability
[18].
(2) Humans communicate during work. They describe
their own and other's tasks to their colleagues [4,18]; in
fact, communication is what makes organizations hang
together [19]. If the behavior of computer systems cannot
be talked about, the new technology hinders this vital
communication. Therefore, interfaces should not only be
interpretable, they should also be verbalizable: rely on the
user's work language as a starting point for designing the
system.
In the following, I shall elaborate on these two points.
5.1. Compulsive interpreters
HCI as a systematic discipline was born in the early eigh-
ties when computers traveled from the computer department
421
into the of®ces, and millions of clerical workers without
prior knowledge of computers were required to use them.
They experienced dif®culties and failures, because, at that
time, programmers built the user interface on the same
concepts, which they themselves had used to construct the
system. Thus, technical concepts were exported into the
of®ces, and they were not welcome there.
In order to get of®ce automation going, the strategy of
`hiding the intestines' was developed. The aim of interface
design was to invent metaphors that users could understand,
but which had not much to do with the internal workings of
the system. Underlying this strategy was an organizational
division of labor: users use systems, software houses make
them, and technicians repair them.
This division of labor had 3 problems:
² Metaphors break down and create confusion.
² Technicians became a scarce resource as the number of
computers grew.
² Users cannot help trying to understand anyway.
The division of labor must therefore be revised. Modern
users to some degree have to be their own support staff, and
systems should not prevent them from performing this func-
tion. However, the decade-old distinction between the
`friendly' user interface and the mysterious intestines is
still a major obstacle. We seem to weave ourselves into a
web of meanings we can neither change nor understand and
which we therefore tend to fear (cf. the Y2K hysteria).
What can be done to make users more self-supporting? If
we look at language, a medium that had to develop and
reproduce itself over many thousands of years without a
technical repair department, we see that self-reliance is
indeed possible with extremely complex systems. One
reason for this is that language can be modi®ed through
use and, since language is its own meta-language, problems
and inadequacies can be discussed and solved by the
language users themselves: we do not have to send for a
linguistic repair-man each time misunderstandings occur
[20]. Therefore, computer system designers ought to take
inspiration from the system type language exempli®es. I
suggest that three formal properties be worth looking into
[21]:
1. Recursive processes that use their own output as input. A
turn in a conversation uses the previous turn as its point
of departure, adds new information, and is itself used by
the next turn.
2. Self-reference. We can talk about the world and the
language in the same language, smoothly shifting from
the topic of a conversation to the conversation itself and
back again. In the computer domain, smooth transitions
between the domain of the system and the system itself
are described in e.g. Refs. [18,22].
3. Self-similarity. It has been argued that syntax and seman-
tics are essentially self-similar. The noun-phrase is
422 P.B. Andersen / Knowledge-Based Systems 14 (2001) 419±424
Fig. 1. Multimedia system for a shop selling third world commodities.
structurally similar to the sentence of which it is a part,
and the structure of clauses is similar to their subordinate
sentence. Likewise, the lexical meaning of a word (de®-
niendum) can be explained by the lexical meanings of a
string of words (de®niens).
It is no coincidence that WWW, the largest, most
dynamic and most distributed computer system in existence,
is both recursive (new pages are produced by means of the
current page) and self-referential (the META tag) [23].
5.2. Compulsive talkers
As mentioned above, work organizations cannot exist
without communication. How else would we be able to
distribute, monitor, and report work, to solve problems of
coordination and methods, and to teach newcomers the
required skills [4]? In some professions, such as maritime
maneuvering and navigation, self-description is mandatory:
the man at the machine telegraph or wheel must describe his
operations loudly so that his colleagues can maintain an
updated knowledge of the current situation. As mentioned
above, verbal communication is self-referential: not only
work, but also communication itself, can be regulated by
communication. An example is a captain telling the forget-
ful helmsman to remember telling the others when he turns
the wheel [24].
Although there are other modalities of communication
than verbal language, verbal language is still the backbone
of organizations, a modality to which one resorts when other
methods fail.
Computer systems are new media for organizational
communication, but they too must obey the law of commu-
nicability. Otherwise the communicative circuit that keeps
the organization going will suffer: what will happen to the
ship if the helmsman is presented with a computerized
helmstand, full of engineering jargon that he cannot even
pronounce, let alone understand?
For this reason, computer systems must be verbalizable.
Computer systems are themselves sign systems, so the
requirement really says that the signs of the system must
be such that they can easily enter into the surrounding
communication. Semantically this means that the basic
meaning distinctions of the system should not be at cross-
purposes with that of the professional language. For exam-
ple, if the basic dimension for classifying tasks is their loca-
tion, then a system classifying them according to temporal
sequence represents a major break with the existing
language [4]. Either the users will ignore the system signs
and invent their own, or they will have to change their own
language. This does not mean that the existing language
should be copied, only that one should take into account
the possible changes of communication in the organization.
If it will change, is the change a desired one?
The notion of communicability can be used as a practical
guideline for interface design: the relations between
elements in a screen image must be describable in a few
simple sentences. If you have to write a whole novel to
capture the meaning of a screen, something is wrong.
5.3. An example
I conclude with a small example taken from a student's
exercise. It illustrates how to enhance communicability by
projecting features from verbal communication onto
computer systems, it shows that the theory can generate
concrete ideas for design, and it illustrates the need for a
medium-independent semiotic framework.
The example is a multimedia system designed for a shop
selling clothes, musical instruments, pottery and jewelry
from third world countries (Fig. 1).
The system contains four main parts:
1. an animated globe;
2. a set of buttons;
3. a page containing text and pictures about the country and
its products; and
4. a music track playing music from the selected country.
The parts are to be interpreted in the following way:
At the location indicated in the globe (1), there is a coun-
try with the name indicated by the depressed button (2) that
is described by the text and pictures to the right (3) and that
cultivate the music played by the loudspeaker (4).
Since the verbal interpretation is easy to formulate, it
does stand up to the criterion of verbalization. However,
improvements can be suggested by including other features
than stark syntactic structure. For example, language
provides facilities for controlling the information structure
of the sentence: theme/rheme, old/new information, and
background/focus. Old information is unstressed, the
theme normally introduces the sentence, and background
information is presented in adnominal constructions, such
as adjectives and clauses.
 P.B. Andersen / Knowledge-Based Systems 14 (2001) 419±424
In the following paraphrase, Zimbabwe is the theme,
located in Africa and playing a certain kind of music is
background information, and has 9610.000 inhabitants
and ¼ is the new information.
² Zimbabwe (2), located in Africa (1) and playing a certain
kind of music (4), has 9610.000 inhabitants and ¼(3)
In order to translate this information structure into the
two-dimensional multimedium, we could suggest the
following changes:
1. Located in Africa is background information ! blurring
the picture of the globe.
2. Music is background information ! decreasing loudness
and ®ltering out overtones.
3. Has 9610.000 inhabitants and ¼ is the new
information ! making it the lightest and sharpest part
of the screen and use color.
4. Zimbabwe is the theme ! use color, but a discrete one, to
show that it connects to (3) but is not the important part of
the two. Remove color from the other buttons to show
that they are not themes at the moment.
We could also use two sentences as in:
² Zimbabwe (2) is located in Africa (1) and plays a certain
kind of music (4).
² This African country has 9610.000 inhabitants and ¼(3)
In this version, the location and the music are presented as
new information in the ®rst sentence. In the second
sentence, the location is background information and the
number of inhabitants becomes the new information.
How would we express this in a multimedium? Here
is one way: before we enter Fig. 1, we see another
screen where the globe is sharp, rotates and is colored,
and the music is loud. When the globe has rotated to
the location of Zimbabwe, we change to Fig. 1. The
globe shrinks, loses color and freezes at the Zimbabwe
location while the loudness of the music decreases and
overtones are ®ltered out.
An account of this exercise needs a semiotic framework
that can talk about signs and meanings without committing
itself to a particular medium. The exercise presupposes that
there are phenomena like backgrounding that can be
expressed as an adnominal construction in verbal code,
but as blurring in a graphical code, and new information
that can be expressed by position and stress in verbal
code, but as movement, location, sharpness, and vividness
in graphics.
6. Conclusion
In this paper, I have argued that Semiotics can serve as a
coherent framework for exploiting the insights of older
423
media while still focusing on the characteristic properties
of the computer medium. Semiotics is also helpful for posi-
tioning design of computer systems in a broader theoretical
and philosophical context.
Finally, I have tried to show that Semiotics, if properly
developed, can also be useful in practical, down-to-earth
design.
References
[1] F. Nake, Human±computer interaction: signs and signals interfacing,
Languages of Design 2 (1994) 193±205.
[2] W.M. Newman, M.G. Lamming, Interactive System Design, Addi-
son-Wesley, Wokingham, UK, 1995.
[3] D.A. Norman, S.W. Draper (Eds.), User Centered System Design
Lawrence Earlbaum, Hillsdale, NJ, 1986.
[4] P.B. Andersen, A theory of computer Semiotics, Semiotic
Approaches to Construction and Assessment of Computer Systems,
Cambridge University Press, Cambridge, 1990/97.
[5] T. Winograd, F. Flores, Understanding Computers and Cognition,
Ablex, Norwood, NJ, 1986.
[6] R. Stamper, Signs, Information, Norms and Systems, in: B Holmqvist,
P.B. Andersen, H. Klein, R. Posner (Eds.), Signs of Work, de Gruyter,
Berlin, 1996, pp. 349±399.
[7] G. SandstroÈm, Towards Transparent Data Bases, Studentlitteratur,
Lund, l985.
[8] P.B. Andersen, Film og multimedier (Film and multimedia), in: B.
Fibiger (Ed.), Design af multimedier (Design of multimedia), A Ê lborg
Ê lborg, 1997, pp. 277±309.
Universitetsforlag, A
[9] J. Monaco, How to Read a Film, Oxford University Press, New York,
1977.
[10] D. Bordwell, K. Thompson, Film Art, Addison-Wesley, Working-
ham, UK, 1979.
[11] D. Piotrowski, Structuralism, computation and cognition. The contri-
bution of glossematics, in: P.B. Andersen, B. Holmqvist, J.F. Jensen
(Eds.), The Computer as Medium, Cambridge University Press,
Cambridge, 1993, pp. 68±91.
[12] P.B. Andersen, A semiotic approach to programming, in: P.B. Ander-
sen, B. Holmqvist, J.F. Jensen (Eds.), The Computer as Medium,
Cambridge University Press, Cambridge, 1993, pp. 16±68.
[13] P.B. Andersen, P. Hasle, P.Aa. Brandt, Machine semiosis, in: R.
Posner, K. Robering, T.A. Sebeok (Eds.), Semiotics: A Handbook
About the Sign-Theoretic Foundations of Nature and Culture, vol.
1, de Gruyter, Berlin, 1997, pp. 548±570.
[14] P.Aa. Brandt, Meaning and the machine: toward a Semiotics of inter-
action, in: P.B. Andersen, B. Holmqvist, J.F. Jensen (Eds.), The
Computer as Medium, Cambridge University Press, Cambridge,
1993, pp. 128±140.
[15] P.B. Andersen, K. Halskov Madsen, Design and professional
languages, in: P.B. Andersen, og Bratteteig (Eds.), Computers and
Language at Work, Institutt for Informatikk, Oslo Universitet, Oslo,
1989, pp. 117±156.
[16] D. Gentner, J. Nielsen, The Anti±Mac Interface, CACM 8 (1996) 70±
82.
[17] C.S. de Souza, Semiotic engineering principles for evaluating end-
user programming environments, in: C.J.P. de Lucena (Ed.), Mono-
gra®as em CieÃncia da ComputacËaÄo, Computer Science Department,
PUC-Rio, Brazil, 1999 PUC-Rio Inf MCC10/99.
[18] P.B. Andersen, M. May, Tearing up interfaces, in: K. Liu et al. (Eds.),
Information, Organisation and Technology. Studies in Organisational
Semiotics, Kluwer, Dordrecht, 2001, pp. 271±298.
[19] N. Luhmann, Soziale Systeme, Suhrkamp, Frankfurt am Main, 1984.
[20] P.B. Andersen, The mass and the individual: language as an example,
424 P.B. Andersen / Knowledge-Based Systems 14 (2001) 419±424

in: J. Andreas (Ed.), The Mass Ornament, Kunsthallen Brandts
Klñdefabrik, Odense, 1998, pp. 125±138.
[21] P.B. Andersen, Dynamic Semiotics, Semiotica, in press.
[22] T.W. Malone, K.-Y. Lai, C. Fry, Experiments with Oval: a radically
tailorable tool for cooperative work, ACM Transactions on
Information Systems 13 (2) (1995) 177±205.
[23] P.B. Andersen, WWW as self-organizing system, Cybernetics and
Human Knowing 5 (2) (1998) 5±42.
[24] P.B. Andersen, P.H. Carstensen, M. Nielsen, Dimensions of coordi-
nation, in: M. Schoop, C. Quix (Eds.), Fifth International Workshop
on the Language-Action Perspective on Communication Modelling,
Aachener Informatik-Berichte (2000-6), Aachen, 2000, 41±60.