See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/305639283

Semiotics of virtual reality as a communication process

Article in Behaviour and Information Technology · July 2016

DOI: 10.1080/0144929X.2016.1212092

CITATIONS READS

27 2,440

4 authors:

Barbara Rita Barricelli Davide Gadia

Università degli Studi di Brescia University of Milan
179 PUBLICATIONS 1,841 CITATIONS 86 PUBLICATIONS 646 CITATIONS

SEE PROFILE SEE PROFILE

Alessandro Rizzi Daniele Marini

University of Milan University of Milan
346 PUBLICATIONS 4,204 CITATIONS 120 PUBLICATIONS 1,931 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Daniele Marini on 12 December 2017.

The user has requested enhancement of the downloaded file.

 Behaviour & Information Technology

ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: http://www.tandfonline.com/loi/tbit20

Semiotics of virtual reality as a communication

process

B. R. Barricelli, D. Gadia, A. Rizzi & D. L. R. Marini

To cite this article: B. R. Barricelli, D. Gadia, A. Rizzi & D. L. R. Marini (2016): Semiotics of
virtual reality as a communication process, Behaviour & Information Technology, DOI:
10.1080/0144929X.2016.1212092

To link to this article: http://dx.doi.org/10.1080/0144929X.2016.1212092

Published online: 25 Jul 2016.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=tbit20

Download by: [Barbara Rita Barricelli] Date: 25 July 2016, At: 23:34
 BEHAVIOUR & INFORMATION TECHNOLOGY, 2016
http://dx.doi.org/10.1080/0144929X.2016.1212092

Semiotics of virtual reality as a communication process

B. R. Barricelli, D. Gadia, A. Rizzi and D. L. R. Marini
Department of Computer Science, Università degli Studi di Milano, Milan, Italy

ABSTRACT ARTICLE HISTORY

The experience of immersive virtual reality (VR) can be considered as a communication process Received 19 January 2016
between human beings, mediated by computer systems, which uses visualisation and other Accepted 8 July 2016
sensory stimulation. In this paper, we analyse how VR characteristics can be explored using
KEYWORDS
semiotic theory and, with methods of generative semiotics, we explore aspects of narrative and Virtual reality; semiotics;
interaction in VR. We propose a semiotic analysis of VR communication focusing on syntax, communication
semantics and pragmatics and considering also some principles of generative semiotics. The
syntactic level is analysed as determined by the characteristics of the visual communication
adopted. The semantic of VR is related to the functional model chosen to realise the virtual
system. The pragmatic of VR is based on the human–computer interaction that changes the
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016

user’s role. We explore how these aspects can be characterised in the context of VR
communication design and what principles can be adopted for a VR application, and we present
an analysis and a classification of the iconic signs that are being used in VR. Moreover, we
present a framework that can be used to classify and describe different kinds of virtual reality
systems and to better understand communication in VR, and we use it to classify eight popular
systems for e-learning and collaboration.

1. Introduction
Here, we aim at analysing virtual reality (VR) from the
semiotics point of view, starting from the classic works
in the field. Central in Peirce’s studies is the relationship
between the concept of virtuality and the concept of sign,
that is, something that once interpreted stands for some-
thing other than itself. Peirce’s most quoted sentence is:
‘I do not say that we are ignorant of our states of mind.
What I say is that the mind is virtual, not in a series of
moments, not capable of existing except in a space of
time – nothing in so far as it is at any one moment ’
(1932, 248). From this position, Peirce draws that the
interpretant of a sign becomes itself a new sign (what
Umberto Eco has defined ‘infinite semiosis’); so a sign
‘is what it is by virtue of its possible later interpretations
– i.e. virtually – so the mind itself is virtual’ (1932, 289).
The centrality of the concept of virtuality in the semiosis
process is particularly important if we consider the pro-
blem from a communication viewpoint. We move from
the assumption that VR can be considered as a com-
munication process between different subjects (two per-
sons or one person and a computer program), mediated
by a computer system, that uses visual communication as
the primary channel and as secondary channels other
sensory systems such as hearing and kinaesthetic
perception.
We recall that in the semiosis process we have differ-
ent abstraction levels to consider, increasing from the
lowest, the syntactic level, followed by the semantic
level and eventually the pragmatic level. Syntax is the
set of rules that are to be applied to build or interpret
sentences made of words chosen from a vocabulary.
While in natural or abstract languages the vocabulary
is made from literary symbols, in VR the symbols are
mainly based on visual elements, so we have to consider
a visual communication system, which is based on signs
that are used to build a representation of the virtual
world. Semantics is the study of the meaning of the
elements of a vocabulary and their composition
(words, sentences, bodies of text). In VR, the semantic
level of the communication process is constituted by
the choice of representation and rhetoric styles. Finally,
pragmatics studies the influence that the context has on
the assignment of meaning. It is the pragmatic level
that deals with the effects of the communication, binding
the message, the sender and the receiver. VR, moreover,
is dynamic: its status can change in time on the basis of
some internal rules or mathematical models and on the
basis of the user interaction. Interactivity happens
between the user and the VR system or can happen
between different users through VR interaction, in a
kind of distributed, coordinated work. Both interactions

CONTACT B. R. Barricelli barricelli@di.unimi.it

© 2016 Informa UK Limited, trading as Taylor & Francis Group
 2 B. R. BARRICELLI ET AL.
can change some internal parameter of the VR represen-
tation changing the way it evolves. Considering for
example human–computer interaction (HCI), the
relationship between humans and computers is based
on the given interaction rules; therefore, we can consider
that the rules of the human–machine dialogue are part of
the pragmatic level (Sjöström and Goldkuhl 2003).
For more complex computer-mediated communi-
cation, such as VR, we present here a novel way to con-
sider the different abstraction levels, to identify rules and
roles that they play. In a communication process, we
search for the best solutions to achieve a given goal
and the rhetorical choices that can be consequently
adopted. To achieve an effective communication, rheto-
ric art is used to organise the discourse, adopting the best
rhetorical figures (e.g. metaphor, allegory, oxymoron,
hyperbola) that altogether should achieve the objective
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
to convince, persuade and induce people to do some-
thing or believe in something. In verbal and written
communication, as well as in cinema, television and
advertising, these aspects have been deeply investigated
and procedures are regularly adopted to steer communi-
cation for entertainment, marketing and even politics.
In these fields, generative semiotics (Greimas 1983) has
produced important results, providing guidelines that
are currently adopted by communication experts. In
generative semiotics, it is the narrative structure that
implements the rhetoric efficacy of communication.
In the field of VR, there is still significant research to
do. Considering the specificity of this communication
medium, mainly experienced through vision and inter-
action, the main purpose of the rhetorical effort is to pro-
vide the sensations of presence and immersivity in some
artificial world, recurring frequently to the suspension of
disbelief. Presence and immersivity, on the other hand,
emerge from the recognition of the content being com-
municated via the VR system: an object, or a system of
complex objects that are intrinsically virtual. A definition
of presence in virtual reality and immersive environment
is given by North and North (2016) ‘how realistically
participants respond to the environment as well as
their subjective sense of being in the place depicted by
the Virtual Environment’.
For many years, the problem to solve for progressing
in the field of Computer Graphics has been the photore-
alism, and this consequently applied to VR. Photoreal-
ism is the ability of the simulation system to convey
details closer and closer to the real world. In the common
opinion, an improvement in photorealism would be the
mandatory way to reach a higher level of recognition,
and as supposed consequence higher involvement
(Pharr and Humphreys 2010). Recent findings in the
field of high dynamic range imaging have pointed out
how the standard photographic imaging, a method we
consider a realistic reproduction of reality, is in the mat-
ter of facts a distorted version of the light field of the
scene (McCann and Rizzi 2011). It follows that our visual
system does not need a perfect representation of the rea-
lity: lower levels of visual sensation have embedded a
mechanism according to which a scene to be interpreted
as the real one has to be verisimile. This goes beyond
being faithful to the physical values of the reality to
depict: the simulation has to be faithful to its spatial rep-
resentation. The question is about how this recognition
happens, and this problem has been already tackled by
semiotics.
A communication process can occur in the presence
of a sender and a receiver that exchange messages
through a communication channel, but it can also
occur inthe absence of a sender. This is the case when
we observe the real world and give a meaning to the
objects and the events that happen around us. In VR
applications, it is this last type of communication that
we have to consider mainly – for example, when the
VR offers the basic functionality to explore (navigate) a
virtual world without or with only a few explicit infor-
mation provided. The receiver of the message in this
case plays the most important role. In the language of
semiotics, this is the case of pure signification, that is,
when a person receives a stimulus and the task is to inter-
pret and thus resolve the problem of assigning a meaning
to it. The signification process is strictly a cognitive pro-
blem: associating signs to objects or events has been
approached by empiric philosophy as the way people
form their knowledge and representation of the world.
The interpretation requires that the player has some pre-
vious knowledge, the so-called encyclopaedia (Eco 1976),
or can build the required encyclopaedia during the
exploration of the VR world.
The building of an encyclopaedia is not only a cultural
problem; research works in cognitive neurophysiology
suggest that brain plasticity plays also a relevant role.
As an example, we can cite the case of Neil Harbisson
– a man born with achromatopsia, a congenital problem
that causes inability to perceive any colours. To deal with
this problem, Neil started a project called Eyeborg in
which he wears a head-mounted colour sensor that
transforms the colour acquired by the sensor into
sound frequencies transmitted to his earing system
through the skull bone.1 The particularity of this case
is that Neil decided to permanently attach the eyeborg
to his head allowing his brain to build a brand new ency-
clopaedia for him to have the sensation of colour. After
some years, this has led to a change in the brain struc-
tures and functionalities (Alfaro et al. 2015). Eco
(1989) observes that the interpretation of a literary text

is the result of an active participation of the reader, who,
on the basis of expectations and previous experiences,
can get different outcomes even unexpected by the
author. The presence of interactivity in VR is a specific
example of an open work: the interacting user by explor-
ing the virtual world builds an encyclopaedia on the fly,
and the final interpretation of the virtual experience can
vary strongly from the intention of the VR author. In
many communication artefacts (e.g. literary, cinemato-
graphic, theatrical), to reduce the ambiguity of interpret-
ation, the presence of a paratext (Genette 1997) can be
effective. The paratext is a collection of information
not directly included in the artefact, like, in the case of
a book, the cover, the title, some comments in the back
cover, or critics and comments published somewhere
else. The paratext helps a reader to catch the meaning
of the text, by providing side information that enriches
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
the reader’s encyclopaedia. An example on how paratex-
tual elements can be associated to an artefact are the
comments provided by people that have already experi-
enced it and published their comments, for example, on
a social network. But such paratextual elements are not
always present in a VR application, so we have to inves-
tigate how the semiotic interpretation is organised, and
we need to explore the semiotic structure of VR. The
high-level description of a communication process can
be explicit (pure communication with sender, channel
and receiver) or implicit (signification with only the
channel, and the receiver). Such description needs to
be expanded to consider the nature of the channel and
in particular the nature of the signs that are adopted,
from the viewpoint of their syntax and semantics. More-
over, the overall organisation of the signs of a VR appli-
cation has eventually to be considered from the
pragmatic viewpoint to get a clear view of the coherence
of the whole rhetorical solution chosen by the designers.
In this paper, we will focus our study on the semiotic
analysis of VR, exploring the nature and the syntax of VR
signs, their semantic content and pragmatic roles. To
restrict our investigation domain, we are considering in
particular immersive VR, that is, systems where a person
is mainly stimulated visually (possibly with stereo view-
ing) and the outside real-world stimuli are dimmed.
Immersive VR can also use other sensorial stimuli such
as auditory, kinaesthetic or olfactory to improve the
immersivity and presence (Bowman and McMahan
2007; Coelho et al. 2006). We will not further explore
these aspects in VR, treated in a previous paper (Marini
et al. 2011).
This research work stems both on a strong study of lit-
erature review and state of the art and on long-time
experience in VR research and development. As to the
theoretical part, we rely on the semiotic theory of Eco
BEHAVIOUR & INFORMATION TECHNOLOGY 3
(1976) and partly on the theory of the French school,
in particular that of Greimas (1983). From there, we
explore the literature on semiotics, applied to narration
in analogical and digital world.
This paper is organised as follows. In Section 2, dedi-
cated to the literature review, we present and discuss how
semiotics has been considered and applied so far in VR.
In Section 3, we present and discuss the semiotic analysis
of VR: how the syntactic level can be organised, what the
available languages are (in the most general meaning of
language), the study of the level of iconicity2 that can
be applied to any Computer Graphics application; the
semantic level, highlighting the importance of human
perception in the critical context of VR, where it is not
a real world that we perceive, but our senses are stimu-
lated by an artificial simulation; the pragmatics of VR
communication, considering in particular the role of
narrative structure. In Section 4, a general framework
of VR is presented allowing us to classify different VR
solutions from the viewpoint of their semiotic nature.
Finally, Section 5 provides some examples of VR appli-
cations in the fields of collaboration and learning, and
uses the framework to classify them.
2. Semiotic studies of virtual reality: literature
review
Semiotics as a discipline to study the process of giving
meaning to computer-based communication has been
largely investigated in the field of HCI – for example,
de Souza et al. (2001), O’Neill (2008), and Valtolina, Bar-
ricelli, and Dittrich (2011), while works on VR are quite
few. At our knowledge, the following are the major works
on this research topic. Aruga et al. (2007) move from
Kant’s theory of human understanding: the recognition
of an ‘a priori’ of a VR arises from the projection into
the brain of an image of an object. However, this theory
is not sufficient to understand VR experience. Peirce’s
triadic description of the nature of sign (representamen,
object, interpretant) is proposed as a more adequate
description of human understanding process without
the need of the ‘a priori’. The authors put their interest
in semiosis as the process of giving meaning to signs pro-
ducing new signs in an unlimited chain. In VR, a real
object is not present, we only have signs (representamen)
that signify a virtual object; to interpret the stimuli of a
flight simulator, for example, the computer system
must produce the same psychological effects of a real air-
craft. In other words, the notion of similarity and mim-
esis is essential to give a meaning to a VR application.
Belanger (2009) explores immersive VR with semiotic
theory. He underlines the importance of interactivity to
involve the user that becomes active in the interpretation
 4 B. R. BARRICELLI ET AL.
process, thus recognising that the communication pro-
cess does not provide a meaning, rather the meaning is
the result of the active interaction of a user, as described
by Eco (1979). Moreover, Belanger clarifies that, differ-
ently from other media, immersivity and presence
(through accurately controlled sensory stimulation)
also contribute to hide the underlying technology from
the users. Immersivity, moreover, has the capacity to
transport the user to a different world by making credible
or believable the artificial world (see Marini et al. 2011
for the concept of verisimile – i.e. similar to the real
thing). Considering the ability of VR to create emotions
in users, he also explores the concept of myth, but does
not extend the study to the rhetoric of VR. Han (2011)
examines the semiotics of Second Life and describes
the nature of the signs, identifies metaphors, connota-
tion, and denotation characteristics and recognises
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
Second Life as a myth-building environment; her work
is actually an example of semiotic analysis of a specific
kind of VR application. Matusitz (2005) explores the
notion of identity, space and deception of virtual worlds
from a semiotic viewpoint. He pinpoints that the cre-
ation of a personal identity in a virtual world is the result
of a semiotic process, which is contextually and histori-
cally contingent. In this sense, the type of communi-
cation that is established with the user through the VR
communication is mainly a pure signification, where
the content can control and possibly deceive receivers
of communication.
Pujol (2006) proposes a semiotic analysis of VR by
exploring its syntax, semantics and pragmatics. In par-
ticular, she focuses on the semiotics of images, and
studies the iconic character of image semiotics, expand-
ing the levels of iconicity proposed by Moles (1969),
including VR as a specific iconicity level. She does not
consider the importance of the narrative character of
VR, while recognising the role of interactivity to the
determination of the pragmatic aspect. Nadin underlines
that ‘Technological artifacts of all kind – from games to
VR labs in which new materials are considered – rely on
various types of semiotic entities, on representations in
the first places, and their interpretation’ (Nadin 2012,
19) . He identifies in ‘languages of interaction’ the critical
research point, and tends to merge VR with other Inter-
net-based technologies. Lemke (1999) proposes an inter-
esting view of semiotics of non-textual communication,
in particular mathematics and geometry, that can be
extended to VR. Besides signs and symbols of mathemat-
ics and geometry, a topological semiotics has to be con-
sidered. The interpretation of topological semiotics
depends on spatial cognition skills that can be developed
with interaction. We can observe that users’ spatial
orientation in navigating a non-natural VR spatial
domain improves with practice. Concluding this short
overview, it is worthwhile to underline why we are
mostly interested to VR rather on general semiotic
aspects of interaction or the web: this is because in
high-level immersive VR (e.g. flight simulators), we
face successful representations of artificial worlds that
mimic the real ones and the interpretation and significa-
tion process is the same as what we perform in the real
world. Thus, the study of semiosis could in future benefit
by using VR as a kind of artificial experimental
environment.
3. Semiotic analysis of virtual reality
As stated before, the aim of this paper is to explore the
semiotics of VR as a communication process and there-
fore to analyse it according to the three abstraction levels
that are typical in a communication process: syntactic,
semantic and pragmatic. These three levels are in a strict
relationship with what we consider the three main
elements of a VR system: structure, model and inter-
action as we defined them in Marini et al. (2011). By
the structure, we mean the organisation into parts, the
shape of the parts, the assembly of the parts, etc., what
in general becomes the 3D geometric model of the equip-
ment, possibly enriched with a dynamic behaviour. By
the function of the equipment, we mean the way each
part works, the way the assembly works or, better say,
how parts and assembly change as the time goes on or
as an operator changes some part. By the underlying
model, we mean what are the physical, chemical, in gen-
eral technical or scientific principles that rule the func-
tion of the equipment.
3.1. Structure: the syntactic level
The structure of VR applications is grounded on visual-
isation and visual communication principles. Structure is
also affected by the choice of rhetoric effects used to raise
desired emotions in users. Visual communication prin-
ciples applied to Computer Graphics allow us to identify
how an effective visual language can be implemented in
VR. The structure can therefore be seen as the core of the
syntax level. VR applications are commonly based on
visual representations of some scenes and/or objects –
real or created by the imagination of designers. The
structure of such representations can be implemented
using graphic solutions, e.g. schematic drawings, 3D geo-
metric models perspective projection, images. Such a
large range of possible solutions offers to VR designers
a wide choice of signs. Signs can be iconic, indexical or
symbolic (Peirce 1932). Iconic signs can be images –
and in this case there is an immediate similarity between

the sign and the referenced object – or can be pictures or
diagrams – where similarity is linked to relations among
parts, or can be metaphors – where similarity is mediated
by the mapping to a different context. This classification
assumes the degree of similarity or resemblance as a cri-
terion but it is unfortunately too simplistic. The notion of
similarity of iconic sign has been criticised first by Eco
(see a discussion in Sonesson 1998) and we also have dis-
cussed this problem in a previous paper (Marini et al.
2011).
In our view, similarity is related to the concept of rea-
lism. We propose that the similarity of an iconic sign has
to be considered as the result of a choice by the designer
to make the sign believable, to make it verisimile, that is,
‘similar to the real thing’. Being similar to the real thing
relies on the ability of the designer to capture the essen-
tial elements of a representation, like in a caricature,
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
creating an illusion to the observer and inducing her/
him to suspend his/ her disbelief. Besides the contri-
bution of similarity, socio-historical conventions can
help in getting the meaning of a sign, especially for sym-
bolic signs. We can have a degree of different abstrac-
tions leading to representations that can be more or
less easily interpreted and that convey the verisimile
attribute. These different levels of abstractions can be
Figure 1. Examples of the 12 iconicity levels.
BEHAVIOUR & INFORMATION TECHNOLOGY 5
classified in terms of the degree or level of iconicity.
The problem of iconicity has been also considered in
the context of user interface design (see de Souza
1993), where the quality of an icon is linked to the diffi-
culty of decoding its symbolic value.
In VR, this problem has a different character, since
icons contribute to get the immersion effect in a living
experience, where the decoding of symbols is just a
part of the whole task and can be affected by a possibly
changing context with strong multisensorial stimuli,
among which vision plays the major role. The visual rep-
resentations are different from textual ones and are based
on visual analogy; in other words, they are an allegory.
An illustration corresponds to a process of denotation
based on displaying signs.
In 1969 Moles devised a taxonomy for illustrations
and drawings in graphic design (specifically in the evalu-
ation of communicability of outdoor advertising), which
is based on an iconicity scale from the most concrete to
the most abstract depiction. Later, in 1992, Anceschi
adapted Moles’ taxonomy to the field of scientific illus-
tration and chose to consider the seven most abstract
iconicity levels only. The Moles’ and Anceschi’s taxo-
nomies of iconicity levels aim at classifying the kind of
allegories that are required to communicate the meaning
 6 B. R. BARRICELLI ET AL.
of the chosen signs. Selecting an iconicity level for an
illustration is therefore a specific rhetoric choice of the
communication designer: a classification of iconicity
levels contributes to a better control over the rhetoric
in the VR communication process.
To apply these seminal works for the field of iconic
analysis of VR, we developed a classification of iconicity
of computer-generated images. The iconicity levels we
considered are 12 (from highest to lowest level of iconi-
city) and they are exemplified in Figure 1 and described
as follows:
(1) The object itself. In Figure 1 (Level 1), the example
given for this level is a car.
(2) 3D model, perspective photorealistic rendered into
a background context; a pot in Figure 1 (Level 2).
(3) Retouched photography. This kind of represen-
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
tation derives from photography of a real scene
or from its painted version based on perspective
projection techniques. The purpose of the illus-
tration is to mimic a reality, also by means of illu-
sory effects. The purpose (pragmatics) of this
illustration in a communication is simply to display
something, an object (like Level 3 in Figure 1) or a
process, the way of use is out of scope, being learn-
ing-to-know the major aim.
(4) Line drawing simplification. Examples of these
kind are illustrations, where an expert graphic
designer simplifies what s/he sees, guiding the
observer to the most relevant aspects of the object
or the process. Contouring and half toning are
the major techniques used to convey the third
dimension and the shape, masking or simply
avoiding to display secondary aspects. The purpose
of this representation is descriptive, since it puts
emphasis to general morphology, while particulars
are less important. One example is given in Figure
1 (Level 4). Other examples of this kind are
caricatures.
(5) Anatomic view. The purpose of this illustration is
to explain the structure, even the internal structure
of an object. Specific techniques can be adopted like
transparent through-view or exploded represen-
tation of the components (like in Level 5 of Figure
1). They are largely used as a graphical solution for
assembling in technical manuals; so their major
purpose is operative (learning the know-how).
(6) Constructive drawings, plane projections. This is
the typical case of technical drawings; also in this
case the major purpose is operative. Measures
and scale are relevant, the graphic designer is free
to choose the signs to be used, and s/he will try
to adopt easily understandable signs; the receiver
of the message should be able to act, e.g. construct-
ing an object, or taking measures. An example is in
Figure 1 (Level 6).
(7) Normalised constructive drawings. Also in this
case the purpose is operative: the observer should
be able to get information relevant to do some-
thing, but the major difference is that the sign
language is formalised. In this representation,
standard symbols are extensively used to convey
information about the structure of an object or
a process. The emphasis of the representation is
on topological rather than on geometrical aspects:
what is important is the function or the structure,
not the shape or the morphology, and larger free-
dom is left to the observer to do something from
the representation; for example, she could be a
designer who is describing the structure or the
process of a complex system (see Level 7 in
Figure 1).
(8) Flow diagrams. This term is used to denote a large
kind of diagrammatic representation, such as
organisation charts, block diagrams and process
diagrams. The purpose of these kinds of represen-
tations is to stimulate the understanding of cause–
effect relationships, time dependency, hierarchical
dependency, etc. Figure 1 shows an example in
Level 8.
(9) Normalised symbols. The adoption of normalised
symbols aims at providing simple and immediate
decoding, when their meaning has been learned,
being strongly conventional. Traffic signs are an
example for this level, depicted in Figure 1 (Level
9).
(10) Logical diagrams. Here, the logical structure is the
major aspects. It can be implemented using nor-
malised or non-normalised symbols (see Level 10
in Figure 1).
(11) Field diagrams. This kind of representation has no
relationship to any structural, topological or mor-
phological property. The observer has to put to
work his/ her imagination, since what is relevant
is how parts of a system are in some way related.
Typical representations of this kind are Venn dia-
gram of set theory, or a dendrogram to describe
taxonomies. The object of the representation is a
kind of underlying conceptual structure of a sys-
tem. Field diagrams are best suited to build a
deep knowledge of a system. Figure 1 (Level 11)
shows a diagram of this kind.
(12) Perfect abstraction. In the last level, iconicity is
totally lost. Perfect abstraction works as an extreme
visual metaphor, the signs used to represent an
object; a process or a system has only a

conventional interpretation. Examples are histo-
grams or statistical charts, where a visual metaphor
drives from a real world to a world of conventional
symbols. In Figure 1 (Level 12), the example pre-
sents a perfect abstraction: train connections of a
railway service.
In some aspects, the levels of iconicity can be con-
sidered as visual rhetoric effects: abstract diagrams such
as logical or flow charts are means to display what is not
visible, to make visible the invisible, that is, the internal
structure of some complex object showing its logical or
functional structure. A given representation can also
take advantage from some manipulations, extending the
visual rhetoric tools. In Figure 2, examples on how to
make visible the invisible through cuts or transparencies
are presented. Other rhetoric effects can be obtained by
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
intermixing different iconicity levels. If we exploit the
possibility of dynamical representations and interaction
in VR, another class of rhetorical effects can be obtained,
for instance by compressing or dilating time, extending
the purpose of making the invisible visible. The adoption
of a mixture of different levels aims at maximising the
expected outcome of the representation, that is, to obtain
the desired pragmatic objective; for instance, in a single
representation, photorealistic techniques can be mixed
with some kind of simplification with field diagrams or
perfect abstractions. An example of this mixture of levels
can be observed in scientific visualisation of e.g. the fluid
flow around an aircraft. The mixture of iconicity levels is
usually organised so that the highest iconic represen-
tations are aimed at describing the context, while the low-
est iconic choices aim at driving the attention to the
desired content. We note that example C in Figure 2 is
also an example of augmented reality, where VR represen-
tation is intermixed with real-world vision.
The iconic-level classification has also been explored
by Pujol (2006) to characterise the system of signs used
in VR. In particular, she extends the levels including
VR itself as an iconic level, but we consider this view
as inappropriate. Following a direct discussion with
Figure 2. Example of rhetorical manipulations: (a) intermixed levels, 3D model and perfect abstraction to show the chemical process
simulation; (b) cut-through; (c) endoscopy.
BEHAVIOUR & INFORMATION TECHNOLOGY 7
Laja Pujol, she recognised that this point needs a better
analysis. Computer Graphics evolved from technical
drawings to video games and computer animation for
entertainment. The driving force has been for a long
time the so-called quest for realism: try to create images
of scenes and objects indistinguishable from the real
thing.
Following a traditional classification (Foley et al.
1990), and extending it with recent advancements in
Computer Graphics techniques (Pharr and Humphreys
2010), the likeliness3 ranges across different visualisation
techniques that we propose in the following 10-level
classification (see Figure 3):
(1) Photorealism with optics, photometric and col-
ourimetric-based illumination models. Methods
based on accurate computer simulation of light–
material interaction can produce photorealistic
images, that is, images that are hard to distinguish
from a photograph of a real object or scene. Col-
ourimetric- and physics-based models can be
approximated using image-based rendering tech-
niques. Image-based rendering can also be con-
sidered as an augmented reality technique: the
light field properties of a scene are captured with
a photograph and mapped to a geometric model,
bringing to a new or different virtual life the
appearance of the scene. An example is given in
Figure 3 (Level 1).
(2) Impressionistic rendering. This technique is
mainly used in digital painting rather than in Com-
puter Graphics. It can be considered as a higher
level of likeliness, given the possibility that a good
graphic designer can reach exceptional results.
Recent image-based rendering techniques can
also be considered as an impressionistic approach,
they are based on information taken directly from
scene photographs or capture of the light field. See
Figure 3 (Level 2) for an impressionistic rendering
example.
 8 B. R. BARRICELLI ET AL.
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
Figure 3. Examples of likeliness levels in computer-generated images.
(3) Global illumination models (interaction of light
among the surfaces of a scene, mirror reflections,
transparency etc.). Global illumination approxi-
mates the equations of light–material interaction,
allowing simulating transparency, mirroring effects
and some optical effects such as caustics. The chess
example in Figure 3 (Level 3) well describes these
kind of models.
(4) Local illumination models (finite and infinite light
sources, point wise or extended light sources) with
real colour are a simpler equation approximation.
Local illumination models allow us to simulate
shining effects and shadows like the ones present
in Figure 3 (Level 4).
(5) Smooth shading for curved shape, with simple illu-
mination models and false or real colour. To dis-
play curved surfaces, bilinear interpolation can be
adopted, or simply interpolating colour values
computed on the basis of the simple illumination
model or by interpolating the surface normals in
any point of the surface. See Figure 3 (Level 5)
for an example.
(6) Flat shading with simple illumination model. Faces
of a shape are filled with colours or grey ; the value
(or shade) of the colour depends on the orientation
of the face towards an imaginary light source pro-
jecting parallel light rays. It is very well suited to
display the shape of the geometry as seen from a
given viewpoint, without any ambiguity. The tea-
pot and ball depicted in Figure 3 (level 6) is an
example for this level.
(7) Line drawing, hidden lines removed. To mask
invisible lines, hidden line removal (like in Figure
3, Level 7) allows us to understand the front- and
back-oriented parts of a complex geometry.
(8) Line drawing plane perspective (central and acci-
dental). Perspective projection allows us to rep-
resent the third dimension in a single drawing. It
is derived from Renaissance painting methods,
and formalised in projective geometry. Perspective

projection can have one (central perspective), two
or three vanishing points (accidental perspective).
In Computer Graphics, it is normally computed
by simulating numerically the image formation in
a pin-hole picture camera. The car line drawing
in Figure 3 (Level 8) is an example for this.
(9) 3D plane orthographic projections. To represent
three-dimensional geometries, the principle of
multiple plane projections is adopted. Derived
from projective geometry, it conveys information
about the third dimension by using multiple pro-
jections on different planes, normally by ortho-
graphic projections or plane parallel projection
like axonometric. Constructions are used as an
example in Figure 3 (Level 9).
(10) 2D line drawing. A 2D line drawing is the typical
representation of technical drawings or simple
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
illustrations. It is normally adopted for represent-
ing plane geometry or to enrich a visual interface.
Another typical representation is a map like the
one in Figure 3 (Level 10).
This classification of likeliness levels in Computer
Graphics can be put in relationship with the iconicity
levels we defined above, even if not with a one-to-one
mapping. Figure 4 illustrates the relationships existing
between the two classifications.
The only level in the iconicity classification that does
not match with a level in the likeliness classification is
Figure 4. Relationships between the iconicity and the likeliness classifications we propose in this paper.
BEHAVIOUR & INFORMATION TECHNOLOGY 9
Level 1, the object itself, that we do not consider when
analysing likeliness since the focus is on evaluating
how verisimile (or believable) a representation of an
object is.
3.2. Model: the semantic level
The model of the VR system is relative to the way the
represented world (real or imaginary) is related to the
real one; this can be a formal representation of the
world, which depends on the desired level of accuracy,
from a kind of impressionistic level up to an accurate
logic, mathematical, physical or chemical model.
The model is part of the semantic level; it incorporates
the meaning of a simulation system. When we associate
semantics to the model of a VR system, we are consider-
ing the whole behaviour, not the individual semantic of
each sign.
The problem of semantics can be tackled from three
main viewpoints (Chierchia 1997; Violi 1990): the philo-
sophical approach, where the objective is to establish
whether a sentence or an inference is true or false and
the truth values are decided on the ground of formal
logic methods; the psychological approach (cognitive
semantics), which aims at understanding the mental pro-
cesses that bring to the comprehension of the link
between messages and human experience, and studies
the relationship between the conceptual content of a
sign and the conceptual structure of human beings; the
strictly semiotic approach, which does not consider the
 10 B. R. BARRICELLI ET AL.
logical or psychological aspects, but is more depending
on sociopragmatics aspects. Sociopragmatics considers
the meaning of a message as bound to the cultural con-
text, the experience and the previous knowledge of the
receiver (Harlow 1990) and is defined by Eco as the ency-
clopaedia (1976).
In the context of VR, the semantic of a message
depends on the possibility that the user can build the
necessary encyclopaedia by getting the knowledge of
the virtual world from his/ her personal knowledge and
previous experience or through the progressive explora-
tion of the virtual world. The semantic, as characterised
by the underlying model, can also be considered as oper-
ational, like in computer programming: the meaning of
the VR system depends on the operations that the system
and the user can perform.
It is a kind of operational semantics that describe
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
how the simulation has to be interpreted. Consider as
an example a VR application to simulate an industrial
chemical plant for maintenance training purpose. We
can distinguish in this case two kinds of models: the
physical model of the structure of the plant, which is
based on the geometry, the structure and the properties
of the elements (we call it the physical model), and the
physical–chemical model of the processes that the plant
implements (in the following we call it the process
model). By the physical model of the structure, we
mean therefore the general modelling of any VR
model of a real object or a complex system, indepen-
dently by its specific function; a model must have the
basic characteristics of each component, based on its
geometry.
Fundamentally, this modelling requires mass proper-
ties necessary to compute mechanical dynamics, to
simulate for instance a fracture, the fall of an object,
the motion inertia, the effects of a collision to a given
level of precision, etc. As an example of such a complex
VR system, we can consider a flight simulator or the
simulation of an industrial chemical plant as in the pro-
ject VIRTHUALIS (Agadir and Marini 2010). The pro-
cess model, on the other hand, can be interfaced to
integrate – e.g. in a simulation of a chemical process
plant, a process simulator with the VR simulation. In
another example, a racing car game, the process
model can be the dynamics of car motion. The accuracy
of the process model depends on the scientific knowl-
edge of the real system to be simulated, and it is linked
to the granularity of the modelling. The level of iconi-
city of a chosen representation has to cope with the
accuracy and details of both the physical and the pro-
cess models. Therefore, the syntactic and the semantic
levels are strictly related. Models can moreover be cate-
gorised as:
. Mathematical : based on physics parameters necessary
to compute kinematics or dynamics (of points, rigid
bodies, fluid, etc.).
. Empirical : the physical behaviour is approximated by
probabilistic or statistic methods or even by simple
animation that approximates the evolution of some
event.
. Impressionistic : in which the model is described only
by a subset of its features, sketching in this way only
some of its more important mechanisms, like in a car-
icature, sufficient to give a meaningful depiction of the
system.
. Abstract, that is, based on a symbolic or logical rep-
resentation of a behaviour.
We can recognise that this simple taxonomy can be
observed in video gaming. In the game industry, it has
been raised the issue of playability. For instance, a
video game of racing motorcycles can be based on an
accurate solution of the dynamics of two wheels motion,
but no player would be able to keep the vehicle in
dynamic equilibrium without a long training. This is
due to the very different sensorial experience in real
vehicles rather than in virtual game. In real vehicle driv-
ing, the training is fed back to the human equilibrium
subsystem, and the trainee gains its ability by trial and
error. Having gained this kind of experience, it is extre-
mely difficult to control the position of a virtual vehicle
by putting to work the real experience: no human equili-
brium subsystem is stimulated and the only feedback
goes through vision and limited haptic perception by
moving a joystick or similar device. Games industry
has therefore developed techniques to approximate the
solution of the physics of motion dynamics to reach
the goal of playability; so the dynamics of motion is
simply approximated, giving the impression of working
with a real two-wheel vehicle. Empirical approaches are
also adopted to simulate other physical systems – such
as explosions, fire and smoke – by the so-called particle
systems.
A statistical description of the evolution of small par-
ticles (simple shapes like small spheres or triangles rep-
resented with suited colours) is programmed by
controlling few parameters such as life duration of a par-
ticle, initial velocity, number of newborn particles per
time unit, presence of perturbation fields such as gravita-
tion and wind. A large variety of natural effects can be
simulated, e.g. fire, smoke, explosion, water pipes rup-
tures, vapour or water emission under pressure. Due to
the high complexity of the context of many video
games and other professional simulations, the impressio-
nistic approach is often necessary. Exactly like

impressionistic painting, the degree of involvement and
likeliness they raise can be very high.
The abstract approach can also be exemplified by
games: the game logic takes control of the evolution of
the game on the basis of user interaction, but is very
rarely based on some mathematical representation of
the game, rather on some artificial intelligence inspired
method. In this way, a player controlled by the system
can behave like a living player who acts following a tactic
and a strategy that mimics possible human tactics and
strategy.
We also have to consider the way human beings can
receive the stimuli that encode the message: the human
perception. Users of a VR system have their own cogni-
tive framework and experience that can be far from being
common to each other. VR is not actual reality, where
people have developed a behaviour based on their per-
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
ception training: to interact with a VR world, a user
needs to learn a new perceptual context, based on non-
natural stimulation processes that act on the sensory sys-
tem in new and unexpected ways, conflicting sometimes
with the usual cognitive framework that, in turn, has
grown up with years of experience in a real environment.
3.3. Interaction: the pragmatic level
Compared to traditional static drawing, in VR the rep-
resentation of the functions of a system has the advan-
tage provided by the dynamics, where a process
evolution and the interaction can be realised by moving
images and by animation. It is the interaction that
implements the pragmatic level; moreover, it contributes
to exploit the dynamic aspect of a VR system, which, on
the one side, depends on the underlying model and, on
the other side, is controlled and steered by a user.
It is by means of interaction that the VR user becomes
an actor of the VR simulation, who plays the role
assigned by the VR designer. The user experience
becomes the participation to a storytelling that has
been put on stage by the designer like a theatre director,
to stimulate emotions and to convey the desired message.
The dynamics of a VR system is a function of a time vari-
able, and differently from a real world, in VR the time
can be dilated or compressed. This particular property
of VR is able at deeply affecting pragmatics by contribut-
ing in the construction of a rhetoric artefact and to
design the desired narrative effect.
The interaction aspect of a VR system depends on a
user action and in this view the user becomes an actor
that takes active part on the virtual story. Before discuss-
ing deeper the semiotic aspect of the narrative nature of
VR interaction, we recall that HCI can be implemented
by adopting a wide variety of techniques and paradigms
BEHAVIOUR & INFORMATION TECHNOLOGY 11
(Preece et al. 1994), ranging from command-based to
direct manipulation and more recently on the use of
specific tools that can capture human movement of
hands and fingers and even whole body. Such devices (
also called phidgets or physical widgets) can be very
sophisticated, including powerful computational capa-
bilities, which can enhance the interaction with the sur-
rounding real world, when used in augmented virtual
reality applications. We cannot discuss this aspect in
this paper, since it goes beyond our scope.
In general, we assume that immersive VR systems
adopt direct manipulation paradigm as well as phidgets
to make the user interaction as immediate as current
technology allows. Interaction techniques and paradigms
can be used in a mixed way, for instance a direct manipu-
lation approach can make extensive use of widgets or
phidgets, or even natural language or body language.
In general, interaction is very demanding in terms of
user understanding, and requires some training. The
concept of affordance proposed by Gibson (1979) and
further developed by Norman (1988) is the most suited
solution to assess effective interaction. Affordance, that
is, the actionable properties between the world and an
actor, can be considered from the semiotics viewpoint:
in this case we can observe a situation where associating
a meaning to the system of signs provided by the VR
application is a typical pure signification problem,
where previous knowledge and experience are the
elements of the user encyclopaedia. A specific rhetoric
structure can be adopted, the metaphor, that simplifies
user interpretation of the meaning of possible com-
mands, possible states of the system, etc. As discussed
early in the paper, interaction is the pragmatic level of
VR semiotic.
In the theory of communication, pragmatics is the
discipline that explores the relationship between signs
and their meaning to be decoded by the participant to
the communication process, in particular solving poss-
ible ambiguities. Each human being has his/ her experi-
ence and during a communication act each participant
has his/ her context, beliefs, knowledge, emotions and
affections. In human communication, each act can be
an assertion, an imperative command or a desire; nor-
mally in speech communication intonation or non-ver-
bal communication (e.g. expression of the face or
gestures) contributes to convey the characteristic of the
communication act. In VR, communication is between
a human being and a computer system, where verbal
communication is almost absent. VR can also be used
as the communication medium between two or more
human beings. In this case, it is necessary to provide
clues for transmitting non-verbal information among
the participants.
 12 B. R. BARRICELLI ET AL.
The use of avatars can be a solution to provide a richer
communication of a single user with a computer system,
as well as video capturing and transmission of facial
expressions. The lack of this kind of information in a
VR communication can be overcome with other
means, like the richness of information conveyed by
the visual and non-visual channels. An important aspect
in pragmatic level is the emotional state of users because
it directly affects their ability to decode information, and
is influenced by experiences or common and/or shared
feelings. Emotions represent also an outcome of the
communication act. Some experiments demonstrated
that in VR, during simulations of critical tasks (e.g. con-
cerning security, maintenance or educational sessions),
the subjects could feel emotions very similar to those
that can be experienced in real episodes (Agadir and
Marini 2010).
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
In a collaborative VR environment, in which different
users are remotely collaborating to the same task with the
mediation of the VR application, the introduction of an
emotional feedback in the communication can improve
performances of the simulation, for instance by changing
conveniently the graphical representation of the users’
avatars, for example, mapping video-captured facial
expression. In the literature, many examples prove that
personalisation plays an important role in enhancing
the quality of virtual collaborative experiences, especially
those focused on learning (e.g. Dascalu et al. 2014, 2015).
Emotions amplify the sensation of presence; so the prag-
matic level has to be coherently designed allowing
emotions exchange among participants and between
users and VR environment. To this aim, emotional
stimuli and feedback must be under control, so that plan-
ning an effective VR means, at the pragmatic level, to
design a kind of ‘storytelling’ defining the boundaries
and degrees of freedom in interaction among users and
with the VR environment. For example, a study (St-Jac-
ques, Robillard, and Renaud 2008) has shown that
anxiety increases the sense of presence in VR; so when
simulating a high-risk industrial site, at the pragmatic
level we have to consider the different answers of the sys-
tem to a normal situation or to an accident, providing all
the possible evolutions of the states of the system follow-
ing users’ reactions and emotions. Degrees of freedom, in
turn, depend on the mathematical model implemented
at the syntactic level. Designing an ‘emotional’ VR
implies the analysis of the scope of the virtual simulation,
so that specific stimuli could be implemented and con-
trolled in the VR story to cause the desired emotional
feedbacks in users.
Storytelling has been extensively explored by semiotic
methods, and it is also studied in video-making for enter-
tainment and advertising (Vogler 2007). We recall some
elements of the semiotic theory of narrative structures by
Greimas (1983) and Genette (1972), trying to recognise
some of its elements in VR systems. To better under-
stand, we consider a VR application for training (e.g.
flight simulation or industrial plant maintenance train-
ing). The first element of a narrative structure is the
plot or storyline; it is the deep structure of the story
based on a given temporal and causal sequence. We
can consider the operational sequence for an aircraft
takeoff as the plot. At a higher level, the trainee who,
by changing the given sequence, creates unexpected out-
comes and a change in the takeoff story evolution can
modify the plot. Here, we see a difference from a
movie story: interactivity offers the opportunity to create
a new story every time the VR application is run. Eco
calls the storyline fabula, while a specific narrative
expression is the plot (in Italian trama and intreccio)
(1979).
Another semiotic aspect of narrative structures is the
focalisation: the focalisator is the person whose view-
point is adopted to present the story. In general, there
are three kinds of focalisation: zero, when no person
viewpoint is chosen and the storyteller knows all about
the inner thinking and psychology of all persons;
internal, when one of the persons of the story plays the
role of the storyteller, external when the viewpoint is
objective, when no psychological aspects of any persons
of the story are described, as in a thriller story. In VR, we
are generally faced with internal focalisation: the user of
the VR system is the focalisator.
A VR application is based on some artificial world that
is called a possible world, a concept deriving from the the-
ory of modal logic. Volli (2005) proposes four kinds of
possible worlds: verisimili that can be easily conceived
and could correspond to some real world, non-verisimili
when a suspension of disbelief is needed, non-conceivable
when some logical rule is broken (think, for instance, of
Alice in Wonderland by L. Carrol), and impossible as a
drawing by Escher. In VR, we have the possibility to cre-
ate all four kinds of possible worlds, and the training
example is that of a verisimile world, while the flying ava-
tar in Second Life is an impossible world. It is interesting
to note that this classification can be extended also to
video games.
The semiotic theory of narrative structure of Greimas
is based on the actantial model that describes the roles
that are played in a story, without considering the effec-
tive characters, since an actant can also correspond to no
character at all, like the magic wand in a fantasy movie,
that plays the actantial role of a helper for the hero. Grei-
mas derived this theory from the works by Propp (1958)
and generalising his findings. In movie production, this
theory has been further extended for example by Vogler

(2007). In VR application, the user plays normally the
actantial role of the hero, but it is difficult to explore dee-
per this aspect without analysing specific VR appli-
cations to identify helpers, opposers, senders, etc. It is
interesting to observe that the fundamental narrative
sequence of a narrative structure, as proposed by Grei-
mas, can be recognised in a ‘getting started’ sequence
of actions to learn how to use a VR interactive system.
First, the user is faced with the manipulation stage
where the user is assigned a goal, the competence stage
where the user acquires the ability to do the actions
necessary to reach the goal, the performance where the
user does the actions needed and the sanction when
the user has obtained the goal and possibly a premium.
4. A framework to classify VR applications
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
As an outcome of our work, we propose a way to classify
different VR applications, based on the above-described
semiotic analysis. The classification will allow us to
characterise semiotic content and characteristics, consid-
ering syntax, semantics and pragmatics. The framework
can help us to understand and maybe also design a VR
system to evaluate its ability to solve the given communi-
cation problem.
To build up the framework, we propose a three-
dimensional space, where the three axes represent the
range of variation of Structure (or syntax), Model
(or semantic) and Interaction (or pragmatics
representation).
Figure 5. The VR framework is a three-dimensional space constituted by eight octants.
BEHAVIOUR & INFORMATION TECHNOLOGY 13
The Structure axis is relative to the syntactic level,
which ranges from symbolic to highly realistic (better
called likely). To identify a position in this axis, we
need to consider the iconicity level and the likeliness
level of Computer Graphics solution adopted as well:
the iconicity level helps in locating the position, while
the likeliness level suggests possible Computer Graphics
solution to obtain the desired iconicity.
The Model axis is relative to the semantic level; it
ranges from mathematical to impressionistic. To identify
a position in this axis, we will consider the detail level of
the underlying mathematical, physical or chemical
model that rule the evolution of the VR world, or the
presence of symbolic or logical model of the evolution.
The Interaction axis is relative to the pragmatic level;
it ranges from abstract to concrete, considering also the
narrative aspect of communication of the system. To
identify a position in this axis, we consider the inter-
action approach and the interaction devices that exercise
different sensory systems.
The VR framework space is therefore organised in a
three-dimensional space constituted by eight octants,
shown in Figure 5 and described in what follows.
4.1. Octant 1 (1, 1, 1)
This first octant corresponds to the combination of Con-
crete (Interaction), Likely (Structure) and Mathematical
(Model). Flight simulators are the best example to
describe this octant, in that they have concreteness of
interaction and navigation, likely structure and
 14 B. R. BARRICELLI ET AL.
mathematical level for the modelling. In this case, the
narrative structure is very complex and is composed of
two phases. The first is when the briefing of the simu-
lation is provided to the person by a trainer who
describes the evolution of the simulation with a zero
focalisation. The second phase is during the actual simu-
lation when we have internal focalisation, centred on the
person. The evolution of the second phase can also differ
from the story told during the briefing.
4.2. Octant 2 (−1, 1, 1)
The octant is characterised by Abstract (Interaction),
Likely (Structure) and Mathematical (Model). CAD/
CAM systems, for example, offer simple interaction
based on keyboard and mouse use but permit to access
high likeliness and quite accurate modelling features.
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
The focalisation is internal.
4.3. Octant 3 (−1, −1, 1)
The third octant combines Abstract (Interaction), Sym-
bolic (Structure) and Mathematical (Model). Scientific
visualisation can exemplify this octant where interaction
is low but can be increased providing steering tools for
changing configuration parameters of the underlying
mathematical model. The structure is symbolic and the
model is very sophisticated. The narrative structure
appears with an internal focalisation but it becomes
external in case the use is only for passive observation
with the steering tools limited to navigation.
4.4. Octant 4 (1, −1, 1)
Octant 4 is the combination of Concrete (Interaction),
Symbolic (Structure) and Mathematical (Model). Indus-
trial plant simulators are the optimal example: the inter-
action is slightly concrete (menu- and widget-based
interface, no sensory devices); the structure of represen-
tation is more symbolic (iconicity level 3 constructive
drawing, likeliness level 4 – mixed representation with
symbolic colour codes); the model is usually a very accu-
rate physics or mathematical model. There is a low level
of immersivity and also a limited narrative content but
focalisation can vary: (a) zero, in case of plant explora-
tion, (b) internal, if the simulation is used for training
(an example is presented in Agadir and Marini 2010)
and (c) external, if the user (typically the designer of
the plant) uses the simulator to discover an existing pro-
blem. This field of application of VR is growing and
growing, thanks to the potentials that technology offers
for simulating critical situations. A study on VR adop-
tion for Risk Perception Simulation is published in
Nedel et al. (2016), where the authors discuss the design
and use of simulators to continuously assess capability
for safe behaviour among workers.
4.5. Octant 5 (1, 1, −1)
To the octant corresponds the combination of Concrete
(Interaction), Likely (Structure) and Impressionistic
(Model). An example for this octant can be found in
some augmented reality applications. Here, the inter-
action required is quite low (limited to simple naviga-
tion), the structure is highly realistic and the model is
rather simple. From narrative point of view, the focalisa-
tion is usually zero. Focalisation can also be external, for
instance in case of augmented surgery application.
4.6. Octant 6 (−1, 1, −1)
The sixth octant combines Abstract (Interaction), Likely
(Structure) and Impressionistic (Model). As examples,
there is realistic simulation of car or home furniture con-
figurators: the user is required to interact in a mostly
symbolic way (selection of items from lists and naviga-
tion to observe the final result), the structure is based
on photorealism and this makes it highly likely, and
the model is impressionistic – for example, dynamics
or kinematics are not in place. In this case, the focalisa-
tion is internal.
4.7. Octant 7 (−1, −1, −1)
Octant 7 is the combination of Abstract (Interaction),
Symbolic (Structure) and Impressionistic (Model). Sys-
tems for museums virtual visits can be used as examples
for this octant. They uses multiple photographs, texture
mapped on virtual shapes to navigate into a scene or
around an object. Their interaction is based on direct
manipulation but with a low degree of immersivity; the
structure is rather symbolic; the model is impressionistic.
The narrative structure is very low level: the user can
simply explore the representation with a zero
focalisation.
4.8. Octant 8 (1, −1, −1)
The last octant combines Concrete (Interaction), Sym-
bolic (Structure) and Impressionistic (Model). A good
example for this octant are some sport video games
using natural interaction devices: concreteness of inter-
action based on the use of controllers (e.g. Wiimote), sym-
bolic (low level) structure and impressionistic level of the
underlying model. The narrative structure can be very

rich, with a variety of actants and different focalisation. In
general, it can be said that focalisation is internal.
5. Classification of learning and collaboration
VR applications
In what follows, we use the VR framework to classify
eight VR applications meant to be used for learning
and collaboration. Table 1 shows each octants ’ charac-
teristics and gives an example of application.
5.1. FlyInside FSX Plugin: Octant 1
With its concrete interaction, likely structure, and math-
ematical model, the FlyInside FSX Plugin for Microsoft
Flight Simulator X4 brings flight simulation into virtual
reality, thanks to the use of Oculus Rift. This particular
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
VR application represents a significative example of
VR learning application that can also be extended to a
collaborative use and can be positioned in Octant 1.
5.2. Improov: Octant 2
Improov5 is a software that supports CAD engineers in
collaborating through virtual reality. Among the features
that it offers, there are online real-time collaboration and
annotations. This makes the application an example of
VR collaboration tool that can also be seen as a learning
tool, thanks to its peculiar features. This software is
characterised by an abstract interaction, a likely structure
and a mathematical model that positions it into Octant 2.
5.3. VMD into Reality Cube: Octant 3
Visual Molecular Dynamics (VMD)6 is a system for mol-
ecular visualisation that displays, animates and analyses
large biomolecular systems using 3-D graphics and
built-in scripting. If ported into the Reality Cube, up to
five persons can be immersed in the VMD system and
interact with it .7 This VR application is an excellent
example of learning VR tool. VMD stands in Octant 3
and therefore presents an abstract interaction, a symbolic
structure and a mathematical model.
Table 1. The octants in the VR frameworks with examples of VR applications.
OCTANT INTERACTION STRUCTURE
1 (1,1,1) Concrete Likely
2 (−1,1,1) Abstract Likely
3 (−1,−1,1) Abstract Symbolic
4 (1,−1,1) Concrete Symbolic
5 (1,1,−1) Concrete Likely
6 (−1,1,−1) Abstract Likely
7 (−1,−1,−1) Abstract Symbolic
8 (1,−1,−1) Concrete Symbolic
BEHAVIOUR & INFORMATION TECHNOLOGY 15
5.4. VIRTHUALIS: Octant 4
The VIRTHUALIS research project8 was aimed at eval-
uating human factors in risk assessment and evaluation
during industrial processes. Specifically, a VR industrial
chemical plant simulator was developed to allow its
users to check the proposed solutions for the treatment
of an emergency, like a fire or an explosion. The appli-
cation can be classified as a VR learning environment.
Its interaction is concrete, the structure is symbolic,
and the model is mathematical; therefore, it can be
seen as positioned in Octant 4.
5.5. MedicAR: Octant 5
MedicAR9 is an example of augmented surgery appli-
cation that, thanks to the use of Google Glass, allows
us to improve the simulation and teaching of procedures
in surgery. This application makes extensive use of the
rhetorical manipulations ‘cut-through’ and ‘endoscopy’.
It presents concrete interaction, likely structure and
impressionistic model and therefore belongs to appli-
cations in Octant 5. Given the primary importance of
VR applications in learning and collaboration for medi-
cal practice, it is worth pointing out that recently many
systems have been developed to use immersivity (two
notable examples are reported in Byrne et al. 2015 and
Srivastava, Das, and Chaudhury 2014).
5.6. Dollhouse VR: Octant 6
Dollhouse VR10 is a VR system that allows experts in
architecture and interior design to design houses alter-
nating top-down and internal views of the virtual spaces.
The application is collaborative and is meant to be used
also for learning purposes. It offers an abstract inter-
action, likely structure and impressionistic model and
is therefore located in Octant 6.
5.7. Etruscans@EXPO: Octant 7
During the Etruscans@EXPO exhibition11 that was held
in Milan (Italy) in 2015, the visitors had the possibility to
MODEL EXAMPLE
Mathematical FlyInside FSX Plugin
Mathematical Improov
Mathematical VMD into Reality Cube
Mathematical VIRHTUALIS
Impressionistic MedicAR
Impressionistic Dollhouse VR
Impressionistic Etruscans@EXPO
Impressionistic The Lab
 16 B. R. BARRICELLI ET AL.
explore a life-size replica of the Etruscan’s Tomb of the
Deathbed, an Etruscan painted tomb, in augmented rea-
lity realised through the use of Google Glass. This appli-
cation is an example in the field of VR for learning. Since
the interaction was abstract, the structure symbolic and
the model impressionistic, this application can be classi-
fied as belonging Octant 7.
5.8. The lab: Octant 8
The Lab12 is a suite of VR mini-games for HTC Vive, a
VR headset. The specific archery game represents a
case of virtual training for early learning in VR. This
can be seen as an example of application in Octant 8,
with concrete interaction, symbolic structure and
impressionistic model.
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
6. Conclusion
We have presented a semiotic analysis of VR systems,
based on some concepts of visual communication –
mainly iconicity level – and of interaction and modelling
of dynamic evolution of the VR simulation. After a pres-
entation of the semiotic studies applied to VR, we have
introduced the three levels of analysis: syntactic, seman-
tic and pragmatic. For the syntactic level, we have pre-
sented an adaptation of a theory on iconicity levels that
considers also dynamics aspects and visual rhetoric
effects.
The discussion on the semantic level, that starts from
the concept of encyclopaedia, is based on the role of the
perceptual system that has to cope with a non-natural
world and on the functions provided by the underlying
models. The adopted modelling approach drives the
understanding of the system in terms of its functions
and operations.
The pragmatic level is strictly linked to user’s inter-
action, which is strongly influenced also by the
emotional involvement of users in VR tasks, who partici-
pate to a specific narration. The structure of the storytell-
ing is therefore part of the pragmatic nature of a VR
system, as well as the focalisation of the story and the
nature of the represented virtual world.
From this analysis, we obtain a framework where a
specific VR application can be located in terms of par-
ameters that allow us to identify the expressive power
of the communication solution provided by VR. At the
same time, the framework allows us to identify the
best-suited VR applications to get a specified communi-
cation objective, by adopting the appropriate techniques
to implement visualisation, interaction and modelling of
the VR application.
We finally showed eight different VR applications for
learning and/or collaboration and we used the VR frame-
work for classifying them.
The framework addresses the communication design
in VR, stressing the importance of being conscious of
how critical is an adequate design of the way communi-
cation is implemented. It represents both a tool for eval-
uating existing VR applications and for supporting
designers of VR systems in their decision-making pro-
cesses. To exploit the framework potentials, designers
and developers have to select the appropriate level of
detail and likeliness in visualisation, interaction and
modelling, choosing the appropriate sensory stimulation
systems, determining the necessary languages to be pro-
posed to perform human–VR communication.
The need of this kind of classification is becoming of
relevant importance due to the current diffusion of con-
sumer-oriented VR devices for interactive and immer-
sive visualisation. In fact, the entertainment industry is
increasingly including technological and technical sol-
utions typical of the VR field in order to produce a
new generation of products, such as immersive video
games, and 360W stereoscopic movies. In this novel con-
text, the proposed framework can contribute in the
identification of the more effective approaches not only
in the VR field, but in all the possible applications
where VR technologies are currently considered as a
potential communication media. The decreasing costs
and prices of VR technology, especially for gaming, is
blurring the boundaries between entertainment and edu-
cational use of VR tools. Serious games represent a viable
delivery system for training activities and, like stated in
Zyda (2005), it might be worth investigating the possi-
bility of incorporating all K-12 science and math edu-
cation in highly immersive, addictive games – moving
from ‘first-person shooter’ to ‘first-person education’
games. Recently, this same trend has been confirmed in
Perry (2016), where the author explains how the future
of VR will be focused more on meeting people, and
thus collaborating, rather than on playing games.
As future works, we intend to use the VR framework
and its dimensions to design new usability evaluation
methods specialised for VR applications. We aim at
designing and testing two types of methods, one predic-
tive to be adopted by usability and interaction experts as
tool in early design phases, and one that involves users.
To maintain the connection with our Semiotic perspec-
tive on VR and communication, we intend to capitalise
on the existing Semiotic Engineering evaluation methods
semiotic inspection method and communicability evalu-
ation method developed by de Souza and Leitao (2009)
and to extend them to consider not only the act of inter-
acting with a system in a specific time and technical

setting, but also the immersion in a virtual world, where
time, space and presence constitute peculiar use settings.
Notes
1. Neil Harbisson’s TED talk (visited May 30, 2016):http://
www.ted.com/talks/neil_harbisson_i_listen_to_color
2. Iconicity in semiotics is defined as the analogy between
the form of a sign and the concept it refers in our per-
ception of the world (Bouissac 1998)
3. As discussed earlier in this paper, realism is not a direct
correspondence of a graphical representation to a given
object, rather it is a question of style and rhetoric choice.
Therefore, to avoid confusion, we classify the different
representation style of Computer Graphics with the
term of likeliness levels, thus with a more direct corre-
lation to the notion of verisimile or believable.
4. FlyInside FSX Plugin (visited May 30, 2016): https://
flyinside-fsx.com/
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
5. Improov (visited May 30, 2016): http://www.middlevr.
com/improov/
6. VMD (visited May 30, 2016): http://www.ks.uiuc.edu/
Research/vmd/
7. Reality Cube (visited May 30, 2016): http://www.rug.nl/
society-business/centre-for-information-technology/
research/hpcv/faciliteiten/realitycube
8. VIRTHUALIS, VI PQ, contract n. 515831-2
9. MedicAR (visited May 30, 2016): http://droide.rs/
stanford
10. Dollhouse VR (visited May 30, 2016):https://hikaru
23ibayashi.files.wordpress.com/2015/08/sigasia15_dhvr.
pdf
11. Etruscans@EXPO (visited May 30, 2016): http://www.
etruscansatexpo.eu
12. The Lab (visited May 30, 2016): http://store.steam
powered.com/app/450390/
Disclosure statement
No potential conflict of interest was reported by the authors.
References
Agadir, N., and D. Marini. 2010. “Decision Making
Assessment in Industrial Process Simulation.” Chemical
Engineering Transactions 19: 403–408.
Anceschi, G. 1992. L’oggetto Della Raffigurazione. Etas.
Architettura e territorio.
Aruga, M., A. Egawa, T. Takeda, S. Ono, and H. S. Egawa.
2007. “A Consideration of Information Content on the
Basis of Semiotics Taking Account of Virtual Reality.” In
IEEE International Conference on Industrial Engineering
and Engineering Management, edited by M. Helander, M.
Xie, R. Jiao and K. C. Tan, 563–567. Singapore: IEEE
Belanger, W. D. 2009. “A Semiotic Analysis of Virtual Reality.”
PhD thesis, Salve Regina University. http://digitalcommons.
salve.edu/dissertations/AAI3376468.
Bouissac, P. 1998. Encyclopedia of Semiotics. New York:
Oxford University Press.
BEHAVIOUR & INFORMATION TECHNOLOGY 17
Bowman, D. A., and R. P. McMahan. 2007. “Virtual Reality:
How Much Immersion is Enough?” IEEE Computer 40
(7): 36–43.
Byrne, J., H. H. Ip, K. Shuk-Ying Lau, R. Chen Li, A. Tso, and
C. Choi. 2015. “InSPAL: A Novel Immersive Virtual
Learning Programme.” Student in Health Technology and
Informatics 219, 129–134.
Chierchia, G.. 1997. Le Strutture del Linguaggio – Semantica. Il
Mulino.
Coelho, C., J. Tichon, T. J. Hine, G. Wallis, and G. Riva. 2006.
“Media Presence and Inner Presence: The Sense of Presence
in Virtual Reality Technologies.” In From Communication
to Presence: Cognition, Emotions and Culture towards the
Ultimate Communicative Experience, edited by C. Riva, M.
T. Anguera, B. K. Wiederhold and F. Mantovani, 25–45.
Amsterdam: IOS Press.
M.-I . Dascalu, C.-N . Bodea, Lytras, M., P. O. de Pablos, and A.
Borlacu. 2014. “Improving e-Learning Communities
through Optimal Composition of Multidisciplinary
Learning Groups.” Computers in Human Behavior 30,
362–371.
M.-I . Dascalu, C.-N . Bodea, Moldoveanu, A., A. Mohora, M.
Lytras, and P. O. de Pablos. 2015. “A Recommender Agent
Based On Learning Styles for Better Virtual Collaborative
Learning Experiences.” Computers in Human Behavior 45,
243–253.
de Souza, C. Sieckenius. 1993. “The Semiotic Engineering of
User Interface Languages.” International Journal of Man–
Machine Studies 39 (5): 753–773. http://dx.doi.org/10.
1006/imms.1993.1082.
de Souza, C. Sieckenius, S. Barbosa, J. Diniz, and R. O. Prates.
2001. “A Semiotic Engineering Approach to HCI.” In CHI
’01 Extended Abstracts on Human Factors in Computing
Systems, Seattle, Washington, CHI EA ’01, 55–56.
New York, NY: ACM. doi:10.1145/634067.634104.
de Souza, C. S., and C. F. Leitao. 2009. Semiotic Engineering
Methods for Scientific Research in HCI. Princeton, NJ:
Morgan and Claypool.
Eco, U. 1976. A Theory of Semiotics. Bloomington: Indiana
University Press.
Eco, U. 1979. Lector in Fabula. Milano: Bompiani.
Eco, U. 1989. The Open Work. Cambridge, MA: Harvard
University Press.
Alfaro, A., Á. Bernabeu, C. Agulló, J. Parra, and E. Fernández.
2015. “Hearing Colors: An Example of Brain Plasticity.”
Frontiers in Systems Neuroscience 9 (56): 1–9. doi:10.3389/
fnsys.2015.00056.
Foley, J. D., A. van Dam, S. K. Feiner, and J. F. Hughes. 1990.
Computer Graphics: Principles and Practice. 2nd ed. Boston,
MA: Addison-Wesley Longman.
Genette, G. 1972. Figures III. Paris: Aux Editions du Seuil.
Genette, G. 1997. Paratexts: Thresholds of Interpretation.
Cambridge: Cambridge University Press.
Gibson, J. J. 1979. The Ecological Approach to Visual
Perception. Boston, MA: Houghton Mifflin.
Greimas, A. J. 1983. Du sens II: essais sémiotiques. Paris: Seuil.
Han, H. C. 2011. “Education, Semiotics, and the Virtual World
of Second Life.” International Journal of Arts Education 9
(2): 53–57.
Harlow, L. L. 1990. “Do They Mean What They Say?
Sociopragmatic Competence and Second Language
Learners.” The Modern Language Journal 74 (3): 328–351.
 18 B. R. BARRICELLI ET AL.
Lemke, J. L. 1999. “Typological and Topological Meaning in
Diagnostic Discourse.” Discourse Processes 27 (2): 173–185.
Marini, D., R. Folgieri, D. Gadia, and A. Rizzi. 2011. “Virtual
Reality as a Communication Process.” Virtual Reality 16
(3): 233–241.
Matusitz, J. 2005. “Deception in the Virtual World: A Semiotic
Analysis of Identity.” Journal of New Media and Culture 3
(1): http://www.ibiblio.org/nmediac/winter2004/matusitz.
html.
McCann, J. J., and A. Rizzi. 2011. The Art and Science of HDR
Imaging. New York: Wiley.
Moles, A. 1969. L’affiche dans la societe urbaine. Paris: Dunod.
Nadin, M. 2012. “Reassessing the Foundations of Semiotics:
Preliminaries.” International Journal of Signs and Semiotic
Systems 2 (1): 1–31.
Nedel, L., V. C. de Souza , A. Menin, L. Sebben, J. Oliveira, F.
Faria, and A. Maciel. 2016. “Using Immersive Virtual
Reality to Reduce Work Accidents in Developing
Countries.” IEEE Computer Graphics and Applications 36
(2): 36–46.
Downloaded by [Barbara Rita Barricelli] at 23:34 25 July 2016
Norman, D. 1988. The Psychology Of Everyday Things. New
York: Basic Books.
North, M. M., and S. M. North. 2016. “A Comparative Study
of Sense of Presence of Virtual Reality and Immersive
Environments.” Australasian Journal of Information
Systems 20, 1–15.
O’Neill, S. 2008. Interactive Media: The Semiotics of Embodied
Interaction. London: Springer.
Peirce, C. 1932. Collected Papers of Charles Sanders Peirce. 1–6
vols. Cambridge, MA: Harvard University Press.
Perry, T. S. 2016. “Virtual Reality Goes Social.” IEEE Spectrum
53 (1): 56–57.
Pharr, M., and G. Humphreys. 2010. Physically Based
Rendering: From Theory to Implementation. 2nd ed. San
Francisco, CA: Morgan Kaufmann.
View publication stats
Preece, J., Y. Rogers, H. Sharp, D. Benyon, S. Holland, and T.
Carey. 1994. Human–Computer Interaction. New York:
Pearson.
Propp, V. 1958. Morphology of the Folktale. Bloomington:
Indiana University.
Pujol, L. 2006. “Arqueologia, museus i ordinadores:
Aproximació semiòtica a l’ús de la Realitat Virtual per la
difusió de l’Arqueologia als museus.” PhD thesis,
Departament de Prehistoria, Universitat Autònoma de
Barcelon.
Sjöström, J., and G. Goldkuhl. 2003. “The Semiotics of User
Interfaces: A Socio-Pragmatic Perspective.” In Virtual,
Distributed, and Flexible Organisations: Studies in
Organisational Semiotics; 6th International Workshop on
Organisational Semiotics. Dordrecht: Kluwer.
Sonesson, G. 1998. “That There are Many Kinds of Iconical
Signs.” Visio 1 (1): 33–54.
Srivastava, K., R. C. Das, and S. Chaudhury. 2014. “Virtual
Reality Applications in Mental Health: Challenges and
Perspectives.” Industrial Psychiatry Journal 23 (2): 83–85.
St-Jacques, B. J., G. Robillard, and P. Renaud. 2008. “Anxiety
Increases the Feeling of Presence in Virtual Reality.”
Presence: Teleoperators and Virtual Environments 17 (4):
376–391.
Valtolina, S., B. R. Barricelli, and Y. Dittrich. 2012.
“Participatory Knowledge-Management Design: A
Semiotic Approach.” Journal of Visual Languages and
Computing 23 (2): 103–115.
Violi, P. 1990. “Discourse Analysis and Semantics.” Text 10
(1/2): 121–123.
Vogler, C. 2007. The Writers Journey: Mythic Structure for
Writers. 3rd ed. Studio City, CA: Michael Wiese.
Volli, U. 2005. Manuale di Semiotica. Roma: Laterza.
Zyda, M. 2005. “From Visual Simulation to Virtual Reality to
Games.” Computer 38 (9): 25–32.