See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/234820740

Real Virtuality: A step change from virtual reality

Article · April 2009

DOI: 10.1145/1980462.1980466

CITATIONS READS

36 4,616

3 authors:

Alan Chalmers David M Howard

The University of Warwick Royal Holloway, University of London
382 PUBLICATIONS 6,257 CITATIONS 404 PUBLICATIONS 3,559 CITATIONS

SEE PROFILE SEE PROFILE

Christopher Moir
The University of Warwick
3 PUBLICATIONS 65 CITATIONS

SEE PROFILE

All content following this page was uploaded by David M Howard on 23 December 2018.

The user has requested enhancement of the downloaded file.

 Real Virtuality: A step change from Virtual Reality
Alan Chalmers∗ David Howard † Christopher Moir‡
The Digital Laboratory, WMG Dept. of Electronics WMG
University of Warwick, UK University of York, UK University of Warwick, UK

Figure 1: The smell process from source, to capture, to delivery.

Abstract Real Virtuality is defined as a true high-fidelity multi-sensory vir-

Humans perceive the world with all five senses: visuals, audio,
smell, touch and taste. Crossmodal effects, i.e. the interaction of
the senses, can have a major influence on how environments are be-
ing perceived, even to the extent that large amounts of detail of one
sense may be ignored when in the presence of other more dominant
sensory inputs. Real Virtuality environments (also known as there-
realityT M ) are true high-fidelity multi-sensory virtual environments
which provide the same perceptual response from viewers as if they
were actually present, or “there” in the real scene being portrayed.
Unlike traditional virtual reality environments, Real Virtuality al-
lows all five senses to be stimulated concurrently in a natural way.
This paper gives an overview of Real Virtuality, describes how such
a system may be achieved, and shows why Real Virtuality is indeed
a step-change from current virtual reality systems.
CR Categories: I.3.3 [Computer Graphics]: Picture/Image
Generation—Viewing Algorithms; I.3.7 [Computer Graphics]:
Three-Dimensional Graphics and Realism - Animation— [I.4.8]:
Image Processing and Computer Vision—Scene Analysis
Keywords: Real Virtuality, there-reality, Virtual Reality, high-
fidelity virtual environments, multi-modal
tual environment that evokes the same perceptual response from a
viewer as if he/she was actually present, or “there”, in the real scene
being depicted. Also known as “there-reality” [Chalmers et al.
2007], such environments are interactive and based on physics. All
five senses are concurrently stimulated to deliver real world modal-
ities naturally and in real time.
This is a leap forward from existing 3D virtual reality solutions.
Real Virtuality opens up a broad remit of applications covering
low carbon economy, healthcare, ageing population, recreational
tourism, gaming, shopping and home entertainment. For example,
Real Virtuality will allow remote business partners to meet “face-
to-face” and families to experience exotic locations throughout the
world without having to actually travel; thus reducing the global
carbon footprint, or allow the elderly, immobile and isolated people
to improve the quality of their lives through experience enrichment.
There are two key challenges that Real Virtuality faces:
• how to capture/model, store, transmit and deliver real world
modalities in real time; and,
• how to facilitate the adoption and widespread dissemination
of such a technology, making it contribute to the global econ-
omy by providing low-cost, high confidence, high quality
multi-sensory knowledge directly to a user’s current location.

1 Introduction 2 Multi-sensory perception

A human’s perception of the world depends on all five senses: vi-
suals, audio, smell, feel and taste. The interaction of the senses,
the so called crossmodal effects, influences our perception of an
environment. A classic example of how our senses can trick each
other, is the ventriloquism effect in which the viewer is fooled into
thinking that the sound source emanates from a visual cue [Howard
and Templeton 1966]. Even our sense of taste can be affected by
other dominant sensory stimuli. For example Zampini and Spence
showed that stale potato crisps tasted fresher when they were ac-
companied by an electronically generated “crispy” sound [Zampini
and Spence 2004]. For this work Zampini and Spence won the 2008
Ig Nobel award for Nutrition.
∗ e-mail: alan.chalmers@warwick.ac.uk
† email: dh@ohm.york.ac.uk
‡ email: christopher.moir@warwick.ac.uk
Most research in sensory perception to-date has been uni-sensory,
focussing on the functional properties of only one sense. It is only
recently that researchers have recognized that, in the real world
(as opposed to a laboratory environment), the human brain sorts
through all sensory input to couple signals that relate to a common
event. This is done concurrently while processing the separate sen-
sory input [Calvert et al. 2004].
In 1993, Stein and Meredith proposed three principles to de-
scribe multi-sensory integration and when this is likely to be
strongest [Stein and Meredith 1993]:
The spatial rule: when the contributing uni-sensory stimuli origi-
nated from approximately the same location;
The temporal rule: when the contributing uni-sensory stimuli
originate at approximately the same time; and,
inverse effect: when the contributing uni-sensory stimuli are rela-
tively weak when considered one at a time.
To deliver a perceptually equivalent “real world experience” to the
user in a virtual environment, it is necessary to deliver the “appro-
priate” level of sensory stimulation for each sense. If it would be
possible to accurately compute all the physics associated with pre-
senting the stimuli, then, as in the real world, the user would simply
process those parts of the stimuli which are necessary for the per-
ception of the environment at that time. Even if we fully understood
all the physics, such physical accuracy is beyond computing capa-
bilities for many years to come. A key feature to increase our ability
to achieve Real Virtuality on current computers is to understand the
limitations of the human brain, which is simply not able to process
all the sensory input we receive every moment of the day. Rather
we selectively process these sensory inputs to build up a useful, but
not necessarily accurate perception of our environment.
2.1 The Perception Equation
The perception of an environment, P, may be described as a func-
tion over time (t) of task (τ ) and preconditioning (ρ ) [Chalmers and
Ferko 2008]:
P(τ , ρ )(t) =
ωV V (t) + ωA A(t) + ωS S(t) +
ωT T (t) + ωF F(t) + ω∆ ∆(t) (1)
where V=Visuals, A=Audio, S=Smell, T=Taste and F=Feel. ∆ is a
“distraction factor” indicating how focussed the user is on the envi-
ronment. For example he/she may be distracted by thinking about
what to have for lunch etc. ωi is the particular perceptual weighting
that each of the senses, and any distraction, has for the perception of
that particular moment, with ∑ ωi = 1. Each of these weightings is
the threshold value below which the perception of the environment
would be different from the perception if one was “there” in reality.
Above this threshold there is no perceptual difference, and thus in
Equation 1, the value of ωi is capped at this threshold [Chalmers
and Ferko 2008].
Key factors that may influence the perceptual importance of dif-
ferent aspects of the real scene are a user’s experience, knowledge
and skills and the activity that the user is undertaking. A high level
of familiarity with the environment (or habituation) may make the
user perceive less [Marsland et al. 2000]. An example of this is
that driving home always seems to take less time than driving to a
new location. Deliberate preconditioning on the other hand, or the
intensity and nature of the task being performed, can force the user
to attend to stimuli that they would otherwise ignore [Nunez and
Blake 2003].
Furthermore, the senses may significantly influence each other. For
example, several researchers have shown that high-quality sounds
coupled with visual stimuli increase the perceived quality of the
visual displays [Storms 1998; Winkler and Faller 2005]. Mas-
toropoulou el al. were able to prove that the combination of
tempo and emotional suggestiveness of music affects the users’ vi-
sual perception of temporal rate and duration [Mastoropoulou and
Chalmers 2003]. Motion has a similar perceptual effect. In [El-
lis and Chalmers 2006] it was shown that a selective render could
exploit the crossmodal interference between a human’s visual and
vestibular system to substantially reduce the quality of parts of an
image without the viewer being aware of any difference.
3 Applications
To be a widely adopted disruptive technology, Real Virtuality must
revolutionise the way in which business is conducted by providing
low-cost, high confidence, high quality multi-sensory knowledge
directly to a user’s current location. An example is purchasing via
the internet, which is currently limited by only providing a view,
or perhaps the sound of the product on offer. Real Virtuality might
enable us not only to view, for example, a brand new Jaguar for
sale and listen to its purring engine, but also to feel the accelerator
while driving in a virtual environment and even inhale the scent of
the interior of this elegant car. An experience as described here
would take place before we make a choice to buy or not, and with
the confidence that the purchasing experience is the same as if we
were there in the showroom, going off for a test ride and examining
the product using all our senses.
There are many other applications for Real Virtuality which could
bring significant, social and economic benefits to users including:
• Tourism
Eg. Visiting an African game park as a family, even if mem-
bers are distributed around the world
• Fully immersive remote meetings or performances
Eg. Selecting your desired listening position at a concert in
the Albert Hall
• Building design
Eg: Designing comfort levels and sensory aesthetics for new
buildings or refurbishments
• Recreating the past
Eg. Exploring site utilisation such as the perception of prehis-
toric cave art as it was being painted
• Remote medical care
Eg. Examining a patient in a remote location from a local GPs
consulting room
• Innovative teaching
Eg. Visiting ancient Rome during History or Latin lessons
• High impact virtual training especially in safety critical areas
Eg. Training pilots to land in “brown out” conditions in the
desert
• Safety in transport
Eg. Enabling drivers to gain experience in rare, but highly
dangerous conditions
• Simulation before introduction of new processes
Eg. Digital mock ups of organisational workflow incorporat-
ing environmental and locational attributes
The diverse nature of the possible applications identified so far, pro-
vide strong indications of the potential for widespread adoption of
the Real Virtuality.
4 Capturing, storing, transmitting and de-
livering the real world
Two different uses are envisioned for Real Virtuality: live interac-
tion with the world in the present; and, the ability to explore en-
vironments in the past or in the future. In the first case it is nec-
essary to capture the real world, using a remote sensing device,
termed a virtual-UT M , shown in Figure 2. This device needs to be
present in the (remote) environment the user wishes to experience.
Figure 2: Virtual-UT M : Device for remote capturing all five senses
Figure 3: The virtual cocoon
It is equipped with sensors for capturing visuals, audio, smell, taste
and feel. In the current plans for the virtual-UT M , the sense of
feel is somewhat constrained including just temperature, humidity
and wind detection, and the ability to shake hands, feel the hard-
ness/texture of objects, move, and feel vibrations.
Any environment in the past, or future, wishing to be explored,
would have to have already been modelled. This model would in-
clude the sources for all the senses and a physics-based simulation
of how the senses are dispersed within the environment. As the user
moves through this modelled environment, the appropriate visuals,
audio, smell, taste and feel are computed and transmitted.
The primary delivery mechanism of Real Virtuality is termed a vir-
tual cocoon, Figure 3. This helmet-like device contains a high dy-
namic range screen, 3D audio headphones, smell and tasting tech-
nology, and temperature, humidity and wind simulation devices.
The user will also be equipped with a glove to interact, such as
shaking hands or feeling the hardness of an object, and a means for
controlling the motion of the virtual-UT M . In addition, a number of
specific user interfaces are envisaged to provide an increased sense
of realism, including a motion platform to replicate the motion of
a vehicle, or, as Figure 4 shows, a means for simulating being on a
dinghy.
The overall bandwidth required to capture, store, transmit and de-
liver real world modalities naturally and in real-time is enormous
Figure 4: Specialist interfaces for Real Virtuality
and beyond the capabilities of modern ICT technology. It is thus
necessary to selectively deliver content, based on the perceptual
importance indicated by the weightings in the Perception Equation
in order to ensure perceptually high-fidelity multi-modal environ-
ments are delivered in real-time. Such selective delivery allows
computational resources to be directed to those areas of high per-
ceptual importance while avoiding computing any detail which will
not be perceived by the viewer. This can save substantial com-
putation and communication [Chalmers et al. 2006; Chalmers and
Debattista 2009].
4.1 Visuals
The human visual system (HVS) receives and processes electro-
magnetic energy in the form of light waves and is sensitive to
the spectrum in the range approximately 400 to 700 nanometers.
The human eye can discern differences between 8 and 12 million
colours; however, we can reliably recall and identify only 6 to
12 colours [Shevell and Kingdom 2008]. The natural world also
presents our visual system with a wide range of intensities. A hu-
man eye can see detail in regions that vary by 1:104 at any given
eye adaptation level. A traditional camera, on the other hand, is
only capable of capturing a limited range of lighting in any scene,
ie. a Low Dynamic Range (LDR). The actual range captured de-
pends on the exposure setting of the camera. The remainder of the
image outside this limited range, is either under- or over-exposed.
Images that can reproduce a large portion of the luminance dynamic
range available in the real world are known as High Dynamic Range
(HDR) images. HDR images offer a more representative descrip-
tion of image-based digital content by storing data with a higher
pixel depth than the more conventional LDR images. The increased
bit depth presented by HDR image formats can account for all the
dynamic range visible by the human visual system. Figure 5 shows
how real world luminances, in false colour on the left, would be
captured at two different exposure levels, in the middle. The resul-
tant HDR image can then be directly displayed on an HDR device,
or compressed in such a way to preserve, as much as possible, the
perceptual fidelity of the original scene, using sophisticated tone
mapping operators, for a typical TV or computer monitor, which
are LDR displays, Figure 5 (right). Real Virtuality requires both
HDR image capture and deliver capabilities.

Figure 5: (left) Real world luminances, (middle) Two different ex-
posure levels, (right) tone mapped image for LDR diplay.
4.2 Sound
The human hearing system consists of the outer ear, middle ear,
and inner ear. The outer ear, or pinna, allows us to locate the
direction of a sound source. The middle ear, consisting of three
small bones (hammer, anvil and stirrup) transmits acoustic vibra-
tions from the ear drum as a mechanical input to the cochlea of the
inner ear. The middle ear can also help protect the hearing system
from loud sounds [Howard and Angus 2006]. In order to recreate
an auditory environment virtually it is necessary to present at the
ear drum, the equivalent physical and acoustic properties of the real
sound source. To achieve the illusion of 3D audio, the individual
nature of the acoustic effect of a person’s head and shoulders (their
head related transfer function) needs to be taken into account and
head tracking used to ensure that when the person moves his/her
head the sound moves as well. Audio has been used extensively
in 3D virtual environments, for example [Begault 1994]. However
less has been done on spatial rendering of sound for 3D environ-
ments, for example [Naef et al. 2002; Murphy et al. 2007; Siltanen
et al. 2007; Tsingos et al. 2004]. For Real Virtuality it is crucial that
the sound is directional, is heard outside the head, and has a sound
quality above the threshold determined by the Perception Equation.
4.3 Smell
Natural smell molecules (for humans), with the notable excep-
tion of a few molecules such as hydrogen sulphide (the charac-
teristic smell of decayed eggs), are carbon-based molecules. Such
molecules are small, which enables them to float in the air and form
an invisible plume of odour; known as volatility. Smell molecules
have two key features: their surface area relative to the volume is
large; and, quantum effects become important [Dodd 2009]. A
human’s ability to smell occurs in the 2-4 cm2 tissue of the ol-
factory epithelium of the nose. This area contains 500-1000 dif-
ferent types of olfactory receptors, which only last for 60 days,
but are continually renewed. These receptors encode information
about the chemical composition of odour molecules into neural sig-
nals, with each type of olfactory receptors tuned to different odour
molecules [Ressler et al. 1994]. Smell in an environment may be
captured manually by sucking the air across a Automated Thermal
Desorption (ATD) tube, Figure 1 (middle). The smell molecules
stick to the fine granular material in the tube. The trapped molecules
can then be determined by first passing them through a gas-liquid-
chromatography (GLC) instrument, which separates the complex
mixture of odorants (many natural odorant mixtures have between
10-600 individual odorant molecules) into constituent molecules.
From the GLC, the molecules pass into a mass-spectrometer, which
produces a resultant histogram of the molecules present. Current
mass-spectrometer devices are not precise and many molecules may
be missed. A human nose of a trained molecular scientist is often
used to identify any smell molecules that the device may not have
detected [Dodd 2009].
Despite the importance of scent in our every day life, little work has
been done to include smell in virtual environments. Real exhibits,
on the other hand, such as the Jorvik Viking Museum in York, have
smells piped to parts of the exhibit for many years, which has been
shown to aid visitors to remember information [Aggleton and Was-
kett 1999]. One of the earliest attempts to include smell in a vir-
tual setting was Morton Heilig’s Sensorama [Rheingold 1992]. De-
veloped in the early 1960s, this mechanical device was capable of
displaying stereoscopic 3D images and included user movement,
stereo sound, wind and aromas which could be trigged during the
film. Unfortunately Heilig was unable to find financial backing for
his device and nothing further was developed. Only much more
recently has smell started to be introduced into virtual environ-
ments for training, for example [Washburn et al. 2003], and ther-
apy, for example [Barfield and Danas 1995; Chen 2006]. Results
from preliminary studies has shown that the introduction of smell
does indeed increase the user’s sense of ”presence” in the virtual
environment [Dinh et al. 1999; Zybura and Eskeland 1999]. In par-
ticular, the introduction of realistic smells, including the smells of
burning rubber and flesh, have been used effectively to treat sol-
diers returning from Iraq with post-traumatic stress disorder [Pair
et al. 2006], and are currently being considered by the MoD within
serious games for training British soldiers [BBC ]. Most recently
an interactive olfactory display was also developed for a ”cooking
game”, in which the duration and strength of a number of prede-
fined smells was controlled and blended in real-time into a number
of recipes by the game [Nakamoto et al. 2008]. In Real Virtuality,
perceptually accurate smell is delivered directly to the user’s nose.
This enables different smells to be added and removed in a straight-
forward manner as the user moves through the environment.
4.4 Taste
The molecules of food are chemicals detected by taste buds, which
are sensory organs situated predominantly on the tongue, but also
a few are on the roof of the mouths of humans. There are five pri-
mary tastes: salty, sour, bitter, sweet and umami (from the Japanese
”tasty”), which corresponds roughly to the taste of glutamate [Abdi
2002]. How we perceive food is also determined by the smell of the
food via the nose, its texture which is determined by mechanore-
ceptors and its temperature detected by thermoreceptors. Smell
and taste combine to form flavour, which may also be related to
other crossmodal interactions [Verhagen and Engelen 2006]. Re-
cent brain imaging studies have shown that there is a difference
in the way in which professional tasters and amateurs appreci-
ate flavour. Sommeliers of wine tasting have a more analytical
approach, with stronger activations in the dorso-lateral prefrontal
cortex, while the non professionals showed much stronger acti-
vations in emotion related areas such as hippocampus and amyg-
dala [Castriota-Scanderbeg et al. 2005].
Despite the fact that research on the perception of taste started as
long ago as the late 1500s, very little work has been done on virtual
taste. A notable exception is the work of Iwata et al [Iwata et al. ].
In 2003 at the SIGGRAPH Emerging Technologies exhibition, they
presented their food simulator: a haptic interface to mimic the taste,
sound and feeling of chewing real food. A device in the mouth sim-
ulated the force of the type of food, a bone vibration microphone
provided the sound of biting, while the chemical simulation of taste
was achieved via a micro injector. Taste in Real Virtuality is also
simulated with a device in the mouth (connected to the virtual co-
coon via a USB cable to prevent swallowing).
4.5 Feel
A human’s sense of feel originates in the dermis layer of our skin.
The dermis is filled with nerve endings of many sensory neurons.
When a subject touches, or is touched by, an object, a force is
imposed on the sensors of the skin. Details of this force are then
conveyed to the brain. There are about 20 different types of ”feel
senses”, the most common of which are heat, cold, pain, and pres-
sure or touch receptors. Some areas of the body contain more sen-
sors than others, making these areas more sensitive to feel. For ex-
ample the tongue has lot of nerve endings of pain and fewer nerve
endings of heat. This explains why it hurts so much if we bite our
tongue and why we can easily burn our mouths if we eat something
hot. Other most sensitive areas are hands, lips, face, neck, fingertips
and feet.
Haptics in virtual environments is a large, active, multidisciplinary
field. However, current haptic devices suffer from a number of
limitations. In particular, they have limited feedback capabilities
compared to the humans tactile sensory system. The human hand
consists of millions specialized tactile sensors all working in par-
allel, whereas current haptic interfaces typically have less than 10
tactile feedback motors. Other limitations of current haptic devices
include high price, high weight and size, bandwidth limitation, la-
tency between a human operator and the force feedback, being de-
signed for a very specific purposes, and instability if the update rate
is much less than 1kHz [Robles-De-La-Torre 2006; Saddik 2007].
Feel in Real Virtuality includes the ability to sense the environment,
such as temperature, humidity and wind, and a limited form of mo-
tion and touch through a virtual hand, see figure 2.
4.6 Case study
To illustrate the concept of Real Virtuality, consider the example of
visiting a game park. In the real world a tourist flies to, for example,
Johannesburg in South Africa, and then drives several hours north
to the game park. To view the animals, the tourist books a game
drive at a certain time. At this time, the tourist climbs onto a truck
with other tourists, figure 6 (left), and is driven around the game
park by a ranger, who explains the fauna and floral as he/she goes.
The “thrill” of a game drive is actually the anticipation of seeing
some animals. The tourists and ranger scan the bush hoping for
that first glimpse of a desired animal. A game drive can last several
hours, after which the tourists are likely to have seen many buck,
but perhaps not the illusive buffalo they had hoped to see.
While it is no doubt the ambition of many people to go on such a
safari once in their lives, there are many difficulties to achieving this
in the real world. Flights to Africa may be expensive and are bad
for the environment; some game parks are currently in war zones;
or may contain malaria. It is vital for the animals that the game
parks achieve revenue from tourists to maintain, for example the
fences which protect the animals from poachers and prevent them
wandering onto farm land. In Real Virtuality a tourist would like
to experience the thrill of the game park from the comfort of their
living room, for example, in England. To achieve this, the tourist
will book and pay for a game drive with the game park. At the
specified time, the game park loads a virtual-UT M onto a game truck
together with other virtual-UT M ’s and perhaps some real tourists as
well. The ranger drives through the park exactly as before. Now the
tourist experiences the game park through the virtual-UT M which
provides in real-time:
• stereo HDR images providing 3D data with real world lighting
• 3D audio giving directional information for the voice of the
ranger, other participants on the game truck and the sounds of
the bush
• the smell of the bush and the members of the game truck
• the taste of, for example, dust in the mouth
• the feel of the heat and humidity of the African climate, the
wind on the face, and the bumpy nature of the ride.
Furthermore, the remote tourist controls where the virtual-UT M is
looking and can talk with the ranger and the other members of the
game truck in exactly the same manner if he/she was there.
5 Uptake of technology
A key aspect to ensuring the success of Real Virtuality is under-
standing what economic value might follow from its use. To the
economist the issue of economic value stems from understand-
ing the conditions likely to lead to firm adoption and subsequent
widespread diffusion of a new technology such as the virtual co-
coon. Economic value is the additional income that accrues to an
economy following from the adoption of a novel technology. This
additional income comes in the form of an increase in all firm prof-
its or an increase in the total income of all workers. Additional
national income usually arises because the new technology allows
firms to improve the efficiency in the way they design, make, dis-
tribute and sell a good or service. Or the new technology facilitates
the production of a superior product that can command a premium
price or process. Additionally increased national income follows
from an efficiency improvement to the way workers, tangible and
intangible capital, are allocated or distributed across firms in dif-
ferent sectors of an economy. Firms are more likely to adopt a new
technology if it is technically superior to technologies that currently
exist and perform the same or like function. Superiority can be in
terms of additional functions, higher quality attributes in the good
or service produced using that new technology, or producing a good
or service at lower cost.
Real Virtuality has the potential of offering value to firms engaged
in the design and build of new products, production systems or
processes that are inherently complex, involve significant elements
of human interaction with a variety of technologies and where the
likely outcome of these interactions is uncertain. New cars, oil re-
finerers, other forms of dangerous and costly structures are all ex-
amples. Superiority could be in terms of cost. Designing something
virtually can cost less than creating physically based prototypes. A
virtual cocoon could also offer greater flexibility in experimenting
with different configurations and integrating different complemen-
tary technologies and skill levels. The early adopter takes on all
kinds of technological, financial and market risk. These carry pos-
sible costs of a failure and rewards for success. Costs are minimised
by experimenting with new technologies in different applications,
and learning from mistakes. Reward typically takes the form of ex-
pected extra profits to adopting firm from the adoption of the new
technology. Profits should be high enough to more than cover the
cost of capital needed to design, make, distribute and sell products
or processes that embody the new technology. Profits should also
offer sufficient reward for taking on risk compared to not taking on
the risk and investing the money in some other way. These are much
easier to achieve if the market the early adopter trades in is not very
competitive. Uncompetitive markets arise either because there are
barriers to stop firms entering the market and competing away early
adopters profits. Alternatively they arise because a single success-
ful firm has some firm specific attribute or competence that is the
source of that firm’s competitiveness and not easily replicated by
 Figure 6: The game park experience: (left) boarding the truck (middle) ranger explaining (right) seeing the animals.
other firms. Sometimes it is better for individual firms not to be an
early adopter themselves but let others make costly mistakes first.
Generally greater economic value arises because of widespread dif-
fusion and subsequent use of new technologies. In the case of Real
Virtuality widespread diffusion is more likely if the application can
be incorporated within an existing technology like a mobile tele-
phone, computer games machine or laptop. Putting a new appli-
cation on an existing technology platform lowers cost in a vari-
ety of ways. It lowers costs of technological integration, and the
knowledge or learning that users need to have, because their under-
standing of the new technology can be based on reference points
that they are familiar with. Using an existing technology platform
reduces the costs of adapting current production to make new com-
puter hardware or write software code that is needed to run the new
cocoon application.
In summary the economic research questions centre on what is new
about Real Virtuality and the Virtual Cocoon, in what respects is
it new, what would be the prospective costs and rewards from a
firm being an early adopter of this technology, how far does these
prospective rewards hinge on the specific characteristics of the firm,
the structure of industry , the intensity of competition in different
markets where a virtual cocoon has possible application and lastly
estimating and explaining the sources of cost and price advantage
of products that embody Real Virtuality technologies.
6 Conclusions
Virtual Reality systems attempt to deliver high levels of interactiv-
ity in realistic looking environments. To maintain such real time
rates, VR typically trade off realism. Furthermore VR systems sel-
dom include more than two or three senses, and most often only
use visuals. By delivering all five senses, Real Virtuality is able
to exploit crossmodal effects which are a key feature of how hu-
mans perceive the real world. When bombarded with all sensory
stimuli, humans cannot cope with everything and thus have to se-
lect the level of each sense they attend to in order to undertake their
current task. This selection, of course, changes over time, but it
does mean that at any point in time we have an imperfect view
of the world compared to the actual present physics. As inatten-
tional blindness clearly shows [Mack and Rock 1998], significant
amounts of information in a scene can literally go unnoticed. It is
precisely the imperfections in our perception that can be exploited
in order to achieve high-fidelity multi-sensory virtual environments
in real time. It is not necessary to compute and deliver full phys-
ical accuracy for all senses interactively, rather we only need to
achieve at least the threshold levels of each of the weights of the
Perception Equation in real time. If we achieve this then the user
will have a perceptually equivalent high-fidelity experience without
being aware of the quality differences within the virtual scene.
The perceptual fidelity of Real Virtuality will enable it to be used,
with confidence, as a means of experiencing the real world in an
accurate, safe and controlled manner. This is particularly important
as travel becomes more expensive and less desirable in order to
protect the environment. Lack of travel opportunities brings some
serious social risks. People will become increasingly isolated from
the rest of the world. Such isolation can lead to suspicion and even
xenophobia which needs to be avoided at all costs.
The possible applications of Real Virtuality are many, however
some key technical and social challenges remain. In addition to
actually capturing/modelling, storing, transmitting and delivering
the real world modalities in a natural manner, for Real Virtuality to
be widely adopted, there needs to be a set of powerful tools to en-
able content to be easily created. Without a rapid growth in content
and thus “experiences” for Real Virtuality users, such a system is
likely to be confined to a few research laboratories and keen early-
adopters. The success of Real Virtuality depends on it delivering
on its promise of “experiencing real world modalities in a natural
manner in a virtual environment” and the technology being afford-
able and accessible to a wide range of users. Each user that joins
the Real Virtuality community should be able to add to the content
and in being a member should help reduce the costs for all other
members.
Acknowledgements
Thank you very much to Helwig Hauser and the organisers of
SCCG’09 for inviting Alan to give this keynote paper. We would
also like to thank the EPSRC for funding the “...towards Real Vir-
tuality” project, Figure 7 (which Jude Brereton has done a great
job of coordinating), our partners in the project: the Universities of
Bradford, Bangor and Brighton, Nic Knott-Craig and Zebula Game
Park, South Africa for hosting us for the case study, George Dodd,
James Covington and Mark Pharoah for the work on smell, Rich
Kasler for the initial design of the virtual cocoon and virtual-UT M ,
Gabriela Czanner for background material on the senses, Alessan-
dro Artusi for the HDR image, Francesco Banterle for late night
LATEXadvice and Eva Zányi for the detailed proofreading of this pa-
per.
References
A BDI , H. 2002. What can cognitive psychology and sensory eval-
uation learn from each other? Food Quality and Preference 13,
445–451.
AGGLETON , J., AND WASKETT, L. 1999. The ability of odors
to serve as state-dependent cues for real-world memories: Can
viking smells aid the recall of viking experiences? British Jour-
nal of Psychology 90, 1–7.

Figure 7: Real Virtuality: As real as if you are here.
www.towardsrealvirtuality.com
BARFIELD , W., AND DANAS , E. 1995. Comments on the use of
olfactory displays for virtual environments. Presence 5, 1, 109–
121.
BBC. http://news.bbc.co.uk/2/hi/uk news/7716725.stm.
B EGAULT, D. 1994. 3-D sound for virtual reality and multimedia.
Ames Research Center.
C ALVERT, G., S PENCE , C., AND S TEIN , B. 2004. The multi-
sensory handbook. MIT Press.
C ASTRIOTA -S CANDERBEG , A., H AGBERG , G., C ERASA , A.,
C OMMITTERI , G., G ALATI , G., PATRIA , F., P ITZALIS , S.,
C ALTAGIRONE , C., AND F RACKOWIAK , R. 2005. The appre-
ciation of wine by sommeliers: a functional magnetic resonance
study of sensory integration. Neuroimage 25, 570–578.
C HALMERS , A., AND D EBATTISTA , K. 2009. Level of realism for
serious games. In IEEE VS-Games’09, IEEE.
C HALMERS , A., AND F ERKO , A. 2008. Levels of realism: From
virtual reality to real virtuality. In SCCG’08, ACM SIGGRAPH,
27–33.
C HALMERS , A., D EBATTISTA , K., AND DOS S ANTOS , L. 2006.
Selective rendering: Computing only what you see. In Graphite
2006, ACM SIGGRAPH, 9–18.
C HALMERS , A., D EBATTISTA , K., M ASTOROPOULOU , G., AND
DOS S ANTOS , L. 2007. There-Reality: Selective Rendering in
High Fidelity Virtual Environments. The International Journal
of Virtual Reality 6, 1, 1–10.
C HEN , Y. 2006. Olfactory display: development and application
in virtual reality therapy. In ICAT’06: Proceedings of the 16th
International Conference on Artificial Reality and Telexistence,
IEEE Press.
D INH , H. Q., WALKER , N., S ONG , C., KOBAYASHI , A., AND
H ODGES , L. 1999. Evaluating the importance of multi-sensory
input on memory and the sense of presence in virtual environ-
ments. In IEEE Virtual Reality 1999, IEEE Press, 222–228.
D ODD , G., 2009. Personal communication.
E LLIS , G., AND C HALMERS , A. 2006. The Effect of Transla-
tional Ego-Motion on the Perception of High Fidelity Anima-
tions. SCCG 2006.
H OWARD , D., AND A NGUS , J. 2006. Acoustics and psychoacous-
tics. 3rd Ed., Oxford: Focal Press.
H OWARD , I., AND T EMPLETON , W. 1966. Human spatial orien-
tation. John Wiley and Sons Ltd.
I WATA , H., YANO , H., U EMURA , T., AND M ORIYA , T. Food
simulator. In ICAT’03: Proceedings of the 13th International
Conference on Artificial Reality and Telexistence, IEEE Press.
M ACK , A., AND ROCK , I. 1998. Inattentional Blindness. Mas-
sachusetts Institute of Technology Press.
M ARSLAND , S., N EHMZOW, U., AND S HAPIRO , J. 2000. Novelty
detection on a mobile robot using habituation. From Animals
to Animats: The 6th International Conference on Simulation of
Adaptive Behaviour.
M ASTOROPOULOU , G., AND C HALMERS , A. 2003. The effect of
music on the perception of display rate and duration of animated
sequences: an experimental study. In TPCG’04: Theory and
Practice of Computer Graphics 2004, IEEE Press, 128–134.
M URPHY, D., K ELLONIEMI , A., M ULLEN , J., AND S HELLEY,
S. 2007. Acoustic modelling using the digital waveguide mesh.
IEEE Signal Processing Magazine 24, 2 (March), 55–66.
NAEF, M., S TAADT, O., AND G ROSS , M. 2002. Spatialized audio
rendering for immersive virtual environments. In Proceedings of
the ACM symposium on Virtual reality software and technology,
ACM Press, 65–72.
NAKAMOTO , T., OTAGURO , S., K INOSHITA , M., NAGAHAMA ,
M., O HINISHI , K., AND I SHIDA , T. 2008. Cooking up an
interactive olfactory game display. IEEE Computer Graphics
and Applications.
N UNEZ , D., AND B LAKE , E. H. 2003. Conceptual priming as a
determinant of presence in virtual environments. In Afrigraph
2003, ACM SIGGRAPH, 101–108.
PAIR , J., A LLEN , B., DAUTRICOURT, M., T RESKUNOV, A.,
L IEWER , M., G RAAP, K., R EGER , G., AND R IZZO , A. 2006.
A virtual reality exposure therapy application for iraq war post
traumatic stress disorder. In IEEE Virtual Reality 2006, IEEE
Press.
R ESSLER , K., S ULLIVAN , S., AND B UCK , L. 1994. A molecular
dissection of spatial patterning in the olfactory system. Current
Opinion in Neurobiology 4, 588–596.
R HEINGOLD , H. 1992. Virtual Reality. Simon & Schuster.
ROBLES -D E -L A -T ORRE , G. 2006. The importance of the sense
of touch in virtual and real environments. IEEE Multimedia:
Special issue on Haptic User Interfaces for Multimedia Systems
13, 3, 24–30.
S ADDIK , A. 2007. The potential of haptic technologies. IEEE
Instrumentation & Measurement Magazine 10, 31, 10–17.
S HEVELL , S., AND K INGDOM , F. 2008. Colour in complex
scenes. Annual Review of Psychology 59, 143–166.
S ILTANEN , S., L OKKI , T., K IMINKI , S., AND S AVIOJA , L. 2007.
The room acoustic rendering equation. Journal of the Acoustic
Society of America 122 (September), 1624–1635.
S TEIN , B., AND M EREDITH , M. 1993. The merging of the senses.
MIT press.
S TORMS , R. 1998. Auditory-Visual Cross-Modal Perception Phe-
nomena. PhD thesis, Naval Postgraduate School, Monterey, Cal-
ifornia.
T SINGOS , N., G ALLO , E., AND D RETTAKIS , G. 2004. Perceptual
audio rendering of complex virtual environments. In SIGGRAPH
2004, ACM Press, 249–258.
V ERHAGEN , J., AND E NGELEN , L. 2006. The neurocognitive
View publication stats
bases of human multimodal food perception: Sensory integra-
tion. Neuroscience and Biobehavioral Reviews 30, 613–650.
WASHBURN , D., J ONES , L., S ATYA , R., B OWERS , C., AND
C ORTES , A. 2003. Olfactory use in virtual environment training.
Modelling and Simulation Magazine 2, 3.
W INKLER , S., AND FALLER , C. 2005. Audiovisual quality eval-
uation of low-bitrate video. In SPIE/IS&T Human Vision and
Electronic Imaging, SPIE, vol. 5666, 139–148.
Z AMPINI , M., AND S PENCE , C. 2004. The role of auditory cues in
modulating the perceived crispness and staleness of potato chips.
Journal of Sensory Studies 15, 9, 347–363.
Z YBURA , M., AND E SKELAND , A. 1999. Olfaction for virtual
reality. Quarter Project Industrial Engineering 543.