Nurul Hasan

Augmented Virtual reality

Augmented Virtual reality

Introduction………..………………………………………………………………………………….………………………………2
Definition……………………………………………………………………………………………………………………………….2
Examples………………………………………………………………………….…………………………………………..3
Sports……………………………………………………………………………………………………………………………….……..3
Handheld video games……………………………………………….…………………………………………………4
Other……………………………………………………………………………………………………………………………4
Technology………………………………………………………………………………………………………………………………5
Hardware…………………………………….………………………………………………………………………………..5
Display…..…………………………………….….……………………………………………………………………………..6
Head-mounted..………………………………………………………………………………………………………..…...6
Handheld………...…………………………………………………………………...………………………………………..7
Spatial…………………………………………….……………………………………………………………………………..7
Tracking……….………………………………………………………………………………………………………………..8
Input devices……………………………………….….……………………………………………………………………..8
Computer…………………………………………….………………………………………………………….……………..8
Application Scenarios………………………….…………………….…………………………………………………..9
Technologies Used…………………………….…….…………………………………………………………………..10
Software and algorithms……………………………………………………………………………….……………………..11
Applications……………………………………………….…………………………..……………………………………………..11
Applications as of 2011……………………………………………………………….……………………………..12
Potential applications……………………..…………………………..……………………………………………..14
Software…………………………………………………..……………..……………………………………………………………..15
Open source……………………………….…………………………………………………………..………………….15
Free closed source……………………………………………………………………………………………………..16
AR Content Management Systems.……………………………………………………………………………..16
In popular culture……………………….……………………………………………………………………………..16
Television, film………………………………………..……………………………………………………………………………..16

University Of Calcutta | Augmented Virtual Reality 1

Introduction

Augmented Virtual reality (AVR) is a live, direct or indirect, view of a physical,

real-world environment whose elements are augmented by computer-generated
sensory input such as sound, video, graphics or GPS data. It is related to a more
general concept called mediated reality, in which a view of reality is modified
(possibly even diminished rather than augmented) by a computer. As a result, the
technology functions by enhancing one’s current perception of reality. By contrast,
virtual reality replaces the real world with a simulated one.

Augmentation is conventionally in real-time and in semantic context with

environmental elements, such as sports scores on TV during a match. With the help
of advanced AR technology (e.g. adding computer vision and object recognition)
the information about the surrounding real world of the user becomes interactive
and digitally manipulable. Artificial information about the environment and its
objects can be overlaid on the real world. The term augmented virtual reality is
believed to have been coined in 1990 by Thomas Caudell, working at Boeing.

Research explores the application of computer-generated imagery in live-video

streams as a way to enhance the perception of the real world. AR technology
includes head-mounted displays and virtual retinal displays for visualization
purposes, and construction of controlled environments containing sensors and
actuators.

Definition

Augmented virtual Reality is considered as an extension of Virtual Reality.

Virtual Reality (VR) is a virtual space where the player immerses himself into that

University Of Calcutta | Augmented Virtual Reality 2

exceed the bounds of physical reality. In the VR, time, physical laws and material
properties no longer hold in contrast to real-world environment. Instead of
considering AR and VR as exact opposite concepts, Milgram et al. claim as the
Reality-Virtual (RV) continuum (Milgram, Takemura, Utsumi and Kishino, 1994).

Ronald Azuma offered a definition in 1997. R. Azuma, A Survey of Augmented

virtual Reality Presence: Teleo. AR is about augmenting the real world
environment with virtual information by improving people’s senses and skills. AR
mixes virtual characters with the actual world. There are three common
characteristics of AR scenes: the combination of real world environment with
computer characters, interactive scenes, and scenes in 3D (Azuma, 1997; Azuma et
al., 2001).

On a graph, the origin R at the bottom left denotes unmodified reality. A

continuum across the Virtuality axis V includes reality augmented with additional
information (AR), as well as virtual reality augmented by reality (Augmented
Virtuality or AV). Unmediated AV simulations are constrained to match the real
world behaviorally if not in contents.

The mediality axis measures modification of AV, AR and mixes thereof. Moving
away from the origin on this axis, the depicted world becomes increasingly
different from reality. Diagonally opposite from R are virtual worlds that have no
connection to reality. (at right) It includes the virtuality reality continuum (mixing)
but also, in addition to additive effects, also includes modulation and/or
diminishment of reality. Mediation encompasses deliberate and/or unintentional
modifications.

Sports

AR has become common in sports telecasting. The yellow "first down" line seen in
television broadcasts of American football games shows the line the offensive team
must cross to receive a first down using the 1st & Ten system. The real-world

University Of Calcutta | Augmented Virtual Reality 3

elements are the football field and players, and the virtual element is the yellow
line, which augments the image in real time.

AR is also used in association football to show the result (or an advertisement) in

the center circle. It is also used to display offside situations.

Similarly, in ice hockey an AR colored trail shows location and direction of the
puck. Sections of Rugby fields and cricket pitches display sponsored images.

Swimming telecasts often add a line across the lanes to indicate the position of the
current record holder as a race proceeds to allow viewers to compare the current
race to the best performance.

As an example of mediated (diminished) reality, the network may hide a real

message or replace a real ad message with a virtual message.

Handheld video games

Nintendo 3DS and PlayStation Vita come with "AR Cards" (fiduciary markers)
allowing to play games which use device camera (Nintendogs + Cats, Tetris Axis,
AR Combat DigiQ)

Other

First-person shooter video games can simulate a player's viewpoint using AR to

give visual directions to a location, mark the direction distance of another person
who is not in line of sight and give information about equipment such as remaining
ammunition. This is done using a virtual head-up display.

Head-up displays in AR cars such as some BMW 7 Series models or within

airplanes are typically integrated into the windshield.

The F-35 Lightning II instead display information in the pilot's helmet mounted
display, which allows the pilot to look through the aircraft's walls as if he was
floating in space.

University Of Calcutta | Augmented Virtual Reality 4

Technology

Hardware

The main hardware components for augmented virtual reality are: processor,
display, sensors and input devices. These elements, specifically CPU, display,
camera and MEMS sensors such as accelerometer, GPS, solid state compass are
often present in modern smart-phones, which make them prospective AR platforms.
University Of Calcutta | Augmented Virtual Reality 5
 Display

There are three major display techniques for Augmented virtual Reality: head-
mounted displays, handheld displays and spatial displays.

Head-mounted

A head-mounted display (HMD) places images of both the physical world and
registered virtual graphical objects over the user's view of the world. The HMD's
are either optical see-through or video see-through. Optical see-through employs
half-silver mirrors to pass images through the lens and overlay information to be
reflected into the user's eyes. The HMD must be tracked with sensor that provides
six degrees of freedom. This tracking allows the system to align virtual information
to the physical world. The main advantage of HMD AR is the user's immersive
experience. The graphical information is slaved to the view of the user. The most
common products employed are as follows: Micro-Vision Nomad, Sony Glasstron,
Vuzix, Lumus and I/O Displays.

Optical see-through HMD conceptual diagram

University Of Calcutta | Augmented Virtual Reality 6

 Video see-through HMD conceptual diagram

Handheld

Handheld displays employ a small display that fits in a user's hand. All handheld
AR solutions to date opt for video see-through. Initially handheld AR employed
fiduciary markers, and later GPS units and MEMS sensors such as digital
compasses and six degrees of freedom accelerometer–gyroscope. Today SLAM
marker-less trackers such as PTAM are starting to come into use. Handheld
display AR promises to be the first commercial success for AR technologies. The
two main advantages of handheld AR is the portable nature of handheld devices
and ubiquitous nature of camera phones. The disadvantages are the physical
constraints of the user having to hold the handheld device out in front of them at
all times as well as distorting effect of classically wide-angled mobile phone
cameras when compared to the real world as viewed through the eye.

Spatial

Instead of the user wearing or carrying the display such as with head-mounted
displays or handheld devices, Spatial Augmented virtual Reality (SAR) makes use
of digital projectors to display graphical information onto physical objects. The
key difference in SAR is that the display is separated from the users of the system.
Because the displays are not associated with each user, SAR scales naturally up to
University Of Calcutta | Augmented Virtual Reality 7
groups of users, thus allowing for collocated collaboration between users. SAR has
several advantages over traditional head-mounted displays and handheld devices.
The user is not required to carry equipment or wear the display over their eyes.
This makes spatial AR a good candidate for collaborative work, as the users can
see each other’s faces. A system can be used by multiple people at the same time
without each having to wear a head-mounted display. Spatial AR does not suffer
from the limited display resolution of current head-mounted displays and portable
devices. A projector based display system can simply incorporate more projectors
to expand the display area. Where portable devices have a small window into the
world for drawing, a SAR system can display on any number of surfaces of an
indoor setting at once. The drawbacks, however, are that SAR systems of
projectors do not work so well in sunlight and also require a surface on which to
project the computer-generated graphics. Augmentations cannot simply hang in
the air as they do with handheld and HMD-based AR. The tangible nature of SAR,
though, makes this an ideal technology to support design, as SAR supports both a
graphical visualisation and passive haptic sensation for the end users. People are
able to touch physical objects, and it is this process that provides the passive
haptic sensation.

Tracking

Modern mobile augmented virtual reality systems use one or more of the following
tracking technologies: digital cameras and/or other optical sensors,
accelerometers, GPS, gyroscopes, solid state compasses, RFID and wireless
sensors. These technologies offer varying levels of accuracy and precision. Most
important is the position and orientation of the user's head. Tracking the user's
hand(s) or a handheld input device can provide a 6DOF interaction technique.
Input devices

Techniques include the pinch glove, a wand with a button and a smartphone that
signals its position and orientation from camera images.

Computer

The computer analyzes the sensed visual and other data to synthesize and position
augmentations.

University Of Calcutta | Augmented Virtual Reality 8

 Overall System Architecture

Application Scenarios

• Provide a personalized guide for on-site visitors using augmented virtual

reality techniques.
• Provide an off-site personalized guide (via Internet or CD-ROM).
• Provide a database of scientific information about a site (cartography,
documentation of each archeological object, references etc.) accessible
remotely via Internet/intranet.

University Of Calcutta | Augmented Virtual Reality 9

 Detailed Component Diagram

Technologies Used

• Client/Server architecture: Wearable computer on client side, fast server,

wireless LAN.
• Position tracking:
– Hybrid tracker: GPS tracking technology, inertia tracking, compass
tracking feeding vision based tracker seeking to identify well known
(static) landmarks.
• Rendering:
– VRML models, multimedia material.
– Occlusion handling of known static objects only.
• Client (Mobile Unit):
– Wearable hardware: very fast processor for real time system
requirements, high speed & resolution video card, video camera &
frame grabber, HMD, large local disk space, WLAN card.
• Server:
– Multi-processor platform for low latency, very large storage capacity,
high speed WLAN card.
• Network: Wireless LAN

University Of Calcutta | Augmented Virtual Reality 10

 • Databases technologies:
– Geographic Information Systems (GIS) technology will be used for
registering the site topology to the DBMS: site cartography to be
obtained through aerophotogrammetry, and GPS.
– Object relational engine to support Object Oriented interfaces.

Software and algorithms

A key measure of AR systems is how realistically they integrate augmentations with

the real world. The software must derive real world coordinates, independent from
the camera, from camera images. That process is called image registration and is
part of Azuma's definition of Augmented virtual Reality.

Image registration uses different methods of computer vision, mostly related to

video tracking. Many computer vision methods of augmented virtual reality are
inherited from visual odometry. Usually those methods consist of two parts. First
detect interest points, or fiduciary markers, or optical flow in the camera images.
First stage can use feature detection methods like corner detection, blob detection,
edge detection or thresholding and/or other image processing methods.

The second stage restores a real world coordinate system from the data obtained
in the first stage. Some methods assume objects with known geometry (or fiduciary
markers) present in the scene. In some of those cases the scene 3D structure should
be precalculated beforehand. If part of the scene is unknown simultaneous
localization and mapping (SLAM) can map relative positions. If no information
about scene geometry is available, structure from motion methods like bundle
adjustment are used. Mathematical methods used in the second stage include
projective (epipolar) geometry, geometric algebra, rotation representation with
exponential map, kalman and particle filters, nonlinear optimization, robust
statistics.

Applications

Applications as of 2011

University Of Calcutta | Augmented Virtual Reality 11

Advertising: Usage of AR to promote products via interactive AR applications is
becoming common now. For example Nissan (2008 LA Auto Show), Best Buy
(2009), and others used webcam based AR to connect 3D models with printed
materials. There are numerous examples of connecting mobile AR to outdoor
advertising

Task support: Complex tasks such as assembly, maintenance, and surgery can be
simplified by inserting additional information into the field of view. For example,
labels can be displayed on parts of a system to clarify operating instructions for a
mechanic who is performing maintenance on the system. AR can include images of
hidden objects, which can be particularly effective for medical diagnosis or
surgery. Examples include a virtual X-ray view based on prior tomography or on
real time images from ultrasound and confocal microscopy probes or open NMR
devices. AR can enhance viewing a fetus inside a mother's womb. See also Mixed
reality.

Navigation: AR can augment the effectiveness of navigation devices. For example,

building navigation can be enhanced to aid in maintaining industrial plants.
Outdoor navigation can be augmented for military operations or disaster
management. Head-up displays or personal display glasses in automobiles can
provide navigation and traffic information. Head-up displays are currently used in
fighter jets. These systems include full interactivity, including gaze tracking. They
are also currently used by car manufacturers as BMW, Corvette and GM in car
windshields, mostly to display meter information, traffic information, ...

Industrial: AR can be used to compare digital mock-ups with physical mock-ups

for efficiently finding discrepancies between them. It can safeguard digital data
together with existing real prototypes, and thus reduce the number of real
prototypes and improve the quality of the final product.

Military and emergency services: Wearable AR can provide information such as

instructions, maps, enemy locations, and fire cells.

Art: AR can help create art in real time integrating reality such as painting,
drawing and modeling. AR art technology has helped disabled individuals to
continue pursuing their passion.

Architecture: AR can simulate planned construction projects.

University Of Calcutta | Augmented Virtual Reality 12

Sightseeing: Guides can include labels or text related to the objects/places visited.
With AR, users can rebuild ruins, buildings, or even landscapes as they formerly
existed.

Time: Historical events such as battle reenactments can be augmented onto

current landscapes.

Collaboration: AR can help facilitate collaboration among distributed team

members via conferences with real and virtual participants.

Entertainment and education: AR can create virtual objects in museums and

exhibitions, theme park attractions, games and books. AR can also be used to make
games that can be played outdoors.

Commerce: AR can be used to display certain products at another way. For

example, the lego-boxes in the lego-store at Schaumburg are fitted with a QR code
which show the product when it's assembled. AR can also be used as an aid in
picking clothing.

Performance: AR can enhance concert and theater performances. For example,

artists can allow listeners to augment their listening experience by adding their
performance to that of other bands/groups of users.

Translation: AR systems can provide dynamic subtitles in the user's language.

Potential applications

Possible extensions include:

 Devices: Create new applications that are physically impossible in "real"

hardware, such as 3D objects interactively changing their shape and
appearance based on the current task or need.
 Multi-screen simulation: Display multiple application windows as virtual
monitors in real space and switch among them with gestures and/or
redirecting head and eyes. A single pair of glasses could "surround" a user
with application windows.
 Holodecks: Enhanced media applications, like pseudo holographic virtual
screens and virtual surround cinema.
 Automotive: eye-dialing, navigation arrows on roadways
 "X-ray vision"

University Of Calcutta | Augmented Virtual Reality 13

  Furnishings: plants, wallpapers, panoramic views, artwork, decorations,
posters, illumination etc. For example, a virtual window could show a live
feed of a camera placed on the exterior of the building, thus allowing the
user to toggle a wall's transparency.
 Public displays: Window dressings, traffic signs, Christmas decorations,
advertisements, interactive dressing room mirrors.
 Gadgets: Clock, radio, PC, arrival/departure board at an airport, stock
ticker, PDA, PMP, informational posters/fliers/billboards.
 Group-specific feeds: For example, a construction manager could display
instructions including diagrams at specific locations. Patrons at a public
event could subscribe to a feed of directions and/or program notes.
 Speech synthesis: Render location/context-specific information via spoken
words.
 Prospecting: In hydrology, ecology, and geology, AR can be used to display
an interactive analysis of terrain characteristics. Users can collaboratively
modify and analyze, interactive three-dimensional maps.

Software

Open source

I. Argon, augmented virtual reality browser by Georgia Tech's GVU Center

that uses a mix of KML and HTML/JavaScript/CSS to allow developing AR
applications; any web content (with appropriate meta-data and properly
formatted) can be converted into AR content; as of November 2011,
available for iPhone only
II. ARToolKit, an open source (dual-license: GPL, commercial) C-library to
create augmented virtual reality applications; was ported to many different
languages and platforms like Android, Flash or Silverlight; very widely used
in augmented virtual reality related projects
III. ArUco, a minimal library for augmented virtual reality applications based
on OpenCv; licenses: BSD, Linux, Windows
IV. Goblin XNA, a platform for researching 3D user interfaces, including
mobile augmented virtual reality and virtual reality, emphasizing games;
written in C#, based on Microsoft XNA Game Studio 4.0, BSD license
V. GRATF, open-source (GPLv3) project, which includes C# library for
detection, recognition and 3D pose estimation of optical glyphs. The project
includes application, which does 2D and 3D augmented virtual reality
University Of Calcutta | Augmented Virtual Reality 14
 VI. mix Augmented virtual Reality Engine (mixare), open-source (GPLv3)
augmented virtual reality engine for Android and iPhone; works as an
autonomous application and for developing other implementations
VII. Open Mobile Augmented virtual Reality (OpenMAR), component framework
for the Symbian platform, EPL license; as of November 2011, website is
down but some information exists elsewhere
VIII. PTAM, non-commercial use only

Free closed source

 metaio Mobile SDK offers free natural features tracking that is available for
both Android and iOS.
 Qualcomm AR SDK (QCAR) is a free AR solution for detection and tracking
of reference images and markers using natural feature detection that is
available for both Android and iOS.

AR Content Management Systems

 buildAR.com, a web based content platform for building geolocation and

natural feature tracking based mobile augmented virtual reality.
 Hoppala Augmentation, a web based content platform for creating
geolocation based mobile augmented virtual reality.

In popular culture

Television, film

A. The television series Dennō Coil depicts a near-future where children use
AR glasses to enhance their environment with games and virtual pets.
B. In the Terminator movie series, all Terminator models, beginning with T-
800 series, use augmented virtual reality systems to "see".
C. The television series Firefly depicts numerous AR applications, including a
real-time medical scanner which allows a doctor to use his hands to
manipulate a detailed and labeled projection of a patient's brain.

University Of Calcutta | Augmented Virtual Reality 15

 D. In the 12993 ABC miniseries Wild Palms, a Scientology-like organization
used holographic projectors to overlay virtual reality images over physical
reality.
E. In the movie Iron Man, Tony Stark (Robert Downey Jr.) uses an augmented
virtual reality system to design his super-powered suit. The suit itself also
uses augmented virtual reality technology.

Reference

www.cse.msu.edu/~stockman/CV/F08Lectures/week14_VR_AR.ppt
www.slideworld.com/pptslides.../augmented-virtual-reality
graphics.usc.edu/~suyay/class/AR.pdf
www-robotics.cs.umass.edu/Papers/percom04.pdf
en.wikipedia.org/wiki/Reality–virtuality_continuum
en.wikipedia.org/wiki/Mixed_reality
en.wikipedia.org/wiki/Augmented_virtuality
en.wikipedia.org/wiki/Reality–virtuality_continuum

University Of Calcutta | Augmented Virtual Reality 16