Professional Documents
Culture Documents
ARIM Unit 2
ARIM Unit 2
By
3
Augmented and Mixed Reality Taxonomy
• Usually, the term mixed reality is used interchangeably with augmented reality.
• However, mixed reality is a broader interpretation that consists of anything of
both the physical world and the digital world.
• Thus, an example of the GPS mapping described earlier would qualify as “mixed
reality” even though it is not considered an “augmented reality” application in
this book.
• It is mixed reality in that it is mixing real-world information (where I am) with
digital information (an abstract map display).
• Another type of mixed reality application is to use a real world object to interact
with a digital world, using that object in the way it is used in the real world.
• Note that all AR applications are mixed reality, but not all mixed reality
applications are AR
4
Augmented Reality Virtual Reality
• Augmented Reality System augments • Totally immersive environment
the real world scene • Visual senses are under control of
• User maintains &sense of presence in system (sometimes aural and
real world proprioceptive senses too)
• Needs a mechanism to combine
virtual and real worlds
5
Technology and Features of AR
Many industries and sectors already use AR for business processes, including:
• Retail. Employees can use AR for onboarding and training sessions. It helps new employees in their future
transactions, such as sales training, touring the sales floor, and preparing for a retail environment. AR can also
help customers test products before purchasing or learn how to use them within their environments. This can
create better engagement or help customers solve problems by providing actionable information in a real-
world context.
• Manufacturing. Technology can offer step-by-step instructions, allowing trainers to provide feedback during
practice for better retention. Using mixed reality also enables employees to learn while on the job, keeping
their hands free to perform work.
• Healthcare. Getting hands-on experience in performing procedures without risk is imperative for healthcare
professionals. AR provides the guidance to practically, yet safely, learn about anatomy and surgeries.
• Military. AR is integrated in combat training to stimulate situational and operational environments so soldiers
have awareness of their time, space, and forces.
• Automobile. AR can help train and allow specialists to explore current and future models, along with their
internal systems.
6
Challenges In AR
1. Hardware issues
• Currently, every available AR headset is a bulky piece of hardware that may be too expensive for
the masses. Also, a majority of AR headsets need to be tethered to a computer, making the entire
experience limited and inconvenient. Alternatively, consumers can use their smartphones or tablets
for AR applications. However, mobile AR faces major issues in displaying visuals accurately. For
instance, mobile sensors such as accelerometer can be disturbed by electric interference, which is
commonly witnessed in urban areas. Additionally, smartphone cameras are built for 2D image
capture and are incapable of rendering 3D images. Hence, the hardware required for AR
technology needs to be enhanced before mass adoption.
2. Limited content
• One of the major challenges with augmented reality is creating engaging content. The content
created for augmented reality devices consists of games and filters used in social networks such as
Instagram and Snapchat. However, creating content that can promote businesses can be extremely
complicated and expensive. Also, augmented reality developers have not created enough high-
functioning use cases that can be used by consumers on a daily basis.
7
Challenges In AR
3. Lack of regulations
• Currently, there are no regulations that help businesses and consumers understand which type of
AR applications can be used and how data can be processed. Hence, the technology can be used
with malicious intent. For instance, a cybercriminal can hijack personal accounts by mining data
output and manipulating AR content. In such cases, consumers may have questions like who could
be held accountable, which mitigation strategies can be used, and how to avoid such incidents in
the future. Hence, one of the significant challenges of augmented reality is creating regulations that
can ensure the privacy and security of consumer data as well as simplify mainstream adoption of
the technology.
4. Public skepticism
• Although augmented reality is a popular topic of discussion among tech experts, consumers are
unaware of the benefits of the technology. Consumers have only used the most popular
applications of augmented reality such as trying out glasses, wardrobe, and accessories. Therefore,
consumers need to be informed about various applications and benefits of augmented reality.
Additionally, a lack of awareness may lead to concerns about privacy and security while using
augmented reality technology. Hence, users’ concerns need to be addressed to accelerate the
mainstream deployment of augmented reality.
8
Challenges In AR
5. Physical safety risks
• Augmented reality applications can be immensely distracting and may lead to physical
injuries. For instance, many people were injured while playing Pokemon Go. Likewise,
augmented reality applications can lead to serious injuries in case they are used in
potentially risky environments such as busy roads, construction sites, and medical
institutions.
• Although augmented reality technology is still in its infancy, its existing applications have
shown that further research and development to address the challenges with augmented
reality can enable large scale deployment of the technology. And once that happens, the
implementation of augmented reality can be witnessed in law enforcement, healthcare,
finance, and other critical areas.
9
AR Systems and Functionality
• Organizations seeking to increase their efficiency by means of AR can begin today with little or very low risk. Many of
the essential ingredients for success are already in place and additional tools are available at comparatively low cost.
• It is helpful to think of AR as being a “plate” sitting upon three key elements: content (data), hardware and software.
1. Content:
▪ The spectrum of data that may be suitable for AR-assisted viewing ranges from small databases of enterprise
assets or resources and extends all the way to massive, continually expanding information repositories, often
referred to as “Big Data.”
▪ Sophisticated analyses must be performed on Big Data to extract benefits that can then be made accessible to AR
technology.
▪ For example, information about facilities or utilities typically includes a street address or latitude/longitude
coordinates that enable it to be correctly displayed in a Web browser or other software, when requested.
▪ A 3D model of a part in a power plant might display a barcode in a field on the screen that permits another
information retrieval system to associate that part with a particular pump or compressor in need of service or
replacement.
• A marker-based AR works on the concept of target recognition. The target can be 3D object, text, image,
QR Code or human-face called markers.
• After detection of the target by AR engine, one can embed the virtual object on it and display it on their
camera screen.
• An object can be recognized by extracting the 2D features from an image captured by a camera. If the
shape or the physical structure of the image is known, the process is known as a model based approach.
• Marker-less AR, also known as location-based AR, uses GPS of mobile devices to record the device
position and displays information relative to that location
• The object recognition process consists of two parts: 2D Vision and 3D Vision
1. 2D Vision
• It extracts 2D features of the objects to be searched
• The extracted and vectorized edges are matched with 2D views of the 3D object models. The pixel images
are pre-processed using a Sobel Filter and a Non-Maxima Elimination and finally vectorized using an Edge
detection algorithm
• Four steps of the 2D vision:
a) The vectorized edges
b) The virtually elongated edges
c) One match of the essential edges
d) One match of the 2D view including neighboring edges.
2. 3D Vision
• In the 3D vision part, the 2D features are compared with CAD data, containing highly visible
edges, faces and texture information
• With correspondences of image features and 3D-model features, hypotheses for the orientation of
the model relative to the camera are generated.
• Each generated hypothesis will be verified by projecting the model into the image plane. This
projection is compared with the extracted edge graph of the input image and the matching of both
graphs is evaluated.
• The best matching hypothesis is taken to determine the recognized object, its Location, and
orientation relative to the camera coordinate system
16
Visualization Techniques for augmented reality
• Visualization can be described as the process of converting abstract data into a visual
representation that is comprehensible by a human observer.
• The visualization process itself is often described step-by-step in one of the various versions of the
visualization pipeline.
• This allows for subdividing visualization methods into sub methods and provides a better overview
and abstraction of these methods.
• Visualizations in real world environments benefit from the visual interaction between real and
virtual imagery. However, compared to traditional visualizations, a number of problems have to be
solved in order to achieve effective visualizations within Augmented Reality (AR).
• AR visualizations have a high potential, however, their success is dependent on their
comprehensibility.
• If heedlessly implemented AR visualizations easily fail to visually communicate their information.
• The complex character of AR environments requires complex visualization techniques to neither
isolate certain structures nor to generate ambiguous presentations.
17
Visualization Techniques for augmented reality
• AR visualization is a powerful tool for exploring real world structures along with additional
contextual information.
• E.g.: By augmenting textual annotations, AR displays are able to provide semantics to real world
objects or places.
• Data flow in a common AR system. Real world imagery is delivered by the system’s video feed
and processed by vision based tracking algorithms. To align virtual and real data, the derived
tracking data has been applied to transform the virtual content. Finally, the rendering is overlaid on
top of the video feed
18
Visualization Techniques for augmented reality
• Data Integration:
▪ A simple overlay of hidden structure on top of the system’s video feed can cause a number of cognitive
problems, caused by the processes involved in creating the impression of depth. Understanding these
causes allows to develop rendering techniques which successfully add and preserve such information in
AR visualizations.
▪ Pictorial depth cues are those that can be found in a single image including:
➢ Occlusion: if the 2D projections of two objects in the environment overlap, objects which are closer
to the observer occlude objects which are further away.
➢ Relative size: more distant objects appear to be smaller than closer objects.
➢ Relative height: objects with bases higher in the image appear to be further away (compare the
stakes of the bridge).
➢ Detail: objects which are closer offer more detail.
➢ Atmospheric perspective: due to dust in the atmosphere, objects which are further away appear
more blurry than those which are nearby.
➢ Shadows: depending on the position of the light source, shadows can be cast from one object onto
another.
➢ Linear perspective: parallel lines converge with increasing distance. Notice how the sidewalks
seem to converge at some infinite place although in reality they appear to be approximately parallel.
19
Visualization Techniques for augmented reality
• Augmenting Pictorial Depth Cues:
▪ By rendering the virtual structure using a camera which uses parameters reflecting the characteristics of
the real camera, the fusion of virtual and real world imagery will automatically provide pictorial depth
cues which match to those present in the real world environment.
▪ Synchronizing the parameter of the virtual and the real camera allows to align real and virtual pictorial
depth cues. The virtual Lego figure in
(a) is correctly perceived next to the real figures, whereas the virtual one in
(b) is correctly perceived behind both. This effect is achieved by aligning depth cues such as perspective
distortion and relative size.
20
Visualization Techniques for augmented reality
• Occlusion Handling:
▪ While renderings from synchronized real and virtual cameras are already able to align depth cues, as soon as
occlusions between real and virtual objects appear, those depth cues are no longer sufficient to produce believable
augmentations (Fig. a). Even though all other depth cues would have been added to the AR display, the virtual object
will be perceived as floating in front of the video image. A believable integration of virtual structure into the real
world environment becomes only possible if occlusions between real and virtual objects have been resolved (Fig. b).
▪ Importance of occlusion cues (a) Even though a number of different depth cues exist, depth order is ambiguous and
perception is wrong if occlusions have been ignored (b) The same rendering as in (a) with occlusion correctly
resolved. This visualization is able to communicate the spatial relationship between its real and virtual content.
21
Visualization Techniques for augmented reality
• Image based X-Ray Visualization:
▪ AR scenes commonly suffer from incomplete virtual representations. Often only the video in combination with the
object of interest is available to the AR system.
▪ In such situations, the AR system requires knowledge about the organization of the scene in order to correctly sort
their elements.
▪ While this information is often difficult to acquire for a generalvisualization, in case of applications using x-ray
visualization to “see through” real world structure, the virtual data can often be assumed as being completely covered
by real world objects.
▪ In this case, the depth order is known, and the AR system can analyze the video stream only in order to preserve
important depth cues.
▪ In the following, we will review the extraction and preservation of image features which have been used to aid depth
perception in x-ray visualizations in AR.
22
Visualization Techniques for augmented reality
• Scene manipulation:
▪ The limitations of the AR system itself has to be considered as well in order to generate comprehensible
visualizations.
▪ In addition, hardware restrictions such as small display sizes, narrow fields of view or the limitations
caused by the egocentric nature of AR influence the comprehension of their visualization.
▪ As a remedy, spatial rearrangements of the objects within the mixed environment have been
demonstrated to be effective.
▪ Techniques used to deliberately modify real world imagery in order to increase the information content
are:
✓ Rearranging Real World Objects
✓ Space Distortion Visualization
23
Visualization Techniques for augmented reality
• Rearranging Real World Objects:
▪ Rearranged AR scenarios consist of real, virtual and relocated real information.
▪ To correctly compose an image out of all three types of information, the rendering algorithm has to fulfill
three requirements.
➢ It must be able to convincingly relocate real-world structures. Therefore, visual information has to
be transferred from its original to the target location after the explosion was applied.
➢ New imagery has to be generated to fill the original locations.
➢ The rendering algorithm has to correctly resolve occlusions between all used data.
▪ Three types of rearranging:
o Dual Phantom Rendering
o Synchronized Phantom Rendering
o Restoration
24
Wireless displays in educational augmented reality
applications
• AR is increasingly being adopted in educational settings, often to help students with complicated
subjects.
• For example, students struggling with geometry can use AR to see and manipulate 3D geometric
forms. Another application of augmented reality in education includes teaching global perspectives
through virtual field trips, enabling students to interactively engage with other cultures.
• Some educational applications are:
• Wireless Head Mounted Displays
• Wireless Handheld Display
25
• AR can have a significant impact on learning environments:
• Student engagement and interest: Student interest skyrockets with the opportunity to
engage in creating educational content. AR technologies can allow them to add to
curriculum content, create virtual worlds, and explore new interests.
• Learning environment: Classes that incorporate AR can help students become more
involved. An interactive learning environment provides opportunities to implement hands-
on learning approaches that can increase engagement, enhance the learning experience, and
get students to learn and practice new skills.
• Content understanding: Lack of quality content focused on education, rather than
entertainment, is a noted concern among teachers hesitant to use augmented reality in
education. However, existing AR technology enables teachers to create immersive
educational experiences on their own to help ensure their students understand curriculum
content.
26
• Collaboration: As AR content is digital, it is easily shared. For example, a group of teachers can
work with their students to continually refine the content. A collaborative learning environment
provides students with increased motivation to learn because they are actively engaged in the
educational content creation process.
• Memory: AR is an excellent tool for bringing lessons to life and helping students remember
essential details. For example, instead of just presenting photographs on a projector showcasing life
in Colonial America, a teacher can use AR technology to create memorable interactive stories.
• Sensory development: AR technology can help teachers create lesson plans with multisensory
experiences. Students benefit from immersive virtual content that incorporates an experiential
learning style in which students carry out physical activities instead of watching a demonstration.
This approach can help with sensory development.
• Cost-effectiveness: The cost of AR equipment is often cited as a barrier to adoption. However, as
smartphone use continues to rise among young Americans, and since smartphones are already
equipped with the hardware needed to run AR apps, augmented reality in education is increasingly
more cost-effective to implement. Additionally, AR can lower educational costs by replacing
expensive textbooks.
27
Mobile projections interfaces
• Projection-based AR is described as a video projection technique, which can extend and reinforce visual
data by throwing images on the surface of 3D objects or space; this belongs to Spatial Augmented
Reality in a broad sense
• Using projection-based AR, it is easy to implement graphical representation that ordinary lighting
techniques cannot express. Unlike general lighting technique, the technique can project high-definition
image or video, and change the object shape visually with the flow of time.
• With the increase in processing power and memory the only bottleneck left is the small display size and
resolution. To keep these devices mobile the size of the screen is restricted and even though the
resolution of such displays is increasing, there is a limit to information presentable on the display.
• The next step is to integrate such a projector directly into a mobile phone.
• These devices are also called projector phones. Up to now several different prototypes both from
research and industry exist. First commercial projector phones are available on the mass market
• Such phones have the capabilities to overcome the problems that arise when exploring large-scale
information on the small display of a present-day mobile phone.
• With these devices one can explore information like maps or web pages without the need for zooming
or panning but up to now the available devices are only projecting a mirror image of the devices display
or images and videos
School of Mech Engg. M.Tech Design 29
Marker-less tracking for augmented reality
• Marker less Augmented Reality (AR) refers to a software application that doesn’t require prior
knowledge of a user’s environment to overlay virtual 3D content into a scene and hold it to a
fixed point in space.
• Marker less AR experiences are possible because of advancements in cameras, sensors,
processors, and algorithms capable of accurately detecting and mapping the real-world.
1. In its most basic form, markerless AR superposes virtual objects into a static, pre-captured 2D
image. it’s straightforward and easy to implement for apps that want to offer offline AR instead
of live experiences.
2. Marker less AR systems that use RGB-D SLAM and sensor fusion approaches are on the
opposite end of the spectrum. Microsoft HoloLens is the most notable example. These systems
integrate information from standard, red, green, and blue (RGB) cameras with state-of-the-art
infrared time-of-flight cameras to construct a 3D map of the user’s surroundings while they use
the application. This feature is a critical component of the SLAM tracking paradigm, as it
enables apps running on these devices to place virtual content within the space concretely.
Pros Cons
• After that, the user may change his/her scope as he/she prefers.
• A pinwheel appears so that a region having the similar color
with the selected one is considered as a wing component of the
pinwheel, as shown in Fig.
• When the camera position changes, its view changes
accordingly, of course.
• Here, multiple regions in the real world may be matched for the
selected color.
• The largest one is chosen as the target in the current
implementation.
• A rough distinction can be made between quantitative methods, qualitative methods, non-user
based usability evaluation methods, and informal methods.
44