You are on page 1of 58

The rise of AR & VR:

why this time is different?


Vasily Ryzhonkov,
Director of Mobile Tech Center,
Skolkovo Foundation
vryzhonkov@sk.ru
http://mobile.sk.ru
The Metaverse is coming…
Realtime networked 3D gaming was a
…and now we got this.
big step 19 years ago…

Source: Quake game by John Source: Neal Stephenson,


Carmack et al., 1996 Snow Crash, 1992
Metaverse (coined in 1994 by Stephenson) is a collective virtual shared space, created by the
convergence of virtually enhanced physical reality and physically persistent virtual space, including the
sum of all virtual worlds, augmented reality, and the internet. (Wikipedia)
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Is this our future?
…and in 2015 Microsoft presented it’s
Fantasts have been dreaming about immersive vision of the future computing platform
computing technologies for a long time… (along with other concepts from
Facebook, Google, Sony, HTC, Valve …)

Source: Minority Report, 2002 Source: Microsoft, 2015


Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
VR/AR are the Next Big Thing

The Next Mega Tech Theme is Virtual Reality (VR).


• We believe virtual reality and augmented reality are the next mega tech themes through 2030.
• We liken the state of virtual and augmented reality today as similar to the state of mobile phones 15
years ago.
• It likely will take a decade before mainstream adoption as necessary improvements in displays and
applications as well as lower pricing are needed to drive demand.
• The first phase will be virtual reality, which we believe will start over the next several years, and is the
primary focus of this report; the second phase will be augmented reality.

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


Technology Definitions

Augmented Reality
Augmented reality, or AR, is when real reality, or RR, is enhanced with virtual
images that appear to coexist with the real world.

Virtual Reality
Virtual reality, or VR, is when you’re immersed in a purely virtual world

Mixed Reality

Mixed Reality (MR) or Diminished Reality, refers to a system that combines


real and virtual objects and information using augmented virtuality, which is
combines some reality with virtual world.

5
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Technology Definitions

What is Total Immersion?


According to Wikipedia, Immersion into virtual reality is a perception of being physically present in a non-
physical world. This combines the sensory perceptions of the human mind to give a surreal impression of
living in another world, constructed of Virtual or Augmented reality. Additionally, Virtual reality glasses
can produce a visceral feeling of being in a simulated world, a form of spatial immersion called Presence.
According to Oculus VR, the technology requirements to achieve this visceral reaction are low-latency and
precise tracking of movements

6
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Ontology of AR / VR
Father of Augmented Reality,
Steve Mann

Source: "Augmented Reality: A class of displays


on the reality-virtuality continuum“, Milgram,
Paul; H. Takemura; A. Utsumi; F. Kishino, 1994

Source: “Mediated Reality with implementations


for everyday life”, Steve Mann, 2002

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


0 WAVE: Infancy
Mirror stereoscope: Sir
Charles Wheatstone
1840 1841 - 1860
Stereoscope: Sir
David Brewster
Sir David Brewster created
a new stereoscope by
utilizing prismatic lenses
Stereoscope: An optical illusion instrument, giving
depth of field through two pictures of the same object
Viewing 3D movies through stereoscope at home
Mainstream 3D Cinema begins
1923 - Present 1922
First 3D film
produced in
Hollywood: The
Power of Love
8
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
1st WAVE: Inventors & Dreamers

1930 1957 1965

“Ultimate Display”, a VR
helmet with two CRT
screens, was created by
Link trainer: Invented by Sensorama: First VR machine Ivan Sutherland. The
Edwin Link for training developed by Morton Heilig, two CRT screens
military pilots, using gives visual and sensory enabled the viewer to
simulation perceptions of virtual reality, experience the 3D
including vibrations, smell images
and touch.

9
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
2nd WAVE: Industrial & Military Applications

1966-68 1972

Thomas A. Furness III developed


the first VR head mounted
General Electric Corporation built one of the first computerized
display cockpit for the US Air
flight simulators. It provided 180-degree FOV (Field-of-View) by
Force, with Bob Sproull.
using three screens surrounding the training cockpit.

1980- Present NASA: Virtual Visual Environment


Display, head mounted unit for VR
graphics and simulation using LCD
display and wide angle lens.
10
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
3rd WAVE: Industrial Applications & Gaming

DHL: Using Augmented Using an iPad, with an


ESI: Improved data handling, application specially
and Virtual Reality, DHL can
warehouse management and developed by Alstom
optimise parcel and load
capacity forecasting using AR Transport, all maintenance
handling, customer delivery
and VR technologies tasks are described with
efficiency and provide
detailed tracking precision superimposing
in real time a 3D virtual
model on the operator’s
Nintendo Virtual Boy : Sega VR: 1994, perception of reality
1994, price - $180. prototype

11
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
4rd WAVE: Next computing platform
The 4th wave of virtual and augmented reality began, when
Facebook bought, the VR pioneering company Oculus for $ 2
Billion US Dollars.

The acquisition sparked a market speculation and generated a


renewed interest in VR and AR technology. The Google Trend
graph depicted in adjacent, shows the interest in the
technology over time since 2014.

Google Glass, launched in 2013 is another example built on


(optical head-mounted display ) the next generation
computing platform using Ubiquitous computing for
displaying real time AR information through lens.

Google Trends: Virtual Reality

12
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Predictions and hype present and in the past
The only outcome I can make is …predictions are always wrong…but they can
indicate media hype and technological advances

Source: Gartner (2008) Source: Gartner (2015)


Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Predictions and hype present and in the past

“Are these expectations


realistic or do they result from
the initial euphoria of an
innovative technology?”, -
Economist, 1996

Source: Frost & Sullivan, 4th Wave in:Bauer (1996), OVUM in Leston et al. (1996),
Machover (1996)

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


7 Failed Virtual Reality Technologies
The Sensorama : Morton Nintendo Virtual Boy : Omnidirectional Treadmills
Heilig, inv - 1957, patent - 1994, price - $180.
1962
The Virtusphere : Ray
and Nurulla Latypov,
1995, price - $100K

VRML, 1995
The Sword of Damocles :
VRML was billed as a 3-
Ivan Sutherland , 1968
Power Glove : Nintendo, D alternative to HTML.
1989, price - $75 The idea was that users
could interact freely with
3-D worlds on the
internet, described by
text and interpreted by
modeling software.
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Why AR & VR didn't get broad adoption?

• Technical imperfection
• Cost
• Bulky & Bad design
• Failed to deliver value to the end user

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What's now and why 2014th is the cornerstone?

Facebook’s Oculus Rift and Sony Playstation.VR is only beginning…

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What's now and why 2014th is the cornerstone?

Critical mass of elements has come together

Technical: Organizational and economic:


• flat-panel displays • critical mass of companies is working together
• cell phone panels (good megapixels) • price and affordability
• optics (wide fov) • ergonomics and design (not bulky)
• mobile CPUs & GPUs
Technical:
• batteries and power management
• computer vision
• wireless
• tracking hardware
• gyroscopes and accelerometers
• content 3d games
• cameras
• latency***
• projectors

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


AR & VR Ecosystem

19
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
AR & VR Ecosystem : Cont.

20
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
AR & VR Ecosystem : Cont.
DISPLAY HARDWARE: [HMD]: AlterGaze, AntVR,
FOVE, AVEGANT, HTC, Oculus, Sony, Panasonic,
VRVANA, Razer, SENSICS, Immersion VRELLA, Light
& Shadows & Mindmaze.

[Wireless/ Mobile VR]: Moar, LG, ZEISS, Wearality,


USENS, Homido, Samsung, GearVR & VR One.

[GAMING]: Activision, Blizzard, CCP, Epic Games, nDreams, Otherworld,


Reload, Owlchemy Labs, Vertigo Games & VR Bits.

[CAVE]: Visbox, Worldviz, Christie and mechdyne.

21
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
VR Ecosystem
Number of VR companies Investments in VR

Top 10 companies in VR based on Total Funding Raised

Source: Greenlight VR, UploadVR


22
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
VR Ecosystem

Total funding raised by all VR companies

VR Investment landscape

Source: Greenlight VR, UploadVR

23
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape
There are more than 60 companies in Russian AR/VR market.
Mostly these companies are digital studios. However, there are high-tech companies with proprietary technology
developed in-house.

24
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape: Cont.

Augmented reality companies:

За 2014 год произошел резкий всплеск


образования компаний,
занимающихся интерактивными
технологиями (40% компаний).

25
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape: Cont.

Virtual reality companies:

26
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape: Cont.

AR / VR continuum: controllers / 3D visualization & 3D content creation / haptic & sensory feedback / images
& video search and analysis / gesture recognition, computer vision etc.

27
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Current Market Analytics :- Revenue Users and Investments

Given the scale of customer adoption, it has


given rise to steady revenue over time, totalling
over 62 $ Million US dollars by 2017 with AR
revenue share consisting of 600 $ Million US
Dollars by 2017. The agencies suggest that a total
of revenue of 160 $ Million US Dollars by 2018,
with head mounted display sales leading to 117 4
Million US Dollars.

28
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Current Market Analytics :- Market Share

Different analytical companies show different


estimates. Piper Jaffray suggests that VR
market can grow up to $7 bln by 2025; some
agencies assess market as $150 bln by 2020
consisting of AR and VR market share.

Upon further analysis, we see that market will


consist of VR Hardware, VR games, VR Film &
Concerts, VR amusement parks, VR
Entertainment, VR speciality products (such as
adult content), VR industrial applications.

29
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Market Analytics : Revenue generation and adoption

Piper Jaffray report:

“We believe technology advances over the next 3-5 years


will continue to drive down the cost of these technologies
and enable significant improvement in tracking and
rendering. We expect CPUs and GPUs of computers to
continue along Moore’s Law which should enable
cheaper computers to eventually run wired headsets
headsets. We expect VR will gradually drop from an all
in entry cost of $1000+ (VR headset $400 + high
powered computer $1,000) to a more reasonable $500+
(VR headset $250 + computer $300) for VR headsets
that offer high levels of immersion”

Conclusion: HMD price will be key driver of the mass


adoption.
30
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Market Analytics : Revenue generation from HMD

Based on the information above, we can conclude that VR HMD will drive the industry forward.
However, content will be king. That’s why a lot of VR companies invest heavily in VR studios, create SDK
for developers. And recently Oculus Rift announced that hundreds of games are in the process of the
development and thousands of developers are involved in this process.
31
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Gaming market dissection

During the third quarter of 2014 , KZero, a


market analytics firm conducted a market
research within the gaming section of VR
sectors.

It was found that RPG style games were most


popular amongst gamers who are older than 30
years old, with simulation and exploration type
games being popular between 20-30 year old
gamer

Based on the red colour depiction within


Simulation, it can be inferred that majority of the
games are in late developmental or in released
stage, whilst RPG (Role Playing Games) are in
early development or alpha stages.
32
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Analysis Conclusion
1. Strong correlation can be seen with other
market technologies like gaming, 3D, motion
tracking, which are interconnected.
2. It has been estimated that, there will be 170
million active users in 2018, accounting to the
total gaming population of 5.2 billion at large.
3. Gaming industry is dominated by software
sector and is believed to remain that way
4. Most opportune and profitable moment in
cycle is between 2017 and 2018, for technology
will be developed and available to bigger mass,
leading to a predominant early majority
generating the highest fixed return per user.
Highest return on adoption is probably
somewhere near the start of next year,
5. To conclude, Augmented Reality is expected to
rise faster than VR, due less complexity,
increased adaptability, more interoperability,
less motion sickness and simulation sickness

33
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
VR/ AR Market drivers and projections

1. An estimated 3000+ million


mobile subscribers are expected in
2020, having a penetration rate of
51% at a global average.

2. This provides Augmented Reality,


a development scope in games,
enterprise application and location
based services.

3. It has been estimated that USA,


Japan and SE Asia will be the most
profitable markets for AR.

4. The key market drivers for AR are,


mobile devices, app stores and
gaming apps.
34
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Challenges and barriers of AR & VR
Economic and organizational risks of Virtual reality:

• Physical Risk. Any discrepancy between what the user sees and feels
could lead to motion sickness.

• Security Risks. For example, if users are able to modify the appearance
of their avatars, they could potentially impersonate other users.

• Privacy Risks. A virtual environment can dramatically expand the scope


of potential privacy violations because every single behavior in a virtual
environment can be tracked, and every element of a virtual
environment can be manipulated.

• Price. Many experts think that the limiting factor for virtual reality is
going to be the price. The reality is a bit more complicated.

• Value for customers. Consumer appetite for VR is hard to predict and


need to be stimulated.
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Challenges and barriers of VR
Barriers from technological perspective: There are four really hard problems that if solved
• GPU and CPU horsepower would make VR far more convincing:
• Input • Tracking
• Tracking • Latency
• Computer vision • Stimulating the human perceptual system to produce
• Display results indistinguishable from the real world
• Audio • Virtual reality sickness
• Haptics
• Sense of empathy
• Etc…

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


Use case: GPU horsepower needed for mobile VR
Goal is to calculate GPU horsepower needed for low-end but realistic mobile VR immersive HMD
Information given: Calculation:

1. Display Resolution: 1080p 1. The GPU needs 50 bytes per polygon, which then is
2. Frames per second: 60-90 multiplied for 12 million polygons, would be 600
3. Depth of resolution: 24 bit Megabytes per second, with 60 frames per second.
4. Texture information: 12 2. In order to calculate the Bandwidth, it will be : 600
million per scene megabytes x 60 frames/sec = 36 GB/sec [Pass 1]. For 90
5. Latency: <20 ms frames/sec it would be 54 GB/sec
6. Pixel Density: 300 dpi 3. According to OpenGL and DirectX best practices, a
minimum of 3 passes on each polygon must be made.
This is done in order to draw the polygon, emulate its
animation properties and to display its texture

NVIDIA
Tesla GPU Pascal GPU
37
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Use case: GPU horsepower needed - Cont.
Information given: Calculation:
1. Display Resolution: 1080p
2. Frames per second: 60-90
3. Depth of resolution: 24 bit 4. Therefore, the bandwidth required by the GPU would be 36 GB/sec x
4. Texture information: 12 million per 3 = 108 GB/sec. For 90 frames it would be 54 GB/sec x 3 = 162 GB/sec
scene
5. Latency: <20 ms 5. Additional considerations: Texture characteristics and attributes, such
6. Pixel Density: 300 dpi as shining, shading, colour and material need to be analysed and
CPU vs GPU Performance incorporated within these parameters
6. In order to rasterize, 1 polygon, it would need around 100 floating-
point operations. Therefore, for 12 million polygons, it would be:
12,000,000 x 100 flops x 60 frames/sec x 3 raster passes,
it would equate to around: 216 GFLOP/ sec
For 90 fps – 324 GFLOP/s

Based on the information above, NVIDIA have suggested GTX 970 to be


the capable GPU according to the requirements.
GTX 970 has 196+28 GB/sec – 3494 GFLOP/s – 145 Wt - - $329 – June,
2014
Top is GTX Titan X – 336 GB/sec – 6144 GFLOP/s – 250 Wt - $999 – 1kg –
Mar, 2015 38
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Use case: Power consideration.
Information given:

1. Power Required: 145 Watts per


Hour
2. Duration: 1, 2, or 8 hours per cycle
3. Battery Type: Lithium Ion

Therefore, energy stored in the battery for scenario 1 would be: 145
Wt*h
Scenario 2: 290 Wt*h
Scenario 3: 1160 Wt*h
For commercial implementation, Lithium Ion gives best performance
and is supported by wide industry standards.
The typical energy density stored in a mobile phone battery is
between 0,9 – 2,3 MJ/L. This equates to 740 Wt*h/L.

39
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Use case: Battery required to power GPU for realistic scene
Scenario 2:
Volume = 0,000391 m3
Equal dimensions = 0,073m

Scenario 3:
Volume = 0,001567 m3
Equal dimensions = 0,12 m

Outcome:
1. For mobile VR system with realistic graphics
working for 1 hour we will need battery with 6
cm x 6 cm x 6 cm dimensions.
2. For mobile VR HMD working at least 8 hours
we will need a battery with 12 cm x 12 cm x 12
cm.

Energy density of Lithium Ion batteries over the years, Battery University, 2014

40
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Complexities within VR & AR

Based on the data obtained from AMD, with Oculus being the benchmark
IMAGE DISPLAY standard, it is evident from the information table above, that a ordinary
Graphics IMAGE RENDERING gaming PC would be unable to suffice the demanding computations for
Sub- DRIVER PROCESSING on CPU this technology.
system SPATIAL AUDIO
PIXEL PERSISTENCE VR Additionally, in order to improve the visual experience the whole sub-
HEAD TRACKING Headset system, starting from input control to image display needs to be optimised
BODY TRACKING for minimal latency and maximum throughput.
HAPTICS
Other
INPUT Peripherals This includes, rendering in 2D, 3D and immersing displays with texture
based rendering.
41
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
CPU and GPU Performance over time
The adjacent CPU/GPU graph suggests, using Moore’s Law,
coupled with Nanotechnology together with advancements
in parallel computation has boosted the overall performance
of GPU in relation to graphic intensive computation.

Based on both graphs, it can be inferred that price of


computing will do down significantly, whilst the performance
gains are expected to be exponential over time.

According to the computation by AI Impact, it suggests that the


price of a gigaFLOPS falls by an order of magnitude roughly
every four years (14 orders of magnitude in 54 years is 3.9 years
per order of magnitude).

Since 1997, each order of magnitude only took three years (5.7
orders of magnitude in 18 years).
42
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Key outcome of the GPU and Battery case
GPU horsepower case Battery case
1. There are cards that already can satisfy minimum
1. Batteries will still be the problematic part of
requirements for realistic immersive VR systems. But
the equation. The energy density/weight
they will work only with high-end laptops and PC for
and energy density/volume ratios are
the next 5 years.
progressing not so fast as energy
2. According to current trends the GPU horsepower will
density/cost ratio. It’s very critical issue for
be increasing, but the power and weight will also be
mobile VR systems and will not be solved
increasing – 250 Wt - $1000 price and 1 kg weight are
for the next 5-10 years according to current
limiting factors for mobile VR.
trends in technology.
3. This leads us to the total cost of VR system for the
end user (gaming market). It will consist of laptop +
high-end GPU + HMD + content = $2000 + $1000 +
$ 1500 + $1000 = $5500 per year per 10 pieces of
content. A bit expensive, isn’t it? The only way I see is rethinking of
4. Mobile VR will be developing with low-end graphics computing architectures. What AMD,
and will not give opportunity to be totally immersed. Nvidia, and Intel already are doing….
5. Mixed systems will be developing where
computational power will be in the backpack of the
gamer
43
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Progress within VR & AR
Recently, AMD has launched Project: LiquidVR® In other developing events, Intel has partnered
dedicated to VR and AR technology. According to with Oculus Rift to form VR alliance in
AMD, GPU software and hardware sub-systems to association with Altspace VR to establish a
tackle the common issues and pitfalls of achieving common platform for better user experience.
presence, such as reducing motion-to-photon latency
to less than 10 milliseconds. Additionally, 6th generation Intel processors
have incorporated Intel RealSense®
“LiquidVR is a silky smooth player” –
Technology which is a gesture based Human-
The Guardian
Computer Interaction model to process and
compute AR/VR information.

Microsoft has also made extensive efforts to


incorporate VR and AR techno in the latest
“GameWorks VR: Smarter graphics for release of Windows 10, and are working on
virtual reality” - CNET releasing the HoloLens (VR HMD) device logy in
2015.

44
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Challenges and barriers of AR

These are AR technological barriers:

• Wide-field-of-view see-through display


• Input
• Mobility The ODG Smart Glasses

• GPU and CPU horsepower


• Battery
• Room-scale tracking
• Walk-around tracking Source: ODG Group

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


Critical Success Factors : Vision 2020

Mobility “AR needs to become more Cheaper


mobile than mobile to Increased
cost of
Vision battery life
compete,“ devices
Immersion – Binocular IO
Usability
Flexibility
Wearability Wider
Portability availability
“While internal market of content
competition is inevitable, the
fight that really matters is Affordable
Manufacturing Elimination
between AR/VR’s installed base
and the huge incumbent of motion
smartphone and tablet market” Affordable sickness
– DIGI Capital Price

46
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
What are key areas for innovation in AR & VR?
VR will kick off graphics into hyperdrive :
• 90 hz stereo, Can't miss frames
• 110 fov
• Resolution of 10k x 10k
• High rendering quality
• New rendering techniques
• GPU performance 2 to 3 orders of magnitude
• Transmission links
• Display ( 100 megapixels and more, thousands of pixels per inch, HDR)
• Rethinking art and animation technology
• Optics
• Rendering engines and content creation platforms
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
What are key areas for innovation in AR & VR?

• Wide field of view (immersion, peripheral cues, 100-110 degrees)


• Adequate resolution (problematic for VR due to wide FOV, one-seventh the pixel density per
degree of a wide-screen TV - 26p, 1080p works, more is better)
• Low pixel persistence (necessary to avoid blurring with eye motion, no more than 3 ms)
• A high enough refresh rate (to avoid flicker with low persistance, 95 Hz seems to be sufficient)
• Global display (illuminates all pixels simultaneously, avoid motion-induced compression,
stretching, and skewing)
• Optics (only one or at most two lenses per eye, not enough to allow simultaneous control of
focal length, viewing distance, size, distortion, various forms of aberration)
• Mobility

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What are key areas for innovation in AR & VR?
• Optical calibration ( the human visual system is amazingly sensitive to deviations, many
problems are impossible to identify until calibration is done )
• Rock-solid tracking (must support translation (position in x, y, z) as well as orientation, position
with at least milimeter accuracy, orientation with at least quarter-degree accuracy, volume at
least 1,5 meters on a side)
• Low latency (20 ms motion-to-last-photon works, 25 ms may be good enough)
• Resolution (The math for resolution together with wide fields of view is brutal; divide 1K by 1K
resolution – about the best an affordable head mounted display is likely to be able to do in
the next year – into a 100 - degree FOV and you get a display with less than 1/50th the pixel
density of a phone at normal viewing distance)
• Input

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


Implications and how it will change any aspect of our life

VR will influence all aspects that it touches:


• Presentation
• Art and creation methods
• Ergonomics
• Productivity (the way we work)
• Locomotion
• Social interaction
• Total immersion and mixed reality

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What is total immersion? Mixed reality as a new media…

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


Total Immersion Required Characteristics

Unique characteristics of immersive virtual reality:


• Head-referenced viewing provides a natural interface for the navigation in three-dimensional
space and allows for look-around, walk-around, and fly-through capabilities in virtual
environments.
• Stereoscopic viewing enhances the perception of depth and the sense of space.
• The virtual world is presented in full scale and relates properly to the human size.
• Realistic interactions with virtual objects via data glove and similar devices allow for manipulation,
operation, and control of virtual worlds.
• The convincing illusion of being fully immersed in an artificial world can be enhanced by auditory,
haptic, and other nonvisual technologies.
• Networked applications allow for shared virtual environments.

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What is total immersion? Mixed reality as a new media…

Technical requirements: Apply that to virtual avatar:


• Face tracking
• Input
• Eye tracking
• Audio
• Head tracking
• Interaction
• Hands tracking
• Haptics
• Body tracking
• Perceptual research :
• Position tracking
psychology and other
human factors

Multimodal interactions are key to VR:


• Feedback loop (see the object, touch it, sense it)
• Correspondence between insights
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
AR & VR as Next Computing Platform

• Drives the perceptual system as its built to be driven


• Different and more powerful mode of interaction
• Proper superset of all existing means of communication
• In the limit, its as big as reality ... or bigger

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What the future of VR & AR will look like?

● Virtual reality will be most popular in the gaming market, because of the high level of immersion
possible in VR, as well as the high demand for interactive experiences.

● Augmented reality will be most successful in the enterprise market, for applications in logistics,
engineering, and automotive.

● Applications such as education, travel, and design are served well by both AR and VR, and success
depends more on specific needs than general application.

● New content mediums, such as interactive VR video, are becoming more popular and catching the
attention of content creators as a potentially viable medium. Existing software solutions, such as CAD
software, can be outfitted to support AR and VR devices, and open a new approach to using the
software.

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


What's next? Possible roadmaps

Scenario 1.
• VR will be the first in mass adoption.
• The roadmap for VR is likely to be pretty straightforward: gaming market,
entertainment market.

Scenario 2.
• AR will be second and after 2020.
• Industrial applications of AR find it’s slow way to the market overcoming
technical & organizational barriers.
• Information glasses succeed.

Scenario 3.
• Console AR revolutionizes living-room gaming.
• Next step – whole new computing platform. Laptops and PCs are replaced
by HMD.

Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru


The future of VR & AR
As we progress into the 21st Century. Augmented
and Virtual Reality will play an integral part of our
lives, helping and assisting us in every way
imaginable.

Given the significant advancements within the


same, this technology is set to become multi-
faceted stream of communication sectors, from
news media to transport and logistics to name a
few.
“Ten years ago, this capability was purely in the realm of futurists and
seemed like a dream. Now, it feels like a very real possibility that will
enable us to experience nature, architecture, history and culture in new “A natural next step for VR is full-immersion home
and meaningful ways, and possibly lead to a new renaissance of art and entertainment” - TechCrunch
discovery”. – The Washington Post
“The Future Is Right In Front Of Our Eyes” -
“Another use case for VR photography could be news journalism” TechCrunch
– ABC News
“Virtual Reality is inevitable” - Palmer Luckey
“The return of virtual reality: 'this is as big an opportunity as the internet” – “The Future Of Virtual Reality Might Be Present” –
The Guardian The Forbes

57
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Our Team 58

Vasily Ryzhonkov,
Director of Mobile Tech Center,
Skolkovo Foundation
vryzhonkov@sk.ru
http://mobile.sk.ru

Joshua Levy
Intern at Mobile Tech
Center, Skolkovo
Foundation

58
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru