Head Mounted Display Optics II

Robert Konrad
Stanford University

EE 267 Virtual Reality

Lecture 8
stanford.edu/class/ee267/
 Lecture Overview
• focus cues & the vergence-accommodation conflict
• advanced optics for VR with focus cues:
• adaptive and gaze-contingent focus displays
• volumetric and multi-plane displays
• near-eye light field displays
• holographic displays
• Maxwellian-type displays

• AR displays
• Microdisplays used for VR/AR
 Magnified Display
• big challenge:
virtual image
appears at
fixed focal
plane!

d • no focus cues

d’
f

1 1 1
+ =
d d' f
Focus Cues – An Important Depth Cue
 Relative Importance of Depth Cues
Cutting & Vishton, 1995
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation
Blur Gradient Driven Accommodation

Rendering cannot replace this!

The Vergence-Accommodation Conflict (VAC)
 Real World:
Vergence &
Accommodation
Match!
Current VR Displays:
Vergence &
Accommodation
Mismatch
 Consequences of Vergence-Accommodation Conflict

• Visual discomfort (eye tiredness & eyestrain) after ~20 minutes of

stereoscopic depth judgments (Hoffman et al. 2008; Shibata et al.
2011)

• Degrades visual performance in terms of reaction times and acuity

for stereoscopic vision (Hoffman et al. 2008; Konrad et al. 2016;
Johnson et al. 2016)

• also: double vision (diplopia), reduced visual clarity

• Q1: How to address the vergence-accommodation
conflict?

• Q2: Should we address the vergence-accommodation

conflict for users of all ages?

• Q3: Can computational displays effectively replace

glasses in VR/AR?
 How to address the VAC?
Adaptive Focus Multiplane Light Field

Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Rolland et al., Applied Optics 2000 Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Maxwellian

Location 1 (red) Location 2 (green) Location 3 (blue)
1

Conventional
display pixel size

Intensity
0

Intensity
AI
0

Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017 Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
 How to address the VAC?
Adaptive Focus Multiplane Light Field

Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Rolland et al., Applied Optics 2000 Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Accommodation Invariant

Location 1 (red) Location 2 (green) Location 3 (blue)
1

Conventional
display pixel size

Intensity
0

Intensity
AI
0

Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017 Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
Current Generation Displays Fix Focus
Lens

d’
d

Display 1 1 1
+ =
Magnified Display d d' f
Adaptive Focus via Screen Actuation
actuator à vary d’

Lens

Display 1 1 1
+ =
Magnified Display d d' f
 Adaptive Focus via FTL focus-tunable
lens à vary f

Lens

Display 1 1 1
+ =
Magnified Display d d' f
 Adaptive Focus - History

manual focus adjustment automatic focus adjustment deformabe mirrors & lenses
Heilig 1962 Mills 1984 McQuaide 2003, Liu 2008

• M. Heilig “Sensorama”, 1962 (US Patent #3,050,870)

• P. Mills, H. Fuchs, S. Pizer “High-Speed Interaction On A Vibrating-Mirror 3D Display”, SPIE 0507 1984
• S. Shiwa, K. Omura, F. Kishino “Proposal for a 3-D display with accommodative compensation: 3DDAC”, JSID 1996
• S. McQuaide, E. Seibel, J. Kelly, B. Schowengerdt, T. Furness “A retinal scanning display system that produces multiple focal planes
with a deformable membrane mirror”, Displays 2003
• S. Liu, D. Cheng, H. Hua “An optical see-through head mounted display with addressable focal planes”, Proc. ISMAR 2008
Padmanaban et al., PNAS 2017
Padmanaban et al., PNAS 2017
Padmanaban et al., PNAS 2017
Padmanaban et al., PNAS 2017
Padmanaban et al., PNAS 2017
Padmanaban et al., PNAS 2017
 Conventional VR Displays cause VAC

Virtual image distance

Stereoscopic distance

vergence
accommodation
 Removing VAC with Adaptive Focus

Virtual image distance

Stereoscopic distance
vergence
accommodation
 Task

4D 0.5D
(0.25m) (2m)

Follow the target with your eyes

 Accommodative Response in Conventional Display
Stimulus
Virtual image
distance

Distance from mean [D]

Stereoscopic
distance
2

−2

0 5 10 15 20
Time [s]
Padmanaban et al., PNAS 2017
 Accommodative Response in Conventional Display
Stimulus
Virtual image User Response
distance Conventional Average
4 mean = 0.29
n = 59

Distance from mean [D]

Stereoscopic
distance
2

−2

0 5 10 15 20
Time [s]
Padmanaban et al., PNAS 2017
 Accommodative Response in Adaptive Display
Stimulus

4
Virtual image distance

Distance from mean [D]

Stereoscopic
distance
2

−2

0 5 10 15 20
Time [s]
Padmanaban et al., PNAS 2017
 Accommodative Response in Adaptive Display
Stimulus
User Response
Dynamic Average
Virtual image distance
4 mean = 0.77
n = 24

Distance from mean [D]

Stereoscopic
distance
2

−2

0 5 10 15 20
Time [s]
Padmanaban et al., PNAS 2017
 Gaze-contingent Focus
• adaptive focus is like real world, but needs eye tracking!
render in retinal blur
virtual image
HMD

micro
display lens

eye
tracking

Padmanaban et al., PNAS 2017

 Gaze-contingent Focus

Padmanaban et al., PNAS 2017

 Gaze-contingent Focus
Oculus Half Dome

Padmanaban et al., PNAS 2017 Video Courtesy of Facebook, F8

 Nearsightedness, farsightedness, and presbyopia

Focal range (range of clear vision)

Normal vision

Nearsighted/myopic

Farsighted/Hyperopic

Presbyopic

25cm Optical
Infinity
Modified from Pamplona et al, Proc. of SIGGRAPH 2010
Importance of Focus Cues Decreases with Age - Presbyopia
Nearest focus distance

0D (∞cm)

4D (25cm)

8D (12.5cm)

12D (8cm)

16D (6cm)
8 16 24 32 40 48 56 64 72
Age (years)
Duane, 1912
 Do Presbyopes Benefit from Dynamic Focus?

Padmanaban et al., PNAS 2017

 Do Presbyopes Benefit from Dynamic Focus?

Padmanaban et al., PNAS 2017

 Do Presbyopes Benefit from Dynamic Focus?

Padmanaban et al., PNAS 2017

 Do Presbyopes Benefit from Dynamic Focus?

Response for Physical Stimulus

Heron & Charman 2004

Padmanaban et al., PNAS 2017

at ACM SIGGRAPH 2016
 Adaptive Focus Summary

• adaptive focus drives accommodation and can correct for refractive errors
(myopia, hyperopia)

• gaze-contingent focus gives natural focus cues for non-presbyopes, but

require eyes tracking

• presbyopes require fixed focal plane with correction

 How to address the VAC?
Adaptive Focus Multiplane Light Field

Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Rolland et al., Applied Optics 2000 Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Maxwellian

Location 1 (red) Location 2 (green) Location 3 (blue)
1

Conventional
display pixel size

Intensity
0

Intensity
AI
0

Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017 Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
Current Generation Displays Fix Focus
Lens

Display 1 1 1
+ =
Magnified Display d d' f
Multifocal Displays present multiple virtual images
fast switching
between fixed
f’s
Lens

Display 1 1 1
+ =
Magnified Displays d d' f
 Multifocal VR Displays

e r
k
lf ic
:
e m
b l
idea introduced
r o
benchtop prototype near-eye display prototype
Rolland et al. 2000
t pAkeley 2004 Liu 2008, Love 2009
•
•
e s
Rolland J, Krueger M, Goon A (2000) Multifocal planes head-mounted displays. Applied Optics 39
Akeley K, Watt S, Girshick A, Banks M (2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH)
•
g
Waldkirch M, Lukowicz P, Tröster G (2004) Multiple imaging technique for extending depth of focus in retinal displays. Optics Express
g
•
•
b i
Schowengerdt B, Seibel E (2006) True 3-d scanned voxel displays using single or multiple light sources. JSID
Liu S, Cheng D, Hua H (2008) An optical see-through head mounted display with addressable focal planes in Proc. ISMAR
• Love GD et al. (2009) High-speed switchable lens enables the development of a volumetric stereoscopic display. Optics Express
• Mercier et al. (2017) Fast Gaze-Contingent Optimal Decompositions for Multifocal Displays. ACM Trans. Graph. (SIGGRAPH Asia)
• … many more ...
 How to address the VAC?
Adaptive Focus Multiplane Light Field

Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Rolland et al., Applied Optics 2000 Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Maxwellian

Location 1 (red) Location 2 (green) Location 3 (blue)
1

Conventional
display pixel size

Intensity
0

Intensity
AI
0

Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017 Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
Light Field Displays Approximate the Wavefront

Retinal Image
Light Field Displays Approximate the Wavefront
Reconstruction of Light Field

Retinal Image

Idea: project multiple different perspectives into different parts of the pupil!
Light Field Displays Approximate the Wavefront
Reconstruction of Light Field

Retinal Image

Idea: project multiple different perspectives into different parts of the pupil!
No eye tracking or retinal blur rendering required!
Number of views needed to drive accommodation
Farther from focus plane

Physical world
retinal blur

The more views,

the more accurate the
blur
 Approaches to generating light fields
Integral Imaging Virtual Retinal Displays Tensor Displays

Lanman et al., SIGGRAPH 2013

Hua et al., Optics Express 2014 Jang et al., SIGGRAPH Asia 2017 Haung et al., SIGGRAPH 2015
 Approaches to generating light fields
Integral Imaging Virtual Retinal Displays Tensor Displays

Lanman et al., SIGGRAPH 2013

Hua et al., Optics Express 2014 Jang et al., SIGGRAPH Asia 2017 Haung et al., SIGGRAPH 2015
 Approaches to generating light fields
Integral Imaging Virtual Retinal Displays Tensor Displays

Lanman et al., SIGGRAPH 2013

Hua et al., Optics Express 2014 Jang et al., SIGGRAPH Asia 2017 Haung et al., SIGGRAPH 2015
 Light Field
Light Stereoscope
Field Cameras

Huang et al., SIGGRAPH 2015

 Light Field Stereoscope
Thin Spacer & 2nd panel (6mm)

Backlight LCD Panel

Magnifying Lenses
Huang et al., SIGGRAPH 2015
 Input: 4D light field for each eye

Target Light Field

Multiplicative Two-layer Modulation Input: 4D light field for each eye
Multiplicative Two-layer Modulation Input: 4D light field for each eye
Multiplicative Two-layer Modulation Input: 4D light field for each eye
 Multiplicative Two-layer Modulation Input: 4D light field for each eye

Reconstruction:
for layer t1
Tensor Displays,
Wetzstein et al. 2012
 Light Field Stereoscope

Traditional HMDs The Light Field HMD

- No Focus Cues Stereoscope
Huang et al., SIGGRAPH 2015
 Light Field Stereoscope

Traditional HMDs The Light Field HMD

- No Focus Cues Stereoscope
Huang et al., SIGGRAPH 2015
 Light Field Stereoscope

Traditional HMDs The Light Field HMD

- No Focus Cues Stereoscope
Huang et al., SIGGRAPH 2015
 Light Field Stereoscope

Traditional HMDs The Light Field HMD

- No Focus Cues Stereoscope
Huang et al., SIGGRAPH 2015
 Vision-correcting Display

printed transparency iPod Touch prototype

Huang et al., SIGGRAPH 2014
 prototype

300 dpi or higher

Huang et al., SIGGRAPH 2014

 How to address the VAC?
Adaptive Focus Multiplane Light Field

Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Rolland et al., Applied Optics 2000 Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Maxwellian

Location 1 (red) Location 2 (green) Location 3 (blue)
1

Conventional
display pixel size

Intensity
0

Intensity
AI
0

Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017 Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
 Holographic Displays Reproduce the Wavefront
Trade off between exit
Computationally Expensive! pupil and field of view
3D Object gram
Holo SLM)
g
min
Inco t
Ligh e(
Object Spherical Plan
point wave

e
Phas
Lens tion
Func

gram
-holo
Sub

t
Ligh
sed
Focu t point)
SLM (obje
c

Image courtesy of: Maimone et. al. 2017

CGH can correct higher order aberrations
 How to address the VAC?
Adaptive Focus Multiplane Light Field

Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Rolland et al., Applied Optics 2000 Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Maxwellian

Location 1 (red) Location 2 (green) Location 3 (blue)
1

Conventional
display pixel size

Intensity
0

Intensity
AI
0

Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017 Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
Blur Gradient Driven Accommodation
PSF Engineering
Q: can we drive accommodation with stereoscopic
cues by optically removing the retinal blur cue?
How do we remove the blur cue?
Aperture Controls Depth of Field

Image courtesy of Concept One Studios

Aperture Controls Depth of Field

Image courtesy of Concept One Studios

Aperture Controls Depth of Field

Image courtesy of Concept One Studios

 Maxwellian-type (pinhole) Near-eye Displays

Point Light
Source
 Maxwellian-type (pinhole) Near-eye Displays

Spatial Light Modulator

Point Light
Source

Severely reduces eyebox; requires dynamic steering of exit pupil

Focal Sweep
Depth Invariant PSF

D
D

D
5

5
0

4
0.

1.

2.

3.
60Hz
EDOF Cameras:
Dowski & Cathey, App. Opt. 1995
Nagahara et al., ECCV 2008
Cossairt et al., SIGGRAPH 2010
 Convolution

*
 Convolution

* =
Deconvolution

* =
Deconvolution

* =
Target
Target Image
 Target
Target Image Conventional Display @ 1D

Conventional
 Target
Target Image Conventional Display @ 3D

Conventional
 Target
AI @ 3D Conventional Display @ 3D

Conventional

AI
Q: can we drive accommodation with stereoscopic
cues by optically removing the retinal blur cue?
Padmanaban et al., PNAS 2017
 Task

4D 0.5D
(0.25m) (2m)

Follow the target with your eyes

 Stimulus
Dynamic
3

2
Distance (D)
1

-1

-2

-3
Gain: 0.85
0 5 10 15 20 0 5
Time (s)
 Measured User Response
Conventional Stimulus Dynamic
Average
3

2
Distance (D)
1

-1

-2

-3
Gain: 0.35 Gain: 0.85
0 5 10 15 20 0 5
Time (s)
 Measured User Response
Conventional Stimulus Dynamic
Average
3

2
Distance (D)
1

-1

-2

-3
Gain: 0.35 Gain: 0.85
0 5 10 15 20 0 5
Time (s)
Now: benchtop Future: multifocal lenses
 How to address the VAC?
Adaptive Focus Multiplane Light Field
Corrects spherical Near-correct optical retinal blur
aberrations Near-correct optical
retinal blur Resolution limits
Requires eye tracking

Render retinal blur
Sugihara et al., SID 1998
Liu et al., ISMAR 2008
Koulieris et al.,SIGGRAPH 2017
Padmanaban et al., PNAS 2017
Spatially-invariant PSFs
Flicker (or bulky) Computationally expensive (for
Rolland et al., Applied Optics 2000 stacked SLM method)
Huang et al., SIGGRAPH 2015
Akeley et al., SIGGRAPH 2004 Lanman et al., SIGGRAPH Asia 2013

Digital Holography Accommodation Invariant

Binocular Location
disparity
Location 1 (red) 2 (green)
driven
Location 3 (blue)
Correct optical retinal blur 1

Conventional
accommodation display pixel size

Intensity
High image quality 0
Supports large demographic
1

Computationally expensive Resolution limits

Intensity
AI
Tradeoff between exit pupil and FOV
Maimone et al., SIGGRAPH 2017, Shi et al., SIGGRAPH Asia 2017
0

Incorrect retinal blur

Waldkirch (Doctoral Dissertation) 2004, Konrad et al., SIGGRAPH 2017
Overview of Optical See-through AR Displays
Ray Ban
Thin Beam Combiner?
Thin Beam Combiner!
Pepper’s Ghost 1862
OST AR – Case Studies
Google Glass
Google Glass
 Meta 2

• larger field of view (90 deg) than Glass

• also larger device form factor

Microsoft HoloLens
 Microsoft HoloLens

• diffraction grating

• small FOV (30x17), but

very good image quality
 https://www.kguttag.com/2019/02/27/h
Microsoft HoloLens 2

ergonomics-but-the-lbs-resolution-
ololens-2-first-impressions-good-
• laser-scanned waveguide display

• claimed 2K resolution per eye (2560x1440),

probably via “interlaced” scanning

math-fails/
• field of view: 52° diagonally (3:2 aspect, 47
pixels per visual degree)

Wall et al. US 10,025,093 2018

Video-based AR: ARCore, ARKit, ARToolKit, …
Challenges: Eye Box vs Field of View
 Challenges: Eye Box vs Field of View

eye box / exit pupil entrance pupil

• need small entrance pupil (small device) and large exit pupil (large
eye box) - pupil needs to be magnified
 Challenges: Eye Box vs Field of View

eye box / exit pupil entrance pupil

field of view

• need small display (small device) but large field of view – image
needs to be magnified
 Challenges: Eye Box vs Field of View

eye box / exit pupil entrance pupil

field of view

• pupil needs to be magnified can’t get both at the same

• image needs to be magnified time – etendue!
 Challenges: Eye Box vs Field of View

eye box / exit pupil entrance pupil

field of view

• possible solutions: exit pupil replication (loss of light), live with small FOV
(not great), dynamically steer eye box (mechanically difficult), ..
 Challenges: Chromatic Aberrations

• thin grating couplers create chromatic aberrations

 Challenges: Chromatic Aberrations

volume holographic couplers, stacked waveguides

e.g. TruLife Optics

• all solutions have their own problems: ease of manufacturing, yield,

robustness, cost, …
 Occlusions
real
object Case 1: Case 2:
virtual in front of real real in front of virtual

virtual
object

à difficult: need à easy: don’t render

to block real light! virtual object everywhere
 Next Lecture: Inertial Measurement Units I
• accelerometers, gyros, magnetometers
• sensor fusion
• head orientation tracking