Seeing the World through Your Eyes

Code (coming soon)

From a handful of portrait photos of a person, we compute a 3D reconstruction of what they are observing using the eye
reflections!

By placing realistic eye models in synthetic scenes, we perform full scene reconstruction using only the eye reflections.

Abstract
The reflective nature of the human eye is an underappreciated source of information about what the
world around us looks like. By imaging the eyes of a moving person, we can collect multiple views of a
scene outside the camera's direct line-of-sight through the reflections in the eyes. In this paper, we
reconstruct a 3D scene beyond the camera's line-of-sight using portrait images containing eye
reflections.
This task is challenging due to 1) the di!culty of accurately estimating eye pose and 2) the entangled
texture of the eye iris and the scene reflections. Our method jointly refines the cornea poses, the
radiance field depicting the scene, and the observer's eye iris texture. We further propose a simple
regularization prior on the iris texture pattern to improve reconstruction quality. Through various
experiments on synthetic and real-world captures featuring people with varied eye colors, we
demonstrate the feasibility of our approach to recover 3D scenes using eye reflections.

How we did it?

The cornea geometry is approximately the same across all healthy adults. Because of this fact, if we
count the pixel size of a person's cornea in the image, we can compute exactly where their eyes are.
Using this insight, we train the radiance field on the eye reflections by shooting rays from the camera,
and reflecting them o" the approximated eye geometry. To remove the iris from showing up in the
reconstruction, we perform texture decomposition by simultaneously training a 2D texture map that
learns the iris texture.
L.radial Inputimage(cropped)
Eg.6

:O
raorigin
Came Cameraraydirection:d Corneahit:0' Projeve(0')
l:ñ. Hitpointineye
Norma '
n:d
le ctio coordinatesystemEvetexturefield
Re f Estimatedeye
texture

Volumetric
x,d'
Sampledpoint&direction
Rendering
Density
Radiancefield Renderedimage Corneamask

However, approximating the eye pose just from the image is always very noisy. To address this issue, we
perform eye pose optimization which is critical for performance as we show below.

Eye pose optimization ablation

Without pose optimization With pose optimization

Texture decomposition ablation

Without texture decomposition With texture decomposition

In the wild attempts

Miley Cyrus Lady Gaga

Using the music videos from Miley Cyrus and Lady Gaga , we attempt to reconstruct what they are
observing. We manage to reconstruct the object that appears in Miley's eyes, and we observe an object
resembling an upper body from Lady Gaga's eyes. However, due to the quality of these videos, the
correctness of the reconstruction is unclear.

The website template is from Nerfies.