Crossroads The ACM Magazine for Students FA LL 2015 V OL .2 2 • NO.1 XRDS.ACM.

ORG
APPRENEUR SCHOLARS
STUDENTS: SHOW US YOUR APPS

APPLY NOW FOR THE

2016 APPRENEUR SCHOLAR COMPETITION
WIN A $5,000 SCHOLARSHIP
GET FREE ACM MEMBERSHIP
ATTEND THE 2016 INTERNATIONAL CES

For more details visit: appreneurscholars.com


Are you looking for
your next IT job?
Do you need Career Advice?
The ACM Career & Job Center offers ACM members a host of
career-enhancing benefits:
• A highly targeted focus on job
opportunities in the computing industry
• Access to hundreds of industry job postings
• Resume posting keeping you connected
to the employment market while letting you
maintain full control over your confidential
information
ACM’s Career
& Job Center
• Job Alert system that notifies you of
new opportunities matching your criteria
• Career coaching and guidance available
from trained experts dedicated to your
success
• Free access to a content library of the best
career articles compiled from hundreds of
sources, and much more!

Visit ACM’s Career & Job Center at:

http://jobs.acm.org
The ACM Career & Job Center is the perfect place to
begin searching for your next employment opportunity!

Visit today at http://jobs.acm.org

 Crossroads
The ACM Magazine for Students
FA L L 2 01 5 V OL . 2 2 • NO . 1

11

begin
5 LETTER FROM THE EDITORS

8 INBOX

9 INIT
Our Virtual Reality Revolution:
This time it’s for real
By Pedro Lopes

11 ADVICE
Writing your Dissertation
By Marius-Tudor Benea

12 UPDATES
Excellence Around the World
By Claudia Schulz

13 MILESTONES
Virtual Reality Device Developments
By Jay Patel

14 CAREERS
In Praise of Side Projects
By Natalie Gordon

16 BLOGS
From Novice to Expert:
The art of science of pedagogy
By Olivia Simpson
Top Image by A. and I. Kruk

Cover Art by
Iwona Usakiewicz / Andrij Borys Associates /
Shutterstock.com

2 XRDS • FALL 2015 • VOL.22 • NO.1

 Virtual Reality

18 46 64

features end
18 FEATURE 46 FEATURE 64 LABZ
L-to-R: Image courtesy of FOVE; Image by Durantelallera / shutterstock.com; Image of VFX-1 by eVRydayVR. Image of Oculus Rift Dev Kit 2 by Ats Kurvet.

One Step Beyond Managing Cybersickness The Virtual Human Interaction Lab
Virtual Reality: Connecting past in Virtual Reality By Andrea Stevenson Won
and future developments By Lisa Rebenitsch
By Diego Martinez Plasencia 65 BACK
52 FEATURE VR Head-Mounted Displays
24 FEATURE What Is the Vergence- By Finn Kuusisto
Substitutional Reality: Accommodation Conflict
Bringing virtual reality home and How Do We Fix It? 66 HELLO WORLD
By Adalberto L. Simeone By Robert Konrad Sorry Kids, Iron Man’s
and Eduardo Velloso Superpowers Aren’t Unique
56 FEATURE By Lara Zupan And Marinka Zitnik
30 FEATURE VR to the Rescue: Avoiding danger
Using Your Own Muscles: with mine disaster simulations 68 ACRONYMS
Realistic physical experiences in VR By Alain Boulay
By Pedro Lopes, Alexandra Ion, 68 POINTERS
and Robert Kovacs 61 PROFILE
Susumu Tachi: The scientist 69 EVENTS
36 FEATURE who invented telexistence
Eccescopy: To look, is to see By Adrian Scoică 72 BEMUSEMENT
By Ken Perlin

40 FEATURE
Lost in the Rift:
Engaging with mixed reality
By Daniel Boland and Mark McGill

XRDS • FALL 2015 • VOL.22 • NO.1 3

 ACM E DI T ORI A L B O A RD
LEARNING CENTER
Editor-in-Chief
Sean Follmer
Stanford University, USA
Departments Chief
RESOURCES Adrian Scoică
University of Cambridge, UK
Issue Editor
FOR LIFELONG LEARNING Pedro Lopes
Hasso Plattner Institut,
Germany
Feature Editors
learning.acm.org Sergiy Bogomolov
University of Freiburg,
Germany
Judeth Oden Choi
Carnegie Mellon
University, USA
Numair Khan
New York University, USA
Talia Kohen
Bar Ilan University, Israel
Hanieh Moshki
HEC Montreal , Canada
Richard Gomer
University of
Southampton, UK
Department Editors
Abhishek Bhattacharya
Amity University, India
Somdip Dey
University of
Manchester, UK
Marius-Tudor Benea
UPB, Romania and UPMC,
France
Tejas Khot
Mumbai University, India
Bryan Knowles
Western Kentucky
University, USA
Finn Kuusisto
University of
Wisconsin-Madison, USA
Darshit Patel
Pimpri Chinchwad College
of Engineering, India
Online Courses
Jay Patel
University of California
Berkeley, USA
from Skillsoft
Claudia Schulz
Imperial College London, UK
Apoorvaa Singh
Online Books
IPEC Ghaziabad, India
Marinka Zitnik
University of Ljubljana,
from Safari, Books24x7, Slovenia
Digital Content Editor
Pedro Lopes
Morgan Kaufmann Hasso Plattner Institut,
Germany
Web Editor
and Syngress Abhineet Saxena
Guru Gobind Singh
Indraprastha University,
India
Webinars News Editors
Doris Lee
University of California,
on today’s hottest topics Berkeley, USA
Marius-Tudor Benea
UPB, Romania and UPMC,
in computing France
4 XRDS • FALL 2015 • VOL.22 • NO.1
A D V IS OR Y B O A RD SUB S C RIBE
Mark Allman, Subscriptions ($19
International Computer per year includes XRDS
Science Institute electronic subscription)
are available
Bernard Chazelle, by becoming an
Princeton University ACM Student Member
Laurie Faith Cranor, www.acm.org/
Carnegie Mellon membership/student
Alan Dix,
Lancaster University
David Harel,
Non-member
subscriptions:
$80 per year
http://store.acm.org/
Weizmann Institute acmstore
of Science
ACM Member Services
Panagiotis Takis Metaxas , To renew your ACM
Wellesley College membership or XRDS
Noam Nisan, Hebrew subscription, please send
University Jerusalem a letter with your name,
address, member number
Bill Stevenson , and payment to:
Apple, Inc.
ACM General Post Office
Andrew Tuson, P.O. Box 30777
City University London New York, NY
10087-0777 USA
Jeffrey D. Ullman,
InfoLab, Stanford
University Postal Information
Moshe Y. Vardi, XRDS (ISSN# 1528-4981)
Rice University is published quarterly in
spring, winter, summer
and fall by Association for
Computing Machinery,
E DI T ORI A L S TA F F 2 Penn Plaza, Suite 701,
Director, Group New York, NY 10121.
Publishing Application to mail at
Periodical Postage rates
Scott E. Delman
is paid at New York, NY
XRDS Managing Editor & and additional mailing
Senior Editor at ACM HQ offices.
Denise Doig
Production Manager POSTMASTER: Send
Lynn D’Addessio addresses change to:
Art Direction XRDS: Crossroads ,
Andrij Borys Associates, Association for
Computing Machinery,
Andrij Borys,
Mia Balaquiot 2 Penn Plaza, Suite 701,
New York, NY 10121.
Director of Media Sales
Jennifer Ruzicka Offering# XRDS0171
ISSN# 1528-4972 (print)
jen.ruzicka@acm.org
ISSN# 1528-4980
Copyright Permissions (electronic)
Deborah Cotton
permissions@acm.org Copyright ©2015 by the
Public Relations Association for Computing
Machinery, Inc. Permission
Coordinator
Virginia Gold to make digital or hard
copies of part of this work
for personal or classroom
ACM use is granted without fee
provided that copies are
Association for not made or distributed
Computing Machinery for profit or commercial
2 Penn Plaza, advantage and that copies
Suite 701 bear this notice and the
New York, NY full citation on the first
10121-0701 USA
+1 212-869-7440
page or initial screen of
the document. Copyrights
C ON TA C T for components of this
General feedback:
xrds@acm.org
work owned by others than
ACM must be honored.
Abstracting with credit
For submission is permitted. To copy
guidelines, please see
http://xrds.acm.org/
otherwise, republish, post
on servers, or redistribute
authorguidelines.cfm requires prior specific
PUBLIC AT IONS BOA RD
permission and a fee.
Permissions requests:
permissions@acm.org.
Co-Chairs
Jack Davidson and
Joseph A. Konstan
Board Members
Ronald F. Boisvert,
Nikil Dutt, Roch Guerrin,
Carol Hutchins,
Yannis Ioannidis,
Catherine C. McGeoch,
M. Tamer Ozsu,
Mary Lou Soffa
LETTER FROM THE EDITORS

Deep Dreams
are Made
of These
O
ver the summer, the Internet went into a collective frenzy for what looked like
psychedelic images more fitting for a late 1960s dorm room, not for the most
popular tech news sites. Images of dogs sprouting other dogs in fractal delight,
and other hallucinatory apparitions took over the Internet. I’m describing Google’s
“DeepDream” image software.1 It seemed like everyone and their uncle were posting crazy,
deepdreamed images on my Facebook feed for a few weeks. Of course these computers
weren’t dreaming, or at least not the way we do. A complex artificial neural network
trained on images, not REM sleep, are generated from unsupervised
is responsible for creating these learning, have existed for a long time
bizarre creations. (ANNs were hot in the 1980s), but now
UPCOMING ISSUES What began as a submission for the we have more data. In many cases the
2014 ImageNet Large-Scale Visual Rec- features the system finds make little
Winter 2015 ognition Challenge, became software or no sense at one level, but contrib-
created to identify objects and pat- ute meaningfully many levels up. But
[December issue]
terns. The DeepDream software takes now deep learning is starting to make
Internet of Things that code and optimizes input images big progress in speech and image rec-
so that its system can find features in ognition. Graphics and HCI research-
Spring 2016 images for easier classification. ers are starting to use these tools to
[March issue] Deep learning has been advancing produce exciting results; in a recent
the field of artificial intelligence at meeting, a colleague suggested using
Digital Fabrication
Google and beyond. These levels of ar- deep learning to classify some of my
tificial neural networks (ANN), which data. Once the HCI people start using
Summer 2016 it, you know it must be a big deal.
[June issue] 1 http://googleresearch.blogspot.
All of this comes out of an AI ap-
Cultures in Computing co.uk/2015/07/deepdream-code- proach that starts to look a lot more
example-for-visualizing.html like how our brain works than tradi-

XRDS • FALL 2015 • VOL.22 • NO.1 5

 often answered by a “we don’t know.”
Looking toward But more and more researchers are
seeing the importance of lots of data,
the horizon, or exposure to events, as central to
these evermore our learning and perception. Philoso-
phers like Alve Noë have described
intelligent the importance of embodiment in
machines mean learning and cognition. AI research-
ers have long aspired to embody these
new, evermore principles in their research, but for
intelligent the past few decades support-vector
networks (SVNs) and machine learn-
applications in ing have prevailed. Now, big data and
almost every robotics might change this.
Looking toward the horizon, these
domain. evermore intelligent machines mean
new, evermore intelligent applications
in almost every domain. But there may
be darker applications: ImageNet can
be useful for labeling YouTube videos,
but it can also be used to identify en-
emy combatants. That is why a number
of AI researchers and technologists
tional machine learning approaches. have recently been pushing a ban on
Of course, we still have little under- AI weapons that have the ability to
standing of how the brain works on a decide who or what to target.2 Some
fundamental level—many of the inter- say an AI arms race may push us in an
esting questions in neuroscience are undesired direction, and that “autono-
MASTER OF SCIENCE IN
COMPUTER SCIENCE
With a focus on network security, the MSCS positions students with
advanced technical skills essential to today’s programmers.
• One-year or part-time formats • Competitive tuition
• Evening classes • Scholarships and financial aid
• Internship and networking • Optional test scores
opportunities
www.merrimack.edu/gradcs
graduate@merrimack.edu • 978-837-3563 NORTH ANDOVER, MA
2 http://futureoflife.org/AI/open_letter_autono-
mous_weapons
6 XRDS • FALL 2015 • VOL.22 • NO.1
mous weapons will become the Ka-
lashnikovs of tomorrow,” easy to pro-
duce and hard to control. Signatories
include Elon Musk, Stephen Hawking,
and AI experts like Stuart Russell, Eric
Horvitz, and Peter Norvig.
Of course, we are not there to-
day. As tech journalist John Markoff
pointed out during a recent interview
at the Computer History Museum in
Silicon Valley, robots can barely open
doors. The recent DARPA robotics
challenge, although clearly display-
ing some impressive strides in the
field, had a human operator in the
loop and still resulted in viral videos
of robots falling over.
Further down the line we will need
to think about not only the ethics of
robot use, but also the rights of ro-
bots as well. In a clear effort to beef
up my nerd cred, I’ve recently become
addicted to “Star Trek: The Next Gen-
eration.” One of the most interesting
story lines on the show, or at least for
an arm-chair computer science ethi-
cist like myself, is that of Lt. Com-
mander Data—one of two androids
in the galaxy—and his quest to un-
derstand what it means to be human.
In the episode “Measure of a Man,”
Starfleet wants to make copies of
Data for military means but to do so
may involve potentially dissembling
him. Data must convince a Starfleet
court that he is alive and sentient.
Just as the current fight for animal
legal “personhood” for captive apes
or circus animals has made us ques-
tion the rights of animals, would my
Roomba or a future version of it need
such freedoms?
Now is the time to think about
these issues. Can we hard code ethics
and morals into these robots? How
can we teach ethics to robots if we
can’t yet teach it to the engineers who
are programming the robots? With
large-scale engineering scandals
like the Volkswagen Diesel exhaust
regulation fraud—the automaker was
caught with systems that falsified
pollutant numbers when tested—it
seems clear that when there are moti-
vations for financial gain ethical lines
become blurred. Before we can have
ethical robots, we have to have more
ethical engineers.
—Sean
— Follmer
Check out the new acmqueue app
FREE TO ACM MEMBERS
acmqueue is ACM’s magazine by and for practitioners,
bridging the gap between academics and practitioners
of computer science. After more than a decade of
providing unique perspectives on how current and
emerging technologies are being applied in the field,
the new acmqueue has evolved into an interactive,
socially networked, electronic magazine.

Broaden your knowledge with technical articles

focusing on today’s problems affecting CS in
practice, video interviews, roundtables, case studies,
and lively columns.

Keep up with this fast-paced world

on the go. Download the mobile app.

Desktop digital edition also available at queue.acm.org.

Bimonthly issues free to ACM Professional Members.
Annual subscription $19.99 for nonmembers.
begin
INBOX

READ THE LATEST ISSUE

ONLINE OR ON THE GO
loving the current
issue of @XRDS_ACM
featuring comp bio!! it’s
so inspiring. read here.
http://xrds.acm.org/
current-issue.cfm
—Natalie Stanley, Ph.D.
student, UNC-Chapel Hill,
Twitter (@natstann)

Editor’s Note:
Thanks Natalie! Don’t
forget you can read the
latest issue online at
http://xrds.acm.org/
current-issue.cfm as well as
in the ACM Digital Library,
where you will find a link
for the digital edition. And
for those of you who like to
read on the go, download
our mobile app from the
iTunes store, Google Play,
and Amazon.

VOLUNTEERING WITH
XRDS
@XRDS_ACM are you
guys interested in HCI
/ Interaction Design
contributions :)?
—Nicolai B. Hansen, Design
researcher, Twitter (@
nbhansen)
@nbhansen YES WE ARE.
What do you have in mind?
—ACM XRDS Magazine,
XRDS is ACM’s magazine for
students by students, Twitter.
Guessing bachelors/masters
is the target?
—Nicolai B. Hansen
Editor’s Note:
If you are interested in
volunteering as a student
editor or blogger, please
contact us at xrds@acm.
org with the subject line:
“Student volunteers for
XRDS.”
=portal&dl=ACM cfm?aid=2748105
—Vaggelis Giannikas,
Research associate at the
University of Cambridge OTHER TWEETS
and Associate director of new in #XRDS_ACM
the Cambridge Auto-ID Lab, The Wrens of Bletchley
Twitter (@VGiannikas) Park http://xrds.acm.org/
article.cfm?aid=2730914
“I am also in the final stage. … #codebreakers
Your article really helped #cryptography
me. Hope I have learned #engimamachine
how to do good research #BletchleyPark
too. Thanks.” —Hacker Jou, Twitter
Post by Ana Oliveira, Tue, (@hackerjou)
Image by Phish Photography

(@XRDS_ACM) THE PH.D. JOURNEY 05 May 2015 14:21:41 UTC;

@XRDS_ACM unsure,
but most of our masters
students here in Aarhus do
Research through Design.
Lessons from my personal
Ph.D. journey now
published in @XRDS_ACM -
http://dl.acm.org/citation.cf
m?id=2752549.2748105&coll
http://xrds.acm.org/article.
How to contact XRDS: Send a letter to the editors or other feedback by email (xrds@acm.
org), on Facebook by posting on our group page (http://tinyurl.com/XRDS-Facebook), via
Twitter by using #XRDS_ACM in any message, or by post to ACM Attn: XRDS, 2 Penn Plaza,
Suite 701, New York, New York 10121, U.S.

8 XRDS • FALL 2015 • VOL.22 • NO.1


INIT
Our Virtual Reality Revolution:
This time it’s for real
H
istory is repeat-
ing itself; we are
reliving the verge
of virtual reality
(VR). The door is open and
we can already peak in-
side—through stereoscopic
headsets made of plastic or
cardboard—into the magi-
cal realm of VR. Only time
will tell whether we will also
relive the death of VR. Al-
though computer scientists,
such as Ivan Sutherland,
first crafted this technol-
ogy in the 1960s, VR was
later propelled by media
hype in the 1990s. However
it fell into a deep hiatus
until 2012. It has only been
three short years since VR
headsets experienced resur-
gence amongst consumers.
VR opens up an immense
space where computer sci-
ence dives in; this special is-
sue will be an eye-opener for
emerging CS students who
It seems
the last VR
revolution
was missing
a critical
component:
Photo Credit TK
an enthusiastic
audience.
XRDS • FALL 2015 • VOL.22 • NO.1
$2 billion is the amount that Facebook paid in
2014 to acquire the VR startup Oculus.
are considering the world of
VR, either as recent gradu-
ates ready to join the bub-
bling industry (think Valve,
HTC, Samsung, Sony, Mi-
crosoft, Facebook, Google)
or as researchers looking
to connect with fellow aca-
demics who research novel
ways of experiencing VR.
Within these pages, XRDS
shines light on both sides of
the coin.
The University of Bris-
tol’s Diego Martinez
Plasencia begins your voy-
age with a step-by-step nar-
ration of VR developments
over the years. He guides
readers from the designs of
consumer headsets, such
as the Oculus Rift, to what
might be the next, scene-
stealing technologies.
VR technology allows
us to see new worlds, travel
quickly to distant places,
and be in full-body remote journey of “substitutional you to put on a very special
meetings. But as our eyes reality.” This is a technique type of virtual reality tech-
widen in surprise, the sen- that uses real-world objects nology: eccescopic goggles.
sation of “being there” is (think of all the everyday With this powerful tool,
broken as soon as we reach objects around you right imagined and provided
out to touch anything. The
answer is immersive physi-
now) as stand-ins for vir-
tual objects. I also address
by Prof. Ken Perlin of New
York University, we embark
cal experiences, those that this challenge with my col- on a voyage that intercon-
span beyond only “look-
ing at VR” and go as far as
leagues from the Hasso
Plattner Institute, Alex-
nects the shores of VR with
the continents of 3-D dis-
“touching VR.” Working andra Ion and Robert Ko- plays, archipelagos of fu-
on a solution to lost physi-
cality is Adalberto Sime-
vacs. We demonstrate new
mobile devices that enable
ture devices, and the poles
of human augmentation
one from the University of physical experiences in the philosophy and other ethi-
Portsmouth and Eduardo
Velloso from Lancaster Uni-
virtual realm, such as walk-
ing on steps or feeling forces.
cal questions.
Staying in touch with
versity, who take you on a Out next stop requires the real world is a challenge
9
 with VR. Daniel Boland
and Mark McGill from the
University of Glasgow ex-
plore how to literally save
all those VR loners from
getting lost, both physically
and socially, by selectively
blending real elements into
the virtual world depend-
ing on how users wish to en-
gage with them.
But with all innovative
technologies come grand
ACM Books. challenges. One of these big
In-depth.
unsolved VR challenges is
“cybersickness.” Lisa Re-
Innovative.
benitsch from Michigan
State University clarifies what
it is, what are the possible ex-
Insightful.
planations, and how can we
expect to alleviate this con-
dition for the future gen-
erations of VR technology.
The VR Book: Human-Centered Robert Konrad of Stanford
University digs deeper into
Design for Virtual Reality painful user experiences
By Jason Jerald, PhD related to 3-D displays, and
teaches us how the principle
Good VR design requires strong communication between behind light-field displays
human and machine, indicating what interactions are might fix the “vergence-
possible, what is currently occurring, and what is about to accommodation conflict.”
occur. A human-centered design principle, like lean At our final stop, Alain
methods, is to avoid completely defining the problem at Boulay from Laurentian
the start and to iterate upon repeated approximations and University shows us how
modifications through rapid tests of ideas with real users. VR is used for altruistic ex-
Thus, The VR Book is intended as a foundation for anyone
and everyone involved in creating VR experiences
including: designers, managers, programmers, artists,
psychologists, engineers, students, educators, and user
experience professionals. VR technology
Available in hardcover, paperback and eBook.  allows us to see
DOI: 10.1145/2792790
new worlds,
For more info please visit travel quickly
http://books.acm.org to distant
places, and be
M M in full-body
&C &C
Association for Morgan & Claypool
Computing Machinery Publishers
remote
meetings.
10
periences. He shares how
a virtual simulator enables
rescue teams to train in re-
alistic, yet safer, extreme
environments, such as
deep mines.
It seems the last VR revo-
lution was missing a critical
component: an enthusias-
tic audience. But this time
around things are different;
VR is out there for every-
body. The cost to experience
VR is rather small, thanks
to simplified solutions such
as the Cardboard or the FO-
V2Go, which leverage your
existing smartphone as a
VR display. Now we have
makers, scientists, and the
public crafting VR applica-
tions; we are witnessing the
birth of new VR devices on
Kickstarter every day, such
as input controllers (Six-
ense and Leap Motion) and
motion platforms (Virtuix
Omni and Vection VR). As
of now, VR is starting to be
what we want it to be; since
it is a constructed reality, VR
should be the place where
things are the way you
want—your reality. Here, in-
side the pages of XRDS, we
share how that reality looks
today: from the challenges
we face (cybersickness, lack
of haptics, etc.) to some
possible solutions (haptic
wearables and light field
cameras). We at XRDS hope
VR has a bright future as a
tool for mankind’s augmen-
tation. The triptych formed
by researchers, public, and
industry will undoubtedly
be instrumental to the suc-
cess of VR.
—Pedro
— Lopes,
Issue Editor
XRDS • FALL 2015 • VOL.22 • NO.1
 March 3, 2015
AMD unveiled its LiquidVR SDK, which enables the development of low latency,
high-performance VR software to run on AMD hardware.

ADVICE

Writing your Dissertation

E
very student, no matter his or
her field of study, has a dream
for life post graduation. Nev-
ertheless, this dream is often
overshadowed by a legitimate fear of
the Ph.D. dissertation and its defense.
The best way to overcome this situa-
tion is to have confidence in the work
you present. If you want to experience
the certitude of well-done work, the fol-
lowing pieces of advice should be con-
sidered when working on a computer
science dissertation.
1. Start writing as soon as possible!
You will never have all the pieces you
need before beginning, but taking this
step early will create a fertile ground for
great ideas and for writing an excellent
dissertation. As soon as you have cho-
sen a subject and your supervisor(s), keep in mind there is someone will- hard to digest. To overcome this, use
don’t hesitate to get a head start before ing to help you. The person who is in many examples and figures and refer
you are too busy. the best position to do so is your su- to them whenever possible. Short code
2. Draft the manuscript. The best pervisor. Avoid, the common mistake examples, which are well integrated
starting point is to create a draft of made by computer science students into the narrative thread, can also be
the manuscript, including its desired everywhere: trying to do everything by used to help you make salient points.
structure, and filling in all the ideas yourself. Regularly ask your supervi- For larger blocks of relevant code, try
you have. This will give you and your sor for help and feedback. You will not to use appendices instead.
supervisor a good overview of the proj- only produce a stronger dissertation, 7. Acknowledge every contribution. It
ect and its progress later on. It will even you will also gain an advocate during is wise to note those who helped you,
spark your imagination for new ideas, its defense. and it would mean a lot to them. This
and will provide you with a “frame- 5. Review your manuscript regularly. will further improve your relationships
work” to easily integrate these ideas The previous advice doesn’t exempt in the long run.
into your dissertation while everything you from having to be honest with If you pay attention to all the tips
is still fresh. yourself. You should review the en- discussed, it is almost impossible to
3. Progressively integrate all your tire manuscript regularly. It will help fail. And the bonus is you will present
new ideas and results. Now, given the you develop confidence in the quality a well-regarded dissertation. Starting
above-mentioned draft of your manu- of your work. Once you discover new early, regularly integrating new con-
script, try to record every new idea or things and you have new ideas or re- cepts, checking your manuscript, and
result as soon as it comes to you. And sults, parts of the original manuscript interacting with your supervisor will
don’t forget to set aside time to do the will become outdated. Your work will not only assure your success, but it will
integration with the rest of the manu- need to be refreshed according to your also give you a better taste for the sub-
Image by A. and I. Kruk

script. After all, with these small steps
and a small dose of perseverance you
can travel long distances.
4. Interact often with your supervisor.
The best piece of advice is to always
new understanding of the studied
problem.
6. Use many examples. Often the
theoretical concepts introduced in
a computer science dissertation are
ject studied. Be careful you might be-
come addicted. Don’t worry, this is one
of those cases when such a situation is
desirable.
—Marius-Tudor
— Benea

XRDS • FALL 2015 • VOL.22 • NO.01 11

begin

UPDATES

Excellence Around the World

This year’s ACM award winners share
how their chapters came out on top.

W
hat makes a student chap-
ter excellent? Is there a rec-
ipe for success? We turned
to this year’s ACM Student
Chapter Excellence Award winners
for answers.
University of Petroleum and Energy
Studies (UPES) ACM Student Chapter,
India. “We have a very disciplined and
specialized hierarchy,” explained Yash
Shanker Srivastava, joint head of the
events and workshop management
committee of the UPES ACM Student
Chapter. In his opinion, this is the
chapter’s secret for effectively orga-
nizing a large variety of outstanding
events. The activities that won them The winners of the Outstanding School Service award, the Tehran ACM Student
the Outstanding Chapter Activities Chapter, holding copies of F1 Journal.
award were mainly targeted at pro-
grammers—not just from UPES, but hour per day in the programming lan- cinnati ACM-W Student Chapter won
from all over the world. An example guage of the participant’s choice. This the Outstanding Community Service
is their “Hour of Code” event, which attracted more than 500 programmers award. The passion of Chawla and the
was held online and intended to be from five continents. The chapter’s chapter’s members is to spark an in-
an international programming chal- second outstanding activity had an terest in computer science. The chap-
lenge for coders (rather than an event even greater number of participants: ter collaborated with a local school,
for programming novices). The UPES more than 2,000 students across In- where they mentored and taught
student chapter set up seven program- dia took part in Prodigy 2015. The school children ways to tackle pro-
ming problems to be solved in seven chapter’s annual technical festival gramming projects. The chapter also
days; coding was scheduled for one featured a variety of coding competi- received sponsorship from Microsoft
tions, workshops on current IT trends, to organize DigiGirlz, an event attend-
and invited talks. “We never had an ed by more than 50 girls from Ohio.
aim for winning the award,” says Sriv- Their goal was to motivate more girls
astava, “we only aimed to do awesome to pursue careers in technology.
work for students and make them love University of Tehran ACM Student Chap-
The secret computing technologies.” ter, Iran. What made the University of

to our success University of Cincinnati ACM-W Stu-

dent Chapter, USA. In order to be recog-
is just not giving nized by ACM, chapter president Priya
up under any Chawla advises students to “ignite
a program that your chapter is pas-
circumstances. sionate about.” The University of Cin-
Tehran ACM Student Chapter stand out
from the other applicants for the Out-
standing School Service award? The
chapter organized a large number of
diverse activities: weekly programming
contests, workshops on web-develop-

12 XRDS • FALL 2015 • VOL.22 • NO.1

 US 4988981 A is the U.S. patent number for Zimmerman
and Lanier’s Data Glove, filed in 1989. It is one of the earli-
est wired gloves for a VR environment.

MILESTONES

Virtual
Reality Device
Developments
ment and typing on a Persian keyboard, Virtual-reality (VR) device designers have many
and programming classes for non-com- variables to consider when creating a new system:
puting students, to name a few. This use case, audience, comfort, and more. In this
meant members of the chapter could collection of milestones, we explore VR devices
“chose what they want[ed] to do and from different points in time and observe how their
whom they want[ed] to engage with,” creators chose to balance varying characteristics.
explained Amirreza Dadfarnia, editor-
in-chief of the chapter’s F1 Journal. This
quarterly publication contains not only
chapter news, but also articles covering
1950 Morton Heilig first describes his idea for
an “experience theatre,” which is an early
example of multimodal technology encompassing all five
hot topics in technology and computer senses into an immersive, cinema experience. Twelve years
science research. “Trust in yourself later, he builds the Sensorama prototype, which includes a
and never fear failure” is Dadfarnia’s display, odor producers, and a moving chair.
advice for becoming a successful
student chapter.
University of Karachi ACM Student
Chapter, Pakistan and Pontificia Univers-
1961 Philco Corporation develops the Headsight
device, a helmet that communicates with
a closed-circuit television system. It is intended as a
idad Javeriana ACM Student Chapter, Co- teleoperation device for the user to share findings when
lombia. The message is clear: In order exploring dangerous environments.
to develop excellence, a student chap-
ter needs a good balance of teamwork,
passion, and discipline. This is also the
advice shared by Madiha Nasim, the
1968 The Sword of Damocles is a head-mounted
virtual reality display, similar to goggles,
which uses a ceiling suspension system to support the
chair of the University of Karachi ACM heavy device on the user’s head. This system supports
Student Chapter, which won the Out- stereo images as well as field-of-view tracking.
standing Recruitment Program award.
“We work as a team, all with positive
attitudes, [...] for the betterment of the
society and after all computer science,”
1990 The first demonstration of Virtuality, a head-
mounted system for arcade-style games, is
released. Virtuality’s VR systems include sit-down pods and
explained Nasim. Alfredo Santamaría stand-up devices with joysticks for gaming.
agrees. He is the publicity leader of the
Pontificia Universidad Javeriana ACM
Student Chapter (for more informa-
tion on the chapter, see the Summer
2012 The first developer kit for the Oculus Rift, a
consumer-oriented head-mounted display,
is made available. Rift has received a great deal of industry
2015 issue of XRDS), which won the and consumer attention for its high-quality resolution, low
Outstanding Website award. His clos- latency, and affordability.
ing words of advice: “Give the best in
everything you do. ... The secret to our —Jay
— Patel
success is just not giving up under
any circumstances.”
—Claudia
— Schulz

XRDS • FALL 2015 • VOL.22 • NO.1 13

begin

The Oculus Rift head-mounted display

has six full degrees of freedom.

CAREERS

In Praise of Side Projects

Editor’s Note: Natalie Gordon’s
BabyList, of which she is founder and
CEO, began as a post on Hacker News
when she asked users to review her
side project. It’s work like this, outside
of our full-time jobs, that provides the
best experiences. Although we’ll likely
fail, it’s the fact that we’re not afraid to
fail in these projects that makes these
experiences richer.
—Bryan Knowles

G
etting your side project to
launch isn’t the end goal.
But not launching is the big-
gest reason that projects fail.
Learning to launch is one of the most
important skills you can develop to
better your career. Choose a project
that is as niche as possible. This way,
it’s more likely that you will actually
get to the hardest part (psychological-
ly), announcing “it’s live” and putting
it in front of real users. Yes you will be
vulnerable, and you will come up with
great reasons to push that uncomfort-
able feeling off, but launch when you’re
still embarrassed. Launch when you
know how incomplete your solution
is. You don’t need a “forgot password” shop. It doesn’t have to be code, but it I had previously launched a Spanish-
feature when you don’t have any users. does have to be something you create English language-learning site, which
You don’t need an awesome logo. Don’t from nothing. taught me first hand every lesson I
fall into the trap of thinking “you only I am four years into my latest side hope to teach you.
get to go live once, so it needs to be per- project. I’m the founder of an online Soon after my Hacker News post, I
fect the first time.” baby registry called BabyList. It start- met a young Filipino designer online.
Image by Eugenio Marongiu / Shutterstock.com

ed with a post on Hacker News; the Within a couple of months we were

WHAT IS A SIDE PROJECT?
Any project that you are doing for fun,
not to make money, is a side project.
That’s not to say your side project can’t
make money, but that’s not why you’re
doing it. Those fun projects that get
done outside of work can have a huge,
disproportionate impact on your
whole career. It could be a library, a
software app, a blog, or even an Etsy
headline read: “I’m a pregnant hacker.
Please review my side project.” But
before that, I was programming full-
time as a contractor for a startup. I
was expecting a baby, and I was creat-
ing my baby registry. I didn’t enjoy the
work I was doing, so I decided to work
on a “fun project”: building a better
baby registry.
This wasn’t my first project to go live
able to launch a very simple version of
the website that looked cute and solved
many of the problems I had faced when
making my baby registry.
It was more than a year later before
I began working on BabyList full-time.
By that point, it was clear there were
users who really cared about the prod-
uct. We were getting better at reach-
ing them online and realized this was

14 XRDS • FALL 2015 • VOL.22 • NO.1


a business that could make money.
Eventually, I joined a start-up accelera-
tor program, 500 Startups. I raised cap-
ital, hired an amazing team of 10, and
continued to scale the business out.
Image by Everything Possible / Shutterstock.com
THE JOURNEY
Envisioning something and building
it yourself, with no boss or professor
guiding you, is tough. But by narrow-
ing your focus you can do something
meaningful for a small group of users.
It’s easier to find a small group of us-
ers who really care about what you’ve
built or are talking about it. Building
a next-generation social network is a
terrible side project; although building
XRDS • FALL 2015 • VOL.22 • NO.1
June 13, 2015
The first-ever virtual reality painting exhibition,
“Tilt Brush,” was held in San Francisco, CA.
a Harvard student directory definitely
passes as niche.
˲˲ You will learn that distributing your
product and attracting users is hard,
harder than you can imagine. Before
you launch, it’s easy to dream about
how your project will take the world by
storm. But once no one shows up, you
will be forced to hustle to attract us-
ers by writing blog posts, answering
forum questions, working on SEO, and
so on. Pro tip: Planning will never work
as well as actually testing out ideas
post-launch.
˲˲ You will fail many times. As painful
as this sounds, it’s a necessary part of
real life. This isn’t a test you can study
for and ace. But since your side project
isn’t your full-time job or education,
it’s the best possible place you can have
to mess up, to learn, and to try again.
˲˲ You will broaden your skills. Your
side project should force you out of
your comfort zone and introduce you
to a wide breadth of skills and tools.
Learn how to write a great email. Ac-
tually run an A/B test. Do an analysis
using Google Analytics. Learn about
SEO. Play with new technologies and
services. Trying something out is far
better than just reading about it.
˲˲ Your side project will open doors.
It could even make you some money.
It will definitely give you a ton of new
ideas for other projects or businesses.
Introducing yourself as the founder
of your project will make you more
distinct as a job candidate, or at least
more interesting at a party.
˲˲ Try working with a potential co-
founder. Collaborating with friends is
great. You will learn the most about
what it’s like to work with someone by
building something together. Do you
have complimentary skills? Are you
able to disagree? Can you make deci-
sions together?
THE DESTINATION
Your side project is not the destination.
It is the journey. What I’ve found moti-
vating, more motivating than anything
else, is the learning of the journey it-
self. I guarantee you’ll learn the les-
sons I’ve mentioned. But first, launch
it. Go public. Say, “I did this!”
Biography
Natalie Gordon is the founder and CEO of BabyList,
an online baby registry. Before that, she graduated
from the University of Waterloo with a degree in computer
science and worked as a software developer at Amazon
for five years.
15
begin
Virtual reality immersion therapy is now a clinically
accepted form of treatment for posttraumatic stress
disorder for suffering military veterans.
BLOGS
From Novice to Expert: The
art of science of pedagogy
By Olivia Simpson
By now, the U.S. STEM problem—the growing gap between
demand and skill in American jobs in STEM fields—is well
recognized.1 One source of the problem is a lack of interest
in the sciences and mathematics among young Americans
that begins at an early age.
Improving our nation’s STEM education could be a
huge step in tackling this problem. In fact, there is lots
of research effort put toward how to better teach science
and math at all levels. How People Learn: Brain, Mind,
Experience, and School is a collection of research on human
learning and effective instructional environments.
The authors note as the nature of information has shifted
in the 20th century, the focus of education must as well:
“In the early part of the twentieth century, education
focused on the acquisition of literacy skills: simple
reading, writing, and calculating. It was not the
general rule for educational systems to train people
to think and read critically, to express themselves
clearly and persuasively, to solve complex problems
in science and mathematics...More than ever, the
16
sheer magnitude of human knowledge renders its cov-
erage by education an impossibility; rather, the goal
of education is better conceived as helping students
develop the intellectual tools and learning strate-
gies needed to acquire the knowledge that allows
people to think productively about history, science
and technology, social phenomena, mathematics,
and the arts. Fundamental understanding about
subjects, including how to frame and ask meaningful
questions about various subject areas, contributes to
individuals’ more basic understanding of principles
of learning that can assist them in becoming self-
sustaining,lifelong learners” [1].
However, the task of helping students develop tools and
strategies is considerably more complicated than teaching
facts. The goal of a science educator is to train students to
think like experts in the field—yet the gap between expert
and novice can be prohibitively large. The so-called “curse
of knowledge” or expert blindness might contribute to
the general idea that science education is bland, rigid,
and impersonal. Experts who have excelled to a level
of competency, which is no longer conscious, may have
trouble connecting to novices with no context for their own
level or mastery or lack thereof. How can experts connect to
novices in a way that is meaningful and engaging?
In direct response to this question, the researchers
behind How People Learn emphasize three key findings.
“Students come to the classroom with preconceptions
about how the world works. If their initial understanding
is not engaged, they may fail to grasp the new concepts and
information that are taught, or they may learn them for
[the] purpose of a test but revert to their preconceptions
outside the classroom.”
Educators must work with the pre-existing knowledge
students bring to the classroom, rather than as
approaching them as empty slates. The constructivist
mindset is that new learning is based on knowledge you
already have, and that therefore learning is very individual.
One way to approach teaching with respect to pre-
existing knowledge is active learning and peer instruction.
Active learning refers to the practice of deviating from
a lecture format and asking students to actively interact
with the material through worksheets, discussion, or
Image by Goran Bogicevic
demonstrations. One striking research study conducted
by Freeman et al. found overwhelmingly positive results
among students who engage in active learning at least
1 Earlier this year, Bill Nye wrote an excellent commentary on the problem
for CNBC.com entitled “Fixing the US STEM Problem.”
XRDS • FALL 2015 • VOL.22 • NO.01
 1962 The year Morton Heilig built Sensorama, a mechanical
contraption predating digital computers, considered to be among
the earliest examples of immersive, multi-sensory technology.
25 percent of class time [2]. In fact, the results were so
overwhelmingly positive there was immediate push to shift
research from whether or not active learning is effective to
which kinds of active learning are most effective.
Peer instruction is one form of active learning that has
been adapted in a number of college classrooms. A basic
peer instruction cycle would work like this:
1. Ask a multiple-choice question to the class. Good
questions are ones that prompt discussion and challenge
the preconceptions of students.
2. Each student will answer on their own.
3. Ask students to discuss their answers with their
neighbors. It might be difficult to engage shy students, but
often asking them to “convince your neighbor your answer
is correct” get the conversation going.
4. After a few minutes, ask students to again answer on
their own.
5. Lead a class-wide discussion based on the correct
answers and the answers you saw.
I have personally been in a class that used peer-
instruction in this way. The class was a subject enough
outside my comfort zone that I actually needed that bit of
encouragement to engage in the discussion. I personally
found having these little in-class quizzes encouraged me to
prepare more for the class, and to openly talk through my
thought process when answering a question. Ultimately, I felt
I gained a much better handle on the material because of it.
“To develop competence in an area, students must:
a.) have a deep foundation of factual knowledge,
b.) understand facts and ideas in the context of
a conceptual framework, and c.) organize knowledge
in ways that facilitate retrieval and application.”
I think the most important point in this key finding is
point c—organizing knowledge in a way that facilitates
retrieval and application. This is the point at which
students step beyond learning facts and move toward
learning tools and strategies. One way educators can
facilitate is by teaching some matter in depth, but
providing many examples of the same concept at work.
Of course, there must be a firm foundational knowledge
base upon which the students organize information. A
big question I’ve encountered among educators is how
to use a finite amount of class time to both impart the
factual knowledge base and build facts into a conceptual
framework. One idea is the “flipped classroom” model.
In this setting, students are asked to do reading or small
homework exercises before class time. The idea is to ask
students to learn the easy things on their own at home
(definitions or basic concepts), while using class time to
XRDS • FALL 2015 • VOL.22 • NO.01
help with the more difficult concepts.
“A ‘metacognitive’ approach to instruction can help
students learn to take control of their own learning by
defining learning goals and monitoring their progress in
achieving them.”
Notice the two things mentioned here: defining
learning goals and monitoring progress in achieving them.
Defining learning goals is not necessarily a natural thing
for a student—especially a novice—to do. This is another
way the educator can help. Explicitly defining and relaying
learning goals can be a great organizational tool for both the
instructor and the student. It is important to have detailed
learning goals at both a course and topic level, and to
explicitly address two or three with every class session. Good
learning goals should involve verbs (and not something vague
and unmeasurable like “understand”) and look something
like “by the end of this lesson/unit/course, you will be able
to...” This provides students a way for self assessment; using
this framework of knowledge to monitor themselves in how
well they are achieving their goals.
Monitoring progress is slightly more complicated, and
should involve many levels of evaluation and feedback.
During class time it is important to give some feedback and
to encourage a “try-fail-receive feedback-try again” cycle
and atmosphere for exploration. These small evaluations
should not be summative. It is always important for
feedback to be timely and at an appropriate level. It is hard
to achieve both things, but it equips students with the
appropriate sense of metacognition crucial for improving
from novice to expert.
All of the above ideas are just a few of the new
techniques being tested and used by instructors in
classrooms all over the country. These techniques could
really transform a classroom for the better—especially
in STEM classrooms that are not especially discussion-
based. The STEM problem is not insurmountable, and
instilling excitement and curiosity in students at an early
age will encourage many to go on to get degrees and
develop the skills this country needs.
References
[1] National Research Council. How People Learn: Brain, Mind, Experience, and School:
Expanded Edition. Washington, DC: The National Academies Press, 2000; http://www.
nap.edu/openbook.php?record_id=9853&page=1
[2] Freeman et al. Active Learning Increases Student Performance in Science,
Engineering, and Mathematics http://www.pnas.org/content/111/23/8410.abstract
Biography
Olivia Simpson is a Ph.D. candidate at the University of California, San Diego working on
algorithms for network analysis.
17
feature

One Step Beyond

Virtual Reality:
Connecting past and
future developments
We have come a long way since the late 1970s, when virtual reality
technology was in its infancy. With an array of new VR technologies
in the market, what might be the future impact on our daily lives?

By Diego Martinez Plasencia

DOI: 10.1145/2809921

V
irtual reality (VR) has gained incredible public visibility in the last three years,
undoubtedly as a result of Palmer Luckey’s vision: the Oculus Rift. What initially
started as a hobby, soon took off, becoming one of the brightest examples of
crowdfunding success. Over the years, Luckey accumulated a large collection of VR
headsets and by the age of 16 he started building his own. On August 1, 2012, he launched
his Kickstarter campaign, pledging a modest $250,000 and aiming to provide cheap and
high-quality headsets to a few VR enthusiasts. Unexpectedly, the campaign raised more
than $670,000 in its first 24 hours, and more than a million dollars within three days.
But Luckey’s most amazing achievement was making the world excited about VR again.
After witnessing the rise and fall The cinematographic industry also fu- develop a VR headset with unforeseen
of so many companies intent on com- eled our expectations to unreachable quality and priced under $300.
mercializing VR in the late ‘80s and levels with their futuristic vision of 3-D From that moment on Oculus has
‘90s (older readers might remember interfaces. After this long and continu- been a runaway success, which ended
the excitement around the Virtuality ous trickle of disappointments, VR en- with Facebook acquiring the company
gaming system, or how you felt when thusiasts were left heart-broken, hope- for $2 billion last year. The rise of Ocu-
you played “Dactyl Nightmare” for the less, and extremely skeptical. lus has also dramatically reshaped the
first time in an arcade). After decades However by the end of 2012, Ocu- stage of the technological market. Re-
of expensive and disappointing head- lus released its first development kit cent market reports foresee VR becom-
mounted displays (HMDs), I still re- only a few months after its campaign ing a mainstream technology in the
member my first experience with a Kai- launch—a big statement against all of near future. Last November, analysts at
ser ProView XL35. It was priced at more this skepticism. Although it was not Sophic Capital predicted VR will reach a
than €20,000 and only rewarded its yet a consumer-ready product, Luckey $7 billion market by 2018. Others, such
user with a field of view of 35 degrees. clearly demonstrated it was possible to as Piper Jaffray, estimate up to $62 bil-

18 XRDS • FALL 2015 • VOL.22 • NO.1

 lion by 2025. All major companies have
acknowledged this potential and are
competing to gain a dominant position
in this growing market (see Figures 1a-
1d). Samsung partnered with Oculus
to release Gear VR, a headset using an
embedded smartphone to create a self-
contained wireless experience. Google
Cardboard, Zeiss VR, and Visus VR have
Image courtesy of FOVE
proposed cases where users can fit their
smartphones to create VR experiences
on the go. Valve’s bet is called HTC Vive
and aims to bring VR to its huge com-
munity of gamers. Sony’s Project Mor-
XRDS • FALL 2015 • VOL.22 • NO.1
pheus will deliver the VR experience for
PS4 users. Companies like NVIDIA and
Apple have filed patents related to head-
sets, but have yet to reveal their inten-
tions. Even free community-based ap-
proaches, such as the Razer OSVR, are
now available.
Smaller companies have also joined
this challenge, adding novel and inter-
esting features to the plethora of com-
mercial VR headsets. In particular,
FOVE includes gaze tracking in their
headsets to determine where you are
looking, allowing possibilities out of
reach for other HMD devices, such as
support for social cues (e.g., eye con-
tact with other avatars), depth of field
effects, or easy selection of small ob-
jects by looking at them (e.g., keys on
a keyboard). I tried FOVE’s prototype
at the most recent South by Southest
festival. Although still an on-going de-
velopment, I was very impressed by its
display quality and gaze tracking ac-
curacy, maybe at the expense of com-
fort. It felt slightly heavier than Ocu-
lus’s DK2. Another ambitious project
is Mind Maze VR, which integrates an
19
feature

world around you to create immersive

Figure 1a. The Oculus Rift soon proved high quality and affordable VR headsets experiences. The flexibility to take on
were achievable. Interest in the Kickstarter campaign showed there was consumer both aiding tasks and immersive gam-
interest in VR. These market opportunities have attracted investment from many ing makes Hololens a very ambitious
technological giants. Many alternative VR devices are on the horizon. approach. Microsoft plans to embed a
Figure 1b. Oculus’ booth at this year’s Consumer Electronics Show (CES), after Window 10 system into the Hololens,
being acquired by Facebook. Figures 1c and 1d. Headsets from OSVR and Sony providing users with all the resources
have also gained public visibility in many trade shows, such as CES and E3. of a computer in a device that can travel
with you and assist you (or amuse you)

Images from E3 2015 courtesy of Entertainment Software Association (ESA). Images from CES 2015 courtesy of Credit Consumer Electronics Association (CEA).Razer OSVR image courtesy of Maurizio Pesce (Flickr).
throughout the day. Whether these
potential developments outweigh the
impact (on your peripheral vision and
social life) of having the Hololens cover
your face, is yet to be seen.

(a)
(c)
array of electroencephalography (EEG)
sensors and also features a depth cam-
era on the front of the device. The cam-
era supports hand interactions and
self-localization of the head. Interest-
ingly, it allows for augmented/mixed
reality experiences, which are experi-
ences that mix the virtual environment
and real world located around the user.
Another consequence of this small
revolution: Oculus’ momentum man-
aged to also bring augmented reality
(AR) back to life and into the market.
This adoption was initially slower,
maybe influenced by what had been—
at best—a slow uptake of Google’s at-
tempt in this field. Although Google
Glass was conceived before Oculus ar-
rived, at the time skepticism against
intrusive devices was strong in the pub-
lic’s mind. Google attempted to keep
their device unobtrusive and aestheti-
cally pleasing. But its small display, lo-
cated off to the side, did not allow wear-
ers to explore the full potential of the
AR experience. This coupled with its
not-so-attractive price (approximately
$1500) and some social concerns (such
as surreptitious video recording), pos-
sibly led Google to abandon the project
earlier this year.
(b)
(d)
The situation for newer AR headsets
was different. With all of the hype sur-
rounding Oculus-like headsets, later
AR solutions were conceived free of the
aesthetic concerns that constrained
Google Glass. These AR headsets em-
brace solutions that cover the user’s
face, but allow for AR to be explored in
full depth. Some of the earliest exam-
ples are OVRVision and Leap Motion,
which both proposed clip-on solutions
to add a stereo video feed of the real
world for Oculus Rift. VRVana’s Totem
and Impression Pi were probably the
first to add built-in cameras to their
headsets, which provided integrated
support to AR.
The big leap arrived at E3 2015, when
Microsoft presented its prototype of a
see-through AR headset: the Hololens.
Unlike the previously mentioned de-
vices, Hololens features a semitrans-
parent screen that allows you to see the
real world with your own eyes, instead
of a digitized video feed, which, even
if stereoscopic, would be no match for
the accuracy of human vision. Virtual
contents can then be added onto your
view of the real world, either in subtle
ways—to help you find your way as you
walk through a city—or overriding the
REVOLUTION OR REFURBISHMENT?
The resurgence of AR/VR hardware
clearly demonstrates technology has
reached a point, in terms of quality and
price, where these devices can be made
available to the public in an attractive
format. Mobile devices can satisfy the
computing and power necessities of
these resource-hungry devices, while
remaining cheap and lightweight. Dis-
play size, refresh rate, and resolution as
well as the advances of sensing technol-
ogies at affordable rates (such as Kinect,
Leap Motion, GloveOne, and a myriad
of other devices) have reached a level of
maturity that makes the current offers
possible. The spark required to actually
push technological giants to take on
this challenge for consumer grade VR
grew stronger with Luckey’s Kickstarter
campaign. After years of seeing VR/AR
through the big screen, people want to
experience it in their homes
However, although the current
range of devices undoubtedly outper-
form the devices available before, it is
also true that the formats proposed are
in no way different than the ones ex-
plored during the 1970s and 1980s by
the research community. Back in 1968,
Ivan Sutherland—who was a pioneer
in computer graphics, AR, and VR—
built the first head-mounted display
that included stereo 3-D graphics and
head tracking. In the following years
different types of headsets, tracking
technologies, tactile and force feed-
back devices, data-gloves, and 3-D con-
trollers were tested. According to this
older research, recent devices are not
new at all. Today’s technology can eas-
ily be categorized according to histori-
cal terminologies: Oculus, Gear VR, or
HTC VIVE (and their 3-D controllers)

20 XRDS • FALL 2015 • VOL.22 • NO.1

are “VR systems”; Totem and Impres-
sion Pi are “video see-through AR sys-
tems”; and Google Glass and Hololens
are “optical see-through AR systems”
(see Figure 2).
These new devices might have made
AR and VR shine again, however they
have not introduced any conceptual
improvements over previous systems.
This would not necessarily be a prob-
lem, but over the years AR/VR has been
described as a “solution looking for a
problem.” Much effort and money has
been spent to pursue the concept of AR/
VR. However, a “killer application” has
not yet been demonstrated, although
many possible applications have been
theoretically considered: virtual tours
to long-ago existing monuments, face-
to-face meetings with people on the
other side of the world, or whole cities
where augmentations guide and help
us in our everyday tasks. Some have
even been put into practice, to some ex-
tent, like training simulators in surgi-
cal and military settings. Still, AR and
VR remain largely unexploited.
Even if the new devices rely on old
formats, and their true application is
yet to be found, it might be that mak-
ing AR/VR technology available to the
masses will become the tipping point to
boost the revolution. After all track balls
and mice remained mostly experimen-
tal equipment in 1968, the potential of
which only becoming apparent when
broadly adopted by consumers purchas-
ing the Macintosh 128K. Even more
telling, back in the time of IBM’s main-
frames many people questioned why
any layperson would need a personal
computer in the first place. And if enter-
tainment is the initial fuel to power the
adoption of AR/VR technology, it will
follow the similar path of personal com-
puter games, which certainly became
the (untold) driving reason for wide-
spread adoption among many users.
TOWARD NOVEL VR FORMATS
Current trends in the AR and VR mar-
kets offer exciting possibilities. Gam-
ing giants such as Valve, Microsoft
Xbox, and Sony are pushing for broad
adoption. Facebook’s Oculus deal will
certainly push for novel VR formats
touching on social networks with the
potential to revolutionize computer
mediated social interaction. While the
XRDS • FALL 2015 • VOL.22 • NO.1
possibility to transform the not-so-suc-
cessful “Second Life” into a more faith-
ful representation of the metaverse de-
picted in Neal Stepheson’s Snow Crash
is equally as exciting.
The social implications of using
these technologies can have an im-
pact, whose importance is yet to be
seen. Users will, at least, wear devices
on their faces, creating interference
with eye contact and facial expression
and affecting potential social interac-
tions. By cutting us off from the world
around us, VR can increase social iso-
lation and raise other concerns. Ironi-
cally, this technology has the potential
to bring us closer to someone on the
other side of the world, but further
away from those closest to us.
This isolation also contrasts with
another concept becoming com-
monplace nowadays: the “Internet
of Things” (IoT). An important part
of the technology market is moving
toward devices that understand and
even anticipate our needs. In the IoT,
a ubiquitous infrastructure around us
would keep track of our agenda as we
move through our day: A public dis-
play in a museum tells you about an
exhibit relevant to your son’s history
assignment; while your mobile device
guides you to the proper location, you
notify your son about your finding,
Figure 2. In the foreground, novel devices in view of traditional VR/AR research.
Ivan Sutherland, VR pioneer, can be seen in the background wearing his headset.
who sees it displayed on the surface of
a cafe table, where he is enjoying a chat
with his girlfriend. In this scenario of
interrelated, seamless, and ubiquitous
technology, current AR/VR technolo-
gies look somewhat like jealous lovers.
By isolating you from the surrounding
world, you are cut off from the remain-
ing infrastructure and only allowed to
enjoy what it can offer you.
Social and technological isolation
could seriously limit the impact AR/
VR technology will have on our lives.
Looking for alternatives that reconcile
immersive AR/VR and the IoT might
play a key role in determining the types
of technologies that can have a deeper
impact on our lives. Present and future
researchers should take on this chal-
lenge to develop such formats or inte-
grated infrastructures.
The presence of touch-enabled sur-
faces, such as tables and displays, is al-
ready becoming ubiquitous around us.
Researchers from Hasso Plattner Insti-
tute are investigating how to adapt this
concept to new formats, such as inter-
active floors with display and touch
abilities. BaseLase, developed by Aar-
hus University, can create interactive
floors of up to 75 square meters using
an integrated unit, which is only 1.3
meters high, and thus could be easily
integrated with public furniture (see
21
feature

Figure 3a. BaseLase from Aarhus University. Projectors with the ability to cover big areas and even to be integrated in
public furniture are becoming available. [Reprinted with permission from Aarhus University.] Figure 3b. RoomAlive from
Microsoft Research. Interactive walls and projection are one of the alternatives to VR that allow ubiquitous augmentation
of the object and spaces we inhabit. Figure 3c. IlumiRoom from Microsoft Research. When coupled with head tracking,
projectors can be used to create immersive 3-D effects, without isolating the user from the real world. [Reprinted with
permission from Brett Jones].

(a) (b) (c)

Figure 3a). Mobile devices with embed-
ded projectors have been proposed as
an alternative to display information
on the objects around us. In this case,
the device is bound to the user instead
of a fixed infrastructure, but is still
part of the IoT.
All these approaches can overlay
augmentations (notifications and infor-
mation) on the world around us and not
occlude users’ faces. However, there are
two important limitations when com-
pared to AR/VR headsets: 1) augmenta-
tions lay on the surface of the world, you
cannot create a virtual pet that seems
to walk on a table; and 2) information is
equally visible to everyone.
Overcoming the first limitation—
creating illusions that pop out from
the surface where they are projected—
usually requires stereo techniques.
However, stereo is an effect that decays
greatly as objects become more dis-
tant. Thus, detecting the user’s head
position and showing the object from
the correct perspective as the user
moves can create compelling 3-D illu-
sions. The Microsoft Research team
lead by Andy Wilson has demonstrated
a variety of projector-based systems ex-
ploring this concept. In particular, the
Beamatron combines a depth camera
and a projector on a pan-tilt platform.
This allows the system to track the
user’s head, the room, and project—for
instance, a racing car that interacts
with the objects in the room. Illumi-
Room and RoomAlive demonstrate
this concept with 3-D augmentations
popping out from any point on a wall or
even a whole room (see Figures 3a and
3b). Taking a more literal approach,
the Tangible Media Group at MIT’s
Media Lab created a shape-changing
tabletop, whose shape changes to ap-
proximate that of the virtual content to
display.
CastAR, from Technical Illusions,
combines projected AR with retro-
reflective materials on the surfaces
to be used as windows to the AR ex-
perience. The information projected
from each user is retro-reflected to his
or her eyes, so it only becomes visible
to the user. CastAR also provides the
possibility to correctly see and inter-
act with the 3-D contents using your
hands or controllers, which is not
possible with traditional optical see-
through HMDs. These headsets pres-
ent content floating at approximately
3 meters in front of us. Our eyes (we
can think of them as a lens with a usu-
al depth of field of + 0.3 diopters) can
blend content with objects, which are
more than 1.6 meters away from us.
Thus, we will be able to blend the aug-
mentations on road signs and cars as
we drive (hopefully more than 1.6 me-
ters in front of us). However, if our eyes
focus on our finger 40 centimeters in
front of us, the contents of the head-
set (optically, at 3 meters) will appear
blurry. In contrast, CastAR augmenta-
tions’ depth is that of the table or wall

Figure 4. Systems embedding displays made of fog provide a look and feel similar to the headset based VR. Figures 4a and 4b.
Mistable allows users to interact with their personal contents on a fog screen in front of them. These contents can be shared on
the central tabletop, or they can be pulled above the tabletop to reveal their 3-D appearance and interact with them. Figure 4c.
Sensabubble uses projectors and soap bubbles filled with smoke to create displays that can float anywhere around the room.

(a) (b) (c)

22 XRDS • FALL 2015 • VOL.22 • NO.1


Figure 5. The fog used in fog displays is made of polar molecules, which react to
electric fields. This property is being explored to control the trajectory of bubble
displays and the shape of fog screens.
where light is retro-reflected. We are
free to approach and interact or touch
the virtual objects on these surfaces.
The need for retro-reflective surfaces
everywhere in a home, and the fact
that it is a headset (although less ob-
trusive than others), clearly limits the
applicability of CastAR. However, it is
certainly an interesting concept and
could allow for some amazing table
game experiences.
The second challenge—showing
content only to specific users (rather
than anyone), while keeping the user
away from any eyewear—has also been
explored. Systems like IllusionHole,
TADs, and Mustard combine masks and
user tracking to provide shared surfac-
es with user specific content, allowing
each user to see something different.
Other systems use intelligent diffuser
materials, such as Lumisty film, which
diffuse light or become transparent
depending on the observer’s location.
Combining these and several projec-
tors, systems like Lumisight and Ulteri-
orScape create tabletops where the im-
age seen by each user depends on which
side of the table they sit.
The University of Bristol’s Interac-
tion and Graphics Group has proposed
several tabletop systems aiming to
take on both challenges simultane-
ously (personalized and 3-D content).
PiVOT is the first system that com-
bines Lumisty film, a projector, and
XRDS • FALL 2015 • VOL.22 • NO.1
an autostereoscopic display, which can
display glasses-free stereo images. Us-
ers sitting around the tabletop share a
common surface they can use to share
and interact. However, when a user
leans forward toward the transparent
region of the Lumisty film, the auto-
stereoscopic display becomes visible.
This second display shows a tailored
view of the tabletop, with specific 2-D
or 3-D content for that user. Another
user leaning forward will get a person-
al view, different from that of the first
user. This allows users to access their
personal information and to transi-
tion from shared to individual tasks
(or simply talk to others) quickly. At
the same time, their attention stays
on the same space where the other us-
ers are, so everyone can keep track of
the group’s developments. A potential
limitation is someone can occlude the
vision of other users while interacting
with personal content that the others
cannot see.
Mistable follows a very similar phi-
losophy, combining a tabletop with an
interactive fog screen between each
user and the display (see Figures 4a
and 4b). Fog screens use a thin curtain
of fog as a projection surface and a pro-
jector. Non-projected areas (black) stay
transparent, while colors projected be-
come visible, seemingly floating in thin
air. This arrangement allows users ac-
cess to the shared tabletop and to see
each other (social interaction). How-
ever, users can access personalized
content in their fog screen, interacting
with it right in front of them and with-
out distracting others. Users can also
move content freely among the spaces,
with specific connotations associated
with each space. For instance, a user
can push a picture onto the tabletop,
for everyone to see it. Alternatively, they
can push it to someone’s fog screen, to
share with him or her only. The space
above the tabletop can be used to pres-
ent floating 3-D objects. As each user
sees the 3-D space through their per-
sonal screen, they can get a tailored
view of the same object. In a more ec-
centric approach, these researchers
have also explored using floating soap
bubbles filled with scented smoke (see
Figure 4c). With 3-D tracking, informa-
tion can be projected onto the bubbles,
creating on demand displays floating
anywhere in your house. The trajectory
of the bubbles can be controlled using
electric fields, a technique also being
explored to create shape-changing fog
displays (see Figure 5).
CONCLUSION
These later systems certainly have not
reached a level of maturity comparable
to that of the incoming commercial AR/
VR headsets. Quite possibly, their best
context of application, i.e., their “killer
application,” still remains unclear and
their price range is out of our reach by
several orders of magnitude. However,
they follow an inspiration that can
bring together concepts from 3-D in-
teraction with unobstructed access to
other present, or future, technologies
around us. And, more important, they
demonstrate new possibilities that
can spark our imagination. And who
knows, maybe these sparks will one
day catch fire and inspire the next revo-
lution in 3-D interfaces, taking us one
step beyond VR.
Biography
Diego Martinez Plasencia is a researcher in 3-D user
interactions (3DUIs) at the University of Bristol. His research
work involves all shorts of technologies, from the traditional
virtual-reality headset or projection systems, to auto-
stereoscopic display technologies and fog screens. In recent
years, his research has focused on creating systems that
allow users to get the benefits of collaborative VR worlds, but
in formats that integrate seamlessly in our day-to-day lives.
© 2015 ACM 1528-4972/15/09 $15.00
23
feature

Substitutional Reality:
Bringing virtual
reality home
Now that virtual reality headsets are finally reaching the wider
consumer market, how can we merge the physical and virtual worlds
to create a unified multi-sensory experience?

By Adalberto L. Simeone and Eduardo Velloso

DOI: 10.1145/2810044

F
or many of us, virtual reality (VR) evokes all sorts of imagery—from neon vector
graphics to dream worlds built to imprison humankind. In its essence, VR as an
artistic medium promises to fulfill a quintessential human desire of exploring places
and living lives that are unattainable in the real world. VR has also found fruitful
practical applications in the fields of architecture, training, design, and rehabilitation. And
with the release of consumer head-mounted displays, such as the Oculus Rift and other
headsets, VR is finally ready to enter our homes. The textbook definition of VR refers to the
artificial simulation of sensory experiences. Modern VR systems do a great job of simulating
visual and auditory sensations, but there is still a long way to go until we are
able to successfully simulate the other real-world counterpart that the user are significantly different than our
senses, often described as the “holy could experience physically? We call furnished (and often messy) homes.
grail” of VR. To truly understand the this “substitutional reality” (SR) [1]. For example, the instruction manual
potential of VR, it is necessary to expe- Each virtual object has a physical ob- for HTC’s Vive headset explicitly rec-
rience the complete package. We are ject that substitutes it, allowing users ommends users to rearrange furni-
interested in looking for alternative to experience the virtual world in the ture in order to make space for the
possibilities for stimulation without same way as the physical world. They tracked area. Other home VR systems
necessarily simulating experiences can navigate the environment without often avoid this problem by limiting
through additional actuators. Our bumping into furniture, or touch and users to their desks, and allowing
approach is to leverage the physical feel virtual objects as if they were real navigation through mice, keyboards,
properties of the world around us to (see Figure 1). and game controllers. However, this
Image by Jennifer Morrow

stimulate the senses that we cannot
yet simulate in VR. What if physical
objects, rather than being obstacles
or impediments, were incorporated
into the virtual environment? What if
every object in the virtual world had a
Research labs responsible for
building VR systems often employ
large, empty rooms that allow users to
move freely without the fear of bump-
ing into furniture or tripping on ob-
stacles. However, these environments
effectively removes two of the most
compelling presence-enhancing fac-
tors in our living rooms—touch and
locomotion.
Previous work on passive haptics
has shown the use of an exact match

24 XRDS • FALL 2015 • VOL.22 • NO.1

XRDS • FALL 2015 • VOL.22 • NO.1 25
feature
Figure 1. Two virtual environments based on the layout of a real room (middle), a medieval courtyard (left), and the bridge
of a spaceship (right). In these substitutional environments, every physical object is paired, with some degree of discrepancy,
to a contextually appropriate virtual object.
(a)
between the physical and the virtual
objects can highly benefit the experi-
ence [2]. However, if virtual reality is a
mere replica of the physical world, is
it easier to just take the headset off?
In SR we seek to embrace this mis-
match between the two environments
and investigate how far a designer can
push this difference until it breaks
the illusion. What if a designer wants
to add objects that do not have a physi-
cal counterpart?
Imagine you are in your living
room, sitting on your couch, and
drinking a cup of tea while watching
TV. You turn off the television, place
your mug on the coffee table in front
of you, and put on a VR headset that
transports you to a medieval court-
yard. All of a sudden, the couch under-
neath you becomes a wooden bench.
The tea cup is now a ceramic mug rest-
ing on a beer keg. Where there used
to be a TV, you now see a sword rack.
Every object in the physical world is
now substituted by a new one, which
matches the context and theme of the
virtual environment, but with certain
variations. Would you still believe you
are sitting on a wooden bench despite
the comfort of a cushioned couch?
Would the difference in size between
the teacup and the beer mug affect
how you manipulate it? Would you
still believe you were drinking beer
as the sweet taste of tea touches your
lips? These are only some of the ques-
tions that this scenario uncovers.
ENABLING TECHNOLOGIES
An SR system is capable of dynami-
cally creating a virtual environment
based on the physical environment in
26
(b)
which it is deployed. There are three
main components to an SR system.
First, it requires a means of captur-
ing the physical environment. The
most basic way of accomplishing this
is to manually create a virtual envi-
ronment that substitutes the physical
one. Individual objects can then be
tracked using computer vision, infra-
red markers, or inertial measurement
units. More sophisticated approaches
include scanning the environment
with a wearable depth camera or even
combining the input from several
cameras using projection mapping.
Second, it requires a means to rec-
ognize, segment, and map physical
objects to their virtual counterparts.
Possibilities range from supporting
users to manually creating such map-
pings to automating the substitution
process. Third, it needs to render the
virtual environment back to the user
through a head-mounted display
while it tracks moving objects. This
can present significant challenges for
system developers as combining sur-
Every object in the
physical world is
now substituted by
a new one, which
matches the context
and theme of the
virtual environment,
but with certain
variations.
(c)
face reconstruction, object tracking,
and 3-D rendering at a frame rate high
enough not to cause nausea is a highly
computationally intensive task. The
feasibility of these components has
been previously demonstrated in
other works and products [3]. A Utah-
based startup, The Void, is working
on an SR experience using a theme
park approach. With a combination
of props, wearable haptic devices, and
environmental effects, the company
proposes to create multisensory expe-
riences. In “the Void,” a physical envi-
ronment is specifically built to match
the virtual environment. The physical
props provide haptic feedback and
the system uses tracking data to pair
them to their virtual counterparts.
In substitutional reality, users are
fully immersed in the virtual environ-
ment. However, other works in mixed
reality also aim to merge the physi-
cal and virtual worlds. For example,
Microsoft’s Hololens overlays digital
content onto physical objects through
a wearable see-through display. On
the other hand, RoomAlive, also by
Microsoft, projects digital content
onto the physical environment using a
combination of projectors and depth
cameras [4]. Although these projects
have different motivations and goals,
the underlying techniques can also be
used for SR systems.
LEVELS OF MISMATCH
Considering the mismatch between
physical objects and their virtual
counterparts, we propose a sequence
of five levels of mismatch. At the bot-
tom level we have replicas, substitut-
ed objects that are 1:1 representations
XRDS • FALL 2015 • VOL.22 • NO.1
of their physical proxy. In the second
level, we have aesthetic substitution,
in which visual properties, such as the
texture or material of the object are
modified. In the third level, we have
addition/subtraction, in which there
are differences in the physical shape
of the object, with virtual objects pre-
senting features that do not exist in
the physical object and vice versa. In
the fourth level, we have functional
substitution, in which there is a mis-
match between the affordances of
the two objects (for example, between
a book and a small box, which are
opened in different ways). Finally, in
the fifth level, there is a categorical
substitution, where there is little to no
resemblance between the virtual and
physical objects. As we move across
these different levels of mismatch,
the more we hinder the believability
of the substitution process.
To better understand the effects of
each level of mismatch we conducted
a user study with two tasks. In the
first, we substituted the same physical
object (a mug) with multiple virtual
representations (see Figure 2).
Our results indicate the material
used to portray an object significant-
ly influences the user’s expectations.
If a box is rendered with a material
Figure 2. In the picture each pair of objects exemplifies a different substitution type. The replica mug is used as baseline
(a). The pair of substitutive objects are: (aesthetic) glass (b) and wooden mug (c), and hot (d) and icy mugs (e); (addition/
subtraction) big mug (f) and small mug (g); (function) basket (h) and lamp (i); and (category) box (j) and sphere (k).
XRDS • FALL 2015 • VOL.22 • NO.1
such as iron, users might expect it
to be heavy. But if the proxy object is
lighter, it will then affect the believ-
ability of the experience. We expect
an SR system should be able to find
the best match according to the re-
quirements of the experience and the
best design guidelines. Thus, if the
physical box is light, the mismatch
can be minimized if it is rendered as
if it is made of tin. If the mismatch
between the actual and the expected
weight is desired, designers can use
it to simulate super-human abilities,
i.e., pairing physical pillows to heavy
boulders in a superhero game. Find-
ing these design guidelines is among
the main objectives of this and future
research.
Ideally, in Ivan Sutherland’s semi-
nal essay on augmented reality, “The
Ultimate Display,” every object simu-
lated by a computer would feel as real
as the actual object. We will probably
have to wait a bit more to enjoy this
“Holodeck”–type VR experience. In
the meantime, the best way to cor-
roborate the illusion is to use physi-
cal props, as The Void is planning to
do. However domestic users cannot
rely on the same resources available
to a commercial endeavor. Thus, we
asked ourselves whether users would
be engaged primarily by using life-
like props, or whether it could be pos-
sible to use approximations in their
stead. For example, not many of us
have a lightsaber at home. Conversely,
a torch or an umbrella are very com-
mon. Can we use objects that can be
manipulated in a similar way and
still be engaged in the VR experience?
Would knowing we are wielding a
torch instead of a lightsaber be im-
mersion breaking?
We set out to answer these ques-
tions in a second study. Different
from our first study, this time around
we fixed the virtual appearance of the
object participants interacted with
to look like Darth Vader’s lightsaber.
However, in reality, participants in-
teracted with three objects: a 1:1
ForceFX’s replica of Darth Vader’s
lightsaber, a torch, and an umbrel-
la. Participants had to hit a floating
sphere with their virtual lightsaber.
After each session, we asked for their
feedback on the object they wielded:
If they felt it distracted or supported
the idea of wielding a lightsaber, how
much they preferred using it, etc. We
found no significant differences be-
tween the actual lightsaber replica
and the torch. When asked if they
could guess which object they held,
27
feature
participants did so correctly only 28
COMMUNICATIONSAPPS percent of the time.
This indicates that true 1:1 repli-
cas are not mandatory to enjoy a VR
experience. Indeed, it is possible to
substitute more common objects in
place of those required by the VR ex-
perience. This is an important aspect
to consider, especially when VR expe-
riences geared toward entertainment
most frequently involve objects that
might be imaginary or very unlikely to
be found in a domestic environment.
Can all objects equally substitute an-
other one needed by the experience?
After analyzing our experiment’s re-
sults, we feel in order to minimize
the risk of breaking the illusion, the
two objects should be similar at least
Access the in the parts the user is most likely to
latest issue, interact with. Thus, a torch is a good
candidate for a lightsaber, whereas
past issues, a book would not be. When we inter-
act with a sword, we are more likely
BLOG@CACM, to focus on the hilt rather than on
the blade. Thus, the object can have
News, and a greater mismatch in the areas less
likely to be interacted with, such as
more. the blade. By extension, objects that
are out of reach can have even greater
mismatch with their paired physical
proxy, or even not exist at all.
FUTURE DIRECTIONS
The exploratory nature of our inves-
tigation uncovered several direc-
tions for future research [5], which we
Available for iPad, broadly categorized as those affecting
perception and the design of SR expe-
iPhone, and Android
riences, as well as the technical chal-
lenges to building SR systems.
One of the overarching goals of this
research is to inform future designers
The core idea of
Available for iOS, SR rests on the
Android, and Windows assumption that
the environment
perceived while
immersed is related
to the physical one in
which the experience
takes place.
28
of the best design practices to follow
when building VR experiences based
on physical environments. The stud-
ies we conducted indicate the idea of
using mismatching objects in place
of the virtual ones they should substi-
tute is a feasible one. There are, how-
ever, several implications that would
be worthwhile to follow. In the model
we presented, we introduced several
categories. Whereas we compared
the effects of different properties at
a coarse grained scale, more work is
necessary to quantify exactly how far
we can push this mismatch.
Locomotion also represents anoth-
er main direction. We mainly focused
on the mismatch between physical
and virtual objects. The same con-
cept of mismatched substitution can
also be applied to furniture and other
architectural features such as doors,
staircases, etc. Based on our results,
we can immediately rule out substi-
tutions greatly diminishing the size
of an object as being potentially dan-
gerous. The core idea of SR rests on
the assumption that the environment
perceived while immersed is related
to the physical one in which the ex-
perience takes place. Subtracting ele-
ments from the virtual environment
would break this assumption. There-
fore, all physical elements should be
represented in the virtual environ-
ment in some way.
The key issue is how do we incor-
porate tangible elements together
with immaterial ones? We created
two wildly different environments
based on the same physical one.
However in order to design the wid-
est variety of environments, some
immaterial elements could still be
needed. For example, if we were to
map a narrow hallway structure to
anything else than an actual hallway,
we could find ourselves in the situa-
tion where part of the hallway is tan-
gible and the other is not. Thus being
able to walk straight through walls
would severely affect the believabil-
ity of the experience, especially if on
the other side of the wall are tangible
objects that we cannot see because of
the virtual wall. Is this scenario likely
and if so, what we could do to mini-
mize or negate its occurrence? An-
other problem is represented by the
XRDS • FALL 2015 • VOL.22 • NO.1
disparity existing between a typical
virtual environment and the physical
environment. If our physical environ-
ment is substituted by a single, room-
sized virtual environment, how do we
transition to another one? Future re-
search could explore the feasibility of
redirected walking techniques to the
cluttered environments that would
be common in SR.
The technical side represents the
next part of the puzzle. So far, we have
authored the environments manually
and tracked participants and objects
with state-of-the-art optical tracking
systems. Our vision of SR is one where
virtual environments are dynamically
adapted to the layout of the physical
environment, through a “substitution
process.” In order to be fully dynamic,
this process will need to reconstruct a
3-D representation of the physical en-
vironment. However, unlike current
research in the field of 3-D reconstruc-
tion, in most cases SR will only require
an approximation of the environment
(for example, a 3-D environment con-
sisting of the bounding boxes of the
objects in the scene). This intermedi-
ate representation can then be adapt-
ed to the particular requirements of
the VR experience: The bounding box-
es identifying a chair can be turned
into a car seat or a tree trunk, a desk
into a stone altar, etc. This is no easy
task and will require research into
both hardware and software solu-
tions. We believe it can be achieved in
steps: Starting from the dynamic sub-
stitution of the immediate surround-
ings of a user’s desk, then proceeding
to the substitution of a single room,
and eventually large environments—
perhaps even outdoor ones. The first
step especially supports what the
gaming press regards as a “killer ap-
plication,” specifically cockpit games
such as flight and racing simulators.
These VR experiences simulate simi-
lar conditions to those we would find
in reality. In most cases, it would not
be possible to walk around the cock-
pit and thus these are ideal for a desk-
top VR game. One of the disadvan-
tages of VR is we lose awareness of the
surrounding environment while im-
mersed and thus activating functions
that require the keyboard becomes
problematic. SR could help by map-
XRDS • FALL 2015 • VOL.22 • NO.1
The textbook
definition of VR
refers to the artificial
simulation of
sensory experiences.
Modern VR systems
do a great job of
simulating visual
and auditory
sensations, but
there is still
a long way to go
until we are able to
successfully simulate
the other senses.
ping the actual cockpit instruments
to objects we have on our desk, so that
pressing a button will encounter pas-
sive feedback, with the potential of
heightening the sense of presence.
In automated mapping of virtual to
physical objects, designers of SR expe-
riences will have to do so without ac-
tually knowing what the physical en-
vironment looks like. The logic of the
experience needs to be described in
abstract terms, i.e., “place this object
near a window if available, otherwise
on the first available planar surface.”
Further, it is conceivable that for the
same experience not all physical en-
vironments will present the same de-
gree of mismatch. For example, if we
consider a medieval experience, the
degree of mismatch will be a lot less
if deployed in an actual castle rather
than in a high-rise apartment. How
can we quantify the mismatch in or-
der to objectively compare the same
experience across different physical
environments?
CONCLUSION
These are exciting times for virtual
reality. Although it failed to become
mainstream in the 1990s, 20 years
later it is experiencing a renaissance.
Although computing technologies
have improved significantly from the
state they were back then, VR still
faces several challenges. To prove its
worth, virtual reality has to evolve
beyond being a display alternative.
Substitutional reality has the po-
tential to untether users from their
desks and immerse them into worlds
where passive haptics strengthen the
illusion. This concept is not just ap-
plicable to entertainment. Imagine
going to a museum where an exhibi-
tion allows you to explore a building
from a different time period, where
props are substituted by period pieces
in the virtual environment. Substitu-
tional reality could help rescue and
response personnel train under harsh
conditions in virtual replicas of real
environments where fire or other ac-
cidents have been introduced. In the
future, substitutional reality could
even be applied to outdoor environ-
ments, where the terrain and flora are
substituted with those from a differ-
ent planet, or perhaps even a different
reality. The possibilities are endless.
References
[1] Simeone, A. Substitutional Reality: A research
agenda. In Proceedings of the first Workshop on
Everyday Virtual Reality (WEVR) at VR 2015. IEEE,
19-22.
[2] Hoffman, H. G. Physically touching virtual objects
using tactile augmentation enhances the realism of
virtual environments. In Proceedings of the Virtual
Reality Annual International Symposium, IEEE,
1998,59-63.
[3] Izadi, S., Kim, D., Hilliges, O., Molyneaux, D.,
Newcombe, R., Kohli, P., Shotton, J., Hodges,
S., Freeman, D., Davison, A., and Fitzgibbon, A.
KinectFusion: Real-time 3D reconstruction and
interaction using a moving depth camera. In
Proceedings of UIST 2011. ACM, 559-568.
[4] Jones, B., Sodhi, R., Murdock, M., Mehra, R., Benko,
H., Wilson, A., Ofek, E., MacIntyre, B., Raghuvanshi,
N., and Shapira, L. Roomalive: Magical experiences
enabled by scalable, adaptive projector-camera
units. In Proceedings of UIST 2014. ACM, 637-644.
[5] Simeone, A., Velloso, E. and Gellersen, H.
Substitutional reality: Using the physical
environment to design virtual reality experiences.
In Proceedings of CHI 2015. ACM, 3307-3316.fdfd
Biographies
Adalberto L. Simeone is a lecturer at the School of Creative
Technologies at the University of Portsmouth (UK). His
research focuses on how to bridge the gap between users
and their access to 3-D interaction and virtual reality
technologies.
Eduardo Velloso is a Ph.D. candidate at Lancaster
University and will take up a position as a Research Fellow
at the Microsoft Research Centre for Social Natural User
Interfaces at the University of Melbourne in January 2016.
His research interests include multi-modal interaction,
eye tracking, and post-desktop user interfaces.
© 2015 Copyright held by Owner(s)/Author(s).
Publication rights licensed to ACM.
1528-4972/15/09 $15.00
29
feature

Using Your
Own Muscles:
Realistic physical
experiences in VR
Leveraging the user’s own muscles to simulate impact and forces
from a virtual reality world allows us to create more immersive
experiences without bulky equipment.

By Pedro Lopes, Alexandra Ion, and Robert Kovacs

DOI: 10.1145/2810243

C
onsumer virtual reality headsets (such as the Oculus Rift, Samsung Gear VR, or
Google’s Cardboard) provide an exciting peek through the window of virtual reality
(VR), but that is where our experience ends. As we reach out to touch these novel
realities we are left with a sense of disillusion, because all we see are 3-D ghosts
without any mass. To give VR users a sense of an object with mass, traditionally large robots
are used that can, for instance, move the user or simulate virtual terrain. However, these
large and stationary robots are out of place in the mobile age: We are used to small and
light, yet powerful, devices in our pockets. After all, consumer-ready VR became a reality Image by Alicia Kubista / Andrij Borys Associates / Shutterstock.com
when the headsets finally met the size and portability requirements of our mobile times.
At the Hasso Plattner Institute’s their fingers, arms, and even the ter- as Lufthansa’s flight simulator.1 These
HCI Lab, our research has focused rain under their feet. This is our con- motion platforms are heavy station-
primarily on bringing a physical tribution to making virtual reality ary machinery that are too impractical
component to mobile VR experi- physical. to ever reach our homes—even if you
ences. Users can reach out and feel could afford a large motion platform,
what is, virtually, there. Because we MOBILE TERRAIN SIMULATION space is sacred these days. This tech-
have engineered these devices to be From the inception of VR in the 1960s nology was acceptable when VR was
wearable, they leave the user uncon- [1], through the industry hype of the a specialist product, and these ma-
strained to freely move in space and 1980s, up until today, many efforts chines shone through their precision
enjoy the virtual experience to its have been geared toward helping users rather than their form factor. In fact,
fullest. In this article we describe a to “feel” virtual worlds. Industry and the biggest players in VR were not gam-
new world of VR, in which users feel research labs responded to this chal-
forces from the virtual world through lenge by using motion platforms, such 1 http://www.virtual-fly.com

30 XRDS • FALL 2015 • VOL.22 • NO.1

XRDS • FALL 2015 • VOL.22 • NO.1 31
feature
Figure 1. The Level-Up motorized stilts allow users walking in a spatial virtual
reality environment to experience physical elevation.
ers, but training centers (e.g., airplane
pilots) and amusement parks (e.g.,
VR rides).
These massive motion platforms
were able to provide the sense of “be-
ing moved” or the sensation of accel-
eration, but weren’t really suited for
simulating a “walk in the park.” Thus,
many researchers designed mechani-
cal devices targeted at rendering the
sensation of stepping onto uneven
ground. Canonical examples of these
were treadmills equipped with indi-
vidually height-adjustable elements
that simulated bumpy terrain and
virtual slopes [2], or the CirculaFloor,
which uses four robot units that place
themselves under the user’s steps [3].
However, in the era of consumer-ready
mobile VR we cannot expect to own a
32
piece of infrastructure such as Circu-
laFloor. It is enough having a Roomba
roam around a house for cleaning pur-
poses, but four jumbo-sized Roombas
moving around the living room for the
sole purpose of emulating VR terrain?
In one of our projects, we took a dif-
ferent approach to ground simulation.
Instead of placing the human into a
robotic device, we mounted motors
under the human. With this approach,
the physical feedback device goes
wherever the human goes—it is mo-
bile. Our computer-controlled stilts, or
“Level-Ups,” allow users to physically
experience elevation. Level-Ups are
designed to allow users to freely walk
in the virtual and physical world, and
produce a higher sense of presence.
While you walk in a virtual world
with just a headset, you encounter vari-
ous obstacles that you cannot interact
with using your physical senses. Howev-
er, with Level-Ups you can actually step
on these objects as depicted in Figure 1.
To get across obstacles, you physically
lift your foot and climb across. In the
physical reality, we see physical eleva-
tion is rendered using our Level-Up mo-
torized stilts. Depicted here, Level-Ups
provide the physical sensation of an el-
evated ground to a user wearing a head-
mounted display tracked by an optical
motion capture system. Every Level-Up
device consists of a boot, mounted onto
a lift table, which in turn is mounted
onto a simple artificial foot. The lift ta-
ble is actuated by a motor, which is ob-
served by an encoder and controlled by
a microcontroller that communicates
via bluetooth with VR applications.
Each boot can render an elevation of
up to 12.5 centimeters. Because Level-
Ups only extend while the user’s foot is
in mid-air, they can be fully extended
or contracted with a compact 40W mo-
tor in half a second, yielding a wearable
free-walking physical feedback device.
THE CHALLENGE OF
MOBILE FORCE FEEDBACK
Level-Ups was, for us, a clear step in the
right direction toward mobile physical
VR, but is it what sci-fi had us dream-
ing about for years? What about feeling
forces from the virtual world?
These dreams of a physically be-
lievable virtual reality aren’t new, but
so far they required heavy gear, such
as exoskeletons. These are large me-
chanical contraptions mounted onto
users; a motor is placed at every joint
of their limbs and thus users are able
to manipulate their limbs while they
experience VR. Think of motion plat-
forms that you carry around. The hu-
man is rather powerful, and therefore
the motors required to move our limbs
must be powerful, too. And powerful
motors are large. Now if you look at
Google Cardboard, Oculus Rift, or any
other headset on the market, you real-
ize their form factor is orders of mag-
nitude smaller than the actuator sys-
tems that would allow us to bring some
physicality to virtual experiences. So
how can we produce enough power to
actually move a human limb, but in a
very concise form factor?
XRDS • FALL 2015 • VOL.22 • NO.1
 At first this sounded as contradic-
tory as Newton’s Third Law, which de-
mands a counter force to every force
you want to apply; this is why an exo-
skeleton has to be a certain size—it
must attach to your arm and apply a
counter force (called force feedback) to
your muscle motion. The same applies
if you want to play virtual soccer: A ro-
botic contraption would need to apply
a force to your foot every time you hit
the massless virtual ball you see on
the headset. So how do you keep the
required amount of force feedback,
but scale down the hardware? Our ap-
proach has been to simply take all the
mechanics out of the equation and
replace them with our own muscles.
Because our technique uses human
muscles to create the force sensation,
we called it “muscle-propelled force
feedback.”
perceived the interaction between the an Arduino microcontroller, a Blue-
involuntary and voluntary tilt as force tooth modem, and an EMS unit (similar
feedback. In fact, these participants to muscle rehabilitation devices found
rated muscle-propelled force feed- in a doctor’s clinic), which outputs a
back as more exciting than vibration maximum 50V/100mA. If you stop and
feedback, which is the standard feed- think for a second about these num-
back modality for mobile devices. bers, you will realize the 100mA needed
The great thing about not using mo- to actuate muscles is two orders of mag-
tors is we were able to miniaturize the nitude more efficient than actuating a
device to fit in a 133 mm × 70 mm × 20 conventional motor. This happens be-
mm casing, weighing only 163 grams. cause, in reality, the electrical muscle
This simple prototype is comprised of stimulation is just a “control signal” to
Figure 2. Our prototype electrically stimulates the user’s arm muscles via the
electrodes shown, causing the user to involuntarily tilt the device. As this user is
countering this force, he perceives force feedback.

LEVERAGING THE USER’S MUSCLES

Muscle-propelled force feedback
brings force feedback to mobile de-
vices. It explores the user’s own muscle
power as a replacement for motors.
To achieve this, we actuate the user’s
muscles using electrical muscle stim-
ulation (EMS), a technique first ex-
plored in physical rehabilitation in the
1960s and 1970s [4], and more recently
in the research of Tamaki, Miyaki,
and Rekimoto [5].
Figure 2 illustrates the use of our
mobile force feedback prototype in
a mobile gaming scenario. The mus-
cle-propelled prototype is mounted
on the back of a mobile phone. The Figure 3. A close-up of our prototype and the electrodes, which carry the medically
player connects to it by attaching two compliant electrical signals that actuate the user’s muscles.
electrodes to the flexor muscles on
the forearm as depicted in Figure 3.
The game requires the user to steer an
airplane through strong side winds
by tilting the device. The game ren-
ders the wind force by tilting the de-
vice against the user’s will. It achieves
this by stimulating muscle tissue in
the user’s arm though the electrodes,
triggering an involuntary contrac-
tion. This causes the user’s arms to
tilt sideways and thus the device to
tilt. Since the airplane is controlled
by tilt, the involuntary tilting threat-
ens to derail the airplane. To stay on
course, players counter the actuation
using the force of their other arm. As
we showed in a study, participants

XRDS • FALL 2015 • VOL.22 • NO.1 33

feature
the muscle, and the body itself has a
large supply of energy that is used to ac-
tually perform the contraction.
After we demonstrated muscle-pro-
pelled force feedback at a few confer-
ences (ACM’s Augmented Human’13,
ACM CHI’13, and IEEE World Hap-
tics’13), some of our peers were really
excited about the idea of having mo-
bile force feedback. While this type of
force feedback works well for “invisi-
ble” forces like wind or magnetism, we
started asking the next question: How
would this feel in virtual reality when a
visible object hits you? Can we create a
believable physical sensation? Can we
let the user feel the virtual world?
REALISTIC PHYSICAL
EXPERIENCES IN VR
It turns out this is harder than it
sounds. When somebody hits you in
“real” physics-based reality, you feel the
point of contact and you feel the impact
Figure 4. Impacto is designed to render the haptic sensation of hitting and being
hit. The key idea that allows the small and light impacto device to simulate a
massive hit is it decomposes the stimulus into tactile feedback (rendered by a
solenoid) and force feedback (rendered by electrical muscle stimulation).
34
We describe a new
world of VR, in which
users feel forces
from the virtual
world through their
fingers, arms, and
even the terrain
under their feet.
transmitted onto your limbs. So how
can we replicate that sensation? One
possibility is to add a robotic arm that
hits you in the “physical world” exactly
when you get hit in the “virtual world.”
Unfortunately that is (1) dangerous as it
might break our bones into pieces, (2)
expensive, and (3) requires large ma-
chinery that you likely don’t want to
store in your basement.
This is why the most conventional
way to emulate the sensation of impact
is to use vibration. There are haptic suits
out there, such as the Tesla Suit,2 which
has embedded vibrotactile and electro-
tactile stimulators touching your skin.
This is a great step forward, however
Newton would certainly not be thrilled
about it since vibration motors only
stimulate the skin’s surface and will not
make your arm move. An impact sensa-
tion is more complex than you might
think. For instance, if someone pushes
you, your skin detects pressure through
its mechanoreceptors at the location
you were touched, your muscles move
and your proprioceptive sense will
tell your brain your muscles are being
moved. So the requirements for impact
emulation pile up: We need to stimulate
the skin, we need to apply pressure, and
we need to inform the proprioceptive
sense by moving your muscles. We need
to do all of this, and keep the hardware
smaller and less dangerous than a mas-
sive robotic arm. The answer is, again,
your own muscles.
To tackle this challenge we built
impacto, a device designed to render
the haptic sensation of hitting or be-
ing hit in virtual reality as shown (see
Figure 4). The key idea that allows our
rather small device to simulate a mas-
sive hit is it decomposes the stimulus
into two distinct components: tactile
sensation and force feedback sensa-
tion. Tapping the skin with a solenoid
renders the tactile sensation; the force
of the impact is added by thrusting
the user’s arm backwards using elec-
trical muscle stimulation. In fact,
because the EMS is able to move the
arm at such a compact form factor, we
can keep the solenoid component to a
minimum size, because all it needs to
do is tap the skin. As a matter of fact,
participants of our study rated impact
simulated using impacto’s combina-
tion of solenoid hits and electrical
muscle stimulation as more realistic
than either technique in isolation.
The sensation of impact plays a key
role in many sports simulations such
as boxing, fencing, football, etc. Imag-
ine all the sport simulators, games,
and digital experiences you had so far,
2 http://www.teslasuit.com/
XRDS • FALL 2015 • VOL.22 • NO.1

Figure 5: By wearing impacto on the feet, the user experiences the impact of
kicking a virtual soccer ball (left). He hits a virtual baseball with impacto’s solenoid
on the prop and electrodes on the wrist extensor muscle (right).
now image that all those impacts be-
come real, and you feel them in your
skin and muscles.
To demonstrate how one could feel
impact in VR experiences, we imple-
mented three VR sport simulators:
boxing, soccer juggling, and baseball.
Boxing is a sport in which impact
sensation is crucial. The opponent’s av-
atar keeps its guard up and attacks pe-
riodically. Users must choose the right
moment to start a successful attack.
When the virtual character attacks, the
impact of the virtual punch is rendered
with impacto—users feel the tactile
sensation on the location where the
avatar hits them and they feel their arm
being thrust backwards due to the mass
of the avatar’s arm colliding with their
own arm. When users attack, again we
render the impact by tapping users at
the location they encountered the oppo-
nent, e.g. their fist or arm, and stimulate
their biceps causing their arm to stop
upon collision.
Users can use multiple impacto units
at the same time, as well as mounting
them on other limbs and muscles (such
as the triceps, quads, etc.). In Figure 5
we mounted a unit to the user’s calves,
which enables a different VR experi-
XRDS • FALL 2015 • VOL.22 • NO.1
ence: soccer juggling. This setup points
the solenoid component at the user’s in-
step (top of the foot) and the EMS unit to
the calf muscles. The electrical muscle
stimulation causes the foot to slightly
bend backward at the moment the ball
hits the foot. Also, by placing impacto
units on arms and legs, users experience
a Thai boxing simulator where they feel
virtual punches and kicks. As in all our
sport simulators, we use a Kinect cam-
era to track the user’s body posture and
an additional wireless accelerometer on
the solenoid to determine the foot’s tilt.
Lastly, we can actuate users, even
while they operate objects. This allows
impact sensation to happen in hand-
held props that the user manipulates
in a virtual world: Imagine hitting a
baseball with your VR bat. Figure 5
shows a prop becoming animated;
here a stick is a stand-in for a baseball
bat. We mounted the solenoid onto
the prop but the EMS unit, in contrast,
stayed with the user; here it stimulated
the wrist muscles exactly when the ball
hits the virtual bat.
CONCLUSIONS
We are finally getting rid of the mass-
less ghosts in VR and are sticking to
the consumer form factor: mobile and
wearable. This is the start of the end
of virtual experiences with no physical
sense of force, impact, and terrain eleva-
tion. Companies have been marketing
their VR products as “experiences like
you are actually there” (e.g., Google’s
360 camera rig) or “feel the sudden im-
pact of a bullet” (e.g., the Tesla Suit), but
these are ghost experiences without any
strong physical sensations. However,
by actuating the users’ own muscles we
can provide strong physical sensations
and fit all of this in a convenient mobile
package that users have come to expect
from technology. Our approach prom-
ises a future of physical VR without
wearing bulky apparatus, such as large
motion platforms or exoskeletons, an
approach that leaves the user untangled
and free to move.
References
[1] Iwata, H., Yano, H., and Fukushima, H. CirculaFloor:
A locomotion interface using circulation of movable
tiles. Computer Graphics and Applications, IEEE 25, 1
(2005), 223–230.
[2] Noma, H., Sugihara, T., and Miyasato, T. Development
of ground surface simulator for Tel-E-Merge system.
In Proc. VR ‘00. IEEE, 2000, 217–224.
[3] Strojnik, P., Kralj, A., and Ursic, I. Programmed
six- channel electrical stimulator for complex
stimulation of leg muscles during walking. IEEE
Trans. Biomedical Engineering BME-26, 2 (1979),
112–116.
[4] Sutherland, I. E. A head-mounted three dimensional
display. In Proc. AFIPS ‘68. ACM, New York, 1968,
757–764.
[5] Tamaki, E., Miyaki, T., and Rekimoto, J. Possessed
hand: Techniques for controlling human hands using
electrical muscles stimuli. In Proc. CHI ‘11. ACM, New
York, 2011, 543–552.
Biographies
Pedro Lopes is a Ph.D. student of Prof. Patrick Baudisch’s
Human Computer Interaction lab in Hasso Plattner
Institut. Lopes creates wearable interfaces that read
and write directly to the user’s body through our muscles
[aka proprioceptive interaction]. He augments humans
and their realities by using electrical muscle stimulation
to actuate human muscles as interfaces to new virtual
worlds. He also enjoys playing improvised music and is the
digital content editor of XRDS.
Alexandra Ion is a Ph.D. student of Patrick Baudisch at the
Human-Computer Interaction lab at the Hasso Plattner
Institute. She is interested in haptics and mechanical
properties in fabrication. Ion has previously worked with
Dr. Michael Haller at the University of Applied Sciences
Upper Austria, Campus Hagenberg on large interactive
whiteboards with pen input.
Robert Kovacs is a mechatronics engineer and artist.
Currently, he is a Ph.D. student at the Human-Computer
Interaction lab at Hasso Plattner Institut and works
on large-scale fabrication. He previously studied at
Free University in Berlin and the University of Novi Sad,
Serbia. Kovacs is the co-founder of Oktopod Studio,
an educational tool for mechatronics, robotics, and
automation based on open-source hardware.
© 2015 Copyright held by Owner(s)/Author(s).
Publication rights licensed to ACM.
1528-4972/15/09 $15.00
35
feature

Eccescopy:
To look,
is to see
Creating a future where dreams walk among us.

By Ken Perlin
DOI: 10.1145/2810052

I
n 2003, I visited Will Wright at Maxis, while he was still working on “The SIMS 2.”
He showed me a box—exactly the size of a computer game CD box—with nice artwork,
text, and system requirements. It was everything you’d expect, except it was labeled “SIM
Everything.” The release date was 10 years in the future. I looked more closely at the system
requirements, and they were far beyond anything that was available then. Will explained this was
always the way he and his colleagues planned new game releases. Right up front they design the
box, the artwork, the characters, and the nice little blurb that goes on the back of the box.

Turning the CD box over in my hand, I
said, “So the box is actually empty?” “No,”
he replied, “The game is already in the
box. You just can’t open it yet.”
A few years later I read Vernor
Vinge’s novel, Rainbows End. I re-
alized everything in it would be at-
tainable in the next few decades.
Computer screens will become su-
perseded by wearables and eventu-
ally by contact lenses. People will be-
come used to seeing virtual objects
superimposed onto the physical
world. Each of us will have our own
personal view of this augmented re-
ality. I call this sort of display “ec-
cescopic,” from the Latin “ecce” and
“scope.” To look is to see.
Will it be a good thing or a bad
thing when virtual objects appear
to inhabit physical space? And how
will it affect our relationship with
the world around us? Will eccescopy
take us even further away from physi-
cal reality, or will it allow us to better
join mind and body?
ECCESCOPY IN POPULAR CULTURE
Of course, the dream of merging the
real and the virtual is far from new. Vi-
sions of an eccescopic reality have a
long history in popular culture. These
visions often highlight real killer apps
for eccescopy: Enhancing the ability of
people to communicate with each other
without disrupting their sense of shared
physical space.
For example, in the 1957 film “For-
bidden Planet,” a machine developed
by the fictional alien race The Krel

36 XRDS • FALL 2015 • VOL.22 • NO.1

 allowed its user to project virtual ob-
jects into thin air merely by thinking
Image by Viktoriya / shutterstock.com
of them. Twenty years later, the first
“Star Wars” film showed something
vaguely similar: An eccescopic depic-
tion of Princess Leia in a beam of light.
Ten years after that, “Star Trek the
Next Generation” introduced the Ho-
lodeck, a completely immersive alter-
nate reality in which everything could
be eccescopic.
XRDS • FALL 2015 • VOL.22 • NO.1
The 1999 film “The Matrix” present-
ed the ultimate extension of this idea. In
the film, a simulation replaced the phys-
ical world. Life was lived entirely within
cyberspace. In such a constructed world,
nothing is real, yet anything becomes
possible. People can have superpowers,
and objects can change form or even
disappear instantaneously.
Yet a direct-brain interface like
the one in “The Matrix” turns out to
be quite difficult to create. The prob-
lem isn’t the physical connection of
electrode arrays to brains. That’s dif-
ficult, but not impossible. In the next
20 years, direct brain/computer inter-
face technology is likely to advance far
beyond what we can do today. Science
has already advanced considerably
in this direction. No, the basic prob-
lem is your perception of reality is
already a construct—one maintained
37
feature
by your brain. For example, you don’t
literally see things the way a camera
does. At any moment in time, your
eyes perceive only a tiny window into
Distinguished reality from which your brain then
constructs a plausible model. It is re-
Speakers Program ally this constructed model that you
“see.” We don’t know very much about
how this construction process works,
http://dsp.acm.org which means we can’t hack into it
with any effectiveness. And even if
we could, a direct brain interface like
the one in “The Matrix” would need
to replace the considerable amount
of image processing done by our optic
nerve. We might also need to simulate
the saccades and other movements
made by our eyeballs as our brain con-
Students and faculty tinually refocuses its attention.
The most reliable way to transmit
can take advantage of visual information to the brain is in
ACM’s Distinguished the form of visible light. Why invent
something new, when you already
Speakers Program have something as powerful as the
human retina?
to invite renowned
ECCESCOPIC PROTOTYPES
thought leaders in IN THE LAB
Around 2002, our research group at
academia, industry
NYU developed an early prototype of
and government Holodust—a kind of eccescopic display,
which created virtual images of 3-D ob-
to deliver compelling jects directly onto a cloud of dust. Since
you can never know the exact position
and insightful talks of each particle in a cloud of dust, our
scheme used two scanning lasers: An
on the most important infrared laser to sweep through the
topics in computing cloud looking for dust particles, as well
as a visible laser along the same opti-
and IT today. cal path that could flash on command.
This dual laser approach gives the abili-
ACM covers the cost
of transportation
for the speaker
to travel to your event. People will become
used to seeing
virtual objects
superimposed
onto the physical
world. Each of us
will have our own
personal view of this
augmented reality.
38
ty to “paint” a 3-D shape onto individual
dust particles floating in the air.
Whereas Holodust creates a glow-
ing image of an object that seemingly
floats in thin air, the 360 degree Light
Field Display at USC is more ecce-
scopic, because it allows the shading
of a virtual object to change as it is
seen from different directions. Unfor-
tunately, that technology relies on a
slanted metal mirror rotating at a very
high speed. If you tried to touch it you
would most likely destroy both the dis-
play and your hand.
THE FUTURE EVOLUTION OF
ECCESCOPIC TECHNOLOGY
Charles Darwin observed every geno-
type requires a viable phenotype. That
is, no mutation can survive unless it
can produce viable offspring. Technol-
ogy is like biological evolution in that
it can’t just magically jump far ahead.
Every step along the path to innova-
tion needs to be useful, otherwise it
will die in the marketplace before en-
abling the next step.
For example, I don’t think we will
first achieve widespread eccescopy
through surgery. Yes, technically we
could give everyone an artificial lens
implant, but until there is a good rea-
son for such an intervention, people
won’t do it. It’s not even that invasive
eye surgery is so exotic. You probably
know many people who have had cata-
racts and are walking around today
with an acrylic lens implant or two.
You don’t know who they are, because
it’s not something people generally
talk about. The operation itself is
relatively simple and safe, requiring
only local anaesthetic and no stay in a
hospital. But it’s only done because it
avoids blindness. A very different val-
ue proposition than, say, implanting
an artificial lens so you can do Google
searches within your eyeball. Most
people won’t opt for invasive surgery
unless it helps them to be more “nor-
mal,” however that word is currently
defined in their culture.
Not too long ago, putting an elec-
tronic auditory enhancement device in
your ear was something you did surrep-
titiously. A hearing aid was something
you tried to hide—ideally you didn’t
want anyone to know you needed one.
Recently there has been a fascinating
XRDS • FALL 2015 • VOL.22 • NO.1
trend in the other direction. A hearing
device on one’s ear may now be per-
ceived as a source of empowerment.
In the case of bluetooth hands-free
cellphones, people don’t try to hide
these devices, they try to show them
off. I think the key distinction here is
between “I am trying to fix a problem”
and “I am giving myself a superpower.”
The former makes you socially vulner-
able, whereas the latter makes you so-
cially powerful.
Yet the thing that strikes me about
both of these set-ups is they interfere
with eye contact. In both cases, you
cannot look directly into the pupil of
the person wearing the head-mounted
display. The pupil is hidden by the dis-
play mechanism, which is literally in
the way. Something tells me this is a
show stopper for widespread adop-
tion. When looking at another person
face-to-face, most people want to see
their eyes. I suspect retaining the abil-
ity to see other peoples’ eyes will be
necessary for widespread acceptance
of an eccescopic future.
LIFE IN AN ECCESCOPIC FUTURE
How different would things look in
an eccescopic world? As different as
books on paper and the Web. Instead
of a world of computer screens (even
the little screens on smartphones),
imagine a world where information
is truly in the air around you. First,
eccescopic interfaces will allow us to
interact with other people directly,
without any screens getting in the
way. Second, they will allow us to
“paint” and otherwise annotate the
physical world around us in ways that
are visible only to some people and
not to others.
Let’s take the first point. Suppose
we are having a conversation about
American history, and a question
comes up, such as: “What was the
name of Thomas Jefferson’s wife?”
In today’s world, at least one of us
would need to break eye contact with
the other to type a query onto a com-
puter screen. Meanwhile, the other
person is probably also visually dis-
engaging—since it is impossible to
maintain eye contact with a person
who is staring at the screen of their
phone. But if we knew the entire
search transaction—both query and
XRDS • FALL 2015 • VOL.22 • NO.1
Why invent
something new,
when you already
have something
as powerful as the
human retina?
response—was accessible wherever
we already happen to be looking, then
there would be no need to break eye
contact. We would develop ways to
query a computer that do not require
loss of eye contact. In an eccescopic
world, “eyes free” methods of entering
text might become not only socially
acceptable, but socially necessary.
That leads to the question of priva-
cy. One objection to everyone having
their own eccescopic display would
be the loss of personal privacy within
the public sphere. Wherever you go on
a city street, somebody will be sure to
record you, and those recordings can
be pieced together to track your every
movement. Yet there are times when
even this can be a good thing.
For example, in 2003, like many
New Yorkers, I attended a protest of
our then president’s decision to go
to war against Iraq. New York City
police routed the crowd of protesters
in a very odd way. We were shunted
off into various side streets, eventu-
ally quite a few of us found ourselves
penned in when policemen on horse-
back charged into the crowd. For the
unfortunate people in front, there was
no way to avoid the kicking hooves of
the horses. The next day, national
newspapers printed the NYPD’s de-
scription of the incident: Hostile pro-
testers attacked police horses, and the
police had done their best to protect
the helpless horses from the danger-
ous and unruly mob.
That was 12 years ago. Today the
police couldn’t have gotten away with
a stunt like this. Too many people in
the crowd would be carrying smart-
phones, each with the ability to in-
stantly upload images of what really
happened before the police had a
chance to take the phones away. In an
eccescopic world there would still be
private spheres, and we would do well
to protect them. But one could argue
a democracy best flourishes when its
shared public spaces are exposed to
the light, not when they are shadowed
in darkness and fear.
Then again, the more we build
our interaction technologies into our
own bodies, the more vulnerable we
become to perception hacking. Once
computer technology is used for per-
ception of the world around us, then
our senses become vulnerable to be-
ing hacked. Our eyes might see things
that aren’t there, our ears might hear
things that don’t exist, or our fingers
might touch objects that are not real.
An entirely new field might arise;
a field of security that protects you
from having your augmented reality
replaced by a chimera.
CONCLUSION
Of course you never really know how
the future will unfold. As the great
user interface imagineer J. K. Rowling
once said: “Predicting the future is a
very difficult business indeed.”
Sometimes innovations simply re-
quire the proper moment to take root.
In 1965, Western Electric ran a maga-
zine advert for a hybrid between a tele-
vision set and a telephone. We now
know, half a century later, such a prod-
uct never took the world by storm. Yet
all those years ago that team of West-
ern Electric designers had hit upon
an essential grain of truth: Eventu-
ally the television and the telephone
would converge in the consumer mar-
ketplace. It just wouldn’t happen un-
til you could carry the technology in
your pocket.
The details may not all be clear yet,
but eventually we will be living in an
eccescopic world, and we need to start
designing for that world.
The game is already in the box.
Biography
Ken Perlin, a professor in the Department of Computer
Science at NYU, is the director of the Games for Learning
Institute, a member of NYU/MAGNET, and founding
director of the Media Research Laboratory. He is a
recipient of the SIGGRAPH Achievement Award and an
Academy Award for Technical Achievement. His computer
graphics work is featured in the Whitney Museum of
American Art.
© 2015 ACM 1528-4972/15/09 $15.00
39
feature

Lost in the Rift:

Engaging with
mixed reality
Virtual reality users are torn between the real and virtual worlds.
Determining how, and when, to show elements of reality in a virtual
view is key to providing usable VR experiences.

By Daniel Boland and Mark McGill

DOI: 10.1145/2810046

A
new virtual reality (VR) user reaches out and feels around in the dark, searching for the
controller yet unable to see where it is. Elsewhere, immersed in their own VR experience,
another user is oblivious to the other person who walks into the room; they are standing
side by side, but unaware of the other’s presence. Mixed reality approaches that allow
these real-world elements to be blended into VR can break the user’s sense of immersion in
their virtual experience. Our research on engagement-dependent mixed reality has solved this
problem, by selectively blending real elements into the virtual world as users wish to engage with
them, which creates a seamless interaction across the continuum of real and virtual reality.
The Oculus Rift’s Kickstarter cam- a number of sizeable problems in try- reality to be mapped to VR. However,
paign triggered a resurgence of in- ing to deliver a VR experience that is perhaps the most pressing concern
terest in VR head-mounted displays usable in the real “consumer” world. currently facing the VR community is
(HMDs). Advances in small form factor Some of these problems are funda- not related to the rendering or display
displays (e.g., the high refresh rate, low mentally technological. For example, of VR experiences, but instead how we
persistence, and high definition pan- the fidelity of the VR experiences in- interact with VR experiences and make
els typically used in mobile devices) creases—as our capability to render VR HMD usage compatible with real-
demonstrated high fidelity VR HMDs and display them does—through world environments.
were now not only technologically fea- more powerful GPUs, better displays,
sible, but a viable and affordable con- and wider field-of-view lenses. Simu- THE GLOVES COME OFF
sumer reality. What followed has seen lator sickness, another major prob- The problem immediately becomes
Image by Creativity103

the likes of Samsung (Gear VR), Sony
(Morpheus), HTC/Valve (Vive), Oculus/
Facebook (DK1/2, CV1), and Google
(Cardboard) battling to be the leader in
this VR renaissance. But there remains
lem for users, is being addressed with
additional sensing such as external
tracking cameras combined with
headset-based inertial motion sens-
ing, allowing for every movement in
apparent when you first wear a VR
HMD. You find yourself in a virtual
world with no perception of reality.
How do you interact with this new vir-
tual world you inhabit? The go-to in-

40 XRDS • FALL 2015 • VOL.22 • NO.1

Photo Credit TK

XRDS • FALL 2015 • VOL.22 • NO.1 41

feature
Figure 1: Milgram et al.’s reality-virtuality continuum. Here we can see augmented
reality (pictured the Microsoft Hololens) and augmented virtuality (pictured the
Oculus Rift DK2) contained within the scope of mixed reality, with examples given
of what the user might perceive at different points along the continuum.
Reality-Virtuality (RV) Continuum
Augmented
Reality (AR)
Mixed Reality (MR)
put mechanisms (game controllers,
mice, and keyboards) immediately be-
come more difficult to use when they
can’t be seen. If you want proof of this,
close your eyes, stand up, spin around
a bit on the spot, and then try to type
something on your keyboard. If you
survive unscathed, you might remark
that it is anything but easy. There are
styles of interaction that don’t need
peripherals, such as gestural interac-
tion. But even this can be hazardous
when you can’t see the real obstacles
that surround you in your household,
such as plant pots, walls, or beloved
family pets. One way of solving this
problem is to develop highly targeted
interactions for specific VR contexts,
like the flight rig in the Birdly VR
Simulator.1 General consumer use of
VR will, however, require generic and
adaptable interaction techniques,
suitable for use in home and office
spaces without significant deploy-
ment or installation.
As such, considerable efforts have
gone into recreating real-world inter-
actions in VR. Hand tracking (e.g., us-
ing the leap motion VR) allows users
to interact with 3-D VR elements in a
natural manner, using their proprio-
ception. Being able to reach out in
1 http://somniacs.co/
42
Gloveone2 uses vibrotactile actuators
embedded in a glove, allowing a lim-
ited perception of virtual objects. In
contrast, the Dexta Robotics Dexmo3
uses mechanical actuators to create a
force-feedback exoskeleton glove, al-
lowing virtual objects to be grabbed
and released. Mid-air feedback also
shows promise, using ultrasound [1]
or air vortices [2]. Given the relative
lack of maturity of these technolo-
gies, interaction with real tangibles in
some form is likely to be required for a
variety of experiences.
With respect to tangible objects,
handheld motion controllers (e.g.,
Oculus Touch, Playstation Move, and
SteamVR) are the most promising cur-
rent approach. They typically provide
Augmented the capability for high-bandwidth in-
Virtuality (VR) put and “hand presence,” meaning
the positional tracking of hands and/
or fingers, at the cost of being able to
naturally manipulate virtual objects
through touch. Other approaches aim
reality and touch objects virtually al- to retain the capability to pick up and
lows common interactions without the manipulate virtual objects. Substitu-
go-between of a peripheral. However tional reality appropriates real-world
the word “touch” is problematic here. tangible objects to represent virtual
Hand-tracking solutions often lack objects, for example turning a torch
haptic feedback when touching a VR into the hilt of a VR lightsaber [3].
object, which pulls users out of the vir- Apart from these, other modalities for
tual experience, making precise ma- interaction are also available. Voice,
nipulation difficult. gaze, and motion tracking all allow
This lack of haptic feedback can some semblance of mapping the ac-
be compensated for in numerous tions of the real world to the virtual.
ways. For example, Neurodigital’s Allowing the user to move in VR can
be achieved using omnidirectional
treadmills, external sensing with re-
directed walking, substitutional reali-
ty on a larger scale, or even room-scale
VR [4].
However, many of the discussed in-
In shutting off teraction techniques share a common
reality, we can problem; the interaction starts in real-
ity, yet the VR user has limited percep-
become extremely tion of that reality.
immersed in LOST IN THE RIFT
VR without Being unable to perceive reality when
distraction. in VR has serious consequences re-
garding usability. Take handheld
But the more controllers; if we put the control-
we shut off reality, ler down, we need to be able to see
where it is in order to pick it back
the more difficult up again. For hand-tracking interac-
certain interactions 2 http://www.gloveonevr.com/
become. 3 http://www.dextarobotics.com/products/Dexmo
XRDS • FALL 2015 • VOL.22 • NO.1
tions, we need to be aware of what
real-world objects might be interact-
There will be HMDs In order to augment virtuality with
reality and achieve this seamless in-
ed with. Waving one’s hands in the in the future tegration of real and virtual, we first
air may make for an immersive ex-
perience, but knocking over a cup of
that will support need a VR HMD that is able to sense
elements of reality. Such sensing
water would make for a rather moist both AR and can be room-wide (e.g., Oculus Con-
one. For movement through real-
world spaces, we need a perception
VR/augmented stellation, HTC/Valve Lighthouse,
or Microsoft Kinect) or mounted
of where we can move. Walking into virtuality modes on the headset itself (e.g., Leap Mo-
a wall impacts your immersion in VR
as well as your head. The problems
and will have the tion VR, Gear VR camera, or Google
Project Tango). These sensing tech-
don’t stop there. There are a variety of ability to transition niques typically support depth cap-
scenarios where a user might want to
perform a necessary interaction with
between any point on ture, person identification, object/
hand tracking, and biometrics across
reality (e.g., typing on a keyboard or the RV continuum. a wide field-of-view. Using multiple
eating some popcorn), or might need sensors, we can start to accurately
to be aware of reality (e.g., your com- map out the room. For example, we
puter spontaneously combusting). can track the VR user (e.g., position
Not only can this make for awkward and orientation, compensating for
situations, it can also leave users feel- user can see nothing but VR), with drifts in on-board inertial sensing),
ing vulnerable with little control over all the points in-between comprising the social context of the room (e.g.,
their personal space. of mixed reality—a blend of real and who is there, what are they attend-
In a survey of 108 existing VR HMD virtual. Depending on how this mix- ing to, and where they are), and the
users, we confirmed there were sig- ing is skewed, it might be interpreted physical context of the room (e.g., ob-
nificant problems regarding interact- as augmented reality (AR), a view of jects, obstructions, and walls). Our
ing with peripherals and communi- reality that is augmented with virtual work was predicated on these sens-
cating the social context of the room. aspects, or augmented virtuality (AV), ing capabilities being widely avail-
Users reported being ineffective at a view of VR that is augmented with able, and was concerned with how we
interacting with real-world objects aspects of reality. Early work in this might use this sensing to enable the
and peripherals. This was unsurpris- area by Metzger [6] proposed a seam- VR HMD user to interact with, and be
ing, but confirmatory, given that users less integration of real-world human aware of, reality. To do this, we had
were unable to see these objects and interfaces with the virtual world, and to answer two questions: How should
peripherals. For social contexts, we our work explores how to realize this. we incorporate reality and when?
found users were unaware of the pres-
ence and proximity of others in the Figure 2. Feedback from real objects can be used to augment virtuality, providing
room. In both cases, users agreed the a low-level control loop that can support high bandwidth interaction, such as
VR HMD should have in-built support typing. The more the user engages with reality, the more real feedback is mixed
for incorporating reality. with virtuality.
In essence, the isolation a VR HMD
provides is both its greatest strength
and most significant weakness. In
shutting off reality, we can become
extremely immersed in virtual reality
without distraction. But the more we
shut off reality, the more difficult cer- Engagement AV
tain interactions become. The problem
space is in deciding when, and how, to
breach that isolation and communi-
cate information regarding real-world
context and objects.
The extent to which we breach this Inferred Engagement
isolation is defined by where we are
on the “reality-virtuality (RV) contin-
uum” [5], also known as the “mixed
reality continuum” (see Figure 1).
The RV continuum is a scale ranging
between completely real (meaning Reality Virtual Reality
the user can see nothing but reality)
and completely virtual (meaning the

XRDS • FALL 2015 • VOL.22 • NO.1 43

feature

scenes would be spoiled by a ghostly,

Figure 3. A VR HMD wearer’s view of engagement-dependent mixed reality.
Here, the user has reached out to interact with an object; in turn, the system has
inferred this engagement and temporarily brought the available objects to be
interacted with into the VR view.
ENGAGEMENT-DEPENDENT
MIXED REALITY
There are clearly scenarios where users
would want to blend real elements into
their virtual experience. The question
then is how to find the appropriate bal-
ance between real and virtual—where
does the user want to be on the RV
continuum? Our work explores how to
intelligently infer which elements of
reality or virtuality the user is engag-
ing with, and adapts the mixed-reality
blending accordingly (for more details
beyond this article, see McGill et. al.
[7] and the associated video4). This
4 www.youtube.com/watch?v=SHdfxuh7_GY
engagement is considered to be the
user’s attention to an element, such as
by a sustained gaze, or the user’s con-
trol of an object reaching for a cup or
typing on a keyboard. This “engage-
ment-dependent” mixed reality allows
a number of pressing VR usability is-
sues to be addressed.
Let us again look at the case of the
keyboard or controller in detail. We
know finding and interacting with
these peripherals unsighted is dif-
ficult. We could incorporate these
objects into mixed reality; however
having such real elements perma-
nently visible in VR would impact im-
mersion—even the most immersive
hovering keyboard. By only including
an object as the user engages with it
(for example by reaching out for it, or
by gaze), the engagement-dependent
mixed-reality approach enables a
high bandwidth interaction such as
keyboard input while allowing users
to otherwise remain immersed in
their VR experience (see Figure 2).
As we infer the user’s engagement,
we can also blend elements where there
is a high likelihood of engagement.
Obvious examples include necessary
objects, such as a cup that is frequently
drank from (see Figure 3), or hazards
such as a wall in the path of the user (as
is the case with the feedback provided
by the HTC Vive headset).
This could also apply in virtuality,
for example incorporating an avatar
of a friend into an otherwise real-
world view. This is particularly useful
for blending people into the VR view
as they enter the room. It can be as-
sumed a VR user wants to be aware
of people entering their personal
space, and so there is an implicit level
of engagement with these people al-
ways blended in slightly as “ghosts.”
As the VR user interacts with others,
they can be blended in more fully, oc-
cluding part of virtuality (see Figure
4). This example, with its degrees of
blending, demonstrates how engage-
ment-dependent mixed reality spans
the RV continuum dynamically.
THE CONVERGENCE
OF VR AND AR HMDS
While much of our work is based
around the concept of a VR HMD
combined with sensors for capturing

Figure 4. When a person enters the same physical space as the VR user, they are faded into the virtual view. When the user
wishes to engage with them, they would become fully opaque. Left: reality; middle: low engagement; right: high engagement.

44 XRDS • FALL 2015 • VOL.22 • NO.1


and incorporating reality, there is an-
other class of mixed-reality displays
currently in use. AR headsets typical-
ly rely on a see-through display where
virtual content can be rendered on
top of the user’s view of reality (as op-
posed to reality rendered virtually).
They occupy the AR part of the RV
continuum, with Microsoft Hololens
being a recent example of the state of
the art.
Because AR headsets have the abil-
ity to transition between a range of
points on the RV continuum, we can
apply our engagement-dependent
mixed-reality approach. This time
the blending would govern when and
how much virtual content is rendered
on top of reality. For example, a lin-
gering gaze on a tourism landmark
might be a cue to incorporate addi-
tional relevant information.
The underlying technologies used
in AR headsets suggest in the future
there will be HMDs that support both
augmented reality and VR/augmented
virtuality modes, and will have the
ability to transition between any point
on the RV continuum. For example, we
can envisage Hololens–like headsets
where there is an additional display
layer that can selectively occlude real-
ity, such that the rendering of virtual
content can be done on top of reality
or on top of a blank, dark canvas—
much like how 3-D TV active-shutter
glasses work today. Such headsets
would be at once empowering and iso-
lating in equal measure, supporting
instantaneous transitions between
virtual spaces and augmented reality
spaces. But how would we make these
transitions seamless? What elements
of reality should get carried back and
forth? And how do we provide users
with a stable mental model for inter-
acting with reality as their immersion
into VR increases? There would be
a need for consistent behaviors and
rule sets regarding transitions across
the RV continuum, be they pertaining
to interactions with reality, aware-
ness of social contexts, and so on. We
suggest an engagement-dependent
mixed-reality model could underpin
such interactions, and such a model is
a necessity if interactions with reality
are to be as seamless and effortless as
users might expect.
XRDS • FALL 2015 • VOL.22 • NO.1
CACM_TACCESS_one-third_page_vertical:Lay
CONCLUSIONS: CAN I SEE YOU NOW?
What we’ve described is a means to
balance the needs of VR users who are
torn between two worlds: reality and
virtuality. Users want to interact with
a virtual world, yet have to route said
interaction via the real world. This bal-
ance between immersion in virtuality ACM
and awareness of reality is key to pro-
viding usable, yet immersive, VR head- Transactions on
sets that can be used in homes, and
offices, as well as other shared, social,
complex spaces without impediment.
Accessible
Our work provides a foundation for
both AR and VR displays, with transi-
Computing
tions in mixed reality governed by the
user’s engagement. This engagement-
dependent approach to mixed reality
provides experiences that better adapt
to user needs, and break down the
isolating digital and physical barriers
raised by head-mounted displays.
References
[1] Wilson, G., Carter, C., Subramanian, S., and Brewster,
S. Perception of ultrasonic haptic feedback on
the hand: localisation and apparent motion. In
Proceedings of the 32nd annual ACM conference on
Human factors in computing systems (CHI ‘14). ACM,
New York, 2014,1133-1142.
[2] Sodhi, R., Poupyrev, I., Glisson, M. and Israr., A.
AIREAL: interactive tactile experiences in free air.
ACM Trans. Graph. 32, 4 (July 2013).
[3] Simeone,A., Velloso, E., and Gellersen, H..
Substitutional Reality: Using the Physical
Environment to Design Virtual Reality Experiences.
In Proceedings of the 33rd Annual ACM Conference
on Human Factors in Computing Systems (CHI ‘15).
ACM, New York, 2015, 3307-3316.
◆ ◆ ◆ ◆ ◆
[4] Machkovech, S. SteamVR: The room-scale VR
world that feels like an “IMAX in your house.” June This quarterly publication is a
13, 2015. Ars Technica UK. http://arstechnica.
co.uk/?p=19673. quarterly journal that publishes
[5] Milgram, P., and Fumio, K. A taxonomy of mixed
reality visual displays. IEICE TRANSACTIONS on
refereed articles addressing issues
Information and Systems 77.12 (1994), 1321-1329.
of computing as it impacts the
[6] Metzger, P.J. Adding reality to the virtual. In
lives of people with disabilities.
Proceedings of the 1993 IEEE Virtual Reality
Annual International Symposium (VRAIS ‘93). IEEE
Computer Society, Washington, DC, USA, 1993, 7-13. The journal will be of particular
[7] McGill, M., Boland, D., Murray-Smith, R., and interest to SIGACCESS members
Brewster, S. A Dose of Reality: Overcoming Usability
Challenges in VR Head-Mounted Displays. In and delegates to its affiliated
Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems (CHI ‘15). conference (i.e., ASSETS), as well
ACM, New York, 2015, 2143-2152.
as other international accessibility
Biographies
conferences.
Daniel Boland applies data science and machine learning ◆ ◆ ◆ ◆ ◆
approaches to HCI. This work was informed by his Ph.D.
thesis on adapting to user engagement. Learn more at www.acm.org/taccess
www.dannyboland.com.
Mark McGill is a Ph.D. student at the University of Glasgow
www.acm.org/subscribe
in the Multimodal Interaction Group, led by Prof. Stephen
Brewster. Outside of VR, his research interests are
in multi-user and multi-view TV. Learn more at www.
markmcgill.co.uk.
© 2015 Copyright held by Owner(s)/Author(s).
Publication rights licensed to ACM.
1528-4972/15/09 $15.00
45
feature

Managing
Cybersickness
in Virtual Reality
If the physical side effects associated with virtual reality are not
managed, the widespread adoption of VR may come to a halt.

By Lisa Rebenitsch
DOI: 10.1145/2810054

S
creenwriters, science fiction authors, you, and I… we all have dreamed of virtual
reality (VR) for decades. Hollywood whet our appetite for VR with films such as “The
Matrix,” “Total Recall,” and “Brainstorm.” As early as 1982, when “Tron” introduced
VR to the masses, the technology seemed well within reach for us all. VR was
present on the small screen as well as in the form of the Holodeck on “Star Trek: The Next
Generation.” By the early 1990s consumers could finally purchase VR gaming devices; there
was the Nintendo Virtual Boy, the Sega VR console, and the Sony Glasstron. The addition of
head tracking enabled dynamic VR applications, while 3-D screens heralded a new interface
mirroring the real world. Beyond entertainment, VR has also been used for military
and aviation training; its versatility lator sickness or visually induced mo- up to the software. Therefore, devel-
has been useful in design, data visual- tion sickness (VIMS), has been well opers simply needed to wait and even-
ization, and more. But just as VR began documented since the 1960–1970s. tually hardware improvements would
to peak, it nearly became extinct. These motion sickness–like symptoms eliminate cybersickness. There was
There are multiple reasons for this, were first reported while using military one major exception: In 1995, Mon-
but one of the most significant is the and aviation simulators. The term cy- Williams, Wann, and Rushton pre-
technology over-promised and under- bersickness is associated with full vir- dicted improving the hardware would
delivered. Military training simulators tual environments, while VIMS is used actually increase cybersickness [1]. To
were extremely expensive to operate. more broadly and refers to any situation better understand how cybersickness
Image by Durantelallera / shutterstock.com

Even as a commercial product, de-
vices proved too costly for consumers.
Despite implementing the most basic
technology, a monochromatic display,
the Nintendo Virtual Boy was a failure.
There was also the issue of comfort.
The Glasstron and Sega headsets were
heavy. Worst of all, using VR devices
made some people ill.
Cybersickness, also known as simu-
that involves looking at imagery, real or
computer generated. Cybersickness is
not the same thing as motion sickness.
They differ with respect to their symp-
toms: Motion sickness is identified pri-
marily by nausea, while cybersickness
involves disorientation and dizziness
usually preceded by nausea.
The general assumption around
1990 was the hardware would catch
works requires delving into human
physiology and how we adjust to vi-
sual imagery.
WHAT IS CYBERSICKNESS?
There are a few common theories ex-
plaining cybersickness, with the most
prevalent being sensory mismatch.
This theory states if there is a mis-
match between the sensory input to

46 XRDS • FALL 2015 • VOL.22 • NO.1

Photo Credit TK

XRDS • FALL 2015 • VOL.22 • NO.1 47

feature

the inner ear and the eyes, a person will

feel motion sickness. An example of
this is when a person is viewing a mov-
ie and sees something moving while
the viewer remains stationary. There
are two other theories that are gaining
favor. The first is the postural insta-
bility theory, which states if someone
cannot maintain a stable posture with
sensory input, he or she will be ill. The
second theory, the rest frame theory,
states the less anticipated the sensory
input, the more ill someone becomes
with the less-anticipated movement.
In other words, you expect a stationary
viewpoint if you are sitting in a station-
ary chair, but a movie gives you a mov-
ing viewpoint. For example, the driver
in a car anticipates the direction of
movement better than the passenger
who does not have control, and there-
fore, will have fewer symptoms.
Several dozen sources for cyber-
sickness related to these theories have
Figure 1. An evolution of VR technology from the 1990s to the present. VR
headsets on the left are from the 1990s, and the headsets on the right are their
modern equivalent.
1990s
Sony Glasstron
(a)
One of the best
things developers
can do is to include
optional settings
that decrease
the likelihood of
cybersickness.
been posed over the last few decades.
Broadly, these sources can be catego-
rized into hardware, software, and the
individual user. Hardware includes
components such as resolution, lag,
size of the screen, and more. Software
refers to the application, speed of
movement, and general appearance.
Current
Sony Personal 3D Viewer
(b)
Individual user factors include age,
gender, and frequency of use of VR.
There is still a great deal of uncer-
tainty concerning what factors affect
cybersickness, and what their indi-
vidual ramifications are. Causative
effects are missing, and therefore,
good estimation models are also lack-
ing. Since there is no good estimation
model of cybersickness, if a developer
wants to know if an application and
system could cause cybersickness,
they need to actually build it. How-
ever the developer would need to find
a varied group of people to test it and
then ask how sick they become. This
is a risky venture; money would be in-
vested into a system that may or may
not be operable.
So why are causative effects still
unclear when cybersickness has been
studied for decades? There are a few
reasons: experimental results are wide
spread, results from different labs can
be hard to compare, and cybersickness
is difficult to measure and quantify.
The simulator sickness questionnaire
used by about 75 percent of the field,
is not ideal [2]. Like most surveys it is
subjective. It is also overly sensitive,
nonlinear, and very time consuming.
Unfortunately, it is currently the best
comparison tool we have.
A relatively recent alternative to
questionnaires is measuring postural
instability (essentially how much some-
one “wobbles”). In general, as a person
becomes more “wobbly,” they will feel
more ill. Postural instability has shown
correlation in multiple experiments,
but the results cannot be applied to a
general setting. The correlation values
vary greatly from lab to lab, and from
experiment to experiment. Postural
monitoring also usually requires the
subject to maintain special postures
and attach sensors to their person.
Asking someone to wear a sticky sen-
sor and assume the Tandem-Romberg
stance (close their eyes, place the feet
heel-toe-heel-toe, and cross their arms)
for 20 seconds every four minutes, is
not the most reasonable request.

VR IS BACK, BUT SO
IS CYBERSICKNESS
Nintendo Virtual Boy Oculus Rift
Recently, there have been significant
(a) (b)
changes to VR hardware: The costs
have come down and the hardware

48 XRDS • FALL 2015 • VOL.22 • NO.1

has started to catch up. The Oculus
Rift has replaced the Virtual Boy, and Table 1. Components of cybersickness are shown below. Options that normally
the Sony Personal 3D Viewer HMZ-T1 lessen cybersickness are on the left, while options that normally increase
has replaced the Glasstron (see Fig- cybersickness are on the right.
ure 1). New multidirectional walking
platforms like Cyberith Virtulizer and
Virtuix Omni will soon be available.
People are excited about VR again and
there is renewed interest in finding
new ways to use VR. But there is still Screen size
much to explore on the human side of
VR. Are the systems comfortable? Are
they easy to navigate? Are they easy
to add to your home? The answers to
Small Viewing Area Large Viewing Area
these questions may be a resound-
(a) (a)
ing yes. Yet, one element may severely
harm the progress of VR and cause it
die out again: cybersickness.
Part of my research on cybersick- Speed
ness involves querying users. When
discussing the phenomenon, 90 Walk Run
percent of audiences have either ex- (c) (d)
perienced illness or know someone
who has become sick while watch-
ing a movie. There is a website, www.
moviehurl.com, entirely devoted to
rating movies for their nausea induc- Textures
ing effects. There are the classical
examples, “The Blair Witch Project”
and the “Bourne” movie series, as well Realistic Textures Generic Repeating Textures
as newer films, “The Hunger Games” (e) (f)
and “Earth to Echo.” Although movies
and VR have several similarities, VR
has a higher incidence and severity
of cybersickness. Researchers study-
ing the problem usually report a 40-60 Time
percent susceptibility rate and claim
more than 90 percent of the popula- Short Duration Long Duration
tion have the potential to be affected. (g) (h)
Personally, I have exited Oculus Rift
demonstrations overhearing people
saying they felt ill, even though they
did not consider themselves to be
prone to motion sickness. This prob- Play position
lem is not going away. Instead, Mon-
Williams, Wann, and Rushton’s pre- Sit Stand
diction is proving true. More people (i) (j)
are using VR and more people are be-
coming ill.
Some marketers have insisted cy-
bersickness will be gone in a few years
due to hardware advancements. Yet it Frequency
has been a problem for 60 years; it is
not going to be solved in a few more.
This misguided belief is based on the
fact that the original hardware was Use VR Frequently Use VR Infrequently
nowhere near sufficient. One of the (k) (l)
problems with this assumption is VR
systems are composed of more than

XRDS • FALL 2015 • VOL.22 • NO.1 49

feature
Figure 2. Multidirectional walking platforms are depicted Virtuix Omni (left) and
the Cyberith Virtualizer (right). Both place a user on a slippery floor that monitors
the direction and speed of steps for use in VR. Slipping is prevented by a brace at
the hips and waist.
(a)
just hardware. What does the applica-
tion look like? Can you see to the hori-
zon? Can you turn sideways? How fast
are you moving? Then, there is the per-
son. Some users are simply more sus-
ceptible to cybersickness. It does not
matter if you put these people on a real
roller coaster or a virtual one. Either
way, they will get sick.
So why is there such a focus on
hardware? The biggest reason is it is
an ideal solution. Solve the hardware
problem, and everyone’s problem is
solved. Unfortunately, we really do
not know how much of cybersickness
is strictly due to hardware. Develop-
ers are focusing intensely on lag and
tracking accuracy, but most studies
on lag looked at delays of 100-200ms.
VR equipment at 10HZ would not be
considered sufficient in the first place.
There is a reason TV screens and mon-
itors are 60HZ or higher. However, lag
does need to be improved. Low frame
rates can cause a flickering effect—
flashing lights are used to diagnose
migraines and for other brain tests.
Jitter is also bad. Just ask anyone who
cannot watch a shaky camera movie.
The goal in VR is to feel as though
there is no difference between the real
and virtual world. Tracking and lag are
only two parts of a whole. Even with
perfect tracking, there will be lag as
the computer must wait for the track-
ing information to be sent and then de-
termine how to use this information in
the application.
50
(b)
So what about the software and
individual side? Why has there been
less focus on these categories? To
start, these are a lot harder to define.
You also cannot place those types of
restrictions on software because any
guideline posed will eventually be bro-
ken by an application. A user will be
a user. You cannot control them. Any
monitoring measure that delays start-
ing or affects comfort is not going to be
well tolerated.
WHAT AFFECTS CYBERSICKNESS?
The difficulty in identifying cybersick-
ness begs the question, what can be
done to prevent it? Much of what causes
cybersickness, how to detect it, and how
to prevent it remains unclear. Many
components of VR such as level of real-
ism, zoom, display type, demographics,
etc. have experiments both supporting
and denying their effects. However,
we do know a few components that
have had reliable results. These are ex-
plained and summarized in Table 1.
Duration. One of the best-reported
components that correlates to cyber-
sickness is duration. It is important
to bear in mind the warning labels on
consoles may only follow some experi-
mental results. Nintendo1 suggests a
break every 30 minutes when using
3-D, but, on average, you can expect to
start noticing cybersickness effects in
1 https://www.nintendo.com/consumer/
systems/3ds/en_la/health_safety.jsp
as little as 15 minutes. If you anticipate
there will be an issue with cybersick-
ness, break up game play into 15-min-
ute active segments buffered by mel-
low inactive segments.
Field of view. Also known as the rel-
ative size of a screen at a set distance,
field of view also has a significant
effect on cybersickness. Doubling
the field of view will approximately
double the frequency and severity of
cybersickness. This is one of the easi-
est settings developers can include. If
someone says they are sensitive to cy-
bersickness, shrink the field of view by
displaying the application in only half
the screen. However, field of view is
regularly affected by the display itself.
Head-mounted displays like Oculus
Rift often have smaller effective fields
of view than large TVs. If the field of
view and interaction are identical
across displays, little to no effect can
be seen because of the display used.
However, this is much more difficult
than it seems. In a head-mounted dis-
play, there is more head motion than
in a large TV, which cannot move. The
instant a user moves with a large TV,
the field of view also changes.
Speed. The next easiest component
to control is forward speed. Generally
users vastly underestimate distances
traveled in VR, so their sensory ex-
pectations are not met. It is easy to
add a speed setting so users will feel
as though they have moved at their
anticipated speed. However, there is a
catch: Users consistently feel worse at
faster speeds. I have added a run but-
ton due to requests, but later had a few
participants avoid using it, with one
going as far as calling it the “sickness
button.” If an application permits
high speeds, a “stamina” meter can
prevent continuous use and decrease
the likelihood of cybersickness.
The effect of rotation speed is less
certain. Most of the experiments on
rotation forced a continual back and
forth motion. How many circumstanc-
es are there that force continual sway-
ing back and forth? There are few other
than boat or carnival rides. The only
exception is that simple up-down (bob-
bing for example), left-right (most turn-
ing for example), and tilt (windmills for
example) rotations are tolerated better
than rotations that combine two or
XRDS • FALL 2015 • VOL.22 • NO.1
more simple rotations (diagonal rota-
tions for example).
Enforce Expectations. VR hard-
ware does not perfectly match the real
world. In short, users can tell what they
are seeing and feeling does not match
what they expect. One way to decrease
this problem is by enforcing real-world
expectations. There are a few ways
to do this. The application can give a
more natural mode of interaction (for
example, multidirectional walking as
shown in Figure 2), seat users so they
can “feel” more of the real world, or
add some visual that always matches
expectations. These visuals can be
lines across the screen, icons, or seeing
parts of the real world. These stabiliz-
ing visuals are often called “indepen-
dent visual backgrounds.” This may be
why augmented reality, which only has
a few virtual components, rarely gener-
ates cybersickness.
Another way to enforce expecta-
tions is to give users some control
over their view and motion. Just like
in a car, the driver with control is less
likely than the passenger to become
sick. We are not sure about the effect
of graphical realism. Some people re-
port an effect, some report no effect,
some report better graphics have an
effect, and some report the opposite.
Personal experience states very low
graphical quality with mostly solid ab-
stract symbols and very high graphi-
cal quality with shadows and complex
imagery are better tolerated.
Habituation. There is one thing that
is working in a developer’s favor—ha-
bituation. If someone uses an applica-
tion regularly, they will slowly build up
a tolerance. We are not sure if using a
different VR application will give the
same effect, or how long it lasts, but
playing video games with substantial
amounts of forward motion such as
platformers and first-person shooters
does correlate with less cybersickness.
Susceptibility. We have discussed
hardware and software, but what
about the person? Past studies suggest
20-40 percent of cybersickness can be
attributed to the user alone. That is a
great deal to ignore. Gender has been
studied repeatedly. Many studies re-
port women are more susceptible to
cybersickness. However, gender also
correlates well with motion sickness
XRDS • FALL 2015 • VOL.22 • NO.1
There have been
significant changes
to VR hardware:
The costs have
come down and the
hardware has started
to catch up.
history—women are more likely to
have a history of motion sickness. Why
is this a problem? The history of mo-
tion sickness also correlates well with
cybersickness. This means if people
are grouped according to motion sick-
ness history, there is no effect of gen-
der [3]. Motion sickness history may
give more information to estimate
cybersickness, while still indirectly in-
cluding gender.
Age. There hasn’t been significant
research on cybersickness and age. Re-
searchers in the field assumed if some-
thing correlated to motion sickness, it
correlated to cybersickness. This as-
sumption may have been wrong. Mo-
tion sickness and cybersickness do
not act exactly the same. With motion
sickness, older people are less likely
to experience symptoms. The results
are preliminary since recruiting older
participants can be difficult. But un-
like motion sickness, older users have
shown a higher incidence of cybersick-
ness in some VR systems [4]. This may
be an effect of video game usage cor-
relating to cybersickness, or it may be
something yet to be discovered.
GUIDELINES FOR DEVELOPERS
One of the best things developers can
do is include optional settings that
decrease the likelihood of cybersick-
ness. The easiest options are to shrink
the field of view, slow forward speed,
give people at least some control over
their movement, and seat users. The
exception to seating people is that, in
general, the more natural or antici-
pated the movement, the less cyber-
sickness there is. This means devices
such as the Cyberith Virtulizer and
Virtuix Omni, which allow walking in
all directions, may be critical for more
immersive (“better”) systems to pre-
vent illness. More extreme methods
would be to prevent any type of jitter
(e.g., shaky camera) by smoothing the
camera or limiting any forward move-
ment other than scene changes, as well
as using head rotation only. The latter
essentially creates a movie where the
audience is in the middle of the 3-D set.
Here are a few more suggestions to
decrease cybersickness that are not
well studied, but come from my per-
sonal experience. The textures in an
application can affect cybersickness.
Participants have more trouble with
generic, repeating textures such as
gravel and concrete as shown in Ta-
ble 1. Conversely, complex non-repet-
itive textures, such as bookcases or
stone walls with random cracks and
ivy, are well tolerated. The distance
between objects also affects cyber-
sickness. Wide-open areas are highly
preferred. Seeing out to the horizon
is well tolerated, while curved nar-
row hallways are not well tolerated.
Last but not least, playing more video
games with lots of forward motion
helps develop habituation, which is
associated with less cybersickness.
Who would like to join me in a game of
“Halo” or “Minecraft”?
References
[1] Mon-Williams, M., Wann, J. P, and Rushton, S. Design
Factors in Stereoscopic Virtual-Reality Displays.
Journal of the Society for Information Display 3, 4
(1995), 207-210.
[2] Graeber, D. A. and Stanney, K. M. Gender Differences
in Visual Induced Motion Sickness. In Proceedings
of the Human Factors and Ergonomics Society 46th
Annual Meeting (Baltimore, September 30-October
4). Human Factors and Ergonomics Society, Santa
Monica, 2002, 2109-2113.
[3] Arns, L. L. and Cerne, M. M. The Relationship
between Age and Incidence of Cybersickness among
Immersive Environment Users. In Proceedings of
the IEEE Virtual Reality (Bonn, March 12-16). IEEE
Washington, DC, 2005, 267-268.
[4] Kennedy, R. S., Lane, N. E., Berbaum, K. S., and
Lilienthal, M. G. Simulator Sickness Questionnaire:
An enhanced method for quantifying simulator
sickness. The International Journal of Aviation
Psychology 3, 3 (1993), 203-220.
Biography
Lisa R. Rebenitsch recently graduated with a Ph.D. in
computer science from Michigan State University.
She joined the software engineering faculty at the
Milwaukee School of Engineering this fall. Her research
interests lie in cybersickness, virtual reality, and user
interfaces in general.
© 2015 ACM 1528-4972/15/09 $15.00
51
feature

What Is the Vergence-

Accommodation
Conflict and
How Do We Fix It?
The next wave of virtual reality technology might turn to light field
displays to solve a dizzying problem.

By Robert Konrad
DOI: 10.1145/2810048

Y
ou’ve probably never heard of the vergence-accommodation conflict, but if you’ve ever
felt nauseous watching a 3-D movie, you’ve felt its effects. Unfortunately, you might
face a similar problem with the current wave of virtual reality (VR) technologies.
Current 3-D movies and VR systems are based on planar stereoscopy, a process of
displaying two different images of the same scene on a planar screen from slightly different
angles—one to each eye. When viewed together, the images fuse and create an impression
of depth and solidity. While planar stereoscopy “tricks” our visual system into perceiving
depth, it also induces a conflict be- you can’t do one without the other. look and focus on the same object.
tween two parts of the human visual More importantly, the distances at However, there is a problem with ste-
system, causing some users to experi- which you converge and accommo- reoscopic displays.
ence pain and even nausea. This con- date are the same because you usually In these displays, each eye is pre-
flict is known as the vergence-accom- sented with a different image and
modation (VA) conflict. one is able to look at objects as if they
In order to understand the VA con- were in front of or behind the actual
flict let me first explain how our visual
system operates in the real world.
Possibly the 2-D display, depending on the dis-
parities for each object. This is great
When a person is looking at the world, most exciting because it gives us the impression of
their eyes respond in two ways. First,
the brain instructs the eyes to rotate
new approach in depth. However, our eyes must focus
on the display surface to keep the
so they are oriented toward the same creating a near images sharp on our retinas. See the
point of interest (this is called ver-
eye display with conflict? Our eyes converge on objects
Image by Alicia Kubista

gence). At the same time, the brain
forces the lens in each eye to focus so
the point of interest appears sharp on
the retina. When looking around the
world these two actions are coupled;
correct vergence and
accommodation is
light field displays.
as if they were some distance away
from the display, while they accom-
modate on the display surface itself.
We are converging and accommodat-
ing to points at different distances,

52 XRDS • FALL 2015 • VOL.22 • NO.1

Photo Credit TK

XRDS • FALL 2015 • VOL.22 • NO.1 53

feature

something our visual system is not fa-

miliar with. It tries its best to keep the Figure 1. In this light field display technique, two rays pass through the pupil,
images fused and in focus, and is able allowing the user to accommodate to the correct plane. The eye focuses on
to do so to a degree, but with negative (a) the 3-D image and (b) display screen.
side effects.
Although the method of displaying
different images to each eye is differ-
ent for VR and 3-D movies, the fun-
damental principle is the same, and
it induces the same conflict. Because
relatively few VR units have been re-
leased, we are not quite sure whether
users will be able to use VR headsets
for extended periods of time or if the
induced symptoms will cause the pub-
lic to back away from VR much like it
did from 3-D. If so, then VR technol-
ogy might have to shift away from the
current stereoscopic method to more
sophisticated displays. Two such dis-
play technologies are multi-focal and
light field displays.
Multi-focal displays are, as the
name implies, displays that allow
one to focus (accommodate) at differ-
ent distances. Such displays are still
stereoscopic in nature, but they now
allow one to converge and accom-
modate at the same distance, thus
removing the VA conflict. There exist to switch between four focal lengths For Akeley and Love, nearly correct
several methods of creating multiple very quickly (less than one millisec- accommodation is achieved regard-
focal planes, but I will only cover a ond). He rendered each frame four less of where the user looks. However,
couple here. times, once at each focal plane, to cre- eye tracking is gaining traction in VR
While at Stanford University, Kurt ate an approximate volumetric dis- and could be leveraged to make a sys-
Akeley proposed a volumetric display, play. If each plane is displayed above tem that dynamically changes its focal
rendering images across multiple the human flicker fusion threshold, state, depending on where the user is
semi-transparent planes in his 2004 then the individual images average looking. Liquid lenses exist today with
paper “A Stereo Display Prototype with together forming one effective image a continuous spectrum of focal lengths
Multiple Focal Distances.” Because for a viewer. This design was not prac- and could be used for this purpose.
light is emitted from sources at “cor- tical because even for only four focal However, current liquid lens technol-
rect” focal distances, the vergence states, 180Hz monitors were needed ogy only allows for small lenses, result-
comes back into line with accommoda- for a frame rate of 45fps. (For refer- ing in small fields of view.
tion. Although the solution is not prac- ence, the fastest monitor on the mar- Possibly the most exciting new ap-
tical due to its physical size, it was a ket today works at 240Hz.) proach in creating a near eye display
successful proof of concept. It showed with correct vergence and accommoda-
that such a display can successfully tion is light field displays. As described
address deficiencies in current VR dis- in the 1996 paper titled “Light Field
play technologies.
Gordon Love proposed a more com-
The overarching Rendering” by Marc Levoy and Pat
Hanrahan, a light field is the radiance at
pact way of creating multiple focal goal is to create a a point in a given position. It is a vector
planes in his 2009 paper “High-Speed
Switchable Lens Enables the Devel-
sampled version of function that describes the amount of
light flowing in every direction at every
opment of a Volumetric Stereoscopic such a light field and point in space. Light field displays are
Display.” He created a volumetric ste-
reo display via high-speed, switchable
project multiple rays capable of projecting a sampled light
field, which is a close representation to
lenses made of birefringent material, from the same point how light propagates through the world
whose refractive index depends on
the polarization and propagation di-
across different and falls onto our retinas.
Light coming from a light source,
rection of light. The lenses were able points on our pupils. the sun or a lamp, can be thought of as

54 XRDS • FALL 2015 • VOL.22 • NO.1

a bundle of rays coming from a single
point source. These rays propagate
through free space (let’s assume air
is free space) until they interact with
some dense object. The light hits the
particles in the object and scatters,
sending light in all directions. You can
now imagine there is another bundle
of light rays propagating in all direc-
tions from that single point. The small
portion of these rays that pass through
our pupil and fall on our retina is what
we see. Our primary mechanism of
physical observation directly stems
from light scattering on surfaces.
In many cases we model the human
eye as a pinhole camera, where the ap-
erture (pupil) is so small that only one
ray of light from any direction is able
to pass through. Although this model
is convenient for many applications, it
is not accurate. When we stop model-
ing our eye as a pinhole, multiple scat-
tered rays propagate from the point of
interest and pass through our pupils
at different positions (multiple rays
from the same point hit our retinas).
When we focus to the distance of that
point, the rays converge on the retina
and become sharp.
Light field displays attempt to mim-
ic this physical phenomena. Different
methods exist, but the overarching
goal is to create a sampled version of
such a light field and project multiple
Figure 2. A rendered light field. Each subimage is a rendering of the scene from
a slightly different angle. When viewed with a light field display, the subimages
fuse, creating a single realistic looking teapot without inducing a vergence-
accommodation conflict.
XRDS • FALL 2015 • VOL.22 • NO.1
rays from the same point across dif-
ferent points on our pupils, in a physi-
cally correct manner. At least two
different rays must pass through our
pupils to trigger a retinal blur, which
drives our accommodative response.
As we can see in Figure 1, when one
focuses on the 3-D object, the rays
converge on the retina and the object
appears in focus. However, when the
eye focuses on the display, the rays do
not converge and the image appears
blurred. Our visual system acts along
the lines of a feedback loop and will
more readily accommodate on the 3-D
object rather than the display. The VA
conflict will not occur.
Although light fields were intro-
duced in the graphics community
close to 20 years ago, we have only
started seeing near-eye light field dis-
plays in the past few years. In 2013,
Douglas Lanman and David Luebke
from Nvidia Research published a
paper entitled “Near-Eye Light Field
Displays,” which described one of
the first complete near-eye light field
displays. The light field is produced
via an OLED microdisplay depicting
interlaced perspectives through a
microlens array. Each perspective is
shown under each individual micro-
lens. Although the design mostly con-
sisted of off-the-shelf components,
the duo was able to achieve nearly
correct vergence, accommodation,
binocular disparity, and retinal blur
depth cues all in a very small form
factor (around one centimeter thick-
ness). Of course there exists a trade
off between the effective resolution
of the display and the number of
achievable perspectives (the more the
better), because the OLED has a fixed
resolution. Nonetheless the results
are impressive.
Professor Gordon Wetzstein’s group
at Stanford University is also advanc-
ing the area of near-eye light field dis-
plays. At this year’s SIGGRAPH, mem-
bers of the group, along with Nvidia
researchers, presented a prototype
of a light field stereoscope that aug-
ments Wheatstone’s original stereo-
scope with modern factored light field
synthesis via stacked LCD panels. The
proposed design is slightly bulkier
than the light field display proposed
in 2013, but boasts significant im-
provements in resolution and retinal
blur quality.
You might think this technology is
far from becoming a reality, but think
again. These displays might be here
sooner than you think in the world
of augmented reality (AR). Have you
heard of Magic Leap (in 2014 Google
invested $542 million) or Microsoft’s
Hololens? Both technologies are
based on the principle of light fields.
The seemingly magical effects these
displays produce work on the same
principle previously described. Of
course, they have novel technology
that is able to project a light field with-
out the resolution tradeoff. How? I
wish I could tell you, but, not surpris-
ingly, neither company is willing to
give away those secrets.
With so many heavy hitters involved
in the development of VR and AR, we
are on the right path to removing pesky
problems like the vergence-accommo-
dation conflict and creating truly im-
mersive VR and AR systems. This next
year is going to be a big one!
Biography
Robert Konrad is a Ph.D. student in electrical engineering
at Stanford University in Professor Gordon Wetzstein’s
group on computational imaging. He is currently focusing
on computational imaging and displays, as well as their
applications to virtual and augmented reality.
© 2015 ACM 1528-4972/15/09 $15.00
55
feature

VR to the Rescue:
Avoiding danger
with mine disaster
simulations
Virtual reality is helping rescue teams prepare for emergency
situations in places they could never ordinarily go, like collapsed
mines deep underground.

By Alain Boulay
DOI: 10.1145/2810050

M
ining is a very dangerous profession. In addition to threats from falling rock deep
underground, there are poisonous gasses deep in the earth that can cause death
and explosive rock blasting hazards that cause unstable rock and mud slides, or
“muck.” Just recently, at Stobie Mine, a few young miners my age were killed due
to a muck slide, which occurs when underground water veins overflow and mix with crushed
rock. Safety is of paramount importance to the mining community, but it is challenging to
prepare for mine emergencies since real disasters cannot be contrived to practice rescue
operations in realistic conditions.
Mines have emergency rescue to work on, nor did I think my re- some well-worn paths for me to fol-
teams that go underground to res- search would lead me to developing low, so I started working on innova-
cue miners in the case of a disaster. virtual reality (VR) simulations for tions in 3-D VR and mining. The town
These emergency teams need to be emergency rescue missions in mines. where I study, Sudbury, is located in
Image by Przemek Tokar / shutterstock.com

trained to rapidly save lives without
becoming victims too. Rescue teams
cannot train in real mine disasters
since it would be unethical to place
them in dangerous situations. The
solution is simulations. This is where
3-D VR is important.
Like many of you, when I started
my master’s degree in computational
science, I had no idea what I wanted
I hoped my background in psychol-
ogy would help me develop insights
into human-computer interaction re-
search, but I was not sure what area I
wanted to specialize in. I now work in
the Computer Human Interac-
tion Lab at Laurentian Univer-
sity, which primarily focuses
on brain-computer interfacing and
3-D VR. My predecessors provided
northern Canada and is world famous
for its nickel and copper mines. The
mining industry’s influence is perva-
sive in the area; the industry helps to
fund our university.
At MIRARCO Corporation, I have
been developing 3-D mine rescue
simulations that can be viewed on
multiple types of VR training plat-
forms. These platforms include 3-D

56 XRDS • FALL 2015 • VOL.22 • NO.1

XRDS • FALL 2015 • VOL.22 • NO.1 57
feature
ACM
ACM Conference
Conference
Proceedings
Proceedings
Now
Now Available via
Available via
Print-on-Demand!
Print-on-Demand!
Did you know that you can
now order many popular
ACM conference proceedings
via print-on-demand?
Institutions, libraries and
individuals can choose
from more than 100 titles
on a continually updated
list through Amazon, Barnes
& Noble, Baker & Taylor,
Ingram and NACSCORP:
CHI, KDD, Multimedia,
SIGIR, SIGCOMM, SIGCSE,
SIGMOD/PODS,
and many more.
For available titles and
ordering info, visit:
librarians.acm.org/pod
58
Figure 1. The iTX multi-gas monitor is widely used in the mining industry to detect
a variety of dangerous gases.
caves, which are large, immersive,
and interactive 3-D viewing spaces
that you can walk right into; 3-D tele-
visions; and even regular computers.
Developing simulations for these de-
vices is a great challenge because the
software requirements for 3-D media
demand proficiency in many differ-
ent programming languages—from
CUDA for NVIDIA GPUs used to run
the simulations, to web-based mul-
timedia scripting languages like Ja-
vaScript or Action Script. Designing a
training system is similar to develop-
ing a sophisticated multiplayer video
game, since simulations may involve
five or six members of a mining res-
cue team interacting within a virtual
environment simultaneously. My fo-
cus has been on improving human-
On some level,
3-D virtual reality
achieves its realism
and pedagogical
benefit by tricking
your brain.
computer interaction and developing
educational software for mine rescue
trainees in 3-D VR systems.
The goal of this work is to prepare
emergency mine rescuers for real
experiences underground, such as
when the smoke from a mine fire is
so thick rescuers cannot see more
than a foot in front of their faces, or
when rescuers must analyze danger-
ous gas levels with special equip-
ment. For instance, one of the train-
ing modules that I have developed
involves practicing the operation
of specialized gas meters. The iTX
multi-gas monitor is a hand-held de-
vice capable of monitoring different
gas concentrations in the air (see Fig-
ure 1). For mine rescue and recovery
operations, the iTX is configured to
monitor carbon monoxide, methane,
and oxygen concentrations. These
gases explode at certain concentra-
tions, and can cause miners and res-
cuers to lose consciousness or die if
directly inhaled. One of the scenar-
Image by Cjp24/Wikipedia
ios available in the 3-D mine rescue
training simulation teaches rescuers
to calibrate the iTX and to take air
quality samples in a virtual under-
ground environment. This prepares
rescuers to use the iTX system in real
XRDS • FALL 2015 • VOL.22 • NO.1
scenarios, to understand the policy
involved in using the device, and to
avoid making lethal errors while on
a rescue mission underground. Many
other modules will be developed in
the future, including ones focused
on the operation of other important
equipment, such as breathing appa-
ratuses that provide fresh oxygen for
rescuers working in an underground
disaster. In combination, these 3-D
VR modules teach rescue workers the
skills they need to graduate with a
government recognized certification
as a mining rescuer.
How can immersive systems like
these achieve the realism necessary
to prepare students for actual emer-
gencies? The realism comes in part
from a variety of hardware interface
systems unique to virtual reality. For
example, many systems include sen-
sors that indicate the position of the
user’s head, hands, or both in order
to be able to facilitate adequate hu-
man-computer interaction between
the user and the 3-D objects on the
screen. While most interaction in 3-D
VR can be done with a joystick and
keyboard, many systems are designed
to permit gestural interfaces for the
users. For instance, head tracking
and motion sensing are very impor-
tant in 3-D virtual cave or panoramic
systems. Other systems employ a vi-
sor or goggles that project stereoscop-
ic images on the lenses rather than on
a large screen, and 3-D televisions are
also available. Each of these different
3-D platforms have unique hardware
needs that must be considered when
designing and programming soft-
ware that could run on them.
3-D VR systems for emergency
mine rescue training produce an im-
mersive experience, allowing users to
experience a state of consciousness
that brings them closer to feeling
present in a non-physical world than
anything else that can be achieved
in conventional human-computer
interaction. These training simula-
tions encourage “tactical immer-
sion,” which more closely resembles
real emergency rescue experiences.
Rescuers are better equipped to learn
necessary skills than they would be
in traditional educational settings
because of the simulations. In 3-D
XRDS • FALL 2015 • VOL.22 • NO.1
It is challenging
to prepare for
mine emergencies
since real disasters
cannot be contrived
to practice rescue
operations in
realistic conditions.
VR training experiences, the user is
surrounded by images and sounds
that simulate real mine disasters,
such as smoke-filled mine stopes
miles beneath the earth or the cries
of a fallen victim. Because the images
and sounds surround the user, some-
times the brain is fooled into perceiv-
ing these virtual objects as physically
present in the user’s space, as in the
case of “active” 3-D VR. In these situ-
ations, users are able to “suspend
their disbelief” and engage in the
rescue operation scenario more com-
pletely, producing a deeper under-
standing of emergency response ac-
tions than would be received in more
standard forms of training. On some
level, 3-D virtual reality achieves its
realism and pedagogical benefit by
tricking your brain.
This is especially true for the tech-
nologies used to display 3-D scenes to
users. The 3-D images we may have
seen as kids, with the polarized or
colorized lenses, allow a fusing of two
stereoscopic images by chromatically
filtering out the left and right images.
This is called “analygraph” 3-D. The
chromatically opposite colors—often
red and cyan—produce the stereo-
scopic 3-D effect. The color filters out
one of the two presented images, and
the visual cortex of the brain fuses
the images back together in a 3-D per-
spective. This is not exactly the same
with “active” 3-D systems.
Active 3-D systems—also called
“active shutter 3-D systems,” “alter-
nate frame sequencing,” or “eclipse
method”—contain “shutter” lenses
that become dark or light in phase
with the video display refresh rate.
Such systems create 3-D images us-
ing glasses that present different im-
ages to each eye by rapidly darkening
and brightening the image on each
lens. For example, the glasses pres-
ent an image to the left eye by dark-
ening the right lens (or to the right
eye by darkening the left lens). This is
done so rapidly that users do not no-
tice the shutter effect, and there is no
interference with the fusion of the in-
dependent images presented to each
eye. These lenses are made of liquid
crystal and require a small voltage
to shutter from light to dark, which
is often drawn from a small battery.
The rate of alteration necessary to
eliminate noticeable flicker is usu-
ally up to 60 image cycles per second
per eye. These kinds of glasses can
be found with different home video
game systems as well as in large 3-D
VR systems.
CONCLUSION
3-D VR has been found to be a prom-
ising tool to train rescuers for emer-
gency situations, from urban fire
response to military training. 3-D
VR emergency training simulations
allow for the creation of training ex-
periences that could not be achieved
in other ways because of safety con-
cerns, cost barriers, or environmen-
tal considerations. The simulation
environment allows users to experi-
ment safely while carrying out dan-
gerous actions, and allows users to
repeat the exercise until they have
gained sufficient confidence and
skill to be prepared for real-life inci-
dents. Although the programming,
software design, and hardware tech-
nology development is challenging,
developing VR simulations offers
programmers the satisfaction of de-
signing sophisticated systems that
may help save lives.
Biography
Alain Boulay completed his bachelor’s in psychology and
philosophy of science at the University of Toronto and the
University of Western Ontario, respectively. He is currently
studying for his master’s in computational sciences,
focusing on human-computer interaction. His interest is
in applying research in human-computer interaction and
ergonomics for emergency rescue simulations for miners.
© 2015 ACM 1528-4972/15/09 $15.00
59
The newest
ACM forum.
Contributions
that cover the vast
information-rich world
where computing
Captivating
is embedded topics.
everywhere. Net Neutrality
and the Regulated
ACM’s Ubiquity is
Internet
the online magazine
oriented toward the The End of Life
future of computing As We Know It
and the people who A Shortage of
are creating it. Technicians
We invite you to The Fractal
participate: leave Software
comments, vote for Hypothesis
your favorites, or
submit your own Your Grandfather’s
contributions.
Oldsmobile–NOT!
Superscalar
Smart Cities

Visit us at
http://ubiquity.acm.org/blog/
 PROFILE   DEPARTMENT EDITOR, ADRIAN SCOICĂ
PROFILE

Susumu Tachi Headline here Run-in dek

The Scientist who
follows here lorem ips
Invented Telexistence
A
l dolorper aci eum vullam, nibh eugait ex eu facin velent adipit,
DOI:10.1145/2810245 cortie molent ut iustrud quip et eriustie delisi. Ignim quissim
exeraestrud esting eu veniam, conum qui ex ero dui blan ut
Susumu Tachi is which leftfeuismolore
a long-standing mincillimpression
umsandre ute con hendrem nulputpatem dolorem
a Professor Emeritus on the values
dolore duisi eum of his
dit family.
veliquam nonsed vulla conse faccum eu facinisl ipisim
at the University Upon enrolling
magnim enibh eugue at the University
vendre of
conseniat. irillutpatum dit loboreet utem ilisl incil
of Tokyo and the Tokyo at the age
Ut nonullam, quam of 18, Tachimagna
iustrud confessed,
feum ulputat ueriureetum quat wisim adit
scientist responsible as per university
iliquam consectet regulations,
dolorperoshe was ut
eliquisit numsan henisl dunt nostrud te dipit wis
for introducing initially
luptat ad signed
tem ditup for
wis liberal
atue euguearts andminibh nit iusto odit in ulputpat lore eu facing
the world to the concept of spent the first year
exer iriuscipis nonsed andmodolorper
a half deciding sustio et luptatue faccum velessit in exero
telexistence (i.e., remote existence): what
odipithenosshould
dolore pursue for his major.
conseniat.Rud tat. He od molorem quat lortisci blam quat. Ut
Humans experience the real-time, was very much
Ommolorer ipitinprate
favorexofeuipit
mathematics
nummy volut lor susto digna core etueraessi
immersive sensation of being in and
nonphysics,
hendip essebut was
modalso euminterested
etum volor ad blam vullaor ipit ad dolor alisim
a different place from where they ineuhuman
faccum beings,
quam,and had
velisi trouble
lorem ipsim. quissenim dolobor si.
presently exist, and being able to reconciling
Andignibh theextwo. The moment
ercillan utem velis of Ugait lore moloborperat amet
interact with the remote environment. enlightenment
nulput nis ad tatue camevelenibh
when heexero first accum velesequisi. Guercidui blaoree
Born in Japan in 1946, Professor heard a reading
odolortie vullaore of Norbert
feugue magnaWiener’s facipsu tummolobore faccum veriuscidunt
Tachi has had the special privilege of a Cybernetics
mmolutat. Ut onwisi
the exradio.
ercinciAt that
el ullaor lutpatincip ex et dipit at. Ut at autpat
career spanning the entire history of moment,
si eros ate Tachi understood
magnim cybernetics
venit dolore vel loreet, consecte vel utate esequam dip
the technological and computational was what he
utpatisim had been looking
zzriuscipsum dip esefor.ting
He exeros nostrud min vel ip el ut dolore
fervor of his country. He is still chose the Department
etue dipsum eum zzrit,ofquat Mathematical
incil utpat tis augait veraesequat nim quisl ut lum
actively conducting research for Engineering and Information
vel in hendreraesed tisl utpat, Physics,
quisit lan ero con hendrem iustio dolorpe rillaortie
the advancement of telexistence by where
hendithe could
lutet, study cybernetics
consent enisit, sectet vel ex eugue
type models.vulluptat
It was augue
all verydolobor
new and sed
leading several research projects. He alongside
ulputat lameasurement
feugue ea faccum physics,
delis nullut tio doluptat
very attractive laorpero
in 1966,” ercil
heutpleasantly
auguer
kindly agreed to speak with me about control theory, bionics,
nulla faccummy nonsent logic circuit ad
acilluptatis siscin voluptat alisl digniamcommy
reminisced.
his professional life and the series theory,
et veros pattern
dio dolor recognition, and other
sed esequiscipis et nostrud
Through tisihis
blaore faciduis
studies nos augue
and developed
of events that led to him stumbling subjects.
num quisim There atuewasdunt no aliquis
computer ciliquisi. Et min exeril
vision, Tachiinbecame
et ver sequatu
convinced ercipis aute
upon the realization and subsequent science
ut illaorecurriculum
ea faccumsan back henim
then, anddolenis ea adit
that accumsandre
we could magna
have a better feu faci
world by
advancement of telexistence. aauguer
lot of its
aditmodern consecrated
accumsandiam do consecte bla faciliquathumans
augmenting duis nibh etuercip
with cyberneticeugiat.
subfields
magna amet, werevenis
in their alitinfancy.
nismodion “Pattern
ea Ut lut lan etuercidunt
machines. He stayed on etumsan
at the hendit
GETTING ON THE ACADEMIC TRACK recognition
aut irit aliquiswasercilis
in an nim
elementary
vel ut non state euisisim vercilla
University of Tokyo facilis modoluptat
to pursue nibh
a master’s
“I was born in Tokyo just after WWII, back then. No backpropagation,
ulluptatummy niatio con hent landre no er sia blandre
and Ph.D. withdolore
Prof.facil ipiscip
Takashi sustrud
Isobe as
and at that time Tokyo had been deep
magnis learning,
dolobor just
sim linear perceptron-
ing enisit alis nos miniat
his eugait ipis
supervisor, the nullan
latter ofhendipit
which wis
he
reduced to ruins as far as the eye nisi bla corperos ad te mod modolorem nimawarded
was irit nos nos nulput
in 1973 atamthedignisi. Agna
age of 27.
could see. The city has been growing vullan vero od elisi blam del ullan ut consectem
After earningnulput
his PhD,utatum volortie
he decided to stay
from the ruins as I grew up,” he euisim quat. Sequat, conullaortie do facidui
at tat, conum
the university fordolobore
two moredoloreet
years, ipit
shared. The most influential figure odionse tie dolobor eetuer acip exero wis moved
and eugiamet labsnonsequat
in order todolorem
initiate niamet
a new
in his upbringing, who pushed him The holy grail of
dolenismolor sisl dolobor tincin ero et, irit nulputate
project: A roboticdit lore
guide eum
dogzzrit.
that would
toward intellectual pursuit, was his
grandfather. Tachi’s grandfather
telexistence is
quatisis nullaore feu facilit ip estinim
velessi. corem
Iduipissecte
aid visually impaired
iriurem
“I first
feugiam iuscidui bla
people.
in utatumsandre
proposed the conceptfeuip of the
was the nephew of estimable Dr. to have robots
Iduipissecte feugiam iuscidui bla ea feugiam,
guide dog robotconulputat,
in 1975,core feugait
performed
Sankichi Satō, who was the first
Japanese professor of surgery at
connected to the
corem iriurem in utatumsandre feuip
ea feugiam, conulputat, core feugait
voloreet volortin
theoretical research
feugait volormethods
engineering
exerat. Wisis ent ulla
regarding
sustrud molortie
by whichex ea at
these
the University of Tokyo, a position Internet and spread
voloreet volortin exerat. Wisis ent ulla lore magna
ideas might at being eugait and
realized, incilfinally
duisse
in which he succeeded German
professor Julius Karl Scriba. After his
all over the globe,
feugait volor sustrud molortie ex ea at
lore magna at ing eugait incil duisse
magna feuip eathe
demonstrated
veriure min
proposed
feugue
et loborti
method to the
vel eugue
feasibility
ssequat.
con
of the
Ut et at.
development
great-grandfather’s death, Tachi’s with people able
magna feuip ea feugue vel eugue con Imthe
of ilit robot
wiscincalled
henimMELDOG,
zzriustrud eum eum
named
grandfather was taken in and brought
to log on and off to
veriure min et loborti ssequat. Ut et at. iuscipit,
after the sed ero diamcon
Mechanical sequamet nibh
Engineering
Photo Credit TK

up during the Meiji era by Dr. Satō in a Im ilit wiscin henim zzriustrud eum eum eugait ex eu(MEL),”
Laboratory facin velent
Tachiadipit,
recalled.quip et
deeply academic home atmosphere, them as they please.
iuscipit, sed ero diamcon sequamet eriustie
After delisi. Ignimstudy,
a preliminary quissimthe project

XRDS • FALL 2015 • VOL.22 • NO.1 61

feature
was conducted on a six-year plan
that established technology related
to the intelligent navigation of mobile
robots, such as routing with the use
of landmarks and obstacle detection.
This work led to the development of
the idea of intelligent disobedience.
Tachi continued, “Right now there
are very good navigation systems,
but back then there weren’t any, so
we had registered landmarks and
searched for a route. Sometimes,
it might be very dangerous for a
person to walk straight due to various
kinds of obstacles, at which point we
realized that it would be better for
the guide dog robot to disobey the
commands of the master and thus, to
protect him. We called that intelligent
disobedience.”
According to Tachi, the project
had other important outcomes, and
in particular it confronted them with
thinking about the problem of human-
machine interaction, as there was
a need for the guide dog robot and
the human to constantly exchange
information about the environment.
VISIT TO MIT, AND THE
BEGINNINGS OF TELEXISTENCE
In 1979, Tachi joined the famous
Professor Robert W. Mann’s lab at
MIT as a Japanese Government Award
Senior Visiting Scientist in order to do
joint research on the guide dog project.
He felt privileged to work at the same
university where Wiener had initiated
cybernetics. After what turned out to
be an extremely productive year, he
returned to Japan in 1980 thinking
There was no
computer science
curriculum back
then, and a lot
of its modern
consecrated
subfields were
in their infancy.
62
about ways to further advance his work.
His aim was to create a simulator device
by which humans could work freely in
an environment with obstacles without
putting themselves in danger, which in
turn led him to conceive the idea
of telexistence.
“I came up with the idea of
telexistence in the early autumn of
that year. When walking down the
corridor, I was suddenly reminded
that all of human vision is just
based on two images projected on
our retinas, and we build the 3-D
world by moving our heads and
exploring the world. So if we replace
that with a virtual image, what we
experience is a kind of virtual world.
Since we perceive the environment
through sensors and reconstruct
the world in our brains, if we gather
that information from a robot and
present it to the human, we can
live through the experience of
being embodied within the robot,
existing in its specific environment.
I immediately went back to my office
and wrote ever ything down, and
simultaneously came up with the idea
of how to design the visual display to
do that. 1980 was the beginning of
telexistence,” Tachi explained.
Despite the simplicity and elegance
of his revelation, Tachi confessed his
ideas regarding telexistence were at
first met with skepticism, and only
after developing a prototype did his
peers truly understand the out-of-
body experience he had been talking
about. “I was immediately impressed
when I saw myself for the first time
using the system I had built. It’s
not like a mirror. When I saw myself
from the third person using that
system, I began questioning whose
consciousness I really was, allowing
me to perceive my body like that,” he
said. Tachi also observed in the age
of the Internet, webcams, and mobile
devices, it’s very easy for a person to
understand these notions, but back
then the only way to explain these
concepts was to invite people to the
lab and have them experience the
prototype system that was built. Only
then did they truly understand what he
meant by telexistence.
He rejoined the University of Tokyo as
a professor in 1989 and has continued
his research in telexistence with more
than a hundred research members
and students in the Tachi Laboratory.
Since he believes only experience
can make people understand the
true meaning of telexistence, he has
constructed TELESAR (TELExistence
Surrogate Anthropomorphic Robot) to
demonstrate the advancement
of the technology. He has invented
several innovative technologies in 3-D,
virtual reality, augmented reality,
and haptics, which have in turn
advanced telexistence.
NEW DIRECTIONS IN TELEXISTENCE
According to Tachi, building technology
that is capable of telexistence will have
a definite positive impact on our rapidly
aging, transportation-reliant society: “If
we could succeed in transmitting haptic
sensations, then society will change, so
that real work can be done remotely, in
environments which would otherwise be
hazardous or hard to reach. We could do
international timeshare, and we could
allow disabled or otherwise old people to
continue working. In Japan, for example,
we have a very aged society, but in many
cases old people can still work, and would
like to keep contributing to society even
if they get somehow physically impaired
with age. They have lifelong experience,
and this experience is very valuable.
Telexistence would liberate them from old
age, and maybe even change the way
we socialize.”
On a final note, Tachi remarked the
holy grail of telexistence is to have
robots connected to the Internet and
spread all over the globe, with people
able to log on and off to them as they
please. He emphasized telexistence is
in a sense the ultimate form of cyborg:
“Using this technology, human beings
can obtain an augmented superhuman
body without losing anything of
their own. They could even be free
of location and timezone. That is,
through the use of their avatar robots
all over the world, human beings would
become virtually ubiquitous.”
Copyright held by Owner(s)/Author(s)
XRDS • FALL 2015 • VOL.22 • NO.1
 Inviting Young
Scientists
Meet Great Minds in Computer
Science and Mathematics
As one of the founding organizations of the Heidelberg Laureate Forum
http://www.heidelberg-laureate-forum.org/, ACM invites young computer
science and mathematics researchers to meet some of the preeminent scientists
in their field. These may be the very pioneering researchers who sparked your
passion for research in computer science and/or mathematics.
These laureates include recipients of the ACM A.M. Turing Award, the Abel Prize,
the Fields Medal, and the Nevanlinna Prize.
The Heidelberg Laureate Forum is September 18–23, 2016 in Heidelberg, Germany.
This week-long event features presentations, workshops, panel discussions, and
social events focusing on scientific inspiration and exchange among laureates
and young scientists.

Who can participate?

New and recent Ph.Ds, doctoral candidates, other graduate students
pursuing research, and undergraduate students with solid research
experience and a commitment to computing research

How to apply:
Online: https://application.heidelberg-laureate-forum.org/
Materials to complete applications are listed on the site.

What is the schedule?

Application deadline—February 3, 2016.
We reserve the right to close the application website
early depending on the volume
Successful applicants will be notified by end of March/early April 2016.

More information available on Heidelberg social media

PHOTOS: ©HLFF / B. Kreutzer (top);

©HLFF / C. Flemming (bottom)
end
LABZ
The Virtual Human
Interaction Lab
Stanford, California
Editor’s Note: In this issue, Andrea
Stevenson Won shares her experience at
Stanford University and her pursuit of
research in virtual reality to help others.
—Somdip Dey
T
he Virtual Human Interaction
Lab (VHIL) at Stanford Uni-
versity uses immersive virtual
environments to study human
interactions. However, the lab’s re-
search is not just limited to classic vir-
64
tual reality in head-mounted displays
like the Oculus Rift. Our research in-
cludes tracking and representing hu-
man behavior. There are three major
areas of investigation. First, how can
virtual reality be used to study hu-
man behavior in the real world? Sec-
ond, how can virtual reality be used
to positively affect the real world?
Third—and this is becoming increas-
ingly relevant as these kinds of sys-
tems become more widely used—how
does being in these environments af-
fect users’ behavior in the real world?
This is my fourth year as a Ph.D.
student at VHIL. Although there are
only half a dozen full-time Ph.D. stu-
dents in the lab, there are also many
master’s students and a large group of
undergraduate research assistants—so
it’s a busy place. I feel really grateful to
be part of such an amazing group. The
graduate students in the department
were helpful to me when I first started
Image by Linda A. Cicero / Standford News Service
and I hope to be able to pass that along
to newer students. I love collaborating
with my fellow students; it is fun to be
around other people who get so excited
about experimental results. There are
also lots of opportunities to collaborate
with people from other departments,
for example, education, psychology,
political science, and medicine. Our re-
search is very cross-disciplinary.
When I started at Stanford I was im-
XRDS • FALL 2015 • VOL.22 • NO.01
 1920xRGBx1080 is the panel
resolution for Sony’s Project Morpheus
head-mounted display.
mediately thrown into experimental
work. This made for a pretty intense
first year, but it was what I was look-
ing for in a lab. My advisor, Jeremy
Bailenson, has a reputation for being
very productive, which means his advi-
sees have a lot of opportunities to get
involved with interesting projects.
Before coming to Stanford my back-
ground was in anatomical models, fo-
cusing on creating digital models for
virtual reality. My M.S. from the Uni-
versity of Illinois at Chicago focused
on biomedical visualization, and I did
a lot of custom work with clinicians
after I graduated. I became interested
in how custom models like these could
be used therapeutically, for exam-
ple, showing patients suffering from
chronic pain a virtual representation
of their own face being painlessly
touched. I became fascinated with
the idea that changing how a person
is represented in media could change
their life in the real world. This idea
led me to pursue my Ph.D. at VHIL.
My main research interest is using
virtual reality therapeutically for pain
patients, by changing how their bodies
appear to them when they view them-
selves in a virtual reality. However, I am
broadly interested in how people inter-
pret the physical world based on cues
they receive through media, whether it
Image of VFX-1 by eVRydayVR. Image of Oculus Rift Dev Kit 2 by Ats Kurvet.
is a super-immersive virtual environ-
ment or a brief text-based interaction.
It’s important to understand both the
good and bad effects of these powerful
media experiences.
In my career, I hope to help alleviate
some of the suffering caused by chron-
ic health conditions by using the abili-
ties of virtual environments to change
people’s perceptions. I look forward to
working with my colleagues at VHIL to
use media to improve human life.
Biography
Andrea Stevenson Won is a Ph.D. student at the Virtual
Human Interaction Lab in the Communication Department
at Stanford University.
XRDS • FALL 2015 • VOL.22 • NO.01
BACK
VR Head-Mounted
Displays
It’s hard to ignore the proliferation of head-mounted displays lately.
Google Glass, Microsoft Hololens, and Oculus Rift are all well-known
products that have entered, or are poised, to enter the market. This
isn’t the first time we’ve seen a frenzy of interest in augmented and
virtual reality (VR) though. By the early 1990s, VR was already a hot
topic in mainstream media and a handful of early consumer products
enjoyed limited success.
VR games had already reached arcades by the early ‘90s, and in
1995, Forte Technologies released one of the earliest VR devices for
the home PC. Due to its relatively competitive price point, and modest
hardware requirements, the VFX-1 was one of the more successful
offers on the market. Sporting the “cyberpuck” controller with built-
in motion sensor, the VFX-1 could be used to play numerous games.
Nevertheless, the VFX-1 suffered from technical shortcomings and a
generally poor user experience that fated it to limited adoption.
In 2012, Oculus VR launched a wildly successful Kickstarter
campaign to help fund further development of its product, the Rift,
quickly surpassing its funding goal. Developer Kit 1 began shipping
in early 2013, the much improved Developer Kit 2 was released in
2014, and the consumer version is set to be released in 2016. With
a wider field-of-view, better resolution, better tracking, and overall
improved technology, the Rift promises to be a much more immersive
experience than any VR hardware that has come before.
Despite the shortcomings of products in the past, this time really
does feel different. Perhaps we will all finally get to experience
movies, games, and other digital media in the new and immersive
ways that we’ve been promised for so many years.
—Finn Kuusisto
Forte-VFX-1 Oculus Rift Dev Kit 2
Release 1995 2014
Price $995 $350
FOV (horizontal) 33.5° 100°
Resolution (per eye) 263x230 960x1080
Weight (pounds) 2.5 1
65
 HELLO WORLD
Sorry Kids, Iron Man’s
Superpowers Aren’t Unique
BY LARA ZUPAN AND MARINKA ZITNIK
E
very superhero has its
own unique set of skills.
Just about any Iron Man-
obsessed fan could tell you
that. They could also give you an
in-depth description of any of their
favorite comic book characters and
respective superpowers. Well, we have
news for you and that obsessed fan;
there are many more characters in
the Marvel Universe than you might
have thought, and most of your
favorite childhood heroes possess
powers similar to those of Stan Lee’s
superheroes.
As it turns out, there are more
than 900 characters hidden within the
Wikipedia category “Marvel Comics
superheroes,” and approximately
85 percent of entries include a
description of superpowers right at
the top of the page. This information
was the basis of our analysis.
In this column, we present the
uncanny resemblance between the
superpowers of your favorite bedtime
story companions and the vast
superhero network provided by the
Marvel Universe and its inhabitants.
SCRAPING WIKIPEDIA
To obtain a list of Marvel Comics
characters, we used a Python library
Wikipedia (http://github.com/
goldsmith/Wikipedia), which gives
us access to Wikipedia data that
can be parsed. The Wikipedia library
wraps the MediaWiki API; this is a
web service that provides convenient
access to article summaries, wiki
features, meta-data, and data, such
as links and images from Wikipedia
pages. After we identified the relevant
characters, we retrieved their
Wikipedia pages and extracted the
text referencing abilities associated
with each character. Figure 1 shows
an example of a portion of a Wikipedia
66
page with the relevant information
about Thor’s known abilities. We then Figure 1: Part of a Wikipedia page
computed the semantic similarity showing Thor’s abilities.
between texts of different abilities
to quantify how similar the abilities
of different superheroes are. Finally,
we constructed the network of comic
book characters; two characters
were linked if their abilities matched
by at least 80 percent according to
Levenshtein-based similarity.
Stan Lee ran out of ideas 641
superheroes ago.
When talking about network
properties, one must first focus on
the number of nodes and edges in
the network. Our network has 752
nodes, with each node representing a
different Marvel superhero. There are
9,496 edges connecting the nodes,
where each edge indicates characters
who share many superpowers.
The network has 18 connected
components of which the largest
component has 641 nodes. Next, we
focused on the analysis of the largest
connected component.
Each character in the network is
on average linked to 30 superheroes.
Based on the definition of our network,
the character with the highest
degree has the most “typical” skill
set. Shatterstar, a mutant hero and
member of the team X-Force, is a
character with the most connections
(see Figure 2). He is closely followed
by Ikaris with degrees of 132 and 128,
respectively. In accordance with this
information, the most generic super
powers include superhuman strength, Many pages, such as this one, were used
speed, agility, healing power, energy to extract information for our network
manipulation, and enhanced cognition. analysis. We considered word tokens
describing the hero’s abilities (see
The longest shortest path, i.e. “Abilities” in the figure) to quantify how
the network diameter, is 15. similar two heroes are to one another
by counting the minimum number of
Aragorn and Apex have the highest
operations required to transform the text
eccentricity scores and are the on abilities of one hero into the text on
furthest apart in our network. This abilities of the other.
makes sense because their skill sets
XRDS • FALL 2015 • VOL.22 • NO.1
 are very different. Aragorn can fly
with the help of his feathered wings,
while Apex is a technopath.
To quantify the degree to which
nodes cluster together, we computed
the average clustering coefficient
of the network. Clustering is an
important property of social networks
where people tend to have friends
who are also friends. This results
in sets of people with many edges
among them, while a randomly
selected equally-sized set of people
would have a much smaller number of
edges connecting them. Interestingly,
the superhero network has a
relatively high clustering coefficient
of 0.48, which suggests the presence
of clustering structure in the
network. We used a modularity-
based measure in Gephi (http://gephi.
github.io) to find clusters of tightly
connected superheroes, which are
shown in Figure 2.
Don’t worry, we didn’t forget to
include Spider Man.
So far, we focused on the
overall structural properties of the
superhero network. Now we will
turn our focus to the elites of the
Marvel Universe. Yes, we are talking
about the superheroes you actually
recognize. Since we defined our
network in terms of shared abilities
rather than joint appearance of
superheroes in comics, many popular
characters were not recognized as
central nodes of the network with
the established centrality measures
such as closeness, betweenness, and
PageRank.
Figure 3 shows a network centered
on Spider Man, Thor, Iron Man, and
Captain America. According to the
large number of edges connecting
Spider Man and Thor, the two appear
to be the most alike in regards to skill
set. They are both supernaturally
strong and experience superhuman
longevity. The superhero with the
most distinct skill set in this group is
Captain America, with only one edge
connecting him to the rest of the
superheroes. Iron Man is most similar
to Spider Man as they are connected
by a few edges, and they both have
genius-level intellect.
XRDS • FALL 2015 • VOL.22 • NO.1
Figure 2: The Fruchterman-Reingold
algorithm was used to layout the
superhero network in Gephi.
Nodes represent superheroes and edges the
intersection of their skills. Different colors
correspond to different clusters in the
network. The representatives of selected
clusters are highlighted. Larger nodes
indicate higher node degrees.
Figure 3: The famous of the super-
hero bunch. The Yifan Hu algorithm
was used to layout the network in
Gephi. (Source of character pic-
tures: Wikipedia)
Iron Man
Captain America
Spider Man
Thor
ROLLOUT
In recent years, the increasing
availability of digitized fiction
and literary works has enabled
computational analysis of these
texts. These approaches have often
been used together with more
traditional techniques of literature
analysis. Recently, ideas coming from
the analysis of complex networks
offer new ways to understand how
properties of fictitious networks
emerge from the interactions between
characters in a non-trivial manner [1].
Here we tried to understand the
structure of the Marvel multiverse
network, where two Marvel characters
are considered linked if they share a
skill set. Related to our work is the
analysis of fictitious social networks
[2–6], where literary texts are
examined to investigate whether
cliques and groups portrayed in these
texts correspond to those that have
been observed in spontaneous human
interactions. For example, analysis of a
social event network constructed from
Hamlet helped to delve into Hamlet’s
dense character network [6]. It is
interesting that these networks are
not random, and they have the same
properties as those typical of human
beings across ages and cultures
such as movie actors or scientific
collaboration networks [2, 6]. Now
it’s your turn. Go analyze your favorite
season of “Star Trek,” and you too
can become the Stan Lee of fictitious
network analysis.
References
[1] Newman, M. E. The structure and function of complex
networks. SIAM Review 45, 2 (2003), 167-256.
[2] Alberich, R., Miro-Julia, J. and Rossello, F. Marvel
Universe looks almost like a real social network.
Feb. 2002. eprint arXiv:cond-mat/0202174,
[3] Gleiser, P. M. How to become a superhero. Journal of
Statistical Mechanics: Theory and Experiment Sept.
2007, P09020.
[4] Mac Carron, P. and Kenna, R. Universal properties of
mythological networks. Europhysics Letters 99, 2
(2012), 28002.
[5] Choi, Y. M. and Kim, H. J. (2007). A directed network
of Greek and Roman mythology. Physica A: statistical
Mechanics and its Applications 382, 2 (2007), 665-671.
[6] Stiller, J., Nettle, D., and Dunbar, R. I. The small world
of Shakespeare’s plays. Human Nature 14,4 (2003),
397-408.
Biographies
Lara Zupan is a high school junior attending Gimnazija
Vic, Ljubljana, Slovenia. She has edited several high
school publications, designed Novi Dijak, a student run
magazine targeting Slovenian high school students,
and is a member of the European Youth Parliament.
She is interested in graphics design, literature, and
visual arts.
Marinka Zitnik is a Ph.D. student in computer science
at the University of Ljubljana. She also did her
research at the University of Toronto, Imperial College
London, Baylor College of Medicine and Stanford
University. Her interests include machine learning,
artificial intelligence, probabilistic numerics, and
bioinformatics.
Copyright held by Owner(s)/Author(s).
67
 ACRONYMS
HMD Head-Mounted Display: An
HMD can be a helmet, goggles, or
spectacles with tiny monitors in front
of each eye to generate images that
the wearer can perceive as 3-D. The
device is generally accompanied by a
head tracker, which allows illustrations
to change according to head
movements to make the experience
seem more real.
IVR Immersive Virtual Reality:
Being physically present in a
non-physical world or a state of
consciousness when an artificial
environment transforms the visitor’s
or immersant’s awareness about the
physical-self.
VA Virtual Artifact: Immaterial
objects such as the Internet, intranet,
cyberspace, or virtual reality, which
exist in a digital environment or the
human mind, can be considered to be
virtual artifacts.
VR Virtual Rehabilitation: The concept
of virtual rehabilitation augments a
therapeutic patient’s training based
on virtual-reality simulation exercises,
making the patient more involved
in desensitization.
VRML Virtual Reality Modeling
Language: Superseded by X3D, and
pronounced vermal, it is a standard
file format for representing 3-D
interactive vector graphics with World
Wide Web implementation in mind.
WITH World in the Hand: This acronym
works as a metaphor for visualized
tracking where the tracker is held in
the hand. This tracker is connected to
the motion of an object shown in a
display elsewhere.
X3D is an XML-based file format for
representing 3-D computer graphics.
X3D provides extensions such as CAD,
Humanoid, and Geospatial to VRML,
and also supports multi-stage and
multi-texture rendering.
68
POINTERS
position and orientation information.
VIRTUAL REALITY
But now you can have accuracy in the
Virtual reality is an artificial,
palm of your hand. A new specialized
computer-generated environment
receiver, called GRID, can deliver
presented to the user for exploration
information—how one’s head rotates
and interaction in such a way that
or tilts—to less than one degree of
the user suspends belief and accepts
measurement accuracy. This could
it as a real environment. The term
greatly enhance VR environments that
“virtual reality” (VR) was coined
are based on a real-world setting by
in 1987 by Jaron Lanier. Called
quickly building a globally referenced
the “engines of empathy,” VR gear
3-D map of one’s surroundings.
allows us to look through someone
http://www.sciencedaily.com/
else’s eyes, experience someone
releases/2015/05/150505083031.htm
else’s story, and develop a sense of
compassion and understanding
for them. The resources listed “Virtual Reality Helps Stroke Patients
explore the potential of virtual Recover Use of Arm”
reality technology to impact various Neuroscience News reports stroke
domains ranging from education patients with hemiparesis—reduced
and entertainment to healthcare and muscle strength on one side of the
the military. body—often under use their affected
—Tejas S. Khot limbs even though they still have some
motor function. Using their healthy
limb may immediately improve the
VIRTUAL REALITY IN “PRINT” ease of their daily activities, but a
long period of non-use of the affected
paretic limb can lead to further loss
“The Rise and Fall and Rise of Virtual
of function. According to a clinical
Reality”
pilot study published in the open
The promises of virtual reality have
access Journal of NeuroEngineering
been made for decades starting in
and Rehabilitation, virtual reality
the late 1980s; yet it is only in the last
could assist arm rehabilitation in
few years that the utopian-seeming
some stroke patients. The researchers
ideas have started to touch reality.
found using virtual reality to increase
This nine-part collection of articles
a patient’s confidence in using their
from The Verge explores the journey
paralyzed arm may be critical
of virtual reality through the years
for recovery.
reviewing current state-of-the-art
http://neurosciencenews.com/stroke-
systems; it includes an interesting
virtual-reality-arm-movement-2112/
tutorial on how to make your
own VR headset.
http://www.theverge.com/a/ “DARPA Explores Virtual Reality as
virtual-reality/ the Future of Cyberwarfare”
By Jason Dorrier
In the 1984 sci-fi novel Neuromancer
“New Centimeter-accurate GPS System
authored by William Gibson, hackers
Could Transform Virtual Reality and
are digitally embodied in a realm
Mobile Devices”
where computers, programs, and
While present VR systems transform
the paths connecting them are
the view for us, they still cannot be
represented as 3-D objects making
used outdoors. Imagine games where
hacking the matrix a video game.
you are running in the backyard
Gibson’s virtual reality Internet might
along with fellow players in a virtual
not be so distant after all. According
environment, blurring the line
to SingularlityHUB, “DARPA’s Plan X
between the real and virtual world.
program wants to make cyberwarfare
For too long, this type of outdoor,
more like warfare in the real world—
multiplayer experience has been
where future cyber-warriors clash
hindered by the lack of precise
XRDS • FALL 2015 • VOL.22 • NO.1

in an immersive virtual reality
simulation of the internet.”
http://singularityhub.com/2014/05/29/
darpa-explores-virtual-reality-as-the-
future-of-cyberwarfare/
USING VR FOR MORE THAN FUN
A company blending the real world
with the virtual...
The next era of computing is not
just about virtual worlds, but about
how the digital and analog worlds
intersect, while placing the user at
the center of that intersection. With
its stereoscopic eyewear and a stylus,
zSpace is turning computer usage
into a non-enveloping VR experience.
It is fully collaborative and it allows
multiple people to view the same
images at the same time, while
remaining separate from the real
world, but not totally isolated from it.
Already, zSpace is changing the way
students learn STEM and bringing a
new dimension to medical training.
http://zspace.com/
Immersive journalism..
Nonny de la Peña’s “Project Syria” uses
VR goggles to place virtual visitors
inside the meticulously researched
world of a Syrian citizen caught in
the current conflict. It is classified
as “immersive journalism” for its
capacity to put people inside the story.
De La Peña has been called “The
Godmother of Virtual Reality.”
http://www.immersivejournalism.com/
XRDS • FALL 2015 • VOL.22 • NO.1
EVENTS
CONFERENCES
IFIP International Conference on
Entertainment Computing
Radisson Blu Royal Garden Hotel
Trondheim, Norway
September 29–October 2, 2015
http://icec2015.idi.ntnu.no/
The 14th IEEE International
Symposium on Mixed
and Augmented Reality
Fukuoka International
Congress Center
Fukuoka, Japan
September 29–October 3, 2015
http://ismar.vgtc.org/ismar/2015/info/
ismar-welcome/top-page
The ACM SIGCHI Annual Symposium
on Computer-Human
Interaction in Play
The Light, Friends House
London, United Kingdom
October 5–7, 2015
http://chiplay.org/
25th International Conference on
Artificial Reality and Telexistence
(ICAT 2015) and the 20th Eurographics
Symposium on Virtual Environments
(EGVE 2015)
Kyoto International
Community House
Kyoto, Japan
October 28–30, 2015
http://www.ic-at.org/2015/#welcome
The Eighth ACM SIGGRAPH
Conference and Exhibition
Kobe Convention Center
Kobe, Japan
November 2–5, 2015
http://sa2015.siggraph.org/en/
The 12th Workshop on Virtual Reality
Interaction and Physical Simulation
Université Lyon 1
Lyon, France
November 4–11, 2015
http://vriphys2015.sciencesconf.org/
28th ACM Symposium on
User Interface Software
and Technology (UIST)
Sheraton Charlotte Hotel
Charlotte, NC
November 8–11, 2015
http://uist.acm.org/uist2015/about
The 21st ACM Symposium on Virtual
Reality Software and Technology
Beihang University
Beijing, China
November 13–15, 2015
http://vrlab.buaa.edu.cn/vrst2015/
index.html
ACM International Conference on
Interactive Tabletops and Surfaces
Vidamar Resort Madeira
Funchal, Portugal
November 15–18, 2015
http://www.its2015.org/
Advances in New Technologies,
Interactive Interfaces and
Communicability
Casa Serrana Hotel
Huerta Grande, Córdoba, Argentina
November 18–20, 2015
http://www.ainci.com/ADNTIIC-2015/
conference_ADNTIIC_2015.html
International Conference Virtual
and Augmented Reality in Education
Technologico de Monterry
Nuevo Leon, Mexico
November 20–21, 2015
http://eventos.ull.es/event_detail/2027/
detail/2015-international-conference-
virtual-and-augmented-reality-in-
education-varea15.html
11th International Symposium
on Visual Computing
Monte Carlo Resort and Casino
Las Vegas, NV
December 14–16, 2015
http://www.isvc.net/
69
 FEATURED EVENTS
CHI PLAY 2015
The Light
Friends House
London, United Kingdom
October 5-7, 2105
As of late, virtual reality and
augmented reality are hot topics
for researchers and companies
alike. With a high demand for
virtual reality gaming systems, the
industry is expected to experience
a surge in the commercial market-
place. Whether the Oculus Rift or
Google’s VR system, called Card-
board, end users can’t get enough
of virtual and augmented reality.
CHI PLAY is an annual sympo-
sium focused on the intersection
of computer-human interaction
and play. The conference brings
together academicians, research-
ers, students, and professionals all
seeking to foster growth in the fu-
ture of digital gaming using virtual
and augmented reality systems. A
game design competition for stu-
dents, along with many workshops,
will be held in the newly developed
“Knowledge Quarter”—an area
where you can find an abundance
of academic, cultural, research, sci-
entific, and media organizations.
The interdisciplinary conference
will be a treat for all those in atten-
dance. As an added bonus attendees
will find numerous tourist attrac-
tions near the venue, which include
museums, parks, galleries, and the
illustrious British Library.
For more information, please
visit http://chiplay.org/.
—Darshit
— Patel
70
CONTESTS & EVENTS
TechCrunch Disrupt
TechCrunch Disrupt is a technology
conference featuring panels and
discussions from TechCrunch
staff as well as guest speakers. The
event also includes a hackathon
and “Startup Battlefield,” in which
15 companies compete for funding
for their idea by pitching their
products in front a panel of judges
and audience. TechCrunch Disrupt
was held in San Francisco from
September 21-23 and will be held
in London on December 7 and 8.
http://techcrunch.com/event-type/
disrupt/
Internet of Things World
The “Internet of Things” describes
the growing scenario in which all
people and objects are uniquely
identified and able to communicate
data over networks intelligently.
Examples include network enabled
heart monitor implants and WiFi-
capable thermostat systems. In
October, Internet of Things World
holds an event in Europe, which
includes keynote speakers, startup
panels, and a hackathon. This year’s
conference and exhibitions took place
in Berlin, Germany. The next event
is scheduled for May 2016 in Santa
Clara, CA.
http://iotworldevent.com/
International Student
Virtual Reality Contest
IVRC is an international contest
focusing on innovation in the field
of virtual reality. The final round
of this contest will take place
from October 24–25 in Japan at
the National Museum of Emerging
Science and Innovation in Daiba,
Tokyo. Each contest submission
begins with a student proposal, with
help from professional researchers.
Winners are selected through
evaluation by judges experienced in
virtual reality research.
http://ivrc.net/2015/en/
Masters of Code Hackathon Series
MasterCard is hosting a series of
regional events to find the “masters
of code.” Participants are challenged
to use MasterCard APIs to develop
an application. Winners of regional
events are invited to take part in a
grand finale event, which will be
held in Silicon Valley. Finalists will
compete for a prize, which includes
$100,000. During the first two weeks
of November, regional competitions
will be held in New York and
London.
http://mastersofcode.com/events/
GRANTS AND SCHOLARSHIPS
Facebook Fellowship Program
Website: https://www.facebook.com/
fellowship
Deadline: November 1, 2015
Eligibility: Graduate students
pursuing research in specified fields of
computer science.
Benefits: Tuition and fees paid,
$37,000 yearly grant for two years,
a visit to Facebook headquarters to
present research, and the opportunity
for a paid internship.
Explanation: Facebook is providing
12 fellowships to graduate students
who are involved in research in the
fields of architecture, computer vision,
data mining, databases, distributed
systems, economics and computation,
human-computer interaction/social
computing, machine learning, natural
language processing, networking/
operating systems, programming
languages/compilers, security/privacy,
and software engineering. Apply with
complete application and references
by November 1, 2015.
Image Courtesy ChiPlay2015
XRDS • FALL 2015 • VOL.22 • NO.1

Nvidia Graduate Fellowship Program
Website: https://research.nvidia.com/
relevant/graduate-fellowship-program
Deadline: January 15, 2016 (tentative)
Eligibility: Full-time Ph.D. students,
who have already completed
the first year of their program at
the time of application, are members
of an active research team, and
majoring in computer science,
computer engineering, system
architecture, electrical engineering,
or a related area.
Benefits: $25,000 grant per award and
technical support from Nvidia.
Explanation: Nvidia provides funding
to Ph.D. students whose research
is centered on graphics, high-
performance computing, and GPUs.
The application portal for the 2016-
2017 session opens this fall.
DAAD Short-Term Research Grant
Website: https://www.daad.
de/deutschland/stipendium/
datenbank/en/21148-scholarship-
database/?detail=50015434
Deadline: Varies from country to
country
Eligibility: Graduates, Ph.D. students,
postdocs from participating countries
Benefits: Recent graduates with a
master’s or bachelor’s receive €750 and
Ph.D. candidates receive €1000, apart
from various allowances, for a period
of 1-6 months.
Explanation: Young academics,
scientists, and Ph.D. students are
invited to apply for this short-term
research grant. Winners must
pursue a research project or a course
of continuing education at select
German institutions, carried out in
coordination with an academic adviser
in Germany.
Image by Sean Pavone
XRDS • FALL 2015 • VOL.22 • NO.1
University of Geneva Excellence
Masters Fellowship
Website: http://www.unige.ch/sciences/
Enseignements/Formations/Masters/
ExcellenceMasterFellowships_en.html
Deadline: March 15, 2016 for
graduate school enrollment
starting September 2016
Eligibility: Candidates who have
completed their bachelor’s degree,
or expect to complete it within six
months, and were in the top 10 percent
of their program.
Benefits: CHF 10,000 to CHF 15,000
for the first year, the fellowship will be
extended for the entire duration of the
course if the candidate is academically
successful in first semester.
Explanation: The Faculty of Science at
the University of Geneva, Switzerland,
has established a fellowship program
to support deserving candidates who
wish to pursue their master of science
degree at the institution.
FEATURED EVENTS
ICAT-EGEVE 2015
Kyoto International
Community House
Kyoto, Japan
October 28-30, 2105
Artificial reality and telexistence
are redefining the boundaries of
human-to-human interaction. With
the help of new devices, we com-
municate virtually with people as if
in person. Rapid advancements in
virtual reality, augmented reality,
mixed reality, and 3-D user interfac-
es have made all of this possible.
ICAT-EGEVE 2015 is a combined
conference, bringing together the
Artificial Reality and Telexistence
conference and the Eurographics
Symposium on Virtual Environ-
ments. ICAT and EGVE are some of
the oldest names in conferences on
VR research. Conference organiz-
ers have joined hands to provide a
unique opportunity to showcase the
latest in virtual reality, mixed reality,
and 3-D user interfaces. This event
is a unique opportunity for research-
ers, developers, and users; it is only
the third time both conferences
have come together. For the more
practically inclined, there are poster
presentations and demos on the
schedule. Strategically organized
just before SIGGRAPH Asia 2015,
this will be a great chance to attend
both the events on a single journey.
Hosted in Kyoto—a city with a
unique blend of Japanese history,
culture, and academia—conference
attendees will have a wonderful time
exploring Japan’s former capital city.
For more information, please
visit http://www.ic-at.org/2015/.
—Darshit
— Patel
71
end

BEMUSEMENT

Ontological Argument 2014

http://xkcd.com/1505/

Psychic
http://xkcd.com/628/

Puzzles: Birthday Fun

When asked about his birthday, a man said:
“The day before yesterday I was only 25 and next year I will turn 28.”
This is true only one day in a year, when was he born?
Find the solution at: http://xrds.acm.org/bemusement/2015.cfm
http://xkcd.com/1311/

SUBMIT A PUZZLE
Can you do better?
Bemusements would like your puzzles and mathematical games
(but not Sudoku). Contact xrds@acm.org to submit yours!

72 XRDS • FALL 2015 • VOL.22 • NO.1

 CONNECT WITH OUR
COMMUNITY OF EXPERTS.
www.computingreviews.com
Association for
Computing Machinery
They'll help you find the best new books
ThinkLoud and articles in computing.

Computing Reviews is a collaboration between the ACM and ThinkLoud.

 CAREERS at the NATIONAL SECURITY AGENCY

EXTRAORDINARY WORK
Inside our walls, you will find the most extraordinary people doing Computer/Electrical Engineering
the most extraordinary work. Yet it’s not just finite field theory, Computer Science
quantum computing or RF engineering. It’s not just discrete Cybersecurity
mathematics or graph analytics. Information Assurance
Mathematics
Foreign Language
It’s all of these and more – rolled up into an organization that leads Intelligence Analysis
the world in signals intelligence and information assurance. Cryptanalysis
Inside our walls you will find extraordinary people, doing Signals Analysis
extraordinary work, for an extraordinary cause: Business Management
Finance & Accounting
The safety and security of the United States of America. Paid Internships, Scholarships & Co-op

U.S. citizenship is required for all applicants. NSA is an Equal Opportunity Employer and abides by applicable employment laws and
regulations. All applicants for employment are considered without regard to age, color, disability, genetic information, national origin, race,
APPLY TODAY
religion, sex, sexual orientation, marital status, or status as a parent.

Search NSA to Download

WHERE INTELLIGENCE GOES TO WORK®