You are on page 1of 8


Ben Newbon
Bsc Computer Science
2 Bowden Lane

For many, virtual reality has been an intriguing topic
since its conception and has produced many
inventions and ideas that have changed the way
humans live their lives. Devices have developed at
a rapid rate over the years starting from the simply
primitive progressing right up to the latest
technological advancements. This paper looks at
how people interact with virtual environments and
how this might change in the near and distant
future as well as how it is changing right now. It
also briefly analyses what has affected the
development of virtual interfaces and why certain
projects have been unsuccessful in the past.
Virtual Reality, immersion, input, motion sensing

The mention of virtual reality often conjures up
images of chunky headsets and large simulators,
though many people do not realise that they may be
using virtual reality everyday. Every time someone
sits at a computer and points with the mouse or
plays with a games console they are interacting
with a virtual environment. For many years children
and teenagers have flocked to amusement arcades
and shot light at a screen or screamed around a
race course by pushing on pedals and turning a
wheel. Now the technology is very much in the
home. A Nielsen Entertainment survey found that
40% of US homes own a PC, game console or
handheld gaming device. Almost a quarter of these,
23%, own all three types of gaming gadget and the
vast majority of gamers, 89%, do their playing via a
Permission to make digital or hard copies of all or part of this work

in-home technology has consisted of using an input

device such as a keyboard or handheld controller
with which the user presses buttons or moves
joysticks and the output is displayed on a screen. In
recent years however, this has been changing.
Console gaming has brought into the home new
technology that immerses the user further into the
virtual environment and this is set to change further
in the imminent future.

2. Background Work
When a person interacts with a virtual environment
they instantly become partially immersed within it,
be it by looking at a screen wherein they are
moving a character with a joystick or wearing a
head-set walking around a virtual world. Complete
virtual immersion would be achieved once the user
is completely unaware of the real world but only the
virtual environment. Though the technology for this
is yet to be anywhere near developed partial
immersion has existed for years, even in the home.
Partial immersion can take several forms and
varying degrees:
Input devices such as keypads and
joysticks are one of the most basic and
minimal forms of immersion on their own
but can be modified to increase this.
Modern gaming consoles tend to use a

for personal or classroom use is granted without fee provided that

are that
not many
made people
or distributed
for profit or
So now
are interacting
advantage and that copies bear this notice and the full citation on
environments, how are they doing it? Most popular
the first page. To copy otherwise, to republish, to post on servers or
to redistribute to lists, requires prior specific permission.

Annual Multimedia Systems, Electronics and Computer Science,

University of Southampton








Figure 1: Using a head mounted display and wired

combination of buttons and joysticks to

maximise input combinations though some
developers are moving back towards more
simple designs to increase accessibility.
This paper will focus on input devices that
enhance virtual immersion and the
technologies that they use.
Output devices such as monitors are the
most basic visual immersion but other more
immersive technologies exist like head
mounted displays and there are also some
that provide haptic feedback, although this
is mostly limited to vibrations.

The first actual video game is believed to have

been created around 1947 by Goldsmith and Mann
using cathode ray tubes with knobs to vary resistor
positions altering the representation of a missile's
path to a target.[14] In 1958 William Higinbotham
created Tennis for Two, the first ever multi-player
video game, which used both a knob to control ball
trajectory and a button to launch the ball over the
net.[2] The same format was used even into the

Hybrids have been created that combine

input and output functionality such as
vibrating controllers.


A decade before video games were invented there
was already a technology being used that has
become highly influential in modern gaming
systems. The light gun was created in the 1930s
appearing first on the Seeburg Ray-O-Light at
amusement arcades in 1936 and consisted of a
light-beam emitting gun and moving targets with
light sensitive tubes on that would fall if hit.[6] This
same idea has been used on many subsequent
video games systems in arcades and home
consoles though usually reversed so that the
sensor is on the gun and light emitted from targets
on the screen.

Figure 3: Tennis for Two - one of the first video


1970's with the Magnavox Odyssey and Atari

systems leading to the world-renowned PONG.
The Atari, being the main console of the early video
gaming era pioneered many of the early input
devices, the concepts of which are still used today.
Primarily, the input devices for the Atari were: a
paddle - a small cylinder that is rotated to move an
object on the screen along one axis; a trackball
like a paddle but spherical, providing movement in
2 axes; a joystick a control column, which can be
moved to produce continuous movement in 2 axes;
and the origins of the gamepad.[8]


Figure 2: The Seeburg Ray-O-Lite machine predecessor to the modern light gun

Another early control development was the joystick,

ideas for which were taken from the airplane and
similarly to the light gun this was originally used at
fair grounds and arcades in non-video games for
instance to control a grabber that picks up prizes. It
has also become an important idea in almost all
recent games consoles.

Between the release of the Famicom in 1983 and

the Playstation in 1994 the main controller format
was a hand-held device, called a gamepad, with 8
to 14 buttons arranged around it or various forms of
modern light gun. The next major development in
input devices was with Nintendo's fifth generation
console the N64, which was the first mainstream
console to feature button and joystick input as
standard. The configuration consisted of a small
central joystick, trigger button, two sets of
directional buttons and an extra three function
buttons. Nintendo's N64 was also the first console
to offer a controller with an add-on, the rumble-pak,
to provide haptic, the sense of touch or feeling,
feedback in the form of vibrations.[22] This was

further developed on by Sony with their dualshock

controller, released in 1997, which featured two
small joysticks, which they called analogue pads,
as well as the 14 buttons in the original
arrangement and new built-in haptic vibration

CLUSTER switches (which are a part of the NES

standard controller). In lores mode the glove reports
position on the hand on the x and y axis and the
buttons (thus emulating a NES controller
completely and allowing one to use the glove with
non-glove-specific games). [13]

4.1 Failed Input Projects

Unfortunately, though it was compatible with many

games, the lack of specifically glove related games
and no official hardware or software support from
Nintendo caused the developers to go bankrupt.
However, overall sales grossed US$88 million since
1989 and the glove will be remembered by many

4.1.1 The Power Pad

Nintendo released an unsuccessful predecessor to
the now popular dance mat for the Famicom
called the Power Pad. This consisted of a fabric mat
with twelve pressure sensors embedded in it that
the user stood on to activate.[7] This was largely
ignored by developers and only one game was
created specifically for it resulting in it being a
general commercial failure.
4.1.2 Virtuality
A system was attempted by one company known as
Virtuality in 1991 whereby the player stood in an
immersive platform with a head-mounted display
and joystick. This was extremely expensive and a
commercial failure however similar systems have
the potential to be developed in the future as
development prices drop and consumer interest
moves on from traditional systems.[3]

4.2 Successful Input Projects

There have been many attempts made over the last
couple of decades by several console developers
as well as third part developers to create novel
input devices and introduce new ways to interact
with games; however only select ones have seen
4.2.1 The Power Glove
The first really significant new development, though
it never became mainstream, was the Power-Glove.
This incorporated the normal control devices on the
back of a glove as well as incorporating
rudimentary tracking devices and finger bend
sensors using a special conductive ink. The
creators credit it as being the first peripheral
interface to recreate human hand movements in
real-time on a TV or a computer screen.[1]
It has two modes "hires" and "lores". In "hires"
mode, the PG reports the position in threespace,
the roll, and configuration of fingers along with the

Figure 4: Mattels Power Glove for use with the

Famicom/Nintendo Entertainment System

people as a milestone in VR interaction, despite it

being based on VPLs dataglove.[10]
4.2.2 The Dance Mat
This evolved from the original Power Pad for the
Famicom and is technically the same concept. A
flexible mat is placed on the floor with pressure
sensors corresponding to the controls for the
specific console: versions are available for the
Playstation, Gamecube and X-Box. The player is
required to step on the pads to produce actions on
the screen. This has been much more of a success
than the Power Pad owing mostly to the clever
marketing and fun games created for it, mostly
centred on dancing.
4.2.3 The Steel Battalion Controller
The most ambitious project undertaken and
successfully released in recent years was the
special controller for Microsofts X-Box console
game Steel Battalion. The game could not be
played without the controller and cost over 130

posing a massive loss risk for the developers. The

cost is justified by the shear massiveness of the
controller and the game achieved a collectors
status since only around 2000 units were made
worldwide. The main unit of the controller consists
of 3 boards with forty buttons and two joysticks
spread between them as well as a foot pedal. The
concept behind this was to create a gaming
environment to further immerse the player, so they
would feel like they were actually controlling a large
machine. However, due to the single minded nature
of the controller and limited availability this has not
had a large impact on industry.

Sega also released a larger equivalent, called the
Sega Activator for the Genesis console, which
consisted of an octagonal ring placed on the floor.
The player stood inside the ring with infra-red
beams from each section interpreting the
movements when interrupted. However, this was
also a commercial failure due to being very difficult
to perform accurate commands with.[20]
A central problem contributing to the failure of these
early attempts to create motion sensitive input
devices was the lack of industry support and
collaboration. There were few games developed
with either of the above devices in mind and
interpreting movements into actions in normal
games did not translate well. Modern motion
sensing projects show more promise, mostly due to
clever marketing and industry support but also
because recent technical advances have radically
brought down the price and the size. [9]

5.2 More Recent Motion Sensing


Figure 5: The enormous controller system for the XBoxs Steel Battalion game.


5.1 Early Motion Sensing Attempts
Atari can be credited with producing the first ever
motion sensitive control system, for the Atari 2600
in the 1980s. The controller looked like one handle
from a motorbike with a button on the top. It used
mercury switches to detect movement but as there
were never any three dimensional games for the
Atari, it only ever produced movement along 2 axes
like any other Atari joystick.[8] Unfortunately the
downfall of Atari consoles to Nintendo and Sega
meant this concept was never followed up.
The U-Force was by far the most ambitious project
made for the Famicom and could have been the
beginning of an entirely new way to interact with
games. It looked like a battleships board which
featured two infra-red sensors and several switches
used to assign inputs to certain hand movements
across the sensors. Unfortunately, it was
expensive, poorly implemented and never gained

In 1998 Microsoft launched a new addition to it's

Sidewinder PC controller range, the Freestyle Pro,
which featured motion sensing capabilities.
Although it was compatible with many PC games it
was most suited to motorcycle or racing games.
Reviews from the time suggest this limited use was
detrimental to sales and prevented the controller
spurring on future development.[12]
The short-lived Sega console, the Dreamcast,
boasted the first handheld motion sensing controller
in the form of a fishing rod. It is able to detect three
basic movements of casting back, casting, and
pulling up as well as providing haptic feedback as if
the fish is actually struggling against you. Though it
is incredibly difficult to find any technical
specifications on the devices used, it most likely
features at least one accelerometer with at least
two dimensional capabilities. The potential for
similar devices to develop was huge but were
ignored due to the narrow scope of fishing games
and general failure of the console following the
release of the Playstation 2.

5.3 The Latest Technology

Since Sony's dualshock controller there has been
little progress or change in the design of input
devices until recent months. Several new seventh

generation consoles have begun to change the way

users interact with their systems immersing them
further into the virtual environment.
5.3.1 Wii
The Wii is Nintendo's latest creation set for release
in Europe on 8th December. It is setting new
precedents for console input devices by using a
controller that knows its location in three
dimensional space. It combines both a pointing
device and motion sensing technology to produce a
controller that can be used in many situations and
to an enormous variety of effects.
The main controller which Nintendo call the Wii
Remote looks just like a normal television remote
but has the option of adding extensions for various
functionalities. It features a directional keypad as
well as 6 buttons, haptic feedback and controller
audio; but its main assets are:

Motion sensing The remote makes use of

multi-axis linear acceleration information
provided from the ADXL330, an integrated
Micro Electrical Mechanical System
accelerometer. There are many types of
accelerometers but this particular one
makes use of microscopic silicon discs and
with sophisticated signal
conditioning similar to the one used in new
Toshiba tablet PCs but providing three axis
information rather than just two.[9][23]

Pointing functionality A sensor bar emits

an infra-red field out directly in front of the
TV. In order to accomplish this, it is
necessary to place the sensor bar in the
same plane as the TV, either on top of or
below the TV is recommended. As the
remote is pointed towards the TV, it
interacts with the infra-red field. Using
triangulation logic, the remote is able to
determine location, angle, and distance; as

you move the remote around, the change in

location/angle/distance is calculated.[11]
All this motion and pointing information is
transmitted to the main console together with any
buttons that have been pressed via bluetooth
wireless technology.
The Wii remote features an expansion port at the
back which can accommodate a selection of
extensions. The most notable of these extensions is
the Nunchuk controller which as well as having its
own accelerometer has extra buttons and a small
joystick providing further functionality to the remote.
Other currently announced extensions include: a
classic controller that is used to play old Nintendo
titles that are incompatible with the format of the Wii
remote; and a zapper, into which the remote slots
and has the appearance of a light gun but uses the
pointing function instead of light from the screen.[5]

Figure 7: Who knows? The Matrix style virtual

immersion may not be all that far away.

5.3.2 Playstation 3

Shortly after Nintendo announced its plans for the

Wii remote Sony announced that its controller for
the Playstation 3 would feature similar functionality.
Apart from looking almost exactly like a wireless
version of the previous dualshock controller, Sony
boasts that it has six degrees of freedom whilst
downplaying the lack of haptic feedback. This new
controller has been dubbed the Sixaxis to signify
the fact that it can sense rotational and translational
acceleration in all three axis of direction, a total of
six axis.[21]
Although Sony have released no official information
regarding the motion sensing technologies behind
the Sixaxis, it appears to involve some form of tilt
sensor; similar to the accelerometer in the Wii
remote but without the infra-red positioning system

Figure 6: The Wii Remote being used with Nunchuk


of the Wii.


It seems plainly obvious that the console industry is
experiencing a paradigm shift with regards to input
devices and game play interactivity.[9] The general
reception of the latest consoles indicate that the
public is finally ready for the next step in interactivity
and virtual immersion but where is this likely to go
in the future?
As with many technologies, a large helping factor is
military involvement. Virtual reality is used by the
military to train personnel, help reduce information
overload, increase understanding of military
processes, and to test and improve new
attempts have been made to create full body suits
that would be like an extension of the Power Glove.
The best example of this was from VPL who
created datasuits, and eyephones amongst
other novel devices.[10] These projects have been
expensive and although the ideas are alluded to in
media and films, they never seem to get off of the
ground. Now an American military engineering
group is investigating creating a body suit for use in
virtual environments to train pilots.[19] The ideas
they are using involve placing the pilot in a
magnetic field and the suit, consisting of many
small stud-like electromagnets, pushes or pulls
against the field when activated. The proposed suit
is mostly planned as an output device so that
trainee pilots receive realistic haptic feedback whilst
using a flight simulator; however there is great
potential for this to be converted to a powerful input
device in the future.
Also connected to pilots is the development of head
mounted display units, known specifically as head
mounted sight systems. Until recent years these
were simply representations of screens close to the
users eyes on a helmet. Now pilots are able to
control missile directions simply by looking in a
particular direction and having the information
passed through sensors to a computer.[24]
One of the most promising, albeit for some slightly
worrying, possibilities is not too far from the vision
popularised in the film The Matrix of people
having virtual environment data passed straight into
their brains. Sony have recently patented an idea

for transmitting data directly into the brain using

ultrasonic pulses to alter neuron impulses and
create false senses. This is however, only in the
theoretical stages of development and Sony claim
that they have not performed any experiments into
the idea yet.[15]
There has been other research into non-invasive
ways to manipulate brain impulses using magnetic
fields;[17] however this has shown to be inaccurate
and potentially unsafe. Invasive methods might in
the future include cybernetic implants although this
will obviously not be popular if non-invasive
procedures are available.
There have not yet been any solid ideas put forward
for this method of implementation to be used as an
input mechanism but logically this would be the

Figure 8: A demonstration of the helmet mounted

sight system

next step from an output system.

The concept of virtual reality has, in terms of
history, only been conceived fairly recently. Its
origins are in the simple mechanical games that
developed in fairgrounds using pulleys and levers.
These had a sense of interaction with physical user
input producing physical responses in the
machinery. Since then technology progressed

inspiring the creation of cathode ray signal

processing by turning knobs and the birth of two
dimensional video games. Combinations of visual
and aural output spurred on gaming development
introducing buttons and joysticks as input
controlling software developed images on screens.
Now wireless motion sensitive control systems
produce visual, aural and haptic responses in fully
three dimensional environments. Each day
technological developments take place that will
increase people's usage of virtual environments
and how they interact with them. The possibilities
for virtual immersion in the near or distant future
are endless and complete immersion may be closer
than anyone expects.

Nintendo Wii. AnandTech. 2006.


16. Michael, Ted. U-Force.

Development Company. 2000.

1. Abrams Gentile Entertainment. Powerglove. .
2. Anderson, John.

Who really invented the

video game?. Creative Computing Magazine,


no.10. 1982.

3. Atlantis
1000CS. 1996.
4. BBC. Games overtake music for US men.
5. Bozon, Mark. E3 2006: Light Gun Shell
Revealed. IGN. 2006.
6. Bueschel, Richarm M. Collector's Guide to
Vintage Coin Machines. Atglen, PA. Shiffer
Publishing Ltd. 1995.

Tennessee. Power Pad

Information. Gamers Graveyard. 1999.
7. Carmel-Veilleux,

9. Tiny springs keep Wii, PS3 under
onsole.controller.ap/index.html . Copyright 2006.
10. Encyclopdia Britannica. Virtual Reality.
13 Dec. 2006.
11. Farhan, Haidar. E3 2006: Playstation 3 and

12. Microsoft Sidewinder
age2.asp. 1999
13. Gardner, Dana L. The Power Glove. Design
News pp63-68. 1989.
14. Goldsmith, Thomas T Jr. Mann, Estle Ray.
assignors to Allen B. Du Mont laboritories.
Cathode-Ray Amusement Device (patent number:
2,455,992). United States Patent Office. 1947.
15. Hogan, Jenny. Fox, Barry. Sony patent takes
first steps towards real-life Matrix. New Scientist
magazine. 2005.


17. Mill, K R. Magnetic brain stimulation: a tool to

explore the action of the motor cortex on single
human spinal motoneurones. University Dept of
Clinical Neurology, Radcliffe Infirmary, Oxford,
UKSept 1991.
18. NOVA Science in the News. Virtual reality bytes
military uses of VR. Australian Academy of
Science. 2002.
19. Sample, Ian. Virtual-reality Body Suit Planned
for Pilots. New Scientist magazine. 2002.
20. Sega


21. Sony
AVAILABLE. Sony Press release. 2006.
22. Tyson, Jeff. How N64 Works. How Stuff Works.
23. Wisniowski, Howard. Analog Devices And
Nintendo Collaboration Drives Video Game
Innovation With iMEMS Motion Signal Processing
Technology. Analog Devices, Inc. 2006.
24. White,
Lewandowski, Ronald J. Haworth, Loran A.
Girolamo, Henry J. Rash, Clarence E. Helmet
Mounted Sighting System - from development into
squadron service.
SPIE proceedings series.
International Society for Optical Engineering
proceedings series. 2001

9. Figures Table

Figure 1. Carlson, Wayne. A Critical History of

Computer Graphics and Animation Section 17:
Virtual Reality. Ohio State University. 2005
Figure 2. Alias~ MrBigglesWorth3. 1936 Seeburg
Figure 3. Reimer, Jeremy. The evolution of gaming:
computers, consoles and arcade. Arts Technica,
LLC. 2005
Figure 4. Drehmer, Guilherme. Powerglove
Figure 5. TNL Developer
Spotlight: Production Studio 4. .2004
Figure 6.
e.jpg. 2006
Figure 7. What is the Matrix?
Figure 8. Kopp, Carlo. Head Mounted Sights and
Displays. Air Power International. 1998.