Electrical and Computer Engineering Dept.

Traditional Applications of VR
 Application
(training,
System architecture education, etc.)
 Types of VR Applications

 VR is having a resurgence in traditional and new markets;

 Traditional markets are medical, entertainment/arts/education and
defense/aerospace;
 Emerging markets are manufacturing, oil/gas exploration, data
visualization;
CyberEdge Information Services Survey 2002
 MEDICAL APPLICATIONS OF VR

 Are needed in order to better train medical students and doctors

alike;
 Allow errors to be made on virtual rather than real patients;
 Allow rare cases to be modeled and studied;
 Alleviate the animal rights controversy and the shortage of
cadavers problem;
 Permit more enjoyable and efficacious treatment for patients
(example rehab);
 Allow more realistic examination (board exams);
 Allow reduction in medical costs (less liability suits, cheaper
training – emergency response).
 MEDICAL VR – Virtual Anatomy

 Everett Koop, former US Surgeon General observed in 1993 that:

“The medical knowledge we have today is 20 times as great as
when I went to medical school, but in that time, the teaching
methods haven’t changed much..”
 Interactive anatomy teaching is needed since cadaver training is
non-repetitive and scarce;
 Visible human dataset is the first completely realistic model for
both male and female; Is a de-facto standard for algorithm
comparisons;
 It is however huge, so cannot be rendered in real time. Needs
model segmentation and decimation. Such models are sold by
Visible Productions Co. (http://visiblep.com/)
MEDICAL VR – Virtual Anatomy
MEDICAL VR – Virtual Anatomy
MEDICAL VR – Anatomic VisualizeR
Medical VR applications
Diagnostic
and triage

Anesthesia
(IV, epidural)

Surgery
(emergency,
MIS, open)

Rehabilitation
Training in Emergency Medical Response to Bio-terrorism
 Training in bio-warfare response is not practical with current
“live” exercise methods. No true casualties, very expensive to
organize – rarely done.
 VR allows casualties to “die”, is much cheaper to organize, can be
done locally, without requiring EMT personnel to travel away from
their township.
Real Response to Bio-terrorism Avatar in protective suit

(Stansfield et al., 2000)

Training in Emergency Medical Response to Bio-terrorism
 Training scenario – airport explosion with casualties and
propagation of Staphilococus Enterotoxin (germ warfare);
 Highly stressful “situation awareness” training where timing
counts. Tasks have to be done correctly and fast.

Simulation view of airport Propagation of toxin

(Stansfield et al., 2000)

Training in EMR - system block diagram

(Stansfield et al., 2000)

Training in EMR - system communication diagram

Adapted from (Stansfield et al., 1998)

Training in EMR -continued
 Intelligent objects help in grasping operations as the user/trainee
lacks dexterity (no finger position measurement);
 Intelligent objects can be queried by the simulation about their
status;

(Stansfield et al., 2000)

 Training in EMR -continued
 Virtual patient is driven by a state machine (finite state automata);
 It changes state based on disease progression, or as a response to
user’s input;
 Responses are specific to disease, example eye reflex is absent for
head trauma, skin color changes for pneumothorax, etc.

(Stansfield et al., 2000)

Training in EMR – Human Factors study
 Human factors evaluation of the simulator was performed in 1999
at Texas A&M University on 23 EMT subjects;
 All subjects were volunteers, and screened for simulation sickness,
medication and poor vision;
 Subjects had no experience with VR;
Training in EMR – Human Factors study
 Had to fill in a subjective evaluation questionnaire (score 1 lowest
– 5 highest)
Training in EMR – Human Factors study

Level of difficulty Satisfaction with

Vital signs handling of objects
(Stansfield et al., 2000)
Diagnostic of Prostate Malignancies
 Prostate cancer second leading cause of death among men – 25%
of patients die;
 Detection of palpable nodules through digital rectal exam (DRE) is
a strong indication of cancer (50% of palpable prostate nodules are
malignant);
 Prostate can be in one of four stages: normal, enlarged, incipient
malignancy (single nodule), advanced malignancy (cluster of
nodules);
 DRE is taught in medical schools in a primitive way (a turret with
rubber models that the student can palpate and blindly diagnose);
 Limited training on patients, no follow up, no online data on
student proficiency, no confidence after graduation, discomfort for
patient, etc.
Rutgers DRE trainer
 A prostate DRE trainer was developed at Rutgers using a
PHANToM, an SGI High Impact workstation, and a mechanical
board to restrict movement

(Burdea et al., 1999)

Rutgers DRE trainer - continued
 The system was programmed using OpenGL for graphics and
GHOST for haptics;

(Burdea et al., 1999)

Rutgers DRE trainer - continued
 Prostate model was simplified to 200 vertices, with each
malignancy constructed of 100 vertices. Malignancies were detected
based on optimized vertex search approach, due to the 1000Hz
requirement of the PHANToM

(Burdea et al., 1999)

Rutgers DRE trainer - continued
 Virtual nodules could be placed randomly on the north-south, east-
west lobes of the prostate.

(Burdea et al., 1999)

DRE Human factors trials
 In order to ascertain efficacy as a trainer, a number of human-
factor trials were performed in 1998;
 There were three groups:
 First group was 22 non-medical student volunteers (16 male and six
female);
 Second group were 4 tired urology residents (after 24 hours
emergency room rotation);
 Third group were other volunteer urology residents from UMDNJ.
 Subjects in the first two groups used the PHANToM for force
feedback, while the third group (control) used traditional rubber
models (turret);
 Only subjects in the second group used the mechanical restricting
board.
 DRE Human factors trials - continued
 First subjects were trained for five minutes on the use of the
PHANToM and then they palpated virtual prostates that were visible
on the screen. The prostates were in wire-frame mode, such that
malignancies were visible;

 During examination mode the screen was blank (they had only
force feedback to base the diagnosis on);

Each subject was given 12 random cases to diagnose, with three-

minute rest periods between groups of four cases;

 Variables recorded were length of time to diagnose (sec.), the

diagnosis given, and the correct diagnosis;
 Diagnosis accuracy vs. case type

Subject Group Exact Diagnosis Benign/Malignant

(four cases) Diagnosis
Non-medical students 46% 67%

Urology residents 33% 56%

Control Group 92% 96%

(Burdea et al., 1999)

 Time to diagnose vs. case type (in seconds)

Subject Benign Benign Malignant Malignant

Group normal size Enlarged (incipient) (advanced)
Non-medical 55 s 60 s 48 s 54 s
students
Urology 53 s 45 s 39 s 48 s
residents
Control 9s 10 s 13 s 16 s
group

(Burdea et al., 1999)

DRE task learning
 Endoscopic Examinations
 Routinely done to detect cancer and other diseases; Done by
inserting a flexible viewing device (the endoscope), which projects
an image of the body on a monitor;

 Depending on the region of interest these procedures are called

“bronchoscopy” (for the lungs), “angioplasty” (for the circulatory
system), or “colonoscopy” for the colon.

 The are invasive procedures, require patient anesthesia, and can

lead to injury, if done badly;

 So endoscopic procedures require training, as well as a minimum

of procedures done annually, to maintain skill.
Endoscopic Examinations - continued
 One training system is the “PreOp” bronchoscopy simulator
developed by HT Medical (now part of Immersion Co.).

VC 8.1
 PreOp Study

 Study done to determine efficacy in training novice physicians;

 An experimental group of five physicians-in-training with no

endoscopy experience and a control group of four experienced
physicians with more than 200 endoscopic procedures each;

 The experimental group had 4 hours of training on PreOp; the

control group had only 30 minutes of familiarization with the
simulator;
 PreOp Study - continued

Then both groups had to inspect the airways of a normal virtual

patient.

 The variables measures were speed (duration of bronchoscopy),

dexterity (number of contacts with the virtual bronchial wall), and
accuracy (number of segments missed).

 The speed of experts was better, but their accuracy was poor (29%
missed segments vs. only 4.5% for the novices).

Results were confirmed on a mannequin (17% missed segments vs.

no miss).
 Endoscopic procedures – Virtual Colonoscopy

 Colonoscopy is a screening procedure that can save lives (90% of

colon cancers found early are treatable);

Advantages of virtual colonoscopy: Non invasive, does not require

anesthesia, is much faster (15 minutes vs. hours);

 An algorithm extracts the colon wall, performs “electronic

cleansing”, and then reconstructs the 3D profile of the colon;

 Another algorithm then allows a virtual camera to fly through the

reconstructed colon looking for polyps.
 Endoscopic procedures – Virtual Colonoscopy

The virtual camera can travel the whole length of the colon, which
is not possible with real flexible endoscopes;

 Has the same detection rate as real colonoscopy for polyps larger
than 5 mm.

 Found 3-mm polyps which were missed by regular colonoscopy

 Endoscopic procedures – Virtual Colonoscopy

Virtual colonoscopy
movie
www.viatronix.com
Medical VR applications
Diagnostic
and triage

Anesthesia
(IV, epidural)

Surgery
(MIS, open)

Rehabilitation
 Intravenous Procedures Trainer

 Insertion of IV needle is high-volume procedure which is also a

leading cause of infections and discomfort for patients.

 Present teaching methods are inadequate and nurses learn on

patients…

 This leads to low retention of nurses (high turnover rates) and

patient complications.
 I.V. Trainer

HT Medical developed the

CathSim simulator;
1-DOF needle simulator with
force feedback.

The case library has six types

of patients from drug user to
geriatric female, to infants.

VC 8.2
Medical VR applications
Diagnostic
(palpation)

Anesthesia
(IV, epidural)

Surgery
(MIS, open)

Rehabilitation
Open surgery - Anastomosys
 Open surgery has advantages and disadvantages; Requires skill
for suturing of two blood vessels – anastomosys;
 Boston Dynamics developed a pilot system for training in VR

(www.bdi.com/Anastomosis.html)

Stereo glasses
Half-mirror

PHANToM arm
PHANToM arm
Anastomosys - continued
 The blood vessels are modeled using the TELEOS spline-based
toolkit;
 Vessels are textured to increase realism;
 Trainee holds real surgical tools (tweezers, needle holder, etc) for
increased task realism. (www.bdi.com/Anastomosis.html)
Blood vessel being deformed Needle insertion task
 Anastomosys – Human factors study

 An experiment was conducted to compare the medical students

and vascular surgeons using the VR surgical trainer;
 The task consisted of repeated insertion of a curved needle through
a simulated vessel;
 The two groups were 12 students (Harvard Medical School), and 9
Boston area surgeons;
 Each group performed the task nine times;
 Results showed that the trainer was able to measure surgical skill
differences and thus may be a tool for (future) board/certification
examinations.
(www.bdi.com/Anastomosis.html)
Anastomosys human factors - continued
 The variables used to gauge performance were:
 Time;
 Accuracy;
Screen view of the trainer
 Angle error;
 Peak force;
 Tissue damage;
 Surface damage;
 Overall score.

(www.bdi.com/Anastomosis.html)
Anastomosys – human factors study - continued
 Surgeons did better than students by having less tissue damage;
 There was also less learning for the surgeons, and more uniformity
as a group. (www.bdi.com/Skills_test.html)

Medical
students
Tissue damage

Surgeons

Trial Number
Anastomosys human factors study - continued
 Individual task performance (normalized over surgeons
performance);

Surgeons
Students
Anastomosys – Human factors study
 Subjects learning process – exemplified by overall score;
(www.bdi.com/Skills_test.html)

Surgeons
Overall score (%)

Medical
students

Trial Number
Minimally-Invasive Surgery (MIS)

 Many current surgical procedures are done “minimally invasive”;

 MIS has advantages for the patient:
 less hospital stay (from one week to less than one day);
 less scarring (three 1-2 cm cuts);
 faster recovery.
 MIS has disadvantages for the surgeon:
 loss of direct (3D) view of the cutting area – has to look at a 2D
monitor;
 loss of tactile feedback, which is filtered by the laparoscopic
instrument (stick with a handle);
 the “fulcrum effect” due to the orifice in the body
MIS - continued

 The fulcrum effect cannot be compensated by increased attention;

 Requires training to automate the proprioception;
 Training should not be done on animals, much less on patients!

Handle motion

Handle

Body surface

Surgical tip
Reversed tip motion
 MIST VR
 The Minimally Invasive Surgical Trainer in Virtual Reality
(MIST VR) developed in UK is a computerized system to train and
asses MIS skills;

 In consists of a PC
coupled with an
Immersion
Laparoscopic Interface
(dual hand version);
 No force feedback is
provided
MIST VR

 The simulation consists of six 3-D manipulation/cutting/burning

tasks of graduated difficulty;
Task time, motion length, errors, are measured transparently and
available remotely to the instructor. (www.vrweb.com/docs/news/mist.htm)
Real laparoscopic task (tissue burning) Equivalent MIST VR task
MIST VR - GUI
(www.vrweb.com/docs/news/mist.htm)
MIST VR - Evaluation

The goal is to have objective measures of surgical skill, and to train

MIS surgeons in an uniform way.
 MIST VR was evaluated to determine:

 Does training on MIST VR make a difference?

 Does it work as a skill evaluator (with possible use in surgical

license examinations)? I.e. is the system sensitive enough to detect
skill differences between experienced and inexperienced subjects?
 MIST VR - The Gallager study

 Used two groups of novice MIS surgeons, with eight subjects

each;
 The experimental group got training on MIST VR – doing two
complete (6 tasks each) sessions in 24 hours. The control group got
no training;
 Following training, both groups were evaluated on their surgical
skills on a non-VR (real) task. Task consisted of cutting correctly a
pattern of lines on a piece of paper, using the laparoscopic
instruments (with fulcrum effect);
 There were 10 trials of 26 cuts each; (Gallager et al., 1999)

Correct cut Incorrect cut

MIST VR - The Gallager study

30 Adapted from (Gallager et al., 1999)

25 Group with MIST VR training

Mean number of correct incisions

20

15 Group without training

10
Maximum number of correct
incisions in a trial =26;
5 STD in Trial 1 was 1.75/3.16;
STD in Trial 10 was 1.99/3.16)
0
Trial 1 Trial 10
 MIST VR - The Taffinder study

 Done at the Imperial College of Medicine in London looking at

the use of MIST VR as an evaluation tool;
 Three experimental groups of 10 subjects each consisted of:
 experienced surgeons (> 100 laparoscopic cases);
 trainee surgeons;
 non-surgeons;
 All groups were put on MIST VR and performed the first five
tasks as training;
 The last MIST VR task, requiring two-hand manipulation was then
used to assess surgical skill.
(Gallager et al., 1999)
The Taffinder study - continued

Quantity Experienced Trainee Non-

Complex task Surgeons Surgeons Surgeons
Efficiency 2.30.3 3.3 1.3 3.6 1.0

Errors 4.8 0.8 6.3 2.3 7.3 2.2

Sub-movements 10.4 3.4 15.9 4.4 18.5 6.1

Time taken (sec) 13.7 2.6 19.3 5.4 21.9 6.7

(Taffinder et al., 1998)

 MIST VR - continued

 It is in current use at the European Surgical Institute (Germany);

 In 2000 over 10,000 trainees took classes there, and reported a
30% improvement in basic skills after using the MIST-VR trainer;
 Most reported that haptics needed to be added
 Laparoscopic Surgical Workstation
 Immersion Co. has recently introduced the “Laparoscopic
Surgical Workstation,” with force feedback for insertion, pitch, yaw,
handle twist and handle grip;
 Works with a PC, being connected over a PCI card.
 Has more realistic graphics (through “LapSym” software).
http://www.immersion.com/medical/docs/LSW_data_sheet.pdf
 Trauma Center simulation using the Wii

Series of surgery games produced by ATLUS (www.atlus.com) for

Nintendo game systems. Trauma Center: Under the Knife” for DS,
“Trrauma Center: Second Opinion” for the Wii.
 Tools used in the game
The Wii Remote allows for precision controlling of the tools. The action is also felt with the
built-in Rumble feature. Each surgical tool is a different direction on the Control Stick. The
player operates with both hands, saving critical time in healing patients. Dialog is missing and
graphics could be improved.
Stitches: use the needle to suture
Surgical Laser: the laser incinerates incisions and other delicate areas.
tumors and viruses.
Bandages: end an operation by applying
Scanner: it's used to find concealed tape to the sutured area.
tumors or to magnify affected areas.

Syringe: this is necessary for injecting

Antibiotic Gel: a potent medication
used for disinfection that can also
heal small wounds.
Scalpel: a blade to make incisions
and excise/remove tumors and
other objects.
various medications.
Forceps: extract harmful materials and
pick up delicate objects like synthetic
membranes.
Hand: take a hands-on approach to
various actions, such as heart-massage

Drain: a long tube for removing and membrane application.

fluid from the area of operation,
primarily blood.
Medical VR applications
Diagnostic
(palpation)

Anesthesia
(IV, epidural)

Surgery
(emergency,
MIS, open)

Rehabilitation
Orthopedic rehabilitation- Ankle

 1.3 million people visited emergency rooms in US in 1998

because of ankle problems;
 1.2 million visits to physicians’ offices for ankle sprains
and 675,000 visits for ankle fractures;
 25,000 people sprain their ankle every day;
Source: American Academy of Orthopedic Surgeons
(http://orthoinfo.aaos.org/fact)
Orthopedic rehabilitation- Ankle

The Rutgers Ankle

rehabilitation system
(2000)
Orthopedic rehabilitation- Ankle Remote monitoring
The Rutgers Ankle station
rehabilitation system
(2002)

Dual platforms

VC 8.3 Telerehab
 2003-2004 study
• 8 patients chronic post-stroke exercised on the
Rutgers Ankle for 12 sessions (4 weeks);
• The first three weeks the therapist was present in the
room. The last week the therapist was in a different
room, controlling at a distance.
Ankle movement repetitions Movement accuracy
 2005 study at Harvard

• 4 patients chronic post-stroke exercised on the Rutgers

Ankle, 2 with VR, 2 without VR (acting as controls);
• Significant differences in walking speed and distance
walked in a 6-minute walk endurance test (Mirelman et
al., 2006)
 Stroke rehabilitation
 There are 4 million Americans that survived a stroke, and
500,000 are added each year. In the chronic phase they do not
receive rehab;
 Holden and colleagues at MIT developed the “teaching by
imitation” method of rehab;

 Tests on 9 patients
underwent 30 one-hour rehab
sessions, three times/week;
 Significantly higher shoulder
flexion and grip strength post
VR-rehab.
Stroke patient trials at Rutgers (2000)

 The group at Rutgers/UMDNJ/NJIT developed game-like

exercises;
 Each VR-based rehabilitation session consisted of four
exercises of N trials each. Each individual exercise concentrates
on one particular parameter of hand movement:
 RANGE
 SPEED
 FRACTIONATION (Individual Movement)
 STRENGTH

(Jack et al., 2000)

Stroke patient trials at Rutgers (2000)

VC 8.4 VC 8.5
Stroke rehabilitation - Rutgers

Type of exercise Finger target

Patient’s hand
Trial number
Finger actual
The good news...
 VR Psychological rehabilitation

 Used to treat eating disorders, post-traumatic stress disorders, and

phobias;
 Patients with phobias may be afraid of heights, confined spaces,
animals, speaking in public or fear of flying.
 Classical treatment of phobias is “exposure therapy” in which
patient gets controlled amounts of stimulus, in order to desensitize;
 Classical treatment is expensive, risky, and poses problems with
patient’s confidentiality.
 VR therapy has advantages:
 on a PC in doctor’s office (privacy);
 safer (like fear of spiders or snakes);
 cheaper (no need to take patient on an actual flight).
 VR Psychological rehabilitation

 Fear of flying therapy developed at Georgia Tech and Emory

University;
 Uses a PC and HMD; The software has a state machine;

Fear of flying
 VR Psychological rehabilitation
 Patient controls progression through a set of progressively harder
situations: taxiing, take off, level flight, aborted landing and landing;
 Controlled study on 45 subjects showed VR to be as effective as
standard exposure with a 92% retention of gains after a year.
 VR Cognitive Rehabilitation
 For patients with Alzheimer’s disease, traumatic brain injury, and
children with attention deficit/hyperactivity disorder;

 For AD/HD trackers are used to register

restlessness. 3-D sound is provided as
distractions. Subjects see a virtual classroom.
A study showed that the system can diagnose
ADHD.
 VR for Post-Traumatic Stress Disorder (PTSD)

 It is found is war veterans as well as those exposed to acts of

terrorism (such as the World Trade Center destruction)
 Studies at USC use “Full Spectrum Warrior” game engine to adapt
it for training of Iraq veterans
 Research in New York at the Cornell Medical Center train
survivors of 9/11

Full-spectrum
warrior
World Trade Center
 VR Education Applications
 Learning is “constructivist,” “constructionist,” and “situated.”
 Constructivist learning involves exploration of pre-built world;
 Example is CyberMath developed in Sweden for college students
 VR Education Applications
 Another example of exploration-based learning is the Virtual
Physics Laboratory to teach high-school students Newtonian and
Quantum Physics;
 Students interact with the simulation using a sensing glove and
virtual control panels. Students are more motivated and understand
3D concepts better compared to other methods of instruction.
 VR Education Applications
 Constructionist learning involves active building of a world;
 An example is the NICE project to teach gardening to elementary
school children; they used a CAVE and ImersaDesk to interact with
VEs showing a virtual garden.
 VR Education Applications
 The system was tested on 52 second-grade students;
 They were divided in groups with a leader that interacted with the
simulation; the garden had avatars to represent team leaders and
intelligent agents (plants).
 The percentage of children understanding gardening concepts went
from 12% (before) to 35% (after), most being the team leaders.
 VR Arts Applications

 VR is a medium for artists to create in;

 It is also a new way to experience and “explore art” – very
useful for art historians;
 It allows “preservation of cultural heritage” in VR;
 It increases access to art for people living in remote locations,
or for the handicapped through “virtual museums.”
 Virtual Florentine Pietà
 Original statue created by Michelangelo in the XVI-th Century
as his tomb monument;
 Recreated in VR by a team at IBM over a period of two weeks,
using stereo scanners and co-registered color digital photos to
produce a mm-level of detail.
Real statue Virtual statue
 VR Arts Applications
 A 3-D viewer (low resolution) allowed interactive frame rates
on a laptop – for art historians;
 Views not possible in a museum;
 The statue was “restored” in VR – adding missing parts;
 It was placed in context – in a virtual mausoleum.
 Virtual Heritage

 Groups replicas of famous archeological, architectural and natural

sites; In 2000 the Virtual Heritage Network
(www.virtualheritage.net) was formed as a UNESCO affiliate.
 The recreation of a virtual heritage model starts with architectural
plans, historical documents and visit to the actual site to acquire
photos (used later for textures).
Light measurements need to be made for input to radiosity
computations.
 Care for cultural sensitivities, as models are placed online..
 Virtual Heritage – SS Sergius and Bachus (Istambul)
 Created by a group at University of Geneva;
 Consists of a 18,000 polygon exterior model and a 59,000 polygon
interior model created using 3D Studio Max; Uses light maps.

Model viewed in VRML 97

 Virtual Heritage – SS Sergius and Bachus (Istambul)
 The VRML viewer assured 30 frames/sec with 2-cm geometrical
accuracy;
 A custom VRML extension (to allow multi-texturing) was
subsequently used to increase realism, but frame rate dropped to 10-
26 fps.

Custom VRML 97
 Virtual Heritage
 Virtual
Notre Dame –
the best
known church
in Europe
(took 200
years to
complete).
 Modeled
using the Epic
Unreal engine
(normally
used in game
creation).
 Virtual Notre Dame
 Virtual tours with a guide who is a friar avatar (1200 textured
polygons). Had predetermined motion sequences.
 An AI engine drives his behavior, based on input from proximity
sensors placed at various locations inside the virtual cathedral
 VR Entertainment Applications

 PC games pose a serious threat to the traditional arcade-based

VR entertainment applications;

Adventure games – Lara Croft Strategy games - StarCraft®

 How about haptics and wind and ambient light?

Ambient LED
light effects

(amBX, Phillips,
2007)
computer-
controlled fans

Vibrating wrist
support
 Games are migrating to cell phones
 Mobile phones have cameras, which means that
GestureTek’s vision gesture recognition runs on them
too.
 “Tilt a World” is the fastest selling video game for cell
phones. It was developed by a team headed by Manjula
Kuttuva (former researcher in my lab at Rutgers).
 VR Entertainment Applications
 Disney has constructed Disney Quest – a building full of arcades;
 Allow feedback effects not possible at home, and multiplayer
games. An example – “Pirates of the Caribbean” – motion platform;
Passive tactile feedback, stereo graphics (SGI);
 VR Entertainment Applications
 Disney also developed the “Virtual Jungle Cruise” on inflatable
rafts. Pneumatic motion platform, sensorized oars and water sprays
add to the feeling of immersion.
 VR Military Applications
 The military has traditionally been a proponent of VR since it
offers many advantages:
 Ability to train remotely as a team (SIMNET) – reduced
transportation/housing costs and reduced environmental impact;
 Aircraft simulators that are programmable and modular, and allow
a shorter life cycle;
 Ability to simulate missions ahead of execution, as well as an
advanced debriefing (after mission) modality;
 Ability to visualize enemy weapon capabilities in order to reduce
casualties;
 Ability to train individual solders in new weaponry, as well as
evaluate trainee performance;
Army use of VR

De-mining trainer

Standard military probe Head Mounted Display

HMD view

PHANToM arm
Army use of VR
Mockup of rocket launcher Head Mounted Display
Stinger trainer

Sensing switch

Stinger rocket trainer developed by TNO (Holland) in early 90s.

Army use of VR

The system is in use in Germany 1999;

Stinger trainer in Germany
It allows team training with two users, a
gunner and a commander;
 Each trainee wears a high-resolution
Gunner Commander
HMD (1280 x 1024) due to the task of
spotting airplanes remotely;
 Each trainee is tracked and their
position is mapped to avatars

(Reichert, 2000)
InterSense Stinger trainer

40 foot Dome (360 degrees horizontal and 70 degrees vertical)

A custom, high accuracy, inertial-acoustic tracking station is mounted inside
of the Stinger Missile tube with two holes drilled for the ultrasonic receiver
microphones.  The IS-900 SoniStrips are mounted under the center projection
tower to provide a 20 foot diameter tracking area
 Small Arms Trainer (CST 300D)
Configuration:
1. Standard LCD projector
2. Standard Hit Camera
3. Optional Night-Vision Filter
4. Optional Ceiling Mount
Joystick controls movement
during tactical scenarios.
Each PSC supports 8 weapons

Provides:
 Individual marksmanship skills training
 Team and squad level tactical training
 Enhances Instructor’s ability to analyze shooters’ performance
 Weapons & Devices
FATS utilizes live weapons converted into simulated weapons.
This provides students with the best, realistic training.
Rifles: M16, M203, SA80, MP5, SA80 - Isw

Machine Guns: M249 “SAW,” M240

Pistols: 9mm, Magnum, Sig, M9

Shotguns

Crew served: AT-4, M72 “LAW”

Kits: MK-19
Small arms A combined arms firing line.
Army use of VR – Platoon leadership training

Needed to improve decision making by young officers, in unusual situations;

Creates mission rehearsal exercises
 Example an accident between a military jeep and a civilian car in Bosnia;
 The only real character is the trainee…

VC 8.6
 Army VR Applications – Battalion-level simulations
 The Simulation Network (SIMNET), to train tank commanders in
a virtual battlefield; Uses dead reckoning, intelligent agents (Semi-
autonomous forces – SAFOR);

 Has stealth vehicles to

look at a battle from any
angle and at any time – like
a 4D history book
(used in the first Gulf War)
 Limited terrain database
50x75 km2
 Army VR Applications
 The Close Combat Tactical Trainer (CCTT) is a replacement of
earlier SIMNET, with a larger terrain database, several weather
conditions, and communication with supporting elements
 Example Aviation CCTT (or AVCCTT)
 Army VR Applications
 The Close Combat Tactical Trainer (CCTT) system consists of
fixed or mobile sites which are modular to allow various helicopter
missions to be simulated

AVCATT
 Navy VR Applications
 The VESUB project to train “officer of the deck”. Consists of a
station that recreates the command station, a suspended and tracked
HMD (CRT-based) which works as virtual binoculars, voice
communication, and AI agents. System is now in use at the
Submarine School in Grotton, Connecticut.

VC 8.7
VESUB Simulation

 The VESUB system was tested in 1998 on 41 subjects

participating in a 3-hour simulation (familiarization, training,
examination);

 Results showed significant task learning on skills like

o issuing correct commands;
o checking range markers;
o using correct commands during emergencies (like man overboard).
 Most learning occurred in novice users (Officers of the Deck);
 Experienced users had no learning (or negative learning), probably
due to conflicts with prior mental image or due to lack of interest.
Naval Artillery Trainer in the British Royal Navy

 Needed to alleviate environmental concerns and to save on

training costs

Close-range naval gun

Traditional shore-based trainer

Naval Artillery Trainer

 System consists of tracked HMD, real gun with sensorized trigger,

and a PC running the simulation. There is also a commander station.
Naval Artillery Trainer

 A third PC is dedicated to the Instructor’s Station System was

placed in service in 2001

Gunner Director view Scenario Control Interface view

Air Force Applications

 Need for smaller, more mobile trainers. Two approaches: The A10
trainer and the Mako virtual cockpit.
 A10 uses real cockpit and side-by-side displays;
 Mako uses HMD, and virtual instruments, glove interaction.

A10 trainer Mako virtual cockpit

Air Force Applications

 Needs to allow remote team training: distribute mission training

(DMT) trains in “dissimilar Air Combat Tactics”