VIRTUAL AND AUGMENTED

REALITY
ECC4351
Module 4: HUMAN FACTORS

Methodology and terminology, user

performance studies, VR health and safety issues
Applications: Medical applications, military applications, robotics
applications.
 MEDICAL APPLICATIONS OF VR
• In recent years increased computer and Internet usage in medicine
has started to change the way health care is delivered.
• The power of computing allows online medical education, patient
databases, presurgery simulation, use of robotics, remote
consultation, digital radiography, expert systems, and so on.
• The resulting increase in the quality of medical procedures makes up
for increased equipment costs.
• Additionally, Internet computing facilitates patients' access to medical
care, either locally or in remote locations where specialists are usually
not available.
• Medical VR is a form of medical informatics, which has seen a steady
growth over the past decade
• VR brings numerous advantages to the medical community.
• These include improved medical training (errors made on virtual,
rather than real patients; modeling of unusual and rare cases), more
realistic certification procedures (objective measures of surgical skill,
for example) etc.,
 Virtual Anatomy
• VR is used to teach anatomy, as it is the basis of medical education throughout the
world
• The majority of current anatomy courses use textbooks and cadavers for
dissection. Unfortunately for the student, textbooks are becoming larger and more
expensive, while access to cadavers is harder to get.
• With the creation of the Visible Human, researchers and educators throughout the
world have necessary database in the development of realistic anatomy course
material.
• Input to the algorithm consisted of the 522 slices of the CT images taken on the
male cadaver. These slices were indexed, then a connectivity algorithm and a
marching-cubes algorithm were used to extract skin, organ, and skeletal structures
Creating animations of 3D organ models based on the Visible Human database is a precursor to developing a
computerized anatomy teaching curriculum.

Anatomical reconstruction of the Visible Man

• The Anatomic VisualizeR provides a virtual dissection room in which
students can explore various aspects of anatomy organized into
teaching modules (skull, ear, thorax, abdomen, and neuroscience).
• Each lesson allows a student wearing active glasses to view an
interactive anatomical model
• Students can create and manipulate cross-section views, measure and
identify structures, change the opacity of organs (for better view of
hidden components), draw lines, link modules, and so on
 Triage and Diagnostics
• Once anatomical models are developed, they can be modified to add
pathology such that students learn to diagnose abnormalities.
• An experienced practitioner will learn from his or her mistakes (on
real patients). Whether diagnostic skill is linked to simple palpation of
a patient or to more complex endoscopic exploration, it requires both
repetition and access to a sufficient number and variety of patients.
• Virtual reality can help by allowing repetition under monitored
conditions as well as the creation of a large variety of pathologies
(including rare cases).
• Diagnosis is also required for triage, a highly stressful process done on
the battlefield or in emergency rooms as a step preceding
administration of treatment.
• In an emergency medical situation tasks need to be done not only
correctly, but also fast.
• An extreme scenario is training to handle a large number of casualties
Emergency Medical Response to
Bioterrorism
• Such training is required in most large population areas such that first
responders learn to diagnose illnesses, administer first aid, or send
patients to specialized medical facilities.
• The current live training methods are inadequate due to high cost and
reduced realism.
• A case in point is the simulated casualty (a volunteer), who, unlike real
bioterrorism situations, never dies. Virtual reality can improve training
since casualty avatars can be programmed to exhibit realistic
physiological responses and if untreated, die.
 Endoscopic Examinations
• These are routinely done to detect cancer or diagnose other diseases
affecting various parts of the body.
• The procedures are performed by inserting a flexible viewing device
(the endoscope) in body cavities and viewing the images of the body
interior displayed on an adjacent monitor.
• Thus the procedure requires good reasoning and manual skills.
• Such skills are currently taught on mechanical models of the anatomy
of interest, on animals, or on patients.
• Later, if they do not perform a certain number of such procedures every
year, they lose these skills
• The foregoing discussion highlights the need for medical simulators that would allow
physicians to make errors without affecting real patients, would allow physicians to
resharpen their skills periodically, and could be used in medical certification.
• The system consists of a real fiber-optics flexible endoscope introduced in a robotic
interface that provides haptic feedback, and a PC running the VR simulation.
• The graphics consists of a textured anatomical model of the airway (based on the
Visible Man database), which responds dynamically to trainee's actions. For example,
the tissue deforms and the "patient" coughs if the tip of the virtual endoscope
touches the airway wall. At the same time, the trainee feels resistance due to the
actuators inside the robotic interface in which the endoscope was inserted. Other
scenarios, such as bleeding, obstructed view due to airway secretions, or the
presence of malignancies, are also simulated.
• Software running on the PC measures the trainee's performance in
terms of time taken for the procedure, errors (hitting the wall or
wedging), number of airway segments left uninspected, and so on.
These are compared with benchmarks obtained from experts
performing the procedure and help quantify the trainee's learning
and skill level.
 Surgery
• The use of VR in surgery affects a number of distinct areas.
• These include training anesthesiologists and nurses in intravenous
procedures, open and minimally invasive procedure training for new
surgeons, surgical planning of complex procedures, navigational and
informational aids during surgery, and prediction of operational
outcomes
• A surgeon training on real cadavers cannot repeat a given procedure
in case of a mistake, as the body organs have been altered.
• Simulators allow surgeons to learn by repetition
 Rehabilitation
• A patient discharged from a hospital following surgery usually has to
undergo rehabilitation in order to regain strength, range of motion,
walking ability, and so on.
• Rehabilitation is also prescribed for people who had fractures or a
stroke or who have cognitive or psychological deficits.
• Thus, depending on the patient population involved, rehabilitation
can be classified as orthopedic, neurological (post-stroke or traumatic
brain injury), and cognitive.
• Virtual reality is starting to be used in all these areas, even though it is
currently mostly in the pilot or clinical trial phases.
• Virtual reality can create an interactive, entertaining environment
(through the use of 3D graphics and video games) and allow
supervised rehabilitation at home.
• This type of rehabilitation, called telerehabilitation, gives a remote
therapist the possibility of monitoring (in real time or not) the actions
of a home-bound patient using the Internet.
 MILITARY VR APPLICATIONS
• The military has long understood the importance of simulation and training
under the doctrine "we train as we fight and we fight as we train.“
• The current trend toward increased technological complexity and shorter
military hardware lifespan requires simulators that are flexible, upgradeable,
and less expensive.
• Remote simulation helps to train without having to transport trainees to the
simulator site.
• Networking is also needed in team simulations, which are more realistic than
single user ones.
• Virtual reality is networkable, flexible, and easily upgradeable, therefore it
ideally matches the needs of military simulation
 Army Use of VR
• The U.S. Army was an early adopter of VR-based training in the form
of simulators for large-scale motorized units
• The current trend for small-scale conflicts (such as the so-called war
on terrorism) emphasizes, however, the need to better train individual
soldiers.
• Single soldier simulators, followed by platoon-level leadership training
is needed
 Single-Soldier Simulators
• This simulators are useful not only for improving marksmanship, but
also for better training in operating expensive weaponry
• Example : Virtual Stinger Trainer
• Another use of single-soldier VR simulators is in training for difficult
missions, such as mine detection and defusing.
• The French Defense Department has developed such a training
system using a PHANTOM interface to provide haptic feedback
The Virtual Stinger Trainer. De-mining training system
 Platoon Leadership Training
• This training is required in order to improve decision making for
young lieutenants commanding these small units.
• Increasingly they are faced with the need to take difficult decisions
under pressure, sometimes in circumstances not mentioned in
training manuals and in unfamiliar surroundings.
• Realistic, highly emotional training Mission Rehearsal Exercises
(MREs) developed by teams of artists and computer scientists are
created for this
 Illustration-MRE

Platoon leadership training for a peacekeeping operation.

 Company and Battalion-Level Simulators.
• A battalion is a military unit. It consists of 300 to 1000 soldiers and is divided into a
number of companies.
• The simulators are used to train armored units and their artillery (heavy military
ranged weapons) and close air support.
• In the early 1980s the Defense Advanced Research Projects Agency (DARPA) began
the development of the first realistic virtual battlefield in order to train tank crews in
joint exercises.
• The primary motivation for this endeavor was a reduction in training costs, but
considerations also included increased safety and reduced environmental impact
(explosions and tank tracks do great damage to the training area).
• The project, called Simulation Network (SIMNET), linked over 200 tank simulators in
the United States and Germany [McDonough, 1993].
Simulation Network (SIMNET)

The Simulation Network virtual battlefield.

 VR Applications in the Navy
• The U.S. Navy started its own studies in the use of virtual reality in the
early 1990s.

• Virtual Environment for Submarine Ship Handling Training (VESUB).

• VR-Based Close-Range Naval Artillery Training.
Virtual Environment for Submarine Ship
Handling Training (VESUB).
• This simulator for "officer on deck" (OOD) training is currently in use
at the Naval Submarine School
• The development of the VR simulator was prompted by the
realization that older shore-based OOD trainers were inadequate.
• As a consequence, junior officers needed to learn on the job during
the limited time of surface steaming, a situation that was risky.
VR-Based Close-Range Naval Artillery
Training.
• This is used in the British Royal Navy to replace shore-based live fire
training.
• The VRbased simulator, which entered service in 2001, is modular in
design, allowing training for 20-mm and 30-mm rapid-fire cannons
and other close-range artillery.
 Air Force Use of VR
• Unlike doctors, pilots (civilian and military) have a long tradition of
simulator-based training owing to the grave consequences human error has
on passengers.
• The traditional flight simulator has been a large domelike structure placed
on a motion platform and housing an identical replica of an airplane cockpit.
• While such training is very realistic (including engine sounds, airflow
turbulence, and other effects), it is also very expensive.

• The Unit Trainer and the Virtual Cockpit.

• Distributed Mission Training (DMT)
 The Unit Trainer and the Virtual Cockpit.
• Those are examples of the newer generation of simulators. They serve
as multitask trainers, allowing both standalone and networked
simulations (necessary to train pilot teams and tactics).
 Distributed Mission Training (DMT)
• This networked simulation environment is being created by the U.S.
Air Force Research Laboratory (Mesa, Arizona) in order to allow team
pilot training
 APPLICATIONS OF VR IN ROBOTICS-
ROBOTS
• Robots are mechanical arms that perform physical actions on their
environment.
• They are characterized by flexibility (versatility), as a given robot can
execute a variety of tasks or execute a given task in a variety of ways.
• Finally, robots are intelligent or self-adapting, so they can react to
changes in the environment and take corrective actions in order to
perform their tasks successfully.
• Robotic manipulators with their programming flexibility are used on
the factory floor, as well as in medical, military, and other application
areas, including VR.
 Robot Programming
• The reachable space, work envelope, of a robot arm is limited by the range of its
mechanical articulations (or joints).
• The robot may, damage itself if it attempts to move past its work envelope, and thus
correct programming is important.
• The current practice is to set software limits on the goal points. These limits create a
virtual work envelope, becoming virtual limits.
• The robot manipulator can also be damaged if it hits obstacles within its work
envelope.
• Normally, a robot relies on its sensors (vision, ultrasonic, lasers, proximity) to detect
such obstacles and avoid them
• As sensor data can be unreliable, software program can be done to generate artificial
repelling forces based on external sensing data.
• Robotic programming is a complex process.
• Industrial robot programming is performed manually (using a teach pendant), offline, or at "task
level."
• A control box for programming the motions of a robot. Also called a "teach box," the robot is set
to "learning" or "teach" mode, and the pendant is used to control the robot step by step. Teach
pendants are typically handheld devices and may be wired or wireless.
• Task-level programming instructs the robot what needs to be done, but not how, and is an area of
research.
• Offline programming creates software (usually in a robotic programming language) that
subsequently needs to be tested running the real robot and requires good programming skills.
Because of these issues, most industrial robots are programmed manually, using a simple teach
pendant. The operator moves the manipulator slowly over its task trajectory by pressing buttons
on the teach pendant and the required via points are stored in memory. This approach has the
advantage of simplicity since it does not require programming skills.
 The system consists of a Mitsubishi PA10 robot (with seven
DOF), a laser range finder, a human operator with the teach
pendant, HMD, and a video tracking system with two CCD
cameras. The MTA (multimodal teaching advisor) runs on a host
PC, which receives voice commands from the operator and is
interfaced with the other controllers. The MTA calculates the
difference between the modeled welding path (based on task
specifications) and that taught by the operator. Subsequently, it
provides remedial advice to the operator through graphics and
audio feedback. Tests showed that the use of the MTA
produced better taught trajectories, while the duration of the
teaching time was reduced.

Teach pendant-based robot programming using a multimodal

teaching advisor
 Robot Teleoperation
• The difficulties related to machine reasoning and unknown
environments has led to the development of two types of robots, for
the industrial and service sectors, respectively.
• Industrial robots perform tasks automatically, but in completely
known (or structured) environments.
• Service robots, such as those used in space exploration, perform tasks
in unstructured environments, but under the remote (or
teleoperation) control of a human.
 Telerobotic system

Whenever a robot is controlled at a

distance, the human operator acts
on a master arm inputting
commands (position, velocity, or
force). These commands are then
sent to the slave robot, which
executes the task and feeds back
information to the operator (such as
video images of the remote site).

Telerobotic system using a virtual slave robot