Contents

Session 1 – Motion Planning

Session Summary ............................................................................................................................ 3
Gerd Hirzinger
Toward Autonomous Free-Climbing Robots ........................................................................ 6
Tim Bretl, Jean-Claude Latombe, Stephen Rock
Whole Quadruped/Hexapod Manipulation.......................................................................... 16
Toru Omata
Dexterous Motion Design for a DD Parallel Robot ......................................................... 26
Masaru Uchiyama, Daisuke Sato
Current Issues in Sampling-Based Motion Planning ....................................................... 36
Stephen R. Lindemann, Steven M. LaValle

Session 2 – Path Planning

Session Summary ......................................................................................................................... 57
Henrik I. Christensen
A Hierarchical Bayesian Approach to the Revisiting Problem
in Mobile Robot Map Building ............................................................................................... 60
Dieter Fox, Jonathan Ko, Kurt Konolige, Benjamin Stewart
Efficient Construction of Globally Consistent Ladar Maps
Using Pose Network Topology and Nonlinear Programming ...................................... 70
Alonzo Kelly, Ranjith Unnikrishnan
Probabilistic Roadmaps of Trees for Parallel Computation
of Multiple Query Roadmaps ................................................................................................... 80
Mert Akinc, Kostas E. Bekris, Brian Y. Chen, Andrew M. Ladd, Erion Plaku,
Lydia E. Kavraki
Topology in Motion Planning .................................................................................................. 90
Howie Choset, Alfred A. Rizzi
aSyMov: A Planner That Deals with Intricate Symbolic
and Geometric Problems .......................................................................................................... 100
Fabien Gravot, Stephane Cambon, Rachid Alami
XII Contents

Session 3 – Medical Robotics

Session Summary ....................................................................................................................... 113
Tomomasa Sato
Microrobotics for Molecular Biology: Manipulating Deformable Objects
at the Microscale ........................................................................................................................ 115
Bradley J. Nelson, Yu Sun, Michael A. Greminger
Modeling of Brain Mechanical Properties for Computer-Integrated Medicine ... 125
Karol Miller, Wieslaw L. Nowinski
Assisted Teleoperation Through the Merging of Real and Virtual Images ........... 135
Alícia Casals
Surgical Robots at TIMC: Where We Are and Where We Go .................................. 145
Jocelyne Troccaz, Philippe Cinquin, Peter Berkelman,
Adriana Vilchis-Gonzales, Eric Boidard

Session 4 – Human-Robot Interaction

Session Summary ....................................................................................................................... 159
Yoshiaki Shirai
Retracted: Human-Machine Collaborative Systems for Microsurgical
Applications ................................................................................................................................ 162
Danica Kragic, Panadda Marayong, Ming Li, Allison M. Okamura,
Gregory D. Hager
Intuitive Human-Robot Interaction Through Active 3D Gaze Tracking ................ 172
Rowel Atienza, Alexander Zelinsky
Designing an Encountered-Type Haptic Display
for Multiple Fingertip Contacts Based on the Observation
of Human Grasping Behavior ................................................................................................ 182
Yasuyoshi Yokokohji, Nobuhiko Muramori, Yuji Sato, Tsuneo Yoshikawa
Face Recognition Using Multi-viewpoint Patterns for Robot Vision ...................... 192
Kazuhiro Fukui, Osamu Yamaguchi
Robot-Robot and Human-Robot Cooperation
in Commercial Robotics Applications ................................................................................ 202
R. Koeppe, D. Engelhardt, A. Hagenauer, P. Heiligensetzer, B. Kneifel,
A. Knipfer, K. Stoddard

Session 5 – Multiple Robots

Session Summary ....................................................................................................................... 219
John Hollerbach
Abstractions and Control for Swarms of Robots ............................................................ 224
Calin Belta, Guilherme A.S. Pereira, Vijay Kumar
Networked Robots: Flying Robot Navigation Using a Sensor Net .......................... 234
Peter Corke, Ron Peterson, Daniela Rus
 Contents XIII

A Multi Agent Distributed Sensing Architecture with Application

to Planetary Cliff Exploration ................................................................................................ 244
Vivek A. Sujan, Steven Dubowsky, Terry Huntsberger, Hrand Aghazarian,
Yang Cheng, Paul Schenker
Multi-robot SLAM with Sparse Extended Information Filters .................................. 254
Sebastian Thrun, Yufeng Liu

Session 6 – Robot Navigation

Session Summary ....................................................................................................................... 269
Ray Jarvis
Safe Robot Driving in Cluttered Environments ............................................................... 271
Chuck Thorpe, Justin Carlson, Dave Duggins, Jay Gowdy,
Rob MacLachlan, Christoph Mertz, Arne Suppe, Bob Wang
Feedback Control of Underactuated Systems via Sequential Composition:
Visually Guided Control of a Unicycle .............................................................................. 281
George Kantor, Alfred A. Rizzi
Obstacle Detection in Foliage with Ladar and Radar .................................................... 291
Larry Matthies, Chuck Bergh, Andres Castano, Jose Macedo,
Roberto Manduchi

Session 7 – Vision
Session Summary ....................................................................................................................... 303
Takeo Kanade
Visual Tracking of Multiple Objects Using Binary Space Partitioning Trees ...... 305
Fabrizio Caccavale, Vincenzo Lippiello, Bruno Siciliano, Luigi Villani
Simultaneous Localization and Mapping with Stereovision ....................................... 315
Il-Kyun Jung, Simon Lacroix
Applying Active Vision and SLAM to Wearables ......................................................... 325
Walterio W. Mayol, Andrew J. Davison, Ben J. Tordoff, David W. Murray
Seeing Through BadWeather ................................................................................................. 335
Shree K. Nayar, Srinivasa G. Narasimhan

Session 8 – Humanoids
Session Summary ....................................................................................................................... 353
Hirochika Inoue
SDR-4X II: A Small Humanoid as a Entertainer in Home Environment ............... 355
Masahiro Fujita, Kohtaro Sabe, Yoshihiro Kuroki, Tatsuzo Ishida,
Toshi T. Doi
Motion Planning for Humanoid Robots ............................................................................. 365
James Kuffner, Koichi Nishiwaki, Satoshi Kagami, Masayuki Inaba,
Hirochika Inoue
XIV Contents

The Human-Size Humanoid Robot That Can Walk, Lie Down and Get Up ........ 375
Hirohisa Hirukawa, Shuuji Kajita, Fumio Kanehiro, Kenji Kaneko,
Takakatsu Isozumi
Exploiting the Global Dynamics Structure of Whole-Body Humanoid Motion –
Getting the “Knack” of Roll-and-Rise Motion ................................................................ 385
Yasuo Kuniyoshi, Yoshiyuki Ohmura, Koji Terada, Tomoyuki Yamamoto,
Akihiko Nagakubo

Session 9 – SLAM
Session Summary ....................................................................................................................... 397
Bob Bolles
Recasting-SLAM – Towards Improving Efficiency
and Platform Independency .................................................................................................... 399
Jong Hyuk Kim, Salah Sukkarieh
Towards Constant Time SLAM on an Autonomous Underwater Vehicle
Using Synthetic Aperture Sonar ........................................................................................... 409
Paul M. Newman, John J. Leonard, Richard R. Rikoski
Towards Lazy Data Association in SLAM ....................................................................... 421
Dirk Hähnel, Sebastian Thrun, Ben Wegbreit, Wolfram Burgard
Building Blocks for SLAM in Autonomous Compliant Motion ............................... 432
Herman Bruyninckx, Joris De Schutter, Tine Lefebvre, Klaas Gadeyne,
Peter Soetens, Johan Rutgeerts, P. Slaets, W. Meeussen
Multi-resolution SLAM for Real World Navigation ..................................................... 442
Agostino Martinelli, Adriana Tapus, Kai Olivier Arras, Roland Siegwart

Session 10 – Robot Perception

Session Summary ....................................................................................................................... 455
Ruzena Bajcsy
Perception of Curvature and Object Motion via Contact Location Feedback ...... 456
William R. Provancher, Katherine J. Kuckenbecker, Günter Niemeyer,
Mark R. Cutkosky
Multi-contact Haptic Interaction with Deformable Objects:
A Multi-rate Approach ............................................................................................................. 466
Federico Barbagli, Domenico Prattichizzo, Kenneth Salisbury
A Model for Machine Perception in Natural Environments ....................................... 479
Hugh Durrant-Whyte, Suresh Kumar, Jose Guivant, Steve Scheding
Multisensory Interaction: Real and Virtual ....................................................................... 489
Dinesh K. Pai
 Contents XV

Session 11 – Design
Session Summary ....................................................................................................................... 501
Bernard Roth
Untethered Micro-Actuators for Autonomous Micro-Robot Locomotion:
Design, Fabrication, Control, and Performance............................................................... 502
Bruce R. Donald, Christopher G. Levey, Craig D. McGray, Daniela Rus,
Mike Sinclair
The 100G Capturing Robot – Too Fast to See ................................................................. 517
Makoto Kaneko, Mitsuru Higashimori, Akio Namiki, Masatoshi Ishikawa
Variable Stiffness Actuators for Fast and Safe Motion Control ................................ 527
Antonio Bicchi, Giovanni Tonietti, Michele Bavaro, Marco Piccigallo

Session 12 – Bio-robotics
Session Summary ....................................................................................................................... 539
Shigeo Hirose
A Biologically Inspired Passive Antenna for Steering Control
of a Running Robot.................................................................................................................... 541
Noah J. Cowan, Emily J. Ma, Mark Cutkosky, Robert J. Full
Learning from Observation and from Practice Using Behavioral Primitives........ 551
Darrin Bentivegna, Gordon Cheng, Christopher Atkeson
Learning Movement Primitives ............................................................................................. 561
Stefan Schaal, Jan Peters, Jun Nakanishi, Auke Ijspeert
A Stochastic Model of Embodied Symbol Emergence ................................................. 573
Yoshihiko Nakamura, Tetsunari Inamura, Hiroaki Tanie

Forum Session 1 – Lessons from the Past 50 Years of Robotics

Lessons from the Past 50 Years of Robotics..................................................................... 587

Forum Session 2 – Open Forum

Open Forum.................................................................................................................................. 595

Erratum
Human-Machine Collaborative Systems for Microsurgical Applications ............... E1
Danica Kragic, Panadda Marayong, Ming Li, Allison M. Okamura,
Gregory D. Hager
Retracted: Human-Machine Collaborative Systems
for Microsurgical Applications
Danica Kragic1 , Panadda Marayong2, Ming Li2 , Allison M. Okamura2, and
Gregory D. Hager2
1 Centre for Autonomous Systems, Stockhom, Sweden,
2 Engineering Research Center for Computer Integrated Surgical Systems and Technology
The Johns Hopkins University, Baltimore, MD 21218 USA

Abstract. We describe our current progress in developing Human-Machine Collaborative

Systems (HMCSs) for microsurgical applications such as vitreo-retinal eye surgery. Three
specific problems considered here are (1) developing of systems tools for describing and
implementing an HMCS, (2) segmentation of complex tasks into logical components given
sensor traces of a human performing the task, and (3) measuring HMCS performance. Our
goal is to integrate these into a full microsurgical workstation with the ability to automatically

ed
“parse” traces of user execution into a task model which is then loaded into the execution
environment, providing the user with assistance using online recognition of task state. The
major contributions of our work to date include an XML task graph modeling framework and
execution engine, an algorithm for real-time segmentation of user actions using continuous
Hidden Markov Models, and validation techniques for analyzing the performance of HMCSs.
ct
1 Introduction
ra

The goal of the Human-Machine Collaborative Systems (HMCS) project is to inves-

tigate human-machine cooperative execution of small scale, tool-based manipula-
tion activities. Our work on HMCS is specifically aimed at microsurgery [6,12] and
cell manipulation, but the basic principles apply to many other fine scale tasks such
et

as opto-electronic assembly and assembling of LIGA parts [1]. The motivation for
collaborative systems is based on evidence [6,15] suggesting that humans operating
in collaboration with robotic mechanisms can take advantage of robotic speed and
precision, but avoid the difficulties of full autonomy by retaining the human com-
R

ponent “in-the-loop” for essential decision making and/or physical guidance [6].
Our approach to HMCS focuses on three inter-related problems: (1) Synthesis:
the development of systems tools necessary for describing implementing an HMCS;
(2) Modeling: given sensor traces of a human performing a task, segmenting those
traces into logical task components and/or measuring the compatibility of a given
HMCS structure to that sequence of components; and (3) Validation: measuring
HMCS performance.
Figure 1 depicts the high-level structure of the human-machine collaborative
systems we are developing. Logically, there are three components: the human, the
augmentation system, and an observer. We assume that a user primarily manipu-
lates the environment using the augmentation system, although unaided manipula-
tion may take place in some settings (dashed line). The user is able to visually ob-
serve the tool and the surrounding environment, and directs an augmentation device

P. Dario and R. Chatila (Eds.): Robotics Research, STAR 15, pp. 162–171, 2005.
 Springer-Verlag Berlin Heidelberg 2005
 Human-Machine Collaborative Systems 163

Fig. 1. Structure of a Human-Machine Collaborative System (left), and the experimental setup
using the Johns Hopkins University Steady Hand Robot (right).

using force and position commands. The system may also have access to endpoint
force data, targeted visual data and/or other application-dependent sensors (e.g.,

ed
intra-operative imaging). The role of the observer is to assess available sensor data
(including haptic feedback from the user) and initiate, modify, or terminate various
forms of assistance. Optional direct interaction between the observer and the user
may also be used to convey information or otherwise synchronize their interaction.
The basic notion of HMCS is clearly related to traditional teleoperation, al-
ct
though the goal in HMCS is not to “remotize” the operator [5] but rather to pro-
vide appropriate levels of operator assistance depending on context. At one extreme,
shared control [3] can be viewed as an HMCS for manipulation tasks in which some
ra

degrees of freedom are controlled by machine and others by the human. At the other
extreme, supervisory control [14] gives a more discrete, high-level notion of human-
machine interaction. Our notion of HMCS essentially incorporates both views, com-
bining them with broader questions of modeling manipulation activities consisting
et

of multiple steps and varying level of assistance, and validating those models against
human performance data.
R

2 Synthesis

Our current system development efforts are motivated by the domain of retinal mi-
crosurgery where tasks occur near the limit of the manipulation capabilities of the
human hand-eye system. A challenging task is retinal vein cannulation: the intro-
duction of a micro-pipette into a vessel of approximately 100 μm in diameter [16].
We augment the surgeon’s physical abilities using the Johns Hopkins University
Steady Hand Robot (JHU SHR, Figure 1). Briefly, the JHU Steady Hand Robot is
an admittance-controlled 7 DOF robot equipped with a force sensing handle at the
endpoint. Tools are mounted at the end-effector and “manipulated” by an operator
holding a force-sensing handle also attached to the end-effector. The robot responds
to the applied force, implementing a means of direct control for the operator. The
robot is ergonomically appropriate for minimally invasive microsurgical tasks and
provides micron-scale accuracy [15]. Two general forms of motion assistance have
164 D. Kragic et al.

been investigated with the JHU SHR: force scaling [13] and motion guidance using
virtual fixtures [8]. In this paper, we focus on the latter.

2.1 Describing Spatial Motion Constraints

“Virtual fixtures” in our implementation provide cooperative control of the manipu-
lator by “stiffening” a hand-held guidance mechanism against certain directions of
motion or forbidden regions of the workspace. Studies on virtual fixtures for tele-
operation have indicated that user performance can increase as much as 70% with
the fixture based guidance [11]. Here, we describe our framework for implementing
virtual fixtures on the JHU SHR. In what follows, we model the robot as a purely
kinematic Cartesian device with tool tip position x SE 3 and a control input that
!

✁ ✂

is endpoint velocity v ẋ R 6 , all expressed in the robot base frame. A human ✄

!

operator guides the robot by applying forces and torques f R 6 on the manipulator
!

handle, likewise expressed in robot base coordinates.

We define SHR guidance geometrically by identifying a space of “preferred”
directions of motion. Let us assume that we are given a 6 n time-varying matrix
D D t 0 n 6, representing the instantaneous preferred directions of motion.
✄ ✁ ✂ ✆ ✞

For example, if n is 1, the preferred direction is along a curve in SE(3); if n is 2 the

preferred directions span a surface, and so forth.
✞

ed ☎
ct
We define two projection operators, the span and the kernel of the column space, as

Span D ✁ ✂ ✟ ✡ D ☞ ✄ D DD✁ ✍ ✂ ✎ D ✍ ✆ and Ker D ✁ ✂ ✟ ✒ D ✔ ✄ I ✕ ✡ D ☞ (1)

ra

where denotes pseudo-inverse for the case where D is (column) rank deficient.
✎

By decomposing the input force vector, f, into two components, f D D f and fτ ✟ ✡ ☞ ✟

f fD D f and introducing a new admittance ratio kτ 0 1 that attenuates the

!

✕ ✄ ✒ ✔ ✆ ✡ ✆ ☞
et

non-preferred component of the force input, we arrive at an admittance control

v ✄ k fD ✁

✘
kτ f τ ✂ ✄ k D ✁ ✡ ☞

✘
kτ D f
✒ ✔ ✂ (2)
R

Thus, the final control law is in the general form of an admittance control with a
time-varying gain matrix determined by D t . By choosing k, we control the overall ✁ ✂

admittance of the system. Choosing kτ low imposes the additional constraint that the
robot is stiffer in the non-preferred directions of motion. As noted above, we refer
to the case of kτ 0 as a hard virtual fixture, since it is not possible to move in any ✄

direction other than the preferred direction. All other cases will be referred to as soft
virtual fixtures. In the case kτ 1 we have an isotropic admittance. ✄ ✆

The development to this point directly supports motion in a subspace, but it does
not allow us to define a fixed desired motion trajectory. If u f x S is the signed ✄ ✁ ✆ ✂

distance of the tool tip from a surface S, we can define a new preferred direction as

Dc x ✁ ✂ ✄ ✡ ✁ 1 ✕ kd D f ✂ ✡ ☞ ✢ ✣ f ✣

✘
kd D u ✒ ✔ ☞ 0 ✞ kd ✞ 1 ✆ (3)

combining the two vectors that encode motion in the preferred direction and correct-
ing the tool tip back to S. The constant kd governs how quickly the tool is moved
 Human-Machine Collaborative Systems 165

toward the reference surface. We note that the division by f is undefined when no ✣ ✣

user force is present. As projection is invariant to scale, we write (3) as

Dc x ✁ ✂ ✄ ✁ 1 ✕ kd D f
✂ ✡ ☞

✘
kd f D u 0
✣ ✣ ✒ ✔ ✞ kd ✞ 1 (4)
and apply (2) with D Dc . ✄

One potential problem with this control law is that when the user applies no
force, there is no virtual fixture because there is no defined preferred direction.
Thus, there is a discontinuity at the origin. However, in practice the resolution of
any force sensing device is usually well below the numerical resolution of the un-
derlying computational hardware computing the pseudo-inverse, so the user will
never experience this discontinuity.
An implementation of this framework in operation is shown in Figure 1 (right).
A CCD camera is attached to the robot and views a curve on a task plane. The view
is processed using the XVision tracking system [4] to determine the tangent to the
reference path at a point closest to the center of the image in real time. The center
is marked with a cross, and a user attempts to move the robot to follow the curve
(similar to tracking a blood vessel in preparation for retinal cannulation). In this
case, if tx and ty are the components of the tangent to the curve in the image plane,
and nx and ny is the vector from the center to the closest point on the curve, then we
have u nx ny 0 0 0 0 and D tx ty 0 0 0 0 More details and extensions of
ed
ct
✁

✄ ✁ ✆ ✆ ✆ ✆ ✆ ✂ ✄ ✁ ✆ ✆ ✆ ✆ ✆ ✂

✍ ✍

this basic task can be found in [9]. Experimental results with users are presented in
Section 4.
ra

2.2 High-Level Task Specification

Most microsurgical tasks consist of discrete, serial, quasi-static steps, each with
a clearly defined outcome. Hence, models for such procedures can be defined by
relatively simple graphs. For example, retinal vein cannulation involves positioning
et

and orienting a needle to the vicinity of the vein, and inserting it when appropriate
until contact is made. Upon contact, puncturing is performed, after which the needle
can be safely withdrawn, as shown in Figure 2.
R

In order to provide appropriate assistance, it must be possible to create models

of procedures, and to attach appropriate types or levels of assistance to each node.
To this end, we have developed a set of tools for defining task graphs. Task graphs
are represented using a specialization of the Extensible Markup Language (XML) 1 .
A task graph modeling system allows a user to interactively design a graph for the
task to be performed and save the graph as an XML file. A task graph execution
engine reads the XML description and coordinates the execution of the graph. The
graph itself references a library of basic primitives for providing user assistance at
each node, and a set of transition conditions. Thus, at each state the graph executor
initiates the appropriate assistance mode and monitors for transition events to sub-
sequent states. Currently, there are three ways of generating a task graph: (1) using
the graphical interface, (2) directly writing an XML file, and (3) automating off–line
task modeling. We now turn to the latter of these methods.
1 www.xml.org
166 D. Kragic et al.

Start Move button1

button2 Orient button1

button2 Insert

error
contact

error

error
error Puncture
button1

error
Hold
error button1

Retract
error
Error button1

Done

Fig. 2. GUI for generating task graphs (left). A graph example for vein cannulation (right).

1
Model 1

1
Model 2

1
2

2
3

3
4

4
5

5
ed
ct
Model 3
ra

Fig. 3. Network for real-time, continuous HMM recognition (left). One model describes the
user motion at any given time. Task descriptions for the human performance experiments
(right): (a) path following, (b) off-path targeting, and (c) avoidance. The black line denotes
et

the virtual fixture reference path, and the path to be followed by the user is dark gray.

3 Modeling and Sensing Human Intent

R

One problem that arises in HMCS is that of providing the appropriate assistance to a
user, based on the intent or context of his or her actions. In order to do so, the system
must have a model of the task being performed, and a means for relating the actions
of the user to that model. We have investigated continuous Hidden Markov Models
(HMMs) [10] as a means of modeling tasks from sensor traces of user execution. We
have also developed a new algorithm for HMM recognition that allows for real-time
segmentation of user actions.

3.1 Offline Task Modeling

The basic hypothesis of offline task modeling is that, for restricted domains such
as microsurgery, it is possible to model tasks using a small vocabulary of primitive
“gestemes” with an associated assistance mode. We have tested our hypothesis us-
 Human-Machine Collaborative Systems 167

ing data acquired from task execution with the JHU SHR and modeled using The
Hidden Markov Model Toolkit2 (HTK). The input data acquired from the robot
consisted of seven variables: Cartesian forces and torques expressed in robot-end-
effector coordinates, and the magnitude of translation between the last reading and
the current one. We assign a linearly sequential left-to-right (SLR) structure to the
HMMs of several gestemes. Two tasks were investigated: (1) a peg-in-hole task,
and (2) a paint task, which are analogous to retinal vessel cannulation and retinal-
membrane peeling, respectively. For the peg-in-hole task, five gestemes were used:
place, position, insert, withdraw and remove. For the paint task, the four gestemes
were: place, position, paint and remove. Five users participated in the experiment
and each user performed ten runs. Each gesteme was trained independently on a
training set, then gestemes were run in parallel to segment a test set (Figure 3).
The obtained segmentation accuracy of the system over all users was around
84% for peg-in-hole task and 90% for paint task. In addition, we found that we
could use gestemes trained for the paint task in the peg-in-hole task, and the result-
ing decrease in recognition performance was only 1.08%. These results, although
preliminary, suggest that for suitably constrained situations, it is plausible that com-
plex tasks can be modeled using a small set of manipulation primitives.

3.2 Online Recognition and Assistance

ed
ct
Given a task model, the question is whether it is possible to determine the appro-
priate task state online. To do this, we have developed an online continuous HMM
ra

recognition algorithm [7]. The structure of each HMM is SLR and each model oper-
ates in parallel. Suppose we have M potential models and each model has N states.
For a given model λ, let φ j λ t be the likelihood of observing vector o1 to ot and !
✁ ✂

being in state j at time t. The partial likelihood of each model can be computed as
et

N
✁

φj λ t
!
✁ ✂ ✄

∑ φi λ t !
✁ ✕ 1 ai j b j ot
✂

✄
✁ ✂ ✆ with φ1 λ 1 !
✁ ✂ ✄ 1 φj λ 1
✆

!
✁ ✂ ✄ 0 ☎ 1 ✞ j ✆ N (5)
i 1
R

The likelihood of state 1 of a model λ in time t ✝ 1 can be defined as

1 M
M m∑1
φ1 λ t
!
✁ ✂ ✄ φN m t !
✁ ✕ 1 ✂ for 1 ✞ λ ✆ M (6)
✂

For each model, we define the sum of the likelihood of all states as the total likeli-
hood at time t as L o1 o2 ot λ ∑Nj 1 φ j λ t , where the model with the highest
✁ ✆ ✆
✁ ✁ ✁

✞
✂ ✄

!
✁ ✂

total likelihood represents the current hypothesis.

✂

We tested the online recognition algorithm with the JHU SHR in a planar en-
vironment, where the task was to follow a sinusoidal path, but avoid a portion of
the path covered by a circle. The real-time HMM recognition output was given ev-
ery 33ms. In our experiment, we included three models/gestemes (HMMs) of user
2 http://htk.eng.cam.ac.uk
168 D. Kragic et al.

actions: (1) silence (user is doing nothing), (2) follow curve (user is following the
curve); and (3) avoid curve (user is not following the curve). The data used was
force in the x and y directions. The force, f , was decomposed into the directions
parallel to reference curve δ and the normal direction τ. f δ , fτ and f δ were
✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞
✁

✞ ✞

the elements of the observation vector used to train the HMMs.

The accuracy of recognition and the sensitivity of our algorithm to training in
differing environments was examined by an experiment where: (1) the same sine
curve was used for both training and recognition, and (2) different sine curves were
used for training and recognition. Here, the average accuracy of real-time continu-
ous recognition among all subjects was larger than 90% in both cases. We believe
that this improvement in recognition over the offline case is due to the additional
context provided by the virtual fixtures.

4 Validation

ed
During task execution, speed and accuracy are generally used as metrics to evaluate
performance. There is typically a trade off between these two metrics, as described
by Fitts Law [2]. In this section, we demonstrate that both speed and accuracy can be
improved simultaneously with the aid of virtual fixture guidance. The experimental
setup makes use of the curve following virtual fixture described in Section 2.1.
ct
4.1 Virtual Fixtures
ra

We present two experiments that examine the effects of virtual fixture guidance
ranging from complete guidance (admittance ratio = 0) to no guidance (admittance
ratio = 1). Detailed results can be found in [8]. The reference path was a 0.39mm
thick sine curve with a 35mm amplitude and 70mm wavelength. For Experiment II,
et

we added a circle of radius 10mm with its center located at the midpoint of the sine
curve. The subjects were provided with instructions as shown in Figure 3, and told
to move along the path as quickly as possible without sacrificing accuracy, consid-
ering accuracy and speed with equal emphasis. Experiment I included five subjects
R

performing the path following task three times with eleven admittance ratios from 0
to 1 (0, 0.1, . . . ). Experiment II included eight subjects performing each task three
times with four discrete admittance ratios corresponding to four guidance levels
(0=complete, 0.3=medium, 0.6=soft, and 1=none). At the run time, the time and
error during motion from Point A to Point B in Figure 3 were recorded. The er-
ror represents the deviation from the reference path. For the off-path targeting task,
we recorded the time each subject needed to get back on the curve after leaving
it. For the avoidance task, we recorded the time needed to avoid the circle, and no
error was measured. For the data obtained in Experiment II, we performed ANOVA
and multiple pair-wise comparisons using Tukey’s method to determine significant
differences. Data are shown in Table 1.
For Experiment I, the data indicate that improvements in error and time have
linear relationships with admittance ratio, as shown in Figure 4. This is to be ex-
pected, since the output velocity of the SHR is linearly proportional to admittance
 Human-Machine Collaborative Systems 169

Fig. 4. Average normalized time and error versus admittance ratio for the path following task.

ratio. For Experiment II, the average execution time and error were used to deter-
mine the improvement in performance with different guidance levels. For the path
following task, a decrease in admittance ratio reduces error, except between medium
and complete guidance. However, a decrease in admittance ratio does not improve

ed
execution time significantly. We note that none of the subjects performed worse in
both time and error when admittance ratio decreased. Complete guidance resulted
with the best performance but more guidance resulted in shorter time and/or higher
accuracy compared to no guidance. For the targeting task, stronger guidance slowed
task execution. However, the execution times for no guidance and soft guidance do
ct
not differ significantly. The analysis also shows no difference in error for all guid-
ance levels. In general, reducing guidance reduces the time and error during target
acquisition. For the avoidance task, only the execution time was considered. The
ra

results indicate that less guidance reduces the execution time.

The results indicate that the selection of virtual fixture admittance is a task-
dependent process. For surgical applications, error reduction is more important than
time required to perform the task. For tasks that require significant interaction from
et

the user, such as object avoidance and off-path targeting, strong guidance will reduce
accuracy and increase execution time. Therefore, a lower amount of guidance is rec-
ommended. Based on the linear relationship between admittance and performance
found in Experiment I, we developed admittance ratio selection parameters that can
R

be used to determine an appropriate guidance level [8]. Using equal weighting for
error vs. time, and path following vs. path avoidance, we found that an admittance
ratio of approximately 0.6 is optimal. However, online admittance tuning is recom-
mended to achieve the full benefit of virtual fixture guidance.

4.2 Online Assistance

In this experiment, 8 subjects were asked to follow the sine curve (model follow
curve) and the edge of the circle (model avoid curve) in three different modes:
(1) HMM+VF where real-time recognition was used to determine when to apply
or remove the virtual fixture, (2) VF: a virtual fixture with constant admittance ratio
kτ = 0.3 was applied, and (3) NG: the virtual fixture was removed (no guidance, k τ
= 1). The error (deviation from the curve) and time were again used to validate the
performance. To create a “gold standard” for the correct segmentation, the subjects
170 D. Kragic et al.

Table 1. Experiment II: Experimental results for eight subjects: rows 1-4 for path following,
rows 5-7 for off-path targeting, and 8-10 for avoidance.
Time (Seconds) Error (Pixels)
Admittance Ratio (kτ ) Guidance Average Standard Deviation Average Standard Deviation
1 0 Complete 18.710 1.357 0.061 0.013
2 0.3 Medium 19.647 1.879 0.078 0.020
3 0.6 Soft 20.425 2.042 0.121 0.030
4 1 None 24.745 5.405 0.229 0.087
5 0.3 Medium 9.754 3.664 1.256 0.456
6 0.6 Soft 7.148 3.178 0.910 0.180
7 1 None 6.256 2.353 0.702 0.437
8 0.3 Medium 14.681 5.629 - -
9 0.6 Soft 11.871 4.959 - -
10 1 None 9.317 2.954 - -

Average error
1.4

1.2

0.8

0.6

0.4
ed HMM+VF
VF
NG
50

40

30

20
HMM+VF
VF
NG
Time
ct
10
0.2

0 0
Follow curve Avoid curve Total task Follow curve Avoid curve Total task

Fig. 5. Mean and standard deviation of average error (left) and time (right) for the follow
ra

curve section of the task, avoid curve section of the task, and total task. HMM+VF indicates
that the virtual fixture was switched off by the HMM when it was detected that the user
intended to move away from the curve, VF indicates a constant virtual fixture with admittance
ratio kτ = 0.3, and NG indicates no guidance (kτ = 1).
et

were asked to press a space bar when they intended to transition from one model to
another. The results are presented in Figure 5 showing the good performance of the
R

combined algorithm (HMM+VF). The HMM+VF provides the user with assistance
when he/she wants to follow the curve, and removes the assistance otherwise.

5 Conclusions and Future Work

We have presented the structure, theoretical and experimental results of our HMCS
project. We have developed a control method for implementing guidance virtual
fixtures with an admittance controlled robot, as well as techniques for automatic
recognition of user activities. These components are integrated through a task graph
and execution engine specifically designed for serial procedures such as those en-
countered in microsurgery.
We are currently integrating these components into a full micro-surgical work-
station for vitreo-retinal eye surgery. We expect to shortly be able to automatically
 Human-Machine Collaborative Systems 171

“parse” traces of user execution into a task model, load the task model into the
execution environment, and thereby provide the user with assistance using online
recognition of task state. The system will be tested using ex-vivo and in-vivo exper-
iments involving evaluation of system performance during operation by surgeons.

Acknowledgments. The authors thank Russell Taylor and Blake Hannaford for their in-
sights related to this research. This material is based upon work supported by the National Sci-
ence Foundation, under Grants No. EEC-9731478, IIS-0099770, ITR-0205318 and Swedish
Foundation for Strategic Research through the Centre for Autonomous Systems.

References
1. J. T. Feddema and T. R. Christenson, “Parallel Assembly of High Aspect Ratio Mi-
crostructures”, SPIE Microrobotics and Microassembly, vol.3834, 1999.
2. P. M. Fitts, “The information capacity of the human motor system in controlling the am-
plitude of movement”, Journal of Experimental Psychology, 47:381–391, 1954.

ed
3. C. Guo, T. J. Tarn and A. Bejczy, “Fusion of Human and Machine Intelligence for Teler-
obotic Systems”, Proc. Int. Conference on Robotics and Automation, 3110-3115, 1995.
4. G. D. Hager and K. Toyama, “The XVision system: A general purpose substrate for real-
time vision applications”, Computer Vision and Image Understanding, 69(1):21-27, 1998.
5. G. Hirzinger, G. Grunwald, B. Brunner and H. Heindl, “A Sensor-based Telerobotic Sys-
tem for the Space Robot Experiment”, Int. Symp. on Experimental Robotics, 1991.
ct
6. R. Kumar, T. M. Goradia, A. Barnes, P. Jensen, L. L. Whitcomb, D. Stoianovici, L. M.
Auer and R. H. Taylor, “Performance of robotic augmentation in common dextrous surgi-
cal motions”, Medical Image Computing and Computer-Assisted Interventions,1999.
ra

7. M. Li and A. M. Okamura, “Recognition of Operator Motions for Real-Time Assistance

Using Virtual Fixtures”, 11th International Symposium on Haptic Interfaces for Virtual
Environment and Teleoperator Systems, IEEE Virtual Reality, pp.125–131, 2003.
8. P. Marayong, A. Bettini and A. M. Okamura, “Effect of Virtual Fixture Compliance on
Human-Machine Cooperative Manipulation”, IEEE/RSJ International Conference on In-
et

telligent Robots and Systems, pp.1089–1095, 2002.

9. P. Marayong, M. Li, A. M. Okamura, and G. D. Hager, “Spatial Motion Constraints: The-
ory and Demonstrations for Robot Guidance using Virtual Fixtures,” 2003 IEEE Interna-
R

tional Conference on Robotics and Automation. (in press)

10. L. Rabiner, “A Tutorial on Hidden Markov Models and Selected Applications in Speech
Recognition”, Proceedings of the IEEE, 77(2):257–286, 1989.
11. L. Rosenberg, Virtual Fixtures. PhD Thesis, Dept. of Mech. Eng., Stanford Univ., 1994.
12. D. Rothbaum, J. Roy, D. Stoianovics, P. Berkelman, G. Hager, R. Taylor, L. Whitcomb,
H. Francis, and J. Niparko. Robot-assisted stapedotomy: Micropick fenestration of the
stapes footplate. Otolaryngology - Head and Neck Surgery, pages 417-426, Nov. 2002.
13. J. Roy and L.L. Whitcomb. “Adaptive Force Control of Position/Velocity Controlled
Robots:Theory and Experiment”. IEEE Trans. on Rob. and Autom., 18(2):121-137, 2002.
14. T. B. Sheridan, Human Supervisory Control of Robot Systems, International Conference
on Robotics and Automation, pp.808–812, 1986.
15. R. H. Taylor, P. S. Jensen, L. L. Whitcomb, A. C. Barnes, R. Kumar, D. Stoianovici, P.
Gupta, Z. Wang, E. deJuan and L. Kavoussi, “A Steady-Hand Robotic Syatem for Micro-
surgical Augmentation”, IJRR, 18:1201–1210, 1999.
16. J. N. Weiss, “Injection of tissue plasminogen activator into a branch retinal vein in eyes
with central retinal vein occlusion”, Opthalmology, 108(12):2249–2257, 2001.
Erratum: Human-Machine Collaborative Systems
for Microsurgical Applications

Danica Kragic1, Panadda Marayong2, Ming Li2, Allison M. Okamura2, and

Gregory D. Hager2

1
Centre for Autonomous Systems, Stockhom, Sweden,
2
Engineering Research Center for Computer Integrated Surgical Systems and Technology
The Johns Hopkins University, Baltimore, MD 21218 USA

P. Dario and R. Chatila (Eds.): Robotics Research, STAR 15, pp. 162–171, 2005.
© Springer-Verlag Berlin Heidelberg 2005

DOI 10.1007/978-3-540-31508-7_62

“The Publisher wishes to include a reference to Dr. Rajesh Kumar's Ph.D. thesis,
entitled.”An Augmented Steady Hand System for Precise Micromanipulation,” Johns
Hopkins University, Baltimore MD (April 2001), which describes a framework for
task level control on the Steady Hand Robot at JHU and which reports demonstrations
of several representative tasks, including retinal cannulation, on dry lab and ex-vivo
phantoms. The Publisher also refers to Figure 2 of the Article and wishes to reference
the depiction of retinal cannulation in Dr. Kumar's thesis, which reflects a task
sequence that was performed for Dr. Kumar's thesis and is depicted in the thesis as
Figure 5.13.”

_______________________________________________
The original version for this chapter can be found at
http://dx.doi.org/10.1007/978-3-540-31508-7_18
_______________________________________________