New Surgical Paradigms through

Intelligent Continuum Robots and Complementary

Situational Awareness

Nabil Simaan
Vanderbilt University

1
Historical Drivers to Surgical Robotics & CAS

Execution

Planning Robotic Manual

Manipulation augmentation Information augmentation

2
 The Surgeons Dilemma

Seeing Preop. imaging

Perceiving Clinical Scene

Sensing
Comprehension
Projection

These challenges exist regardless of surgical technique

What happens when we add robot assistance?
Why use intelligent surgical robots?
3
Robotics: Reduced Trauma at a Cost

Surgeon
Anatomy
experience

Sensory Situational
Situational
perception awareness
awareness

4
Robotics: Reduced Trauma at a Cost

Surgeon
Anatomy
experience

Sensory Situational
perception awareness
Open Surgery

Anatomy Comprehension
gap

Sensory & Situational

Perception gap awareness

Robot-Assisted Surgery
5
Robotics: Reduced Trauma at a Cost

Surgeon
Anatomy
experience

Sensory Situational
perception awareness
Open Surgery

Anatomy Immersion Surgeon

Comprehension
(VR) experience
gap

Situational
Haptics Computer awareness
Navigation
Robot-Assisted Surgery
6
Robotics: Reduced Trauma at a Cost

Surgeon
Anatomy
experience

Situational
awareness

Why is this not enough?

Open Surgery

Anatomy Immersion Surgeon

(VR) experience

Situational
Haptics Computer awareness
Navigation
Robot-Assisted Surgery
7
1st Reason: Many Systems Common Limitations

NeuroMate OrthoDoc (ISS) RoboDoc (ISS) Computer Motion Zeus system

Titan SPORT

ISI
single
Hansen Magellan Mako Mazor
port
da-Vinci Si ISI

Limited/No sensory feedback

No use of contextual/sensory data
No intelligent behaviors
Limited human-robot partnership
MedRobotics 8
2nd Reason: Emerging Surgical Paradigms
N.O.T.E.S.

Thyroidectomy

New paradigms impose new

constraints
Constraints of manipulation
Constraints of perception

Challenges of situational
awareness

9
 Related Work: Single Port/NOTES
Intuitive Surgical

Nanyang Tech. University

S.S. SantAnna
KIST+SAIT
Purdue Univ.

Intuitive
Surgical

Imperial
Nagoya Univ. College

Univ. of Tokyo Univ. of Nebraska

Intuitive Surgical Hanyang Univ.

Titan SPORT Tech. Munich Univ.

JHU/Columbia Carnegie Mellon

Swanstrom et al. Columbia/Vanderbilt UM-SJTU Waseda Univ. DGIST

2003 2004 2005 2006 2007 2008 2009 2010 2011 2013 2014

10
Traditional Challenges of Multi-Port MIS
MIS

Tool dexterity Incision constraint

Multiple tools Hand-eye
Visualization coordination
Sensory deficiency

11
Additional Challenges of NOTES/SPAS
NOTES MIS

Constrained workspace
Tool dexterity Incision constraint
Multiple tools Hand-eye
Visualization coordination
Sensory deficiency

12
Additional Challenges of NOTES/SPAS
NOTES MIS

Constrained workspace
Tool dexterity Incision constraint
Complex interaction
Multiple tools Hand-eye
Tortuous access
Visualization coordination
Multiple contacts
Sensory deficiency
Imperceptible constraints

13
Additional Challenges of NOTES/SPAS
NOTES MIS

Constrained workspace
Tool dexterity Incision constraint
Complex interaction
Multiple tools Hand-eye
Tortuous access
Visualization coordination
Multiple contacts
Sensory deficiency
Imperceptible constraints
High DoF slave robots
Flexible robots
Complex telemanipulation correspondence

14
Additional Challenges of NOTES/SPAS
NOTES MIS

g-prox Endo-stitch, Eagle Claw

Constrained workspace
Tool dexterity Incision constraint
Complex interaction
Multiple tools Hand-eye
Tortuous access
Visualization coordination
Multiple contacts
Sensory deficiency
Imperceptible constraints
High DoF slave robots
Flexible robots
Complex telemanipulation correspondence
Access wound closure
15
 Challenges:
Examples From
NATURAL ORIFICE SURGERY

16
 Prelude to NOTES: MIS of the Throat

endoscope
instrument

laryngoscope
Need: Intracorporeal Knot Tying for
Functional Reconstruction
In collaboration with R. H. Taylor, Ankur
Kapoor, P. Kazanzides & Paul Flint JHU

Challenges
Distal dexterity
Increased precision
Constrained workspace

17
Example: Single Port Access

P. D.
Allen Fowler

16x

Challenges:
Complex telemanipulation
correspondence
Perception Constraints

8x
18
 Example: Access Constraints - TURBT
Resectoscope Flexure Actuationcone

R. Goldman

Actuationunit

Distaleffector Resectoscope
A. Bajo

Challenges
Firstsegment Secondsegment
Confined space

Fiberscope Constrained access

Laserprobe Perception barriers
Resectoscope

Forceps

Goldman, Bajo & Simaan, IEEE TBME 2013

19
 Tran-Nasal MIS of the Upper Airways
Current Approach: TOS: Challenges
Constrained access
R. Goldman
Safe access
Laryngoscope Perception barriers
Dr. James Netterville

Our Approach: TNROS: A. Bajo

Nasopharyngeal tube

Broncho-
Needle Fiberscope scope
Before injection
Office-Based interventions
Lower Cost+ Higher throughput
No general anesthesia
Patient comfort Trans-nasal Filler Injection After injection
Patient feedback Otology Head & Neck Surgery, 2014, 2015 20
Additional Challenges of NOTES/SPAS
NOTES MIS

Constrained workspace
Tool dexterity Incision constraint
Complex interaction
Multiple tools Hand-eye
Tortuous access
Visualization coordination
Multiple contacts
Sensory deficiency
Imperceptible constraints
High DoF slave robots
Flexible robots
Complex telemanipulation correspondence
Access wound closure
21
Additional Challenges of NOTES/SPAS
NOTES MIS

Constrained workspace
Tool dexterity Incision constraint

What is the Solution?

Complex interaction
Tortuous access
Multiple contacts
Multiple tools
Visualization
Hand-eye
coordination
Sensory deficiency
Imperceptible constraints
High DoF slave robots
Flexible robots
Complex telemanipulation correspondence
Access wound closure
22
Prerequisite: Surgeon Expertise

Anatomy Immersion Surgeon

(AR) experience

Situational
Haptics Computer awareness
Navigation

23
Solution: Complementary Situational Awareness

Immersion Surgeon Robot task

Anatomy
(AR) experience under-
standing

Situational
Haptics Computer awareness
Navigation
Robot Robot co-
complementary manipulation
perception

24
 Complementary Situational Awareness
Complementary Motivating Visible to user
situational awareness Example:
(CSA): operational
Simultaneous perception constraints Camera
and use of the beyond
environments and human
operational constraints for perception Perceived by robot
task execution.

Robots can act as our

partners, not only in
manipulation, but in
perception and control.

http://nri-csa.vuse.vanderbilt.edu/joomla/

In collaboration with H. Choset (CMU) and R.H. Taylor (JHU) 25

Intelligent Robotic Telemanipulation Slaves

Intelligent surgical robotic slaves use pre-

operative and intraoperative sensing and
information to assist the surgeon in achieving a
surgical objective

Sensing
26
 INTELLIGENT
SURGICAL SLAVES:
Examples

27
 Prelude to NOTES: MIS of the Throat

endoscope
instrument

laryngoscope

Manipulation augmentation
Distal dexterity
Increased precision

Otology Clinics of America 2008 , IJRR2009

In collaboration with R. H. Taylor, Ankur Kapoor, P. Kazanzides & Paul Flint JHU 28
Intelligent Surgical Slaves: Examples (1/7)
Stiffness Contact Active
probing detection & compliance
localization

IEEE TRO 2008, 2010 IEEE TRO 2012 IEEE TRO 2014
IJRR 2009, ASME JMR2010

Sensory augmentation
Force sensing Force/motion control Tissue
exploration
Stiffness estimation
Constraint detection

Manipulation assistance
Compliant motion
Force/motion control IJRR 2015 Robotica 2011

Automated exploration
29
Intelligent Surgical Slaves: Examples (2/7)
Stent deployment, Vision and online OCT sensory data
force sensing for assistive telemanipulation

H. Yu
W.
Wei

IEEE TRO 2009, ASME JMED 2010 K. Joos

8x 8x
ASME JDSMC 2012 Optics Express 2015 IEEE TMECH 2016

Sensory augmentation Manipulation augmentation

Force sensing Intraocular dexterity
Depth perception Telemanipulation assistance 30
Intelligent Surgical Slaves: Examples (3/7)
J.
Zhang

Manipulation augmentation
Steerable electrode arrays
Optimal insertion plans

Perception augmentation
Force sensing
Failure onset detection
ASME JMED 2009, ASME JMR 2012, IEEE
Otology Head & Neck Surg. 2009, TMECH2014
31
Intelligent Surgical Slaves: Examples (4/7)
Robust insertion without exact registration stylet
modiolar wall
J.
Pile
(PEA)
Perimodiolar
electrode array
F/T
sensor

Insertion
tool

Mastoide
ctomy

Perception Augmentation Manipulation Augmentation

Force sensing Adaptive insertion path plans
Detection of onset of failure Hybrid position/admittance control for
safe insertion
ASME JMED 2009, ASME JMR 2012, IEEE
Otology Head & Neck Surg. 2009, TMECH2014 32
Intelligent Surgical Slaves: Examples (5/7)

Perception
Augmentation
Sense abnormal
conditions
Detect ONSET of
failure
Pile & Simaan Robotica, 2016 33 33
Intelligent Surgical Slaves: Examples (6/7)
Resectoscope Flexure Actuationcone

R. Goldman

Actuationunit

Distaleffector Resectoscope
A. Bajo

Access
Constraint

Perception Augmentation
Access constraints 4x
Manipulation Augmentation Goldman, Bajo & Simaan,
Dexterous ablation IEEE TBME 2013
34
Intelligent Surgical Slaves: Example (7/7)
Perception Augmentation
Compliant motion for safe
insertion R. Goldman

Manipulation Augmentation
Dexterous intervention
A. Bajo

Nasopharyngeal tube Dr. Kelly Groom

Broncho-
Needle Fiberscope scope
Before injection
Office-Based interventions
Lower Cost+ Higher throughput
No general anesthesia
Patient comfort Trans-nasal Filler Injection After injection
Patient feedback Otology Head & Neck Surgery, 2014, 2015 35
 TECHNOLOGICAL
ENABLERS FOR SURGICAL CSA

INTRINSIC FORCE SENSING

CONTACT DETECTION & ESTIMATION
COMPLIANT MOTION CONTROL
HYBRID MOTION CONTROL FOR CONTINUUM
ROBOTS
CONTINUUM ROBOTS FOR DEEP ACCESS

36
Multi-Backbone Continuum Robots
Configuration Space:
Configuration
space

, ,
Joint Task
space space

Simaan 2003, 2004, 2005, IJRR 2009

37
Interaction Sensing: Wrench Estimation
i1
l2 l6 l3 B0 lb
l1 B0 B0 l
b2
l9 l2 l3
l1 l4 b2
l5
l2 l5 2 l l3 lb l1 b
l b3 b l6 b3 a 1
b1 la b2 l6 i2 l4
l5 8 l7 1 b3 l
i1 b1 l1 3 l6 b3 i3
l4 b2 l5 l4
General Complex Singular Complex Hyperbolic Congruence
n p2
b2 p1 l5
p3 r2 b2 B0
l2 l5 l4 r1
l4 B l3 l r l2 l3
l1 P0 B0 1 3 l3 l2 l4
b1 0 l6 b3 l1 b1
b3 l l1b1 l6
l3 5 l2 l5 l4 n
B0 l6 b2 l6
b3 (a) (b)
Flat Pencil Plane Singularity Bundle Singularities
Simaan & Shoham IEEE TRO 1999 Kai Xu

Xu & Simaan IEEE TRO 2008

Xu & Simaan IEEE TRO 2010
38
 Force Sensing: Experimental Validation

palpation dissection

Force sensing errors are within 5 in

Safer interaction with anatomy
direction and 0.34 grams in magnitude
39
 A. Bajo

CONTACT DETECTION
& ESTIMATION OF
CONTACT LOCATION

40
Interaction Sensing: Contact Detection
Detect multiple contacts
Estimate the location of
contacts
Use information for safe A. Bajo
manipulation

Bajo & Simaan IEEE TRO 2012 41

Estimation of Contact Location
k argmin r
k k rk
s .t . 0 Lk Contact estimated
k
on disk number

Disk
number
where
actual
contact is

Localize contact within an error of

1 disk height (3 mm)
42
 Roger
Goldman

COMPLIANT MOTION
CONTROL

43
 Configuration Space Stiffness
Generalized
Statics: U U
force
Generalized force Config. space
perturbation perturbation

Assumption: Ln>>Lb Roger

Goldman

Joint-level
stiffness

Second order
stiffness effects Active stiffness Passive stiffness
Goldman, Bajo & Simaan, TRO2014
44
Compliant Motion Control Algorithm
Generalized force:
Configuration
Space error

Assumption: fast
controller

Control law

Liyaponov 1 T
Stability 2

T
choose A K1

Compliant motion
Control Law
45
Compliant Motion Controller
Generalizedforce, Sensedactuationforce

kth segment:
Elasticenergy Estimateduncertainty
Augmented
Generalizedforce: uncertainty

Control Estimated
Law gen. force

SVR
estimationof
modeling
uncertainties
46
 Andrea

HYBRID FORCE/MOTION Bajo

CONTROL USING
INTRINSIC WRENCH
ESTIMATION

47
Hybrid Motion/Position Control with Intrinsic Sensing

Motion
controller

Force
controller

Wrench
Estimation

Bajo & Simaan, IJRR2015 48

 Hybrid Motion/Position Control with Intrinsic Sensing

Motion
controller

Force
controller

Wrench
Estimation

Bajo & Simaan, IJRR2015 49

 ENABLING SITUATIONAL
AWARENESS DURING
TELEMANIPULATION

N. Simaan H. Choset R. Taylor P. Kazanzides A. Deguet

P. Chalasani A. Srivatsan L Wang R. Roy E. Ayvali

50
Using Stiffness Features for Registrations
Constraints
Prior Model Registr
{S} Nonlin.
Kalman ation &
Filter model A. H.
Robot L. Wang
update Srivastan Choset
Interaction data
{R}
Constrained Kalman filter for registration using exploration data

Before registration Registered

First reported work of registration

while simultaneously using
stiffness and geometry information
51
Exploration for Updating Virtual Fixtures
A-priori Deformed
model Reality

Force controlled exploration

Exploration data Deformable registration CPD Model mediated telemanipulation

First reported update of virtual fixture

geometry based on force controlled
exploration data
L. Wang P. Chalasani R. Taylor 52
Conclusion
Online sensing of interaction is the key
Need sensory technologies
Need new telemanipulation frameworks for shared awareness
and execution of tasks
Need new frameworks for enabling intelligent surgical
robots to form understanding of the environment and task
during task execution complementary situational awareness

53
Acknowledgement of Funding

NSF Career NSF Grant NRI-Large CISE NSF CMMI

Grant # IIS- CBET # IIS-1063750 # 1537659
1063750 #0651649

NIH Grant # NIH Grant # NIH Grant # NIH Grant #

U19A1067773 PI: 7R21EB0077 7R21EB0077 1R21EB0156
D. Brenner 79 79 23-01A1

54
 Thanks to A.R.M.A. Graduate Students

Samrat Bhattacharyya
Kai Xu
Jason Pile
Wei Wei
Haoran Yu
Jian Zhang
Long Wang
Alessio Salerno
Aditia Bhowmick
Roger Goldman
Giuseppe Del-Giudice
Saleem Hamid
Rajarshi Roy
Youhua Chen
Rashid Yasin
Jienan Ding
Nima Sarli
Andrea Bajo
Zhangshi Liu

55
Thanks to Collaborations

Spiros John T Peter Dennis Howard Fine Stanley Chang

Russell
Taylor, Manolidis Roland Allen, Fowler M.D. Robert Wood Ophthalmology
Beth Israel Otolaryngo Ph.D. CS Johnson Columbia
Ph.D. Surgery
Hospital logy Columbia Medical
Johns Columbia Center
Hopkins NYU

Rob James Gaelyn George Karen Joos Duke Herrell

A.
Labadie Netterville Garrett Wanna Ophthalmology Urology
Rivas
Vanderbilt Vanderbilt

Otolaryngology Head & Neck

56