2018 IEEE 14th International Conference on Automation Science and Engineering (CASE)
Munich, Germany, August 20-24, 2018
Ultrasound–Guided Involuntary Motion Compensation of Kidney
Stones in Percutaneous Nephrolithotomy Surgery
Ishara Paranawithana, Member, IEEE, Hsieh-Yu Li, Member, IEEE, Liangjing Yang, Member, IEEE,
Terence Sey Kiat Lim, Shaohui Foong, Member, IEEE, Foo Cheong Ng, U-Xuan Tan, Member, IEEE
Abstract— Percutaneous Nephrolithotomy (PCNL) is a min-
imally invasive percutaneous surgical procedure used for large
kidney stone removal under ultrasound and fluoroscopy guid-
ance. During the surgery, precise control of handheld 2D ultra-
sound probe is required but highly challenging as it depends on
operator's experience, judgement and dexterity. To complicate
the problem, kidney stone moves away from its 2D ultrasound
image plane due to respiratory movement of the patient. This
makes locating the kidney stone extremely challenging, if not
impossible, further limiting the success of the initial needle
puncture. Therefore, there is a need to bring automation
to the intraoperative workflow to compensate out–of–plane
motion of the kidney stone. Maintaining simultaneous control of
appropriate contact force during visual tracking is also essential
to ensure accurate percutaneous access to the target calyx.
This work proposes a visual servoing framework to address the
aforesaid problems. Our proposed visual servoing framework
comes in the form of two stages namely; pre–scan and real–
time visual servoing. Probe holding robotic manipulator firstly
scans a small region around the target to construct 3D volume
data, followed by out–of–plane target tracking using image
correlation–based block matching algorithm. A position based
admittance control scheme is developed to address the latent
need of maintaining an appropriate contact force between the
probe and patient's body during visual servoing. Experimental
results show that proposed framework is able to track out–of–
plane motion of kidney stone with a position error of only one
frame while regulating the environment force feedback with
a maximum error of 0.2N. By incorporating automation to
existing surgical workflow, we hope to positively impact the
way minimally invasive surgeries are performed.
I. INTRODUCTION
Percutaneous Nephrolithotomy (PCNL) is a commonly
practiced minimally invasive surgery to remove large renal
stones. Clinical studies have shown that this operation pro-
cedure is better than open surgery due to faster recovery
time, less morbidity, less pain and better preservation of
renal function [1]. Furthermore, recent studies suggest that
PCNL is able to achieve higher stone-free rates with less
complications compared to open surgery [2].
*Research supported by CGH-SUTD HealthTech Innovation Grant.
Ishara Paranawithana, Hsieh-Yu Li, Shaohui Foong and U-Xuan Tan
are with the Pillar of Engineering Product Development, Singapore Uni-
versity of Technology and Design, Singapore (phone: +65 6303-6600;
email: ishara paranawithana@sutd.edu.sg, hsiehyu li@sutd.edu.sg, foong-
shaohui@sutd.edu.sg, uxuan tan@sutd.edu.sg)
Liangjing Yang is with the Zhejiang University/University of Illi-
nois at Urbana-Champaign Institute (ZJU-UIUC Institute), China (email:
liangjingyang@intl.zju.edu.cn)
Terence Sey Kiat Lim and Foo Cheong Ng are with the De-
partment of Urology, Changi General Hospital, Singapore (email: ter-
ence lim@cgh.com.sg, foo cheong ng@cgh.com.sg)
978-1-5386-3593-3/18/$31.00 ©2018 IEEE
PCNL is a keyhole surgery performed through a small
incision of 1cm under the guidance of ultrasound imaging
and fluoroscopy. During this procedure, a needle is punctured
into the pelvis of the kidney to create a tract between the
target calyx and the skin. Successful execution of the needle
puncture requires a lot of preoperative and intraoperative
planning. Respiratory movement of the kidney together with
involuntary motion of the surgeon makes renal stone tracking
extremely challenging. The current PCNL procedure relies
heavily on manual control of the ultrasound probe, hence,
ability to gain accurate initial access to the target depends
highly on operator's experience, judgement and dexterity.
Therefore, several needle punctures are often required for
successful percutaneous access which increase the risk of
bleeding and causing damage to the nearby organs. Despite
the advancements in image-guided surgical robots, involun-
tary motion compensation of both patient and surgeon during
PCNL surgery still remains a challenge.
Fig. 1. (a) Surgeon checking the ultrasound image of kidney stones and
(b) performing a needle puncture to gain initial access to the target calyx
of the kidney.
Ultrasound imaging is accepted as one of the most suited
imaging modality for diagnostic procedures due to low
cost and radiation free features. However, the quality of
ultrasound image greatly suffers due to uncertainties of the
scanning method. The probe has to be kept at the target in a
certain orientation for a considerable time until the surgeon
makes a successful needle puncture as depicted in Figure
1. Two important requirements have been identified for the
success of the PCNL procedure, namely: 1) maintaining
the ultrasound probe position and orientation to correctly
target a calyx or kideny stone despite the involuntary motion,
and 2) achieving an appropriate contact force between the
ultraound probe and scan area of the patient to obtain good
ultrasound images. To achieve this, we propose an ultrasound
image–guided visual servoing method for tracking out–of–
1123
plane motion of the kidney stone influenced by respiratory
movement of the patient during PCNL surgery. Furthermore,
an admittance control scheme is proposed to maintain an
appropriate contact force between the ultrasound probe and
patient's body when the operator releases the probe after
initial manual positioning. This not only provides satisfactory
image quality but also reduces burden imposed on the
surgeon.
The rest of the paper is organized as follows. Relevant
literarure is reviewed in Section II to identify the limitations
in the existing systems. Section III details the methodology
while section IV gives a description of the experimental setup
and conditions. Finally, results and discussion is presented
in Section V before concluding the paper by summarizing
the contribution and prospective future work.
II. LITERATURE REVIEW
Minimally invasive percutaneous interventions have im-
mensely advanced in the recent past due to extensive research
carried out in the development of ultrasound image–guided
robotic systems. However, automation is not readily brought
into PCNL surgical procedure because of the challenges
unique to kidney stone removal under the influence of
respiration and limitations of conventional 2D ultrasound
imaging.
Standard 2D ultrasound imaging provides only a cross–
sectional view of the organ, thus target tracking with B-mode
image can be separated into two problems namely; 1) in–
plane motion and 2) out–of–plane motion [3], [4]. Tracking
a target during in–plane motion (translational motion in x,
y axis and rotational motion in z axis) is relatively less
challenging because target is not missing from the current
frame [5], [6]. However, conventional methods do not work
in out–of–plane motion (translational motion in z axis and
rotational motion in x, y axis) as the target disappears from
the current frame when kidney is susceptible to respiratory
motion. Therefore, the surgeon has to combine a series of
2D images to form an imaginary 3D view of the anatomy.
The use of 3D ultrasound probe to obtain two orthogonal
image planes is one way to tackle the out–of–plane motion
problem [7]. However, 3D probes which usually have a larger
field of view are not commonly used in urological inter-
ventions compared to obstetric ultrasonography. Ultrasound
based visual servoing system for lithotripsy was developed
by Koizumi et al. [8], [9], [10] using two conventional
2D ultrasound probes. In this case, two ultrasound probes
were mounted on the end effector of the robotic platform
in such a way that their scanning planes are perpendicular
to each other. One of the limitations of this work is that
3D tracking can fail if the displacement of the target kidney
stone between two consecutive frames is more than 20%
of the beam thickness. In addition, this approach cannot be
readily integrated to most commercial ultrasound systems at
present as they do not allow two probes to work concurrently.
While some of the methods require substantial hardware
modifications, there are techniques that leverage direct visual
servo to track out–of–plane motion of an anatomical target.
1124
Merbaki et al. [11] proposed a visual servoing method that
utilizes single 2D ultrasound probe. However, this approach
is based on image moments and requires contour detection.
An intensity based ultrasound visual servoing approach was
proposed by Nadeau et al. [12], [13]. This is an important
contribution that addresses the need for hybrid vision–force
control. The out–of–plane visual servoing is realized through
the computation of interaction matrix using image gradients.
A proportional force control law is applied to maintain the
contact between probe and the skin. Nevertheless, in general,
it is difficult to guarantee robust control against complex
contact disturbances which is common in most surgical pro-
cedures including PCNL. A pre–scan based visual servoing
method for tracking out–of–plane motion was developed by
Nakadate et al. [3], [4]. This is an interesting work that
demonstrates the practicability of using single conventional
2D ultrasound probe without significant modifications to the
present ultrasound systems. However, the effect of force
control in intraoperative ultrasound imaging is not discussed
in this work.
Gilbertson et al. developed a handheld force controlled
device to maintain a constant force between the patients body
and ultrasound probe for the improvement of image stability
[14]. Abayazid et al. employed an implicit force control to
align the ultrasound probe with curved surfaces to assure
the maximum contact range to acquire images of sufficient
quality [15]. In [16], velocity, force and image information
were used with a shared controller for ultrasound carotid
artery diagnosis. An interesting research on simultaneous
control of contact force between the probe and the body was
carried out recently by Chatelain et al. to optimize the image
quality during ultrasound diagnostic procedures [17], [18].
While they serve as important contributions in force control
for ultrasound diagnostic procedures, the problem of tracking
out–of–plane motion due to respiration is not addressed.
While both fields have advanced remarkably, hardly any
previous research considered this important contribution of
combining out–of–plane visual servoing and robust force
control for automated target tracking during initial percu-
taneous access of the surgery. Therefore, this paper proposes
a hybrid vision–force control framework to bridge this gap
by developing an elegant workflow that seamlessly integrates
to the existing systems. Doing so provides great assistance
to the surgeons and lifts off the burden from challenging
scanning task.
III. METHODOLOGY
A. Design Concept
To address the limitations and gaps discussed in Section
II, we propose an integrated control framework to track out–
of–plane motion of the kidney stones in PCNL surgical
procedure. Although, we choose the target application as
PCNL surgery, in which involuntary movement of the patient
is predominantly due to respiration, we envision the proposed
method can be generalized to many different surgical tasks
to compensate large involuntary movements. Furthermore,
the proposed method is capable of offering great ease of
integration and operation because of two reasons. Firstly, this
method can be readily implemented on any existing standard
2D ultrasound system without any hardware modifications.
Secondly, the active probe holding robotic manipulator takes
care of maintaining an appropriate contact force amidst
contact disturbances. This minimizes the need for human
interaction and manual control of the ultrasound probe,
allowing the surgeon to focus more on executing accurate
needle puncture during the PCNL surgery.
The proposed methodology for out–of–plane motion
tracking consists of two major components namely, pre–
scanning and Real–Time Visual Servoing (RTVS). Figure
2 illustrates the proposed framework for out–of–plane
motion tracking in a graphical representation. The next
two subsections describe out–of–plane motion tracking
framework in detail followed by a discussion on the
admittance control scheme used to maintain sufficient
contact force between ultrasound probe and the patient's
body.
Fig. 2. Overview of out–of–plane motion tracking framework, including
Pre–Scanning and Real–Time Visual Servoing (RTVS) stages.
B. Out–Of–Plane Motion Tracking
1) Pre–Scanning:
Pre–scan is the first step of out–of–plane motion tracking
framework used to construct the missing 3D information
around the target kidney stone. In this process, firstly, the
surgeon has to manually place the ultrasound probe tenta-
tively at the center of the target. Then the robotic manipulator
which holds the 2D ultrasound probe scans a small area
around the target kidney stone. The purpose of pre–scan is
to record several consecutive B–mode ultrasound images at
regular intervals to construct volume data with their position
information.
PCNL surgery is typically done when the patient is in
prone position as the lower pole calyces of the kidney
are mostly, if not always subcostal [2]. Therefore, proper
selection of tracking axis for pre–scan is an important
consideration in PCNL procedure. In order to create
3D volumetric data, there are four common scanning
methods available at present namely; parallel, pivotal, tilt
and rotational scanning [19]. However, tilt and rotational
scanning methods are presented with side/end firing trans-
rectal (TRUS) probes which are commonly used in prostate
imaging [20]. A large region of interest can be scanned with
a small angular displacement by tilting the conventional
1125
probe in a fan–like geometry using pivotal scanning
method. However, the resolution of the acquired images
tend to degrade with the depth. This is a very important
consideration when selecting a suitable scanning modality
for the proposed application as target kidney stone can be
in anywhere inside a calyx. In contrast, parallel scanning
method records a series of parallel 2D images by linearly
translating the probe on patients body without significantly
affecting the image quality with the depth. Hence, this
work used parallel scanning approach for pre–scan and
subsequently in Real–Time Visual Servoing (RTVS). The
following two steps detail the pre–scan operational workflow.
Step 1: Manual Locating
The Surgeon/sonographer manually scans the kidney area
based on the information of preoperative images and place
the ultrasound probe at the center of the target. This is
considered as the initial (home) position.
Step 2: Recording 3D Volume Data
The robotic manipulator moves the ultrasound probe by
a distance of −LbN/2c from the initial position. Host PC
records pre–scan data while moving the probe by a distance
of L(N − 1) to scan a small region across the target kidney
stone. In the pre–scan, N consecutive frames at a regular
interval of L are recorded to construct the 3D volume. After
completing the pre–scan, robotic manipulator goes back to
its initial position.
There remains a unique challenge in developing a method
for out–of–plane motion tracking of kidney stones in PCNL
surgery. As the calyceal anatomical structure around the
target kidney stone can be symmetrical, the images acquired
from pre–scan to the left and right, while center being
the target are identical. Although, it is not an issue for
one directional visual servoing, it is problematic for bi–
directional out–of–plane tracking. Therefore, authors have
proposed a more practical approach to avoid the symmetrical
problem by scanning the target area at an angle of 450 with
respect to horizontal scan–line.
2) Real-Time Visual Servoing (RTVS):
Once the pre–scan is completed, the proposed system
starts real–time tracking of out–of–plane motion of target
kidneys stones. Inter–frame block matching is performed
between the current frame and all N frames recorded from
the pre–scan to find the best matched frame to the current
frame. Sum of Squared Difference (SSD) is used as the
similarity measure for the image correlation analysis. A
rectangular region of interest (ROI) which includes the target
kidney stone is selected for both current frame and pre–
scanned frames to reduce the computational complexity of
the block matching process. Calculation of SSD can be
expressed as in (1)
m
n X
X
SSD(k) = {Ik (i, j) − Ic (i, j)}2 , (1)
j=1 i=1
k = {1, 2, ..., N } ,
where Ik (i, j) and Ic (i, j) are the pixel intensity of the k th
frame and current frame respectively. mxn is the size of the
rectangular ROI used.
The best matched frame k match is chosen by evaluating
the index of the frame which has the lowest SSD(k) value.
Hence, the position error of the current frame (P ), (current
location of the probe with respect to the initial position)
along z–axis is estimated by
n l N mo
P error = k match − L.
2 admittance Y (s) is therefore designed as a first order system
A predictive model is then applied to compensate the time
delay between image processing and motion control loops.
Then, the position error and the time delay (Z), of the probe
is estimated as
Z = P error + P delay ,
where P delay = V (tdelay − T ) is the position error caused by
the time delay. V is defined as the velocity of the probe in
the previous frame, tdelay and T are delay time in the TCP/IP
loop and the sampling time, respectively. tdelay is realized
through empirical evidence in the experiments.
Based on the estimated position error (Z), the velocity
command V z is given to the probe holding robot manipulator
as in (4)
V z = γZ,
where γ is the gain of the vision controller. The goal of this
method is to find the local minimum of SSD values instead
of calculating an exact value or a distance. Thus, inter–
frame block matching is relatively robust for tracking out–of–
plane motion of kidney stones compared to the conventional
tracking methods.
C. Position–based Admittance Control Scheme
The aim of admittance control is to control the dynamics
of the contact surface. The control scheme for the environ-
ment contact is shown in Figure 3, where F y , F y out are the
desired force and output force, respectively. The force/torque
sensor only senses the environmental force denoted by F y en
makes use of a 4th order low pass filter (LPF) with a
cut–off frequency of 2Hz. P y , V y , P y out , and V y out are
the desired position, desired velocity, position output and
velocity output, respectively.
Fig. 3. Proposed position-based admittance control scheme used to control
the contact force between probe and the body.
The admittance controller, Y (s), can be described as in
(5)
Py Py 1
Y (s) = = ≡ (5)
dF F y − F y en Bds + K d
where dF is the force difference between the desired force
and the environmental force. B d and K d are the positive
constants that represent desired damping and stiffness, re-
spectively. Using a low pass filter, the environment force
(2) is delayed with a higher order transfer function. The target
to prevent divergence due to the inappropriate parameters.
The admittance can be employed to achieve a desired force
response with a low overshoot and small errors by tuning B d
and K d as in [21], [22]. The robotic manipulator is designed
with position control, not utilizing data of torque. Hence, the
(3) dynamic interaction between the robot and the environment
can be regulated smoothly and the robot will move until the
environment force is the same as the desired force.
IV. EXPERIMENTAL SETUP
Experiments are conducted on a kidney phantom to eval-
uate the effectiveness and robustness of the proposed frame-
work qualitatively and quantitatively. The experimental setup
of the proposed system is illustrated in Figure 4. In our
previous work, we developed a collaborative robot to assist
PCNL surgery [23]. Control of 2–DOFs of the end effector
(4) are considered in this paper, targeting the main objective of
out–of–plane visual servoing while maintaining appropriate
contact force with patient's body. Rest of the 4–DOFs are
position controlled to minimize the disturbance.
B–mode ultrasound images are acquired at a rate of 30
frames–per–second from a 4MHz phased–array 2D ultra-
sound probe (PA7–4/12) connected to ultrasound diagnos-
tic system (SonixTouch; Ultrasonix Medical Corporation,
Canada). A frame grabber (DVI2USB 3.0; Epiphan Sys-
tems Inc., Canada) is used to capture the images from
the DVI output of the imaging device. The resolution of
the captured images is 1024x768, subsequently, a smaller
ROI of 400x280 is used for inter-frame block matching to
reduce the computational load. The end effector of the robot
manipulator is equipped with a force/torque sensor (Nano 25;
ATI Industrial Automation Inc., USA). The control system
is implemented in CompactRIO Real Time (RT) controller
(cRIO–9035; National Instruments Inc., USA) which consists
of Xilinx Kintex–7 70T Field–Programmable Gate Array
(FPGA). A multi–modality kidney phantom with an embed-
ded stone (Yezitronix Group Automation Inc., Canada) is
used to evaluate the experiments. The size of the kidney
phantom and the stone is 15x15x9cm3 and 5x8x10mm3
respectively. Image processing is performed using MATLAB
running in a PC (Intel Core i5, 2.5GHz) with Windows 7
operating system. Terminal PC and CompactRIO controller
are interfaced through TCP/IP connection.
Number of frames recorded from the pre–scan (N ) is set
to 15 and distance between two consecutive frames (L) is
maintained at 3mm. The kidney phantom is fixed on an
1126
actuated platform at an angle of 450 for diagonal scanning
using 1–DOF probe holding manipulator. To simulate the
respiratory movement, the kidney phantom is moved along
the visual servoing axis following a periodic motion with a
frequency of 0.1Hz and peak-to-peak amplitude of 40mm.
Before starting the experiments, vision controller gain (γ)
is set to 0.8. In the admittance control scheme, desired
force (F y ) is set to 0.8N. PI control is applied for velocity
loop with a bandwidth of 15Hz. Besides that, P control is
employed for position loop with a bandwidth of 3Hz. The
target admittance is designed by tuning B d and K d to 2Ns
and 0.001N/m respectively for optimal performance.
Fig. 4. Experimental setup and the robotic system.
V. RESULTS AND DISCUSSION
A. Qualitative Observations
To evaluate the performance of the proposed out–of–
plane tracking framework, we considered a series of B-mode
images captured at a regular interval of one second, with
and without employing the tracking algorithm explained in
Section III-B-2. Figure 5 shows a graphical illustration of
the kidney stone, calyces and the acoustic shadow during
half cycle of the respiration. Kidney stones are usually
observed as bright white spots in the B–mode ultrasound
images because of the hyper echogenicity properties of the
stone compounds. To verify the exact location of the kidney
stones, surgeons usually use very important B–mode image
feature known as acoustic shadow. Therefore, it is important
to qualitative analyze the shape and the position of the
acoustic shadow with and without tracking scenarios. Figure
5a) shows an example where both kidney stone and the
acoustic shadow disappear from the image at the end of
inspiration, with tracking algorithm disabled. However, using
the proposed tracking algorithm, acoustic shadow is always
kept visible on all the images as illustrated in Figure 5b).
Another experiment is conducted to examine the effect of
applied contact force on the quality of the image obtained. In
this experiment, the operator scans the target region manually
using the 6–DOF robot and acquires a reference image which
is clear and of acceptable quality. This step is analogous to
how a surgeon would typically obtain the initial image of the
1127
Fig. 5. B–mode images during half cycle of respiration with tracking
a) disabled and b) enabled. Second, third and fourth images from the top
corresponds to beginning of inspiration, end of inspiration and beginning of
expiration, respectively.
kidney stone. The reference frame is chosen in such a way
that all the important interest points in the kidney phantom
including kidney stone, acoustic shadow and surrounding
calyces are visible. After the operator releases the US probe,
a fixed contact force is applied and visual servoing is
performed. The contact force is then progressively varied
in each trial from 0N to 1.4N.
A set of images acquired during visual servoing is com-
pared against the reference frame and corresponding SSD
values for k match for each trial are reported in Figure 6. It
is observed that SSD value swiftly decreases as the contact
force increases, however, it tends to slow down and stabalize
after 0.8N. Lower SSD value translates to higher confidence
of achiveing better image quality with respect to the chosen
reference frame. Thus, this result suggests that 0.8N is an
appropriate force to conduct visual servoing considering the
conditions of the phantom used for experiments. Moreover, it
can be confirmed by qualitative empirical observations of the
ultrasound images as image quality tends to get better with
a larger contact force. A contact force of 1N was used on
abdominal training model as well as ex-vivo porcine samples
in [17] and similar performances were reported in [13] which
Fig. 6. Sum of squared difference (SSD) value of kmatch frame is plotted
against the applied contact force in each trial; Minimum SSD value is
reported at 0.8N.
makes our result consistent with the current development.
Figure 7 shows the variation of SSD with respect to
recorded pre-scan data of 15 frames while regulating the
contact force to 0.8N. If the local minimum of the curve
occurs at the middle frame of the pre-scan (k=8), the position
error can be correctly determined using the proposed method.
A clear minimum is observed at k=8 as depicted in Figure
7. This result further demonstrates that 0.8N of contact
force is a good fit and does not negatively impact tracking
performance of out–of–plane visual servoing.
Fig. 7. Variation of SSD values for pre–scan of 15 frames; A convex
shaped plot is obtained while a clear minimum is achieved at k = 8.
B. Quantitative Evaluation
Figure 8 shows tracking performance of the ultrasound
probe over a period of 30 seconds. It corresponds to k match
results obtained from SSD based inter–frame block matching
algorithm. It can be observed that ultrasound probe is able
to follow the simulation of the respiratory motion with a
maximum error of one frame. Hence, k match value fluctuated
only between two adjacent frames resulting effective tracking
without losing important visual information. A typical kidney
stone removed using PCNL surgery is about 20mm or larger
in diameter. Hence, minimum distance resolution between
1128
Fig. 8. Variation of Kmatch value over a period of 30 seconds; frame number
8 (k=8) is chosen as the initial center position before starting target tracking.
Maximum tracking error of one frame (3mm) stays within the kidney stone
region (1cm) and is acceptable for PCNL surgery.
two consecutive frames is selected as 3mm by considering
the size, shape and the important visual features of the
kidney stone. The shape and the structure of the target is
a dominating factor when the distance is large and speckle
noise is more appealing if the distance is small [4]. By
choosing a relatively larger distance, simple yet computation-
ally efficient SSD–based tracking can tolerate a satisfactory
level of noise which could be present in the form of tissue
deformation and in–plane motion of the kidney stone.
Results of the proposed admittance control scheme which
helps to keep the probe in contact with the kidney phantom in
a perpendicular direction to the tracking axis, are presented in
Figure 9. The applied force is regulated by the environment
force feedback by setting the desired force as the bottom–
line. The reasons for small fluctuations around the desired
force can be identified as follows. The scanning surface of
the kidney phantom is softer than real human skin, thus
making force regulation on the kidney phantom challenging.
This introduces complex contact disturbances which are not
common with human skin. Noise of the force/torque sensor
and the friction between phantom surface and probe also
add slight uncertainties to the force feedback. However,
the proposed method is able to maintain an appropriate
contact force during out–of–plane motion tracking without
significantly affecting the tracking performance.
VI. CONCLUSION AND FUTURE WORK
A visual servoing framework for out–of–plane motion
tracking of kidney stones in PCNL surgery is proposed in this
paper. The main objective of this study is to develop a robust
system to compensate involuntary motion of both patient
and surgeon while incorporating an autonomous method to
maintain an appropriate contact force between ultrasound
probe and the patient's body. By doing so, we expect to
provide better real–time image guidance to the surgeon.
Therefore, bringing automation to such procedures greatly
enhances the accuracy and success rates of the surgery while
reducing the possibility of human error.

Fig. 9. Performance of the position-based admittance control scheme;
measured environmental force is regulated at 0.8N with a maximum error
of 0.2N.
The proposed method can be further extended to several
clinical applications where out–of–plane motion problem is
present due to large respiratory movements. This includes,
but not limited to needle biopsies of lung and liver and many
other minimally invasive procedures discussed in [9], [24].
Potential future work will focus on conducting experiments
on real animal/human tissue to track out–of–plane motion of
kidney stones under the influence of respiration and employ-
ing an online estimation technique to build up 3D volumetric
data during pre–scan while the patient is breathing as normal.
Thereby, we envision to expose new possibilities in the
way minimally invasive percutaneous interventions are being
performed.
ACKNOWLEDGMENT
The authors would like to thank Assistant Professor Lujie
Chen from Singapore University and Technology and Design
for providing ultrasound diagnostic system and necessary
equipment. Further, we extend our gratitude to Ms. Jennifer
Lauv from Massachusetts Institute of Technology and Mr.
Zhong Hoo Chau from Singapore University and Technology
and Design for the mechanical design of the manipulator
used during the course of this research.
R EFERENCES
[1] A. Koroglu, T. Togal, M. Cicek, S. Killc, A. Avas, and M. Ersoy, ”The
effects of irrigation fluid volume and irrigation time on fluid electrolyte
balance and hemodynamics in percutaneous nephrolithotripsy,” in
International Urology Nephrology, vol. 35, no. 1, pp. 1-6, 2003.
[2] R. Ko, F. Soucy, J. D. Denstedt, and H. Razvi, ”Percutaneous
nephrolithotomy made easier: a practical guide, tips and tricks,” in
BJU International, vol.101, no.5, pp. 535–539, 2008.
[3] R. Nakadate, A. Safwan, H. Ishii, A. Saito, A. Takanishi, and M.
Hashizume, ”3D tracking of respiratory liver movement by a robot
assisted medical ultrasound,” in Computer Aided Surgery, Springer,
Tokyo, pp. 17–25, 2016.
[4] R. Nakadate, J. Solis, A. Takanishi, E. Minagawa, M. Sugawara, and
K. Niki, ”Out–of–plane visual servoing method for tracking the carotid
artery with a robot–assisted ultrasound diagnostic system,” 2011 IEEE
International Conference on Robotics and Automation, Shanghai, pp.
5267–5272, 2011.
[5] S. Golemati, J. S. Konstantina, and S. Nikita, ”On the use of
block matching for the estimation of arterial wall motion,” 2008 8th
IEEE International Conference on BioInformatics and BioEngineer-
ing, Athens, pp. 1–5, 2008.
1129
[6] P. Abolmaesumi, S. E. Salcudean, Wen-Hong Zhu, M. R. Sirouspour,
and S. P. DiMaio, ”Image–guided control control of a robot for med-
ical ultrasound,” in IEEE Transactions on Robotics and Automation,
vol.18, no.1, pp. 11–23, Feb 2002.
[7] C. Nadeau, A. Krupa, J. Petr and C. Barillot, ”Moments–based
ultrasound visual servoing: from a mono to multiplane approach,” in
IEEE Transactions on Robotics, vol.32, no.6, pp. 1558–1564, Dec.
2016.
[8] N. Koizumi, S. Warisawa, M. Nagoshi, H. Hashizume, and M. Mit-
suishi, ”Construction methodology for remote ultrasound diagnostic
system,” in IEEE Transactions on Robotics, vol.25, no.3, pp. 522–
538, June 2009.
[9] D. Lee, N. Koizumi, K. Ota, S. Yoshizawa, A. Ito, Y. Kaneko, et
al., ”Ultrasound–based visual servoing system for lithotripsy,” 2007
IEEE/RSJ International Conference on Intelligent Robots and Systems,
San Diego, CA, pp. 877–882, 2007.
[10] N. Koizumi, J. Seo, D. Lee, T. Funamoto, A. Nomiya, K. Yoshinaka,
et.al., ”Robust kidney stone tracking for a non–invasive theragnostic
system: servoing performance and safety enhancement,” 2011 IEEE
International Conference on Robotics and Automation, Shanghai, pp.
2443–2450, 2011.
[11] R. Mebarki, A. Krupa, and F. Chaumette, ”2-D ultrasound probe
complete guidance by visual servoing using image moments,” in IEEE
Transactions on Robotics, vol.26, no.2, pp. 296–306, April 2010.
[12] C. Nadeau, A. Krupa, J. Gangloff, ”Automatic Tracking of an Organ
Section with an Ultrasound Probe: Compensation of Respiratory
Motion,” Int. Conf. on Medical Image Computing and Computer
Assisted Intervention, MICCAI’11, Toronto, Canada, 2011.
[13] C. Nadeau and A. Krupa, ”Intensity-Based Ultrasound Visual Servo-
ing: Modeling and Validation With 2-D and 3-D Probes,” in IEEE
Transactions on Robotics, vol. 29, no. 4, pp. 1003-1015, Aug. 2013.
[14] M. W. Gilbertson, and B. W. Anthony, ”Force and position control
system for freehand ultrasound,” in IEEE Tranactions on Robotics,
vol.31, no.4, pp. 835–849, Aug. 2015.
[15] M. Abayazid, P. Moreira, N. Shahriari, S. Patil, R. Alterovitz, and
S. Misra, ”Ultrasound–guided three dimensional needle steering in
biological tissue with curved surfaces,” Medical Engineering and
Physics, vol.37, no.1, pp. 145–150, 2015.
[16] W.-H. Zhu, S. E. Salcudean, S. Bachmann, and P. Abolmaesumi, ”Mo-
tion/force/image control of a diagnostic ultrasound robot,” Proceedings
2000 ICRA. Millennium Conference. IEEE International Conference
on Robotics and Automation Symposia Proceedings San Francisco,
CA, pp. 1580–1585 vol. 2, 2000.
[17] P. Chatelain, A. Krupa, and N. Navab, ”Optimization of ultrasound im-
age quality via visual servoing,” 2015 IEEE International Conference
on Robotics and Automation (ICRA), Seattle, WA, pp. 5997–6002, ,
2015.
[18] P. Chatelain, A. Krupa, and N. Navab, ”Confidence–driven control of
an ultrasound probe: target–specific acoustic window optimization,”
2016 IEEE International Conference on Robotics and Automation
(ICRA), Stockholm, pp. 3441–3446, 2016.
[19] A. Fenster, D. B. Downey and H. N. Cardinal, ”Three–dimensional
ultrasound imaging,” Physics in Medicine and Biology, vol.46, no.5,
pp. 67–99, 2001.
[20] H. Bassan, T. Hayes, R. V. Patel and M. Moallem, ”A novel ma-
nipulator for 3D ultrasound guided percutaneous needle insertion,”
Proceedings 2007 IEEE International Conference on Robotics and
Automation, Roma, pp. 617–622, 2007.
[21] G. Zeng, and A. Hemami, ”An overview of robot force control,” in
Robotica, Cambridge University Press, New York, vol.15, no.5, pp.
473–482, 1997.
[22] N. Hogan, ”Impedance control: an approach to manipulation–Part
I ,” in Transactions of the ASME Journal of Dynamic Systems,
Measurement and Control vol.107, pp. 1–7, 1985.
[23] H.-Y. Li et al., Stable and Compliant Motion of Physical Human-
Robot Interaction Coupled with a Moving Environment using Variable
Admittance and Adaptive Control, IEEE Robotics and Automation
Letters,, vol. 3, no. 3, pp. 2493-2500, July 2018.
[24] J. Hong, T. Dohi, M. Hashizume, K. Konishi, and N. Hata, ”An
ultrasound–driven needle–insertion robot for percutaneous cholecys-
tostomy,” in Physics in Medicine and Biology, vol.49, no.3, pp. 441–
455, 2004.