Visual Servoing of a Parallel Robot System

1
A. Traslosheros ,
1
J. M. Sebastin,
2
L. ngel,
3
F. Roberti,
3
R. Carelli
atraslosheros@etsii.upm.es, jsebas@etsii.upm.es, langel@etsii.upm.es,
froberti@inaut.unsj.edu.ar, rcarelli@inaut.unsj.edu.ar.

1
Universidad Politcnica de Madrid, DISAM, Madrid, Espaa.
2
Universidad Pontificia Bolivariana, Bucaramanga, Colombia
3
Instituto de Automtica, Universidad Nacional de San Juan, San Juan, Argentina.

Abstract This paper describes the visual control of a parallel
robot called RoboTenis. The system has been designed and
built in order to carry out tasks in three dimensions and
dynamical environments, thus the system is capable to interact
with objects which move up to 1m/s. The control strategy is
composed by two intertwined control loops: The internal loop is
faster and considers the information from the joins, its sample
time is 0.5 ms. Second loop represents the visual servoing system
which is external loop to the first mentioned, second loop
represents the study purpose, that it is based in the prediction of
the object velocity which is obtained form visual information
and its sample time is 8.3 ms. Lyapunov stability analysis, system
delays and saturation components has been taken into account.
Keywords Parallel robot, visual control strategies, tracking,
system stability.
I. Introduction
The vision systems are becoming more frequently used in
robotics applications. The visual information makes possible
to know about the position and orientation of the objects
presented in the scene and the description of the environment
with a relative precision. Although the above advantages, the
integration of visual systems in dynamical works presents
many topics which are not solved correctly yet, thus many
important investigation centers [1] are motivated to
investigate about this field, such as in the Tokyo University
([2] and [3]) where fast tracking (up to 2 m/s) strategies in
visual servoing are developed. In order to study and
implementing the different strategies of visual servoing, the
computer vision group of the UPM (Polytechnic University of
Madrid) has decided to design the ROBOTENIS. Actually the
implemented controller make possible to achieve high speed
dynamical works. In this paper some experiments made to the
system are described, the principal purpose is the high
velocity (up to 1 m/s) tracking of a small object (black ping
pong ball) with three degrees of freedom. It should be
mentioned that the system is designed with higher attributes
in order to response to future works. Typically the parallel
mechanisms possess the advantages of high stiffness, low
inertia and large payload capacity; however, the principal
weakness is the small useful workspace and design
difficulties. As shown is fig. 1, mechanical structure of
RoboTenis system is inspired in DELTA robot [4].
Kinematical model, Jacobian matrix and the optimized model
of the RoboTenis system have been presented in other
previous woks [5]. Dynamical analysis and the joint
controller have been presented in [6] and [7]. The dynamical
model is based upon Lagrangian multipliers thus, is possible
to use forearms of non-negligible inertias in the development
of control strategies. Two control loops are incorporated in
the system: the joint loop is a control signal calculated each
0.5 ms, at this point dynamical model, kinematical model and
PD action are retrofitted. The other loop is considered
external since is calculated each 8.33 ms; this loop uses the
visual data and is described in detail along this work.

Fig. 1. RoboTenis System
II. DESCRIPTION OF THE SYSTEM
This section describes the experimental environment, the
elements that are part of the system and the functional
1-4244-0830-X/07/$20.00 2007 IEEE.

characteristics of each element. For more information: [8].
For more information see [8].
A. Experimentally Environment
The objective of the system resides in the tracking of a
ping pong ball along 600 mm. Image processing is
conveniently simplified using a black ball on white
background. The ball is moved through a thread (fig. 2), and
the ball velocity is close to 1 m/s.


Fig. 2. Work environment
B. Vision System
RoboTenis system has a camera located on the end-
effector as the fig. 2 shows. This camera location merge two
important aspects; when the robot and the object are relatively
faraway each other, the field of view of the camera is wide
but some error are generated in the ball position
measurement; when the ball and robot are near each other, the
field of view of the camera is narrow but the precision of the
ball position measurements is increased. The above
mentioned will be usable in future works as catching or
hitting to the ball. Thus some characteristics of the
components are the following.

Camera. The used camera is the SONY XC-HR50 and its
principal characteristics are: High frame rate (8.33ms) and
240 x 640 pixels resolution; the used exposure time was 1 ms;
progressive scan CCD; relatively small size and little weight:
29x29x32mm and 50gr, see fig. 3.


Fig. 3. Camera on the RoboTenis system

Data Acquisition Card. The data acquisition system is
composed by the Matrox Meteor 2-MC/4 card and is the
responsible of visual data and is used in double buffer mode,
the current image is sampled meanwhile the previous image is
being processed.
Image Processing. Once images are acquired; the visual
system segments the ball on white background. The centroid
and diameter of the ball are calculated using a subpixel
precision method. The 3D position of the ball is possible to
calculate through preliminary camera calibration. The control
system requires the velocity of the ball which estimated
through Kalman filter ([9] and [10]) from the ball position.
C. System of Positioning Control
The positioning system is composed by DSPACE 1103
card which is the responsible of: the generation of the
trajectories, the calculation of kinematical models, dynamical
models and control algorithms. The motion system is
composed by AC brushless servomotors, Ac drivers and
gearbox (for more information see [8]).
D. Characteristics and Functions
The visual controller is conditioned by some
characteristics of the system and there are others inherited by
the own application; some of them are:
- High uncertainly in data from the visual system. The
small sample data (8.333 ms) makes bigger the errors of the
velocity estimation. For example, if the ball is located 600mm
from the robot (camera) then the diameter of the ball it will be
measured in 20 pixels and, if the estimation of the position
has an error of 0.25 pixels, then the estimated distance error
will be 8mm approximately, in the same order the estimation
of the ball velocity error will be 1 m/s. The errors above cited
origin an elevated discontinuity and the control action
required exceeds the robot's capacities; the Kalman filter
implemented helps to solve the problem.
- The target speeds for the robot should be given
continually. In order to avoid high accelerations, the
trajectory planner needs continuity in the estimated velocity.
For example, an 8 mm error in one sample rate (8.333ms) will
demand acceleration equivalently to 12 times the gravity. The
system guarantees a soft tracking performance by means of a
trajectory planner; the trajectory planner is specially designed
for reference shifting, thus the target point can be changed at
any moment.
- Some additional limitations have to be considered in the
system. The delay between the visual acquisition and the joint
motion is estimated in two visual servoing sample rate
(16.66ms). This delay is attained to some principal reasons as
image capturing, image transmission and image processing.
The maximum velocity the end-effector of RoboTenis is
2.5m/s.

III. VISUAL CONTROL OF THE SYSTEM
The coordinated systems considered are shown in the fig.
4; they are
w
,
e
, and
c
which represent the word coordinate
system, the end-effector robot system and the camera
coordinate system respectively. Other notations defined are:
c
p
b
represents the position of the ball in the camera coordinate
system,
w
p
e
represents the position of the robot end effector in
the word coordinate system;
w
p
e
is obtained by means of the
direct kinematical model. The rotation matrix are constant
and known as
w
R
e
,
w
R
c
,
e
R
c
;
e
T
c
is obtained by means of the
camera calibrations.

Fig. 4. Coordinated considered systems.

Although several alternatives exists [11], the controller
selected is based in the ball position. Schematic control can
be appreciated in the fig. 5, the error function is obtained
though the difference between the referenced position (
) (
*
k p
b
c
)
and the measured position (
) (k p
b
c
) this difference must be
constant because the goal is the tracking of the ball. Once the
error is obtained, the controller calculates the velocity of the
end effector. By means of the trajectory planner and the
Jacobian matrix, all the joint motions are calculated. k
indicates the sample time considered.

( ) k e ( )
* c
b
p k
+
-
( ) u k ( ) k q
( )
c
b
p k
Vision
system
ROBOT
Jacobian
matrix
Controller

Fig. 5 The Visual servoing uses the ball position
A. Visual Servoing Model.
In the fig. 5 can be observed that the position error can be
expressed as follows:
) ( ) (
*
k p p k e
b
c
b
c
=

(1)
If ball position is referenced in the coordinate system of
the camera then the position of the ball can be expressed as:
)) ( ) ( ( ) ( k p k p R k p
c
w
b
w
w
c
b
c
=

(2)
If (2) is substituted into (1) then we obtain:
)) ( ) ( ( ) (
*
k p k p R p k e
c
w
b
w
w
c
b
c
=

(3)
The system is supposed stable and in order to guarantee
that the error will decrease exponentially, thus we choose:
0 ) ( ) ( > = con k e k e

(4)
Deriving (2) and supposing that w
c
R
is constant, we obtain:
)) ( ) ( ( ) ( k v k v R k e
c
w
b
w
w
c
=

(5)
Substituting (3) and (5) into (4), we obtain:
)) ( ( ) ( ) (
*
k p p R k v k v
b
c
b
c T
w
c
b
w
c
w
=

(6)
Where
) (k v
c
w
and
) (k v
b
w
represent the camera and ball
velocities respectively. Since
[ ] ) ( ) ( k v k v
c
w
e
w
=
the control law
can be expressed as:
[ ] ) ( ) ( ) (
*
k p p R k v k u
b
c
b
c T
w
c
b
w
=

(7)
The equation (7) is composed by two components: a
component which predicts the position of the ball (
) (k v
b
w
)
and the other contains the tracking error (
[ ] ) ( ) (
*
k p k p
b
c
b
c

).The
ideal control scheme (7) requires a perfect knowledge of all
its components, which is not possible, a more realistic
approach consist in generalizing the previous control as
[ ] ) ( ) ( ) (
*
k p p R k v k u
b
c
b
c T
w
c
b
w
=

(8)
As is shown in (8), the estimated variables are represented
by the carets which are used instead of the true terms. The
basic controller is shown in the fig. 6.

( )
* c
b
p k ( ) k e
+
-
+
+
( ) u k
( )

w
b
V k
+
+
( )
w
b
p k
( ) k p
c
w
( ) k q
( )

c
b
p k
( ) k p
e
w
-
+
e
c T
w
c
p R
ROBOT
Vision
system
Velocity
estimator
Jacobian
matrix
Forward
kinematics
T
w
c
R
T
w
c
R
Fig. 6. Visual servoing control system architecture.
B. Adjusting The Parameter.
Fundamental aspect in the performing of the visual
servoing system is the adjustment of thus, will be
calculated in the less number of sample time periods and will
consider the system limitations. This algorithm is based in the
future positions of the camera and the ball; this lets to the
robot reaching the control objective (
) ( ) (
*
k P k P
b
c
b
c
=
).

The future position (in the k+n instant) of the ball in the
word coordinate system is:
Tn k v p n k p
b
w
b
w
b
w
) ( ) ( + = +

(9)
Where T is the visual servoing sample time period (8.333
ms) and the future position of the camera in word coordinate
system in the k+n instant is:
Tn k v p n k p
c
w
c
w
c
w
) ( ) ( + = +

(10
)
The control objective is reaching the target position in the
shorter time as be possible, thus if we substitute: the target
position, the future ball and camera position into (2):
[ ] 0 ) ( ) (
*
= + + n k p n k p R p
c
w
b
w
w
c
b
c

(11)
Substituting (9) and (10) into (11), we obtain (12).
[ ] Tn k v k p Tn k v k p R p
c
w
c
w
b
w
b
w
w
c
b
c
) ( ) ( ) ( ) (
*
=

(12)
It has been indicated that
) (k v
e
w
=
) (k v
c
w
, if (2) is considered
the control law is:
[ ] ) (
1
) ( ) (
*
k p p R
Tn
k v k u
b
c
b
c T
w
c
b
w
=

(13)
If (8) and (13) are compared, we can obtain the
parameter as:
Tn
1
=

(14)
The equation (14) gives a criterion for adjust as a
function of the number of samples required (n) for reaching
the control target.
C. Implemented Algorithm
The visual servoing architecture proposed in fig. 6, doesnt
consider the physical limitations of the System such as delays
and maximum system operation components. In fig. 7 we
propose the visual control structure that considers the
limitations above mentioned.

The term
r
z

represents a delay of r periods respecting

the control signals. If we consider that the visual information
(
) (k p
b
c
) has a delay of 2 sampling times (r=2) with respect to
the joint information, then at an instant k+n, the future
position of the ball can be obtained by:
) ( ) ( ) ( ) ( r n T r k v r k p n k p
b
w
b
w
b
w
+ + = +

(15)
The future position of the camera is given by (10). Using
the (14) is possible to adjust the for the control law by
considering the following aspects:
-The wished velocity of the end-effector is represented by
(13). In physical systems the maximal velocity is necessary to
be limited. In our system the maximal velocity has a direct
effect in the minimal number of sampled time periods (T)
which the target function can be achieved. The number of
sampled time periods (T) is not the same in all joints, thus this
data in the more restrictive joint (the bigger time) will be used
in the parameter calculation. It is desirable that the velocity
of the robot can be continuous.
Position
filter
( )
* c
b
p k ( ) k e
+
-
+
+
( ) u k
( )

w
b
V k r
( ) k q
( )
c
b
p k r
( ) r k V
c
w

( )

c
b
V k r
+
+
( )

c
b
p k
+
+
( )
w
b
p k r
( ) r k p
c
w
( ) k p
c
w
( ) k p
e
w
-
+
e
c T
w
c
p R
( )

c
b
p k r
ROBOT
Vision
system
Velocity
estimator
c T
w
R
c
w
R
r T
-
+
r
z

r
z

Jacobian
matrix
Forward
kinematics
T
w
c
R

Fig. 7. Visual servoing control proposed architecture.

D. System Stability.
By means of Lyapunov analysis is possible to probe the
system stability; it can be demonstrated that the error
converges to zero if ideal conditions are considered;
otherwise it can be probed that the error will be bounded
under the influence of the estimation errors and unmodeled
dynamics.
For the stability analysis we consider (5), and (7), to obtain
the close loop expression:

0 ) ( ) ( = + k e k e

(16)
We choose a Lyapunov function as:
) ( ) (
2
1
k e k e V
T
=

(17)
0 ) ( ) ( ) ( ) ( < = = k e k e k e k e V
T T

(18)
Equation (18) implies that
0 ) ( k e
when
k
; but if
u v
e
w

is not true then:

) ( ) ( ) ( k k u k v
e
w
+ =

(19)
Where
) (k
is the velocity error vector which is produced
by the bad velocity estimations and unmodeled dynamics. We
consider (19) and, (16) can be written as:
) ( ) ( ) ( k R k e k e
w
c
= +

(20
)
We consider again a Lyapunov function as:
) ( ) (
2
1
k e k e V
T
=

(21)
) ( ) ( ) ( ) ( ) ( ) ( k R k e k e k e k e k e V
w
c T T T
+ = =
(22)
The sufficient condition for 0 < V

is:

> e

(23)
If we consider that
0 ) ( k
(this has two means: The
velocity controller is capable to make the system to reach
u k v
e
w
) (
and there is not error in the velocity estimations)
then
0 ) ( k e
; otherwise we can conclude that when (23) is
not fulfilled, the error does not decreases and remains finally
bounded by:

e

(24)

Fig. 8. 3D ball movements
IV. EXPERIMENTAL RESULTS
In this section, the experiments results are related to visual
tracking object up to 1 m/s. The results show the performance
of the visual servoing algorithm proposed for the RoboTenis
system. The control objective consists in preserving a
constant distance ([0, 0, 600]
T
mm) between the camera and
the moving target. The ball is hanged to the structure and is
moved by manual drag (see fig 2). Different arbitrary
trajectories have been performed. As example, in fig. 8 is
represented an experiment in which the space evolution of the
ball is shown.
A. Performance Tracking Indexes.
The arbitrary movement of the ball makes complicated the
systematic study of the experiments; in consequence two
tracking indexes have been defined. The tracking indexes
have been defined according to the tracking error (difference
between the target and real position) and the estimation of
the ball velocity. Two indexes are:
Tracking Relation (table I) is defined in (25) as the relation
between the average of the tracking error module and the
average of the estimated velocity module of the ball. This
relation is expressed in mm/1/mm/s and allows isolating the
result of each experiment from particular features of motion
of the ball made in other experiment.

=
=
=
N
k
b
w
N
k
k v
N
k e
N
relation Tracking
1
1
) (
1
) (
1

(25)
Average of the tracking error in function of the estimated
ball velocity. In order to makes an easier comparison; the
estimated ball velocity has been divided in 5 groups which
the tracking error is shown in the table II.
Table I. Tracking relation using a proportional and
predictive control laws
ALGORITHM Tracking relation
Proportional 40.45
Predictive 20.86

Table II. Grouped Average tracking Error
Velocity /
Algorithm
V<200
200-
400
400-
600
600-
800
V>800
Proportional 6.3 13.7 20.1 26.2 32.5
Predictive 4.2 8.1 9.5 11.3 13.5
B. Studied and Compared Control Laws
Two control laws have been used:
Proportional control law: It does not consider the
predictive component of (8), thus:
[ ] ) ( ) ( ) (
*
k p k p R k u
b
c
b
c T
w
c
=
(26)
Predictive control law: Considering the predictive
component of (8), thus:
[ ] ) ( ) ( ) (
*
k p k p R v k u
b
c
b
c T
w
c
b
w
=
(27)
The Table I and Table II show the indexes result when the
two control laws are applied. The Numerical results presented

have been obtained from the average of 10 experiments of
each control algorithm. The results show a better performance
of the system when the predictive control algorithm is used,
thus a smaller tracking relation error and smaller grouped
average tracking error is observed. Figures 9 and 10 show the
tracking error for the control laws which were used fin order
to carry out the tracking task.

Fig. 9. Proportional control law. (a) Tracking error

Fig. 10. Predictive control law. (a) Tracking error.
V. CONCLUSIONS
In this paper a novel visual servoing structure is presented,
this control strategy is used in a parallel robot in order to
reach a high velocity tracking of an object which moves in
unknown trajectories. RoboTenis has been the first parallel
robot known which uses a visual tracking control system in
dynamic environments. The control strategy is based in
obtaining the smaller number of time samples in which the
control target function is able to achieved. The existing delays
in the system, the saturations in the velocity and acceleration
are considered in de control model thus, this consideration
makes possible that RoboTenis can move faster than its
characteristics allow it. The Lyapunov stability of the
proposed system was probed under some ideal and non ideal
conditions. When the conditions are ideal, it was probed that
the error converges to zero otherwise, if the conditions are
nonideal, the error will be finally bounded.
The experiments were carried out in order to illustrate the
high performance of the system was; a ball tracking was
successfully achieved and the error was smaller than 20mm,
the visual loop has a sample time of 8.33 ms. In future Works,
new control strategies are considered in order to attain a
tracking velocity of 2m/s. With the same purpose, the
interaction of the system with the environment and (or) works
as caching or hitting the ball are desirable.
For more information consult:
http://www.disam.upm.es/vision/projects/robotenis/
References
[1] 1Kragic, D., Christensen, H.I. (2005). Advances in robot vision.
Robotics and Autonomous Systems. 52 (1), 1-3
[2] Kaneko, M., Higashimori, M., Takenaka, R., Namiki, A., Ishikawa, M.
The 100 G capturing robot - too fast to see. IEEE/ASME Transactions
on Mechatronics. Volume 8, Issue 1, pp 37 44. March 2003
[3] Senoo, T., Namiki, A., Ishikawa, M. High-speed batting using a multi-
jointed manipulator. 2004 IEEE International Conference on Robotics
and Automation, ICRA '04. Volume 2, pp 1191-1196, 26 Apr 1
May, 2004.
[4] R. Clavel,, DELTA: a fast robot with parallel geometry. 18
th

International Symposium on Industrial Robot, pp. 91-100, 1988,
Sydney, Australia.
[5] ngel, L., Sebastin, J.M., Saltarn, R., Aracil, Sanpedro, J. (2005).
RoboTenis: Optimal Design of a Parallel Robot with High Performance.
IEEE/RSJ International Conference on Intelligent Robots ans Systems
(IROS). 2-6 August 2005. Alberta, Canad.
[6] ngel, L., Sebastin, J.M., Saltarn, R., Aracil, R., Gutirrez, R. (2005).
RoboTenis: Design, Dynamic Modeling and Preliminary Control.
IEEE/ASME AIM2005, 24-28 July 2005, Monterey, California USA.
[7] ngel, L., Sebastin, J.M., Saltarn, R., Aracil. RoboTenis System.
Part II: Dynamics and Control. 44 IEEE Conference on Decision and
Control and European Control Conference (CDC-ECC05), Sevilla,
2005.
[8] ngel, L. Control Visual de Robots Paralelos. Anlisis, Desarrollo y
Aplicacin a la Plataforma Robotenis. Tesis Doctoral de la Universidad
Politcnica de Madrid. Diciembre 2005
[9] Gutirrez, D. Estimacin de la posicin y de la velocidad de un objeto
mvil. Aplicacin al sistema RoboTenis. Proyecto Fin de Carrera de la
E.T.S.I.I. de la Universidad Politcnica de Madrid
[10] Gutirrez, D., Sebastin, J.M., ngel, L. Estimacin de la posicin y
velocidad de un objeto mvil. Aplicacin al sistema Robotenis. XXVI
Jornadas de Automatica, 6-8 septiembre 2005, Alicante
[11] Hutchinson, S.A., Hager, G.D., Corke, P.I. (1996): A tutorial on visual
servo control. IEEE Trans. Robotics and Automation, 12-5 651-670.
[12] Carelli R., Santos-Victor J., Roberti F. and Tosetti S., Direct visual
tracking control of remote cellular robots, Robotics and Autonomous
Systems. Volume 54, pp. 805-814 (2006).