You are on page 1of 12

36 Int. J. Mechatronics and Automation, Vol. 3, No.

1, 2013

Rotary object dexterous manipulation in hand:

a feedback-based method

Qiang Li*
Research Institute for Cognition and Robotics (CoR-Lab),
25, 33615, Bielefeld, Germany
*Corresponding author

Martin Meier
Neural Information Group/CITEC,
21-23, 33615, Bielefeld, Germany

Robert Haschke and Helge Ritter

Research Institute for Cognition and Robotics (CoR-Lab),
25, 33615, Bielefeld, Germany

Bram Bolder
Honda Research Institute Europe,
Carl-Legien-Strasse 30, 63073 Offenbach/Main, Germany

Abstract: We focus on manipulating the object of unknown shape, weight and friction properties
in hand task. Unlike the traditional method considering the manipulation as multi times grasp
planning, we propose the robust and reactive manipulation strategy. Firstly, we design the
compact closed-loop control (local controller) to manipulate object in the neighbour area of
its current configuration. Then, we employ global planner to manage the role of fingers
(active/passive) and modify robot hand to new comfort grasp posture for new local manipulation.
A physics-based simulation setup which is similar with real robot hand is used to prove our
manipulation concepts – using four fingers manipulate one rotary object in hand. Only
commercial available sensors feedback values are used in simulation in order to facilitate the
algorithm transferring to real robot platform. Result shows the object can be manipulated in ideal
and simulated artificial noise cases.

Keywords: multi-fingered hand manipulation; reactive plan; finite state machine.

Reference to this paper should be made as follows: Li, Q., Meier, M., Haschke, R., Ritter, H. and
Bolder, B. (2013) ‘Rotary object dexterous manipulation in hand: a feedback-based method’,
Int. J. Mechatronics and Automation, Vol. 3, No. 1, pp.36–47.

Biographical notes: Qiang Li received his PhD in Pattern Recognition and Intelligence Systems
from Shenyang Institute of Automation(SIA), Chinese Academy of Sciences (CAS) in 2010. In
2009, he was awarded the HRI stipend and started his postdoctoral researching at CoR-Lab
of Bielefeld University. Currently, he is a Scientific Researcher in neuroinformatics group at
Bielefeld University. His research interest include: mobile robot navigation and control,
dexterous manipulation of multifingered robot hand, visual/tactile servo control and optimal
parameter estimation based on filter theory.

Martin Meier received his diploma in Informatics in the Natural Sciences in 2010. Since then, he
joined the neuroinformatics group at the University of Bielefeld as a PhD student and is currently
working in the CRC ‘Alignment in Communication’. His major interests are neural networks for
robot control and perception.

Copyright © 2013 Inderscience Enterprises Ltd.

Rotary object dexterous manipulation in hand: a feedback-based method 37

Robert Haschke is heading the Robotics Group within the Neuroinformatics Group at Bielefeld
University. He received his PhD in Computer Science from Bielefeld University in 2004. His
research focuses on reactive and adaptive control algorithms to endow anthropomorphic robot
hands with manual intelligence, i.e., the ability to autonomously acquire and refine manipulation
skills from observation and exploration. His research also focuses on software integration aspects
to realise cognitive control architectures for complex robot systems.

Helge Ritter is the Head of the Neuroinformatics Group at the Faculty of Technology, Bielefeld
University. His main interests are principles of neural computation and intelligent systems, in
particular cognitive robots with ‘manual intelligence’. In 1999, he was awarded the SEL Alcatel
Research Prize and in 2001 the Leibniz Prize of the German Research Foundation DFG. He is the
co-founder and the Director of the Bielefeld Cognitive Robotics Laboratory (CoR-Lab) and
coordinator of the Bielefeld Excellence Cluster ‘Cognitive Interaction Technology’ (CITEC).

Bram Bolder received his diploma from the Physics Department of the Bergische University
at Wuppertal, Germany in 2000. He received his PhD degree from the Physics Department
of the Ruhr-University at Bochum, Germany, in 2005. He worked at the Institute for Neural
Computation at the Ruhr-University from 2000 till 2005. Since then, he is a Senior Scientist at
the Honda Research Institute Europe and is working in the field of intelligent robotics and
intelligent driver assistance systems. His research interests are system architecture for intelligent
artefacts, computer vision and tracking.

This article is a revised and expanded version of a paper entitled ‘Object dexterous manipulation
in hand based on finite state machine’ presented at IEEE ICMA, Chengdu, 5–8 August 2012.

1 Introduction avoided the complex analysis of geometric relations and

apply state-of-the-art motion planning methods like RRT
Object dexterous manipulation in multifingered robot hand
(Yashima, 2004) and PRM (Saut et al., 2007) to the
is one important research issue in humanoids community.
manipulation problem. Along this line, firstly, they employ
This task requires that the object is manipulated and
physics-based simulation (Miller and Allen, 2004) to model
changed its pose to new configuration, which is one
the grasping and manipulation processes. Then grasp poses
common requirement in our human’s everyday life (i.e.,
optimised w.r.t. certain quality criteria (Miller and Allen,
rotate one object for vision inspection, change grasp posture
1999) become arranged in a pose graph (Phoka and
and position for more comfortable manipulation). In
Sudsang, 2009) to plan manipulation sequences by
the machine learning and motion plan community,
RRT/PRM. Xue et al. (2008) assume that all possible grasp
multifingered robot hand manipulation is always selected as
points were detectable by physics simulation approach and
productive research background because multifingered
dynamically plan the contact points sequence by optimising
robot hand’s high dimension continuous joints variable
the grasp quality. They tackled the problem of screwing a
cause the known ‘curse of dimensionality’. When the high
light bulb. The pure planning method’s drawback is that
dimension planning problem is combined with the
offline planning can not consider the uncertainty during the
uncertainty interaction (between the hand and the object)
course of implementation, so such plan can lead to disaster
problem, the situation is becoming more complex.
manipulation result at the run time. Kragic et al. (2001)
Roughly, there are three major research lines proposed
show the potential application of integrating the planning
to solve manipulation task. The first method is based on the
idea and monitoring implementation. They added feedback
analysis representation of dexterous manipulation. These
at the monitoring level to perform the simple known
theoretical approaches assume various things to be known:
geometry object grasp and transport task by using the
the hand kinematics, object properties like shape, mass and
simulation interaction to serve as the forward predict model.
mass distribution, the contact locations and friction
The previous two research line depends on prior
coefficients, and the local surface geometry of both the
knowledge in different level, and researchers always assume
object and finger tips. Based on this knowledge, it is
that this knowledge will be given the robot manually. The
possible to compute joint-level finger trajectories in an
third research line comes from automation feedback control
offline fashion, and even determine slipping and rolling
opinion, which is quite possible to lead to fully autonomous
motions of the fingertips (Murray et al., 1994). This class of
manipulation. This research line depends on the
method only can cope with the known model’s object
self-contained sensors to detect robot hand proprioceptive
state and observe the interaction with environment, then the
Comparing the analysis method, another important
robot hand can learn the manipulation skill by the trial and
research line is to adopt planning idea. Some researchers
error method. Currently, this research line is still in its initial
employ the optimisation technology to find the high quality
stage, and much work still focus on the known object
grasp points (Roa et al., 2008), even the manipulation
manipulation and unknown object inaccurate manipulation.
sequence (Phoka and Sudsang, 2010). Other researchers
i.e., employing the fast tactile feedback, Ishihara et al.
38 Q. Li et al.

(2006) propose a control law to spin a pen of known shape Based on the primitives, complex tasks can be easily coded
at an impressive speed. Tahara et al. (2010) point out a into programming. This behaviour organisation structure
method to manipulate objects of unknown shape. They use a based on the primitives can facilitate:
virtual object frame determined by the triangular finger-tip
1 the hierarchical task planning (Off and Zhang, 2012)
configuration of a three-fingered hand to derive a control
law to manipulate the object’s pose. However, without 2 the autonomous exploration learning for manipulation
explicit sensory feedback, their method is limited in task.
This paper is arranged as following. In Section 2, we
Another challenge in object in hand manipulation task is
summarise the needed assumption in our algorithm. In
the limited robot hand workspace. In order to realise the
Sections 3 and 4, we describe local controller and global
large-scale manipulation of the object, it is necessary to
planner respectively. In Section 5, we discuss dexterous
employ the finger gaits to perform the object’s regrasp
tasks, DMPL and explain how to code the finger/fingers
while it is held in hand. Phoka and Sudsang (2009) used
primitives into programming and how to hierarchical
heuristic approach to cluster the large number of discrete
construct robot hand skill based on the exist finger/fingers
contact points to find the representative stable regrasp
primitives. In Section 6, we introduce the physical
points, and this feasible regrasp points will be organised in
simulation setup and give the simulation results. In
the form of graph. In this way, it is possible to search and
Section 7, we remark our algorithm robustness for the
find a path from initial grasp state to the final grasp state.
unexpected disturbance. Finally, Section 8 summarises our
Furukawa et al. (2006) used three fingers robot hand, fast
work and suggests future work.
vision feedback and the predefined grasp posture to re-grasp
the thrown object. They use vision to estimate the contact
position of the middle finger.
2 Assumptions of control strategy
We follow the third research line and go beyond its state
of the art. Concretely, we propose to employ the 2.1 Point contacts
feedback-based manipulation strategy to realise the
unknown object large-scale manipulation in hand. The The real contact geometry between the fingertip and the
whole manipulation process is divided into two stages: a object is complex and difficult to model, even if geometric
local manipulation and a globally acting regrasp shape information is available. Usually, the contact force is
manipulation. The local controller reactively moves the distributed on a larger contact area. In this paper, we assume
object by a small amount only. Regrasp planning is that there is only one contact point on every finger and the
employed to adapt the grasp configuration. Subsequently, distributed contact force is concentrated on this point. We
local manipulation is continued. In feedback-based do not explicitly model friction properties. However, the
manipulation strategy, we need vision feedback to estimate physical simulation adopts a Coulomb friction model
the object pose, joints angle and tactile feedback to estimate approximating circular friction cones by four-sided
the contact position and contact force along the contact pyramids. The contact is assumed to be compliant, due to
normal vector. The feedback value can be easily accessed elasticity of either the body or the finger tip. This
from the modern robot hand system, i.e., shadow robot assumption is important to realise the contact force
hand. Currently, we obtain this feedback from a physical controller on top of a joint position controller employing a
simulation, which is used to show the feasibility of the linear spring model.
approach. To confirm the applicability of our method in
noisy real-world feedback, we add artificial noise to the 2.2 Micro manipulation assumption
accurate sensor readings obtained from simulation. In our control strategy, we assume that all contact locations
Our contribution in this paper is one practical stay fixed for every control cycle, i.e., contact frames Ci
manipulation strategy. Especially: relative to the object frame O and relative to the finger tip
1 To propose local controller to manipulate the object in frames Fi do not change (see Figure 4). This also implies,
the neighbour area of object current configuration. This that the contact points on the object and on the finger tip
local controller includes one position planner, one force move with identical velocity. Obviously, this assumption
planner and one force/position composite controller will be often violated in practice, i.e., by occurring slip.
However, employing the feedback from observed contact
2 To use FSM to manage the finger gaits to realise the locations we can determine changes of the grasp
large-scale manipulation. configuration (including sliding and rolling) after each
3 To discuss about the implementation of dexterous control cycle.
manipulation primitives library (DMPL) and introduce
how to define the formalised the finger/fingers 2.3 Robot hand kinematics, object property
primitives as the structural task.
We assume the manipulated object is rotary and surface
smoothly. We also assume the known robot hand kinematics
tree model from palm to phalanges part. These are
Rotary object dexterous manipulation in hand: a feedback-based method 39

reasonable assumptions because most of commercial robot compute the contact locations pi′ associated to the target
hand provides such data. Combining this kinematics tree pose and subsequently obtain joint angles realising those
model and known model of tactile sensors array on the contact locations employing the inverse hand kinematics (Li
fingertips, we can online estimate the robot hand forward et al., 2011). Because the local object and finger tip
kinematic model from palm to the contact frame while the geometries as well as grasp stability measures are not
fingertip is contacting the object. explicitly taken into account, the actual grasp configuration
might have changed after application of the computed hand
pose. This corresponds to sliding or rolling contacts or even
3 Reactive manipulation strategy to a loss of a contact. In order to maintain stable contacts
Conventional grasp and manipulation planning methods anyway, we apply a force-control scheme additionally to the
(Cole et al., 1989; Zribi et al., 1999) uncoupled the planning position-controlled object manipulation. All components of
from the control stage. The planning stage strongly the closed-loop control system are summarised in Figure 2
depended on global knowledge about the geometry of the and will be detailed in the following. The only exception is
object and fingertips. Some work also explicitly considers the servo controller for the finger joints, which is described
spherical finger tips to facilitate the geometry-based in Roethling et al. (2007).
planning process (Tahara et al., 2010). Furthermore, the
friction coefficients for all contacts are required to evaluate 3.1 Obtaining object pose and contact locations
grasp stability to obtain optimally stable grasps.
The minimal requirements for deliberative object pose
In real world scenarios, especially when handling
control are the knowledge of current object pose and coarse
unknown objects, this information is not available. Here, we
contact point locations. We assume that the object pose can
propose to employ tactile feedback to estimate contact
be estimated from vision, i.e., employing markers or modern
positions and forces and introduce a manipulation strategy
3D vision approaches. However, we do not consider these
solely based on this feedback. If friction properties and joint
vision aspects within this paper, but obtain the object’s pose
torques are not available anymore, we cannot actively
from physical simulation in our experiments.
control rolling and slipping anymore, because internal
The contact locations can be obtained from tactile finger
forces cannot be designed. However, as we will show, local
tip sensors providing enough spatial resolution. For
object manipulation is possible without explicitly designing
example, our shadow robot hands are equipped with tactile
all details of physical hand-object interaction.
sensors comprising 34 tactile elements (tactels), thus
Based on an estimation of the current pose O of the
providing a spatial resolution of approx. 3 mm (cf.
object and its target pose O′, we derive the required object
Figure 3). From the known shape of the finger tip and the
motion M to realise the target pose within the next control
known hand kinematics, we can easily compute the location
cycle (Figure 1). The box in the figure shows the one step
of a contact spot relative to the coordinate frame of the
plan. Knowing the current contact locations pi, we
palm, employing the forward kinematics. This is enough for
determine the current grasp configuration, i.e., contact
hand-only manipulation.
positions relative to the object’s frame. Assuming a static
grasp configuration within each control cycle, we can easily

Figure 1 Incremental manipulation of object pose O (see online version for colours)

Figure 2 Components of closed-loop control scheme

40 Q. Li et al.

Figure 3 Tactile sensor covered by a conductive foam contact force feedback (obtained directly or indirectly), but
(see online version for colours) only the force magnitude is available from tactile sensors.
Following concepts from Tahara et al. (2010), the central
idea is to plan the force direction such, that the resultant
moment will be zero, and to plan the force magnitudes
along these directions such that the resultant force applied to
the object becomes zero.
Obviously the resultant moment is zero, if the contact
force directions of all fingers intersect in one point. As
illustrated in Figure 4, we chose the intersection point as the
Figure 4 Planner employs centroid p of contact points

(see online version for colours) centroid p of all contact points pi, i.e., p = 1
pi .
Accordingly, normalised contact force direction vectors can
be computed as follows:
p − pi
fˆi = , (3)
p − pi

where fi denotes the contact force of ith finger. The force

magnitudes fi are constrained by

∑ ∑ f ⋅ fˆ = 0,
i =1
fi =
i =1
i i (4)

which defines a system of linear equation. Denoting the

matrix of normalised force directions with Fˆ ∈ \ 3× N and
the vector of desired force magnitudes with F ∈ \ N , we
3.2 Contact position planning can summarise the latter equation in matrix form:
Denoting the current and targeted object pose with O and O′
Fˆ ⋅ F = 0 (5)
and the current contact position pi resp., we can easily
compute the transformation matrix M describing the We find a positive solution (fi > 0) to this equation using
required finite object motion: singular value decomposition (SVD). As the problem is
O′ = O ⋅ M ⇔ M = O −1 ⋅ O′. (1) under determined, there exists a non-zero null space of Fˆ .
The desired solution can be expressed as a linear
Assuming, that contact positions do not move relative to the combination of the null space basis. The superposition
object within the control cycle (micro manipulation coefficients should be selected as small as possible to
assumption), we can calculate the new contact positions p′i improve the grasp stability under the condition of the
(w.r.t. the palm). The contact positions pio expressed manipulability of the object.
relative to the object frame stay fixed:
3.4 Composite position + force controller
pio = O ′ ⋅ pio = O ⋅ M ⋅ O −1pi . (2)
This controller’s task is to map the desired translational
From this, we can compute the required positional changes motion of contact positions as obtained from the position
Δpi = p′i − pi for all contact points as input to the inverse planner as well as the contact force deviation obtained from
hand kinematics. the force planner to a composite control signal determining
the resulting translational motion of each fingertip.
Conventional solutions for a simultaneous position and
3.3 Contact force planning
force control are the hybrid position/force controller and the
A mere kinematic consideration of the problem is not indirect force control. The hybrid position/force controller
sufficient. In order to maintain a stable grasp and to not decouples the control problem based on the task constraint
break the object, we have to control contact forces as well. defined by the contact frame: Force is controlled along the
Conventional contact force planners strive for a globally surface normal of the contact, while position is controlled in
optimal contact force distribution ensuring grasp stability, the tangent plane.
i.e., all contact forces staying within corresponding friction However, in our scenario, both components cannot be
cones, the totally applied force exactly resisting external separated, because there is a motion component along the
forces (i.e., gravity), and limiting local contact forces. This force direction and vice versa. That is why we prefer the
general solution is meaningful only if the contact force is indirect force control scheme and propose the composite
controllable. However, we assume that there is no 3D position + force controller. The schema diagram is shown in
Rotary object dexterous manipulation in hand: a feedback-based method 41

Figure 5. The control signals vector (3D) u1 and u2 from T: thumb; F: forefinger; M: middlefinger; R: ringfinger.
both branches are additively superimposed to form the ‘Exploration’ means that the finger serves as active role and
composite translational motion u of a single finger tip send contact the object with very small contact force and explore
to the inverse kinematics module. The position controller is its neighbour area. ‘Grip’ means that the exploration finger
realised as a P controller parameterised by diagonal gain change it role to grip finger and contact the object with
matrix k Pp , while the force controller is a PI controller desired contact force planed by contact force planner.
‘Rotate’ means that grip fingers locally rotate the object. In
parameterised by diagonal gains matrix k Pf and k If . A PI state space:
controller is used in order to guarantee a higher priority
level for force control. TM is transform matrix from the S1: All are grip fingers
contact frame reference frame, which can transform the
S2: TAM grip fingers and R exploration finger
scalar force magnitude deviation Δf into the vector-valued
error represented in the reference frame. The magnitude S3: TFR grip fingers and M exploration finger
deviations along the tangent surface of contact frame were S4: TMR grip fingers and F exploration finger
left as the unmodelled error. The stiffness coefficient kstiff
transfers the force deviation to a displacement error. The S5: FMR grip fingers and T exploration finger
subscription i is used to express the time instant. The whole
controller can be summarised as The whole object manipulation process is shown in
Figure 6, which can be formalised as a FSM. This is a
( ∫ )
′ closed statechart and it is possible to transit from current
u = k Pp ⋅ Δp + kstiff ⋅ TM ⋅ 0, 0, k Pf ⋅ Δf + k If ⋅ Δf (6)
state to the other arbitrary state in the statechart. Such state
transition must pass by the medium state-four finger holding
Figure 5 Composite position + force control scheme the object.
FSM starts from all fingers grasping the object in a
comfort manipulation posture and all fingers are defined as
grip fingers. Every finger use the same role switching
process, so only ring finger switch is analysed here as an
example. Initially manipulation state is S1, and action 4
(A4) is taken after high level task is given (i.e., rotate the
object 10 degree around the X axis in the reference
4 Regrasp planner frame). After this action is finished, the process goes
Local manipulation controller can only modify the object into state 2 (S2) and ring finger serves as exploration finger
pose in a small scale since the workspace of multifingered and contacts the object with a small enough desired force
hand is limited. It is necessary to employ a global planner to (i.e., 0:1). TFM are grip fingers and their desired
change grasp posture to a new ‘comfort’ configuration in force/position are planned by the local force/position
which new fine manipulation can be possible. Our global planner to stably grasp object. Small enough contact force
regrasp planner was inspired by human manipulating object of exploration finger is used in order to not damage the
experience – using three passive fingers continuously other three finger stable grasping (they assume the contact
rotating the object a small scale (i.e., 10 degree), one finger force of ring finger is 0). It also provides the contact
actively regrasping object in the appropriate point which information about the exploration finger, which is helpful
can facilitate the new cycle passive fingers rotation. In this for searching new feasible grasp point. At state 2 (S2),
paper, we focus on using four fingers to design global action 2 (A2) is taken and TFM–three grip fingers are
regrasp planner in order to prove the regrasp concept used to rotate the object. This is a local rotation
feasibility although we are admit that pinky is very useful as manipulation, we can use the local manipulation algorithm
auxiliary to make grasp motion more smoothly. It can be developed in 3. The exploration finger still contacts the
structured as Figure 6. The global planner will work in object with small contact force and try to explore its
discrete domain. It will not process the low level sensor neighbour feasible contact points in the object surface. After
feedback but the abstract representation of raw feedback. a small rotation is finished, action 3 (A3) is taken and ring
We use state-action space defined in the semantic level to finger releases its exploration finger role and changes to be
describe the planner. In action space: the grip finger. Its desired contact force is changed from
small enough to the desired contact force planed by local
A1: TFMR Rotate A2: TFM Rotate contact force planner (for four finger grasp). Contact point
A3: R Grip A4: R Exploration can select the previous grasp point on the object because of
A5: TFR Rotate A6: M Grip the specialising of rotary object. In this case, manipulation
process reaches back to state 1 (S1). More deliberate
A7: M Exploration A8: TMR Rotate
‘comfort’ grasp posture problem is formulated as
A9: F Grip A10: F Exploration
exploration grasp point optimisation which can be found in
A11: FMR Rotate A12: T Grip
our another work (Li et al., 2012).
A13: T Exploration
42 Q. Li et al.

5 DMPL and structural task description primitives to realise the complex manipulation task
according to transition conditions (events) in structural
5.1 The definition of DMPL environment. We show one human-robot hand interaction
In order to facilitate the global planner implementation, it manipulation statechart which cover the tasks which can be
will be helpful to refine the abstract primitive sets from the realised by DMPL in Figure 7. Firstly, we assume the object
task on hand. DMPL provides such an abstract primitives is fixed. This is a reasonable assumption because the object
set which can be defined in discrete domain and serve as the can be held by human and transferred to the robot hand in
basic element for the high level dexterous manipulation human robot interaction application. The robot hand can
planning. Currently, these primitives include: grasp the object by telling it predefined contact point on the
object. Before the robot hand contact the object, it will use
• Finger position control in Cartesian space. the finger position control primitive to drive the all
This abstract primitive is used to drive the finger to manipulation fingers to approach the setting points. All
approach one setting spatial point without the fingers position control primitives are in the orthogonal
interaction with the object. It can be instantiated by states. Once contact is detected by the tactile sensor on the
assigning a spatial position parameter or a vision fingertip, the fingers composite control primitive is
feature. employed. When all fingers are in the composite control
primitive and stable grasp condition are met, the
• Finger composite control in Cartesian space. manipulation will switch to the unfrozen object state.
This abstract primitive is used to drive the finger to Human can release the object, and robot hand will hold the
contact one setting spatial point with a setting contact object in the setting configuration (predefined pose of the
force. It can be instantiated by assigning a spatial object). Then human can send command to robot hand to
position and contact force parameter. This primitive is perform task (i.e., small scale object manipulation, finger
mainly used in the object manipulation. regrasp, and large-scale object manipulation, etc.). Such
tasks are exclusive in current software version. At last,
• Finger hybrid control in Cartesian space. human can send command to end task.
This primitive is similar with composite control This manually planner can work well in no unexpected
primitive, but using different control law to realise the event happen. It, however, cannot robustly deal with the
instantiation primitive. Composite control primitive following dynamic, open-end manipulation environment.
superimposed the output of force/position controller.
• Irregular object surface and unknown manipulation
Hybrid control primitive, however, uncouple the force
and position planning in the contact frame. It plans
contact point position in the contact tangent surface and The principle of hard coding object manipulation is to
plans contact force in the normal vector. This primitive transfer human manipulation intelligence to robot hand.
is mainly used in finger exploring on the object surface. Human model the manipulation process and translate
Using the previous three primitive, object 3D small this manipulation process into programming language.
scale motion – one composite manipulation action This method can work very well in the known object
(object-centred) can be realised. This action can be (human manipulated). It is difficult to generalise the
instantiated by assigning the desired object pose and setting manipulation process to the unknown object. i.e., if the
the manipulation fingers. robot hand has no idea about the object surface
geometry, it will not know when it has to leave the one
Figure 6 High level: global regrasp planner surface to stride across the edge of the object to contact
a new surface. Which surface it should contact in order
to keep the stable grasping. On the other hand, when
the operation environment is open end, the robot hand
does not know what is the next task. The programmer
can not predict all tasks the robot hand need to
performed and coding it in the compile time. When the
new task is coming, the programmer has to re-write

• Unexpectedly fingers leave the object surface.

The phenomenon happens because of the conflicting

between the kinematics planning domain and dynamics
implementation domain. Sometimes although planning
5.2 Structural task description in kinematics level is correct, the execution in
dynamics level is not so stable because of poor measure
DMPL have been coded in C++ successfully. Programmer
accuracy and model error
can manually write action sequence based on such
Rotary object dexterous manipulation in hand: a feedback-based method 43

Figure 7 Human robot hand interaction object manipulation statechart

Figure 8 Data structure definition sequence for every finger and for desired object motion.
With this data structure, it is very convenient and intuitive
to code the FSM described in Figure 6. Every subtask
(action) can be structured as: desired entry state, desired
output state, action primitive, transition condition (for
finger), and desired object motion and how to interpolate
this desired motion into tiny step (see in Figure 1). The
source code of DMPL and structural task description can be
checked out in Li (2012).

6 Simulation

In order to solve such problems, we propose to use the The object manipulation algorithm is validated in a physical
structural task description command and auto-replan simulation experiment. We use the Vortex physics engine to
mechanism. Auto-replan mechanism will replan task obtain real-time contact information (i.e., contact position
sequence while the contingency happen between the and contact force magnitude), and the object’s pose (object
anticipation state and the current perception state, then the position and orientation). Currently two geometric
replan result will be automatically ‘translated’ into the primitives, namely sphere (radius = 2.5 cm), cylinder
structural task description command. We have finished (radius = 1.4 cm, height = 9 cm) are evaluated. The tested
defining and coding such structural command, and how to objects are sized middle-scale compared to the robot hand,
model the manipulation behaviour, auto replan task and so rolling and slipping between the fingertips and the object
‘translate’ the task into the task description will be our will occur during the course of the manipulation. Object
future work. In order to formalise the structural task shape and size information and friction information are not
description, the main data structure is defined as in Figure 8. available to the manipulation strategy. The simulation
Class FI state is used to describe the finger state to show scenario is shown in Figure 9 resembling our real robot
whether the finger contact the object, whether the finger is setup to facilitate future transfer into real world, once the
active finger or passive finger. Class PTask is primitive task required tactile feedback is robustly available from finger
definition. This class define: which primitive task is tip sensors. The controller gains k Pp , k Pf , k If are manually
implemented, what is the entry state and output state while set to guarantee the stability of the object manipulation in
the primitive is implemented and what the transition all dexterous simulation.
condition of this primitive task is. CTask is complex task The whole manipulation process comprises three
which are composed of the primitive task temporal phases:
44 Q. Li et al.

1 Grasping of the object, when it is fixed in the world. position offset between the no noise case and noise case,
This is necessary to achieve a successful grasp without which are caused by the biased actively stable grasp with
kicking the object off. noisy measurement in unfreezing object process.
2 Unfreeze the object and stabilise the grasp employing Figure 10 Object motion parallel to contact normals (y-axis)
active force control in order to prepare manipulation. (see online version for colours)
3 Actually manipulate the object, i.e., change its pose
relative to the palm of the hand.
Only the latter manipulation phase is considered in this
paper. Without loss the generality, the orientation of the
object is aligned to the world reference frame, such that the
principal axes of cylinder or box are parallel to the z-axis of
the world frame. The object is grasped with the thumb
opposing three fingers. The contact normals of all contacts
are roughly aligned to the y-axis of the world frame.
Simulation experiments are used to validate our
manipulation strategy in four aspects.
1 To show it can deal with the unknown object geometry,
friction coefficient.
2 To show it can deal with the uncertainty measurement
from the object’s pose and contact position/force
magnitude. According to the equations (1) and (2), Figure 11 Object motion around (y-axis) (see online version
matrix M represents the uncertainty of object’s pose for colours)
measure and pi represents the uncertainty of contact
position. So the desired linear velocity calculation is
effected by the two uncertainty in position planner. We,
however, focus the uncertainty on one parameter, i.e.,
contact point position in the simulation since the two
uncertainty are coupled relation.
3 To show the ring finger role switching process.
4 To show one periodical of finger role switching.

Figure 9 Simulation scenario (see online version for colours)

Figure 12 Tracking a figure eight (see online version

for colours)

• Simulation 1: The object’s transition/rotation and

complex manipulation trajectory tracking simulation
with/without the superimposed artificial measurement
noise to the values obtained from the physics engine.
Motion scale is 0.5 cm/0.2 rad along/around the y-axis and
tracking figure – ‘8’ (height and width are 1 cm and 0.5 cm
resp.) The standard deviations of the added Gaussian noise
are: 0.5 cm for contact positions (the spatial accuracy of our
available tactile sensor is 0.3 cm), and 0.3 for contact force
magnitude (desired contact force is 1.0). The exemplary
simulation results are shown in Figures 10, 11 and 12 in
order to observe noise affection. There are tiny initial
Rotary object dexterous manipulation in hand: a feedback-based method 45

Simulation results with/without noise measurement show • Simulation 4: One periodical of finger role switching.
that although we did not explicitly modelling the fingertip The contact force result is shown in Figure 17. It
rolling on the object, our proposal manipulation strategy showed that ring finger, middle finger, forefinger and
deal with this unexpected phenomenon by the compact thumb served exploration finger in sequence to
control-closed-loop and realise the unknown object local implement the continuous rotation of the object. The
manipulation (simple and complex trajectory tracking). green solid line means the desired contact force and
Noisy feedback simulation also shows that the complex blue solid line is the actural contact force
object manipulation tracking performance is not as good as
the ideal manipulation. Tracking error (maximum) is less
Figure 13 Stochastic experiment object manipulation in-hand
than 20%. Multi simulation results, changing the controller
(see online version for colours)
parameters in equation (6) show that the composite
controller parameters have the direct affection on the tracing
error accuracy. An adaptive controller parameters regulation
algorithm will be helpful in increasing the tracking accuracy
comparing with this manual setting parameters.

• Simulation 2: The sensitivity evaluation of object

manipulation to the measurement error. The object
(cylinder) is moved 0.5 cm along y axis (step
responding test).
In this simulation, we continuously change the variance of
contact position (step = 0.1 cm) and contact force
magnitude (step = 0.1) to check the manipulation accuracy
under the difference noise measure and the manipulation Figure 14 Manipulation error average value (see online version
stochastic performance. The qualitative and quantitative for colours)
tests are simulated to check the sensitivity of our strategy to
measurement parameters uncertainty. In Exp1, we add no
force magnitude noise but change the error variance of
contact position from 0.1 cm to 1 cm. In Exp2, we add a
fixed position noise (variance is 0.5 cm) and the force
magnitude noise whose variance change from 0.1 to 1.
Experiments are run ten times for every parameter. The
qualitative test result is shown in Figure 13. The x axis
means the test sequence and y axis means the successful
times. Manipulation failure analysis shows the reasons of
failure are:
1 unsuccessfully grasp in the first stage (19 times)
2 the finger collision (22 times)
Figure 15 Manipulation error variance (see online version
3 finger move from one contact surface to another one for colours)
Not considering grasp failure factors, our manipulation
strategy successful rate is 87:3%. The quantitative results
are shown in Figures 14 and 15. Manipulation error
accuracy is lower than 20% and the contact force magnitude
measurement error has important affection on the
manipulation error variance.

• Simulation 3: Exploration finger contacts the object

with the desired force.
In Figure 16, ring finger is exploration finger and it contact
the object with small contact force – 0.1. Other three fingers
use the contact planner to design the desired contact force.
The green solid line means the desired contact force and
blue solid line is the actural contact force.
46 Q. Li et al.

Figure 16 Exploration (ring) finger contact the object practice this depends on the friction properties. If the
(see online version for colours) planned contact force direction is located inside the friction
cone, the reactive contact force plan can be realised.
Otherwise slip between the fingertip and the object will
occur. In the next control cycle, the new desired contact
force direction and magnitude will be planned based on the
update contact situation.
From equation (6), it can be seen that the control signal
is composed from two components. The integral component
of the force controller guarantees that force trajectory
tracking is prior level at the steady state – at the expense of
losing positional control accuracy. The contact point motion
will stop at an equilibrium position with a tiny positional
error (10–2 mm) but almost no force deviation. If the integral
contribution would be missing, both force deviation and
position deviation would exist simultaneously. We also
consider about the contribution of two components. Position
controller parameter k Pp can be used to modified the
Figure 17 Finger roles switching (see online version tracking velocity. The higher value can improve the tracking
for colours) accuracy, but can cause the vibration of object
manipulation. The stiff coefficient kstiff can be use to modify
the compliance of the grasp. The lower value can improve
the smooth performance of manipulation but can cause the
decreased manipulation accuracy performance. So an online
adaptive controller parameters regulation algorithm is more
useful in order to balance the manipulation accuracy and
tracking velocity.

8 Summary
We propose a reactive control strategy to realise
manipulation motions for unknown objects. In contrast to
traditional manipulation strategies, which require a lot of
information about the object and plan in an offline fashion,
our plan method developed in local manipulation level and
global manipulation level is in an online fashion and
employs minimal sensory information. Abstract primitives
are refined to facilitate the implementation of global
manipulation planning. FSM is employed to manage
manipulation state transition and improve the object
7 Discussion manipulation robustness. In physical simulation
In the proposed strategy, both contact position planning and experiments, we proved the feasibility of the manipulation
contact force planning are reactive. Both planners work strategy.
independently of each other and their results are combined Several research issues have not been considered in this
by the composite position/force controller, which thus can paper but they are also important for successful object
be regarded as a plan coordinator. manipulation.
If unexpected disturbances change the object’s pose, the 1 Gravity issue. In our simulation, we only assume the
position planner will drive the fingers reactively to light object.
counteract these disturbances. Consider for example the
grasp stabilisation process, when the initially frozen object 2 Finger collision avoidance.
is released to freely move. Because the desired contact force 3 Guarantee of positive force magnitudes.
during the initial grasping stage is chosen arbitrarily,
unbalanced forces will trigger unexpected object motions, So far, every finger motion is controlled independently. A
when the object is released. The fingers will move along the finger motion coordination strategy should be used to avoid
surface of the object to counteract this motion. collisions between fingers and thus to improve the
In the contact force planning process, we assume that robustness of the manipulation strategy. The contact force
the desired contact force magnitude is realisable. But, in magnitude of all fingers should always be positive, because
Rotary object dexterous manipulation in hand: a feedback-based method 47

only ‘pushing’ forces can be applied. However, in our Miller, A.T. and Allen, P.K. (2004) ‘Graspit! A versatile simulator
simulations we only observed null space basis vectors for robotic grasping’, Robotics & Automation Magazine,
IEEE, Vol. 11, No. 4, pp.110–122.
having identical sign, such that positive force magnitudes
could be chosen in all cases. But no strict mathematical Murray, R.M., Li, Z.X. and Sastry, S.S. (1994) A Mathematical
Introduction to Robotic Manipulation, CRC Press, Boca
proof is given here. These three issues will be addressed in
Raton, Florida.
our future work.
Off, D. and Zhang, J. (2012) ‘Continual HTN planning and acting
in open-ended domains – considering knowledge acquisition
opportunities’, in Proceedings of the 4th International
Acknowledgements Conference on Agents and Artificial Intelligence, pp.16–25.
Qiang Li gratefully acknowledges the financial support Phoka, T. and Sudsang, A. (2009) ‘Contact point clustering
approach for 5-fingered regrasp planning’, in IEEE/RSJ
from Honda Research Institute Europe for the project International Conference on Intelligent Robots and Systems,
‘Autonomous exploration of manual interaction space’. IEEE, pp.4174–4179.
Phoka, T. and Sudsang, A. (2010) ‘Regrasp planning of
three-fingered hand for a polygonal object’, in IEEE
References International Conference on Robotics and Automation, May,
Cole, A.A., Hsu, P. and Sastry, S.S. (1989) ‘Dynamic regrasping
by coordinated control of sliding for multi-fingered hand’, in Roa, M.A., Surez, R. and Rosell, J. (2008) ‘Grasp space generation
IEEE International Conference on Robotics and Automation, using sampling and computation of independent regions’, in
Scottsdale, AZ, USA, May, Vol. 2, pp.781–786. IEEE/RSJ International Conference on Intelligent Robots and
Systems, pp.2258–2263.
Furukawa, N., Namiki, A., Taku, S. and Ishikawa, M. (2006)
‘Dynamic regrasping using a high-speed multifingered hand Roethling, F., Haschke, R., Steil, J.J. and Ritter, H.J. (2007)
and a high-speed vision system’, in IEEE International ‘Platform portable anthropomorphic grasping with the
Conference on Robotics and Automation, IEEE, pp.181–187. bielefeld 20-dof shadow and 9-dof tum hand’, in
International Conference on Intelligent Robots and Systems,
Ishihara, T., Namiki, A., Ishikawa, M. and Shimojo, M. (2006)
San Diego, California, USA, October, pp.2951–2956.
‘Dynamic pen spinning using a high-speed multifingered
hand with high-speed tactile sensor’, in IEEE International Saut, J.P., Sahbani, A., El-Khoury, S. and Perdereau, V. (2007)
Conference on Humanoid Robots, pp.258–263. ‘Dexterous manipulation planning using probabilistic
roadmaps in continuous grasp subspaces’, in IEEE
Kragic, D., Miller, A. and Allen, P. (2001) ‘Realtime tracking
International Conference on Intelligent Robots and Systems,
meets online grasp planning’, in Proc. 2001 IEEE
International Conf. on Rehabilitation Robotics,
pp.2460–2465. Tahara, K., Arimoto, S. and Yoshida, M. (2010) ‘Dynamic object
manipulation using a virtual frame by a triple soft-fingered
Li, Q. (2012) ‘Autonomous exploration of manual interaction
robotic hand’, in IEEE International Conference on Robotics
space’ [online]
and Automation, pp.4322–4327.
(accessed January 2012).
Xue, Z., Zollner, J.M. and Dillmann, R. (2008) ‘Dexterous
Li, Q., Haschke, R., Bolder, B. and Ritter, H. (2012) ‘Grasp point
manipulation planning of objects with surface of revolution’,
optimization by online exploration of unknown object
in IEEE International Conference on Intelligent Robots and
surface’, in 12th IEEE-RAS Intl Conf on Humanoid Robots,
Systems, pp.2703–2708.
Osaka, Japan, 29 Novermber.
Yashima, M. (2004) ‘Manipulation planning for object
Li, Q., Haschke, R., Ritter, H. and Bolder, B. (2011) ‘Simulation
re-orientation based on randomized techniques’, in IEEE
results for manipulation of unknown objects in hand’, in IEEE
International Conference on Robotics and Automation,
International Conference on Robotics and Biomimetics.
Vol. 2, pp.1245–1251.
Miller, A.T. and Allen, P.K. (1999) ‘Examples of 3d grasp quality
Zribi, M., Chen, J. and Mahmoud, M.S. (1999) ‘Control of
computations’, in Proceedings on 1999 IEEE International
multifingered robot hands with rolling and sliding contacts’,
Conference on Robotics and Automation, IEEE, Vol. 2,
Journal of Intelligent and Robotic Systems, February, Vol. 24,
No. 2, pp.71–77.