Robotic Welding Intelligence and Automation

Lecture Notes
in Control and Information Sciences 299
Editors: M. Thoma · M. Morari

Springer
Berlin
Heidelberg
NewYork
Hong Kong
London
Milan
Paris
Tokyo
T.-J. Tarn S.-B. Chen C. Zhou (Eds.)
Robotic Welding,
Intelligence and
Automation
With 234 Figures and 36 Tables
13
Series Advisory Board
A. Bensoussan · P. Fleming · M.J. Grimble · P. Kokotovic ·
A.B. Kurzhanski · H. Kwakernaak · J.N. Tsitsiklis
Editors
Prof. Tzyh-Jong Tarn Prof. Dr. Changjiu Zhou
Washington University Singapore Politechnic
School of Engineering and School of Electrical and
Applied Sciences Electronic Engineering
Ones Brookings Drive Dover Road 500
63130 St. Louis, MO 139651 Singapore
USA Singapore
Prof. Shan-Ben Chen

Shanghai Jiao Tong University
Institute of Welding Engineering
200030 Shanghai
People’s Republic of China
ISSN 0170-8643
ISBN 3-540-20804-6 Springer-Verlag Berlin Heidelberg New York
Cataloging-in-Publication Data applied for

A catalog record for this book is available from the Library of Congress.
Bibliographic information published by Die Deutsche Bibliothek
Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed biblio-
graphic data is available in the Internet at <http://dnb.ddb.de>.
This work is subject to copyright. All rights are reserved, whether the whole or part of the mate-
rial is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilm or in other ways, and storage in data banks. Duplication
of this publication or parts thereof is permitted only under the provisions of the German Copyright
Law of September 9, 1965, in its current version, and permission for use must always be obtained
from Springer-Verlag. Violations are liable for prosecution under German Copyright Law.
Springer-Verlag is a part of Springer Science+Business Media
springeronline.com
© Springer-Verlag Berlin Heidelberg 2004
Printed in Germany
The use of general descriptive names, registered names, trademarks, etc. in this publication does
not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
Typesetting: Data conversion by the editors.
Final processing by PTP-Berlin Protago-TeX-Production GmbH, Berlin
Cover-Design: design & production GmbH, Heidelberg
Printed on acid-free paper 62/3020Yu - 5 4 3 2 1 0
Preface
Robotic welding systems have been used in different types of manufacturing.

They can provide several benefits in welding applications. The most prominent
advantages of robotic welding are precision and productivity. Another benefit is
that labor costs can be reduced. Robotic welding also reduces risk by moving the
human welder/operator away from hazardous fumes and molten metal close to the
welding arc. The robotic welding system usually involves measuring and identify-
ing the component to be welded, welding it in position, controlling the welding
parameters and documenting the produced welds. However, traditional robotic
welding systems rely heavily upon human intervention. It does not seem that the
traditional robotic welding techniques by themselves can cope well with uncer-
tainties in the welding surroundings and conditions, e.g. variation of weld pool
dynamics, fluxion, solid, weld torch, and etc. On the other hand, the advent of in-
telligent techniques provides us with a powerful tool for solving demanding real-
world problems with uncertain and unpredictable environments. Therefore, it is in-
teresting to gather current trends and to provide a high quality forum for engineers
and researchers working in the filed of intelligent techniques for robotic welding
systems. This volume brings together a broad range of invited and contributed pa-
pers that describe recent progress in this field.
This volume is mainly based on the papers selected from the 2002 International
Conference on Robotic Welding, Intelligence and Automation (RWIA’2002),
December 9-12, 2002, Shanghai, China. We have also invited some known
authors as well as announced a formal Call for Papers to several research groups
related to welding robotics and intelligent systems to contribute the latest progress
and recent trends and research results in this field. The primary aim of this volume
is to provide researchers and engineers from both academic and industry with up-
to-date coverage of new results in the field of robotic welding, intelligent systems
and automation.
The volume is divided into four logical parts containing twenty-five papers. In
Part 1 (6 papers), the authors introduce the recent development of welding auto-
mation. Various applications such as vision sensing and control of welding proc-
ess are discussed. In Part 2 (4 papers), the authors deal with some intelligent tech-
niques for robotic welding. In Part 3 (7 papers), the authors describe their work on
intelligent robotic systems which may lead to new development of intelligent ro-
botic welding systems. Finally, in Part 4 (8 papers), the authors introduce some
emerging intelligent techniques and their applications, which may contribute sig-
nificantly to the further development of intelligent robotic welding systems.
We would like to thank Professors J.L. Pan, B.S. Xu, S.Y. Lin, H.G. Cai, S.W.
Xie, T.H. Song, L. Wu, Y.X. Wu and T. Wang for their kind advice and support to
VI Preface
the organization of the RWIA’2002 and the publication of this book; to Wenjie
Chen, Bing Wang, Wenhang Lei and Surong Zhang for their precious time to de-
vote all RWIA’2002 correspondences and to reformat the most final submissions
into the required format of the book; to Jun Ni, research associate of Singapore
Polytechnic and PhD candidate of Shanghai Jiaotong University, for his editorial
assistance, last but not least to Dr. Thomas Ditzinger for his advice and help dur-
ing the production phases of this book.
Tzyh-Jong Tarn, Washington University at St. Louis, USA

Shan-Ben Chen, Shanghai Jiao Tong University, China
Changjiu Zhou, Singapore Polytechnic, Singapore
October 2003
Contents
Preface V
T.-J. Tarn, S.-B. Chen, and C. Zhou
Part 1: Intelligent and Automatic Techniques for Welding

Dynamic Process
Process Stability of Automated Gas Metal Arc Welding of Aluminum 1
S. Emil
Nd:YAG – Laser – GMA – Hybrid Welding in Shipbuilding

and Steel Construction 14
U. Jasnau, J. Hoffmann, and P. Seyffarth
Computer Vision Sensing and Intelligent Control

of Welding Pool Dynamics 25
S.B. Chen, D.B. Zhao, Y.J. Lou, and L. Wu
Surface Shape Reconstruction of Weld Pool during Pulsed GTAW

from Its Single Image 56
D.B. Zhao, J.Q. Yi, S.B. Chen, and L. Wu
A Novel Intelligent Monitor for Manufacturing Processes 63

M. Ge and Y. Xu
3-D Grasp Analysis and Synthesis Using the Ray-Shooting Technique 80

Y.H. Liu, D. Ding, and M.L. Lam
Part 2: Intelligent Techniques for Robotic Welding

Laser Visual Sensing and Process Control in Robotic Arc Welding
of Titanium Alloys 110
H. Luo and X.Q. Chen
VIII Contents
Intelligent Technologies for Robotic Welding 123

S.B. Chen, T. Qiu, T. Lin, L. Wu, J.S. Tian, W.X. Lv, and Y. Zhang
The Seam Tracking System for Submerged Arc Welding 144

H. Zhang, X. Ding, M. Chen, B. Da, and C. Zou
Visual Measurement of Path Accuracy for Laser Welding Robots 152

D. Du, B. Sui, Y.F. He, Q. Chen, and H. Zhang
Part 3: Intelligent Robotic Systems

Map Building and Localization for Autonomous Mobile Robots 161
Y.L. Ip, A.B. Rad, and Y.K. Wong
Control and Stabilization of the Inverted Pendulum via Vertical Forces 190
D. Maravall
Vision Based Motion Planning of Humanoid Robots 212

M.F. Ercan and C. Zhou
Evolution of Locomotion Controllers for Legged Robots 228

A. Boeing and T. Bräunl
Gait Planning for Soccer-Playing Humanoid Robots 241

Z. Tang, C. Zhou, and Z. Sun
Fuzzy-Neural Impedance Control for Robots 263

Z. Xu and G. Fang
On Singular Perturbation Based Inverse Dynamics Control

for a Two-Link Flexible Manipulator 276
X. Dai, L. Sun, and H. Cai
Part 4: Intelligent Techniques and Its Applications

Wavelets and Biorthogonal Wavelets for Image Compression 281
D. Hong, M. Barrett, and P. Xiao
Adaptive Neuro-Fuzzy Control of Systems with Unknown Time Delay 304

F.H. Ho, A.B. Rad, Y.K. Wong, and W.L. Lo
Contents IX
Adaptive Robust Fuzzy Tracking Control for a Class

of Nonlinear Systems 327
Y. Yang and C. Zhou
A Complete Algorithm for Attribute Reduction 345

B. Wang and S.B. Chen
Preliminary Probe of Embedded Real-Time Operating System

and Its Application in Welding Control 353
W. Chen, S. Chen, and T. Lin
Estimation of Oxygen Concentration Using Dynamic Recurrent Neural

Network Based on PCA 358
D. Sun, F. Ye, and X. Gu
Robust Stability of Interval Lur'e Systems: A Bilinear Matrix

Inequality Approach 365
J.T. Sun
Thresholding Segmentation and Classification in Automated Ultrasonic

Testing Image of Electrical Contact 372
H.D. Chen, Z.Y. Cao, Y.W. Wang, and J. Xue
Author Index 381

Process Stability of Automated Gas Metal Arc
Welding of Aluminium
Emil Schubert
Alexander Binzel Schweisstechnik GmbH & Co. KG, Gießen/Germany,

Kiesacker 7-9, D-35418 Buseck, Germany
schubert@binzel-abicor.com
Abstract. For modern transport systems Aluminum is getting of growing

importance. To fully utilize this potential economic, effective and stable joining
technologies have to be available. However, the special features of Aluminium
materials like high heat conductivity, oxide formation, pore and hot cracking
susceptibility require special measures for high quality welding especially for thin
sheet metals. New developments in gas metal arc welding power sources, torches
and wire feeders for Aluminum welding to overcome these problems will be
described. Besides the technological factors system aspects and their influence an
process stability and manufacturing up-time will also be discussed. Actual and
potential future applications of these new developments will be demonstrated.
1 Introduction
The advantages of the material aluminium (low density and high stability at the
same time) are more and more used as a possibility for weight reduction,
especially in the field of transportation. Fig. 1. shows the use of aluminium in
automobiles during the period from 1975 until today and gives an outlook to the
year 2005 showing that the employment of aluminium will become twice as large
until then [8].
However, in order to produce automobiles out of sheets, profiles and cast parts
suitable joining processes have to be available to enable an effective, efficient
joining of the aluminium materials with sufficient process stability. Gas metal arc
welding processes such as the GMA , TIG- and Plasma welding process can be
used in this field for producing sealed and continuous welding seams [6]. In
addition the combination of these processes with the Laser welding process can be
used to realize new possibilities for welding structures [3]. Especially in the field
of thin sheets the handling of the soft filler metals as well as the minimized heat
input represent the most important challenge for the gas metal arc welding
processes. Additionally, the system aspects (i.e. integration of the welding process
into the production cells including torch change, cleaning stations, robot control
etc.) play an important role in the field of production technology.
T.-J. Tarn et al. (Eds.): Robotic Welding, Intelligence and Automation, LNCIS 299, pp. 1−13, 2004.
 Springer-Verlag Berlin Heidelberg 2004
2 2 E. Schubert
80 74
68
70 63
60 55
steel and iron
50
elastomers / others
40 polymers
Aluminium
30
other non-ferrous
20 1413 1415 metals
13 13
10 10
10 6 5 4 6 4 6
3 4
0
1975 1985 1995 2005
Year of construction
Fig. 1. Employment of aluminium alloys in the automotive industry
Therefore, possibilities to increase and maintain the process stability during gas
metal arc welding will be described in the following considering the special
characteristics of the aluminium materials, and examples for applications will be
given for explanation. Starting point are new developments in the field of welding
power sources and torch technology which will open the field of thin sheets for the
inert-gas welding in the future. Another core component is the feeding of the filler
metal as well as the system integration of all components into the industrial
manufacturing cell as already mentioned above.
2 Potentials and Problems of Aluminium Welding
The low density of 2,7 g/cm³ at stabilities of up to 300 N/mm² (depending from
the type of alloy) are opening for the aluminium alloys a large potential for
application in the field of light-weight construction. Due to other physical
characteristics (high heat conductivity, hydrogen absorption, large interval of
solidification) and without going too deep into detail regarding these special
characteristics - problems such as pipe cavities, pores and hot cracks as well as
thermal distortions are occurring very frequently when welding these materials.
To minimize the distortion problem it is generally possible to weld with an energy
per unit of length as low as possible and to distribute the heat input by suitable
welding cycles. Especially in the field of pulsed arc welding the heat input can be
minimized (see chapter 3).
The problem of pore formation [2] is caused by the high affinity of aluminium
towards hydrogen. The hydrogen is dissolved in the melt pool and will be
Process Stability of Automated Gas Metal Arc Welding of Aluminium 33
segregated due to the considerably lower solubility in the solid aluminium,

accumulating in form of pores. To avoid this problem the hydrogen has to be
prevented from dissolving on one side and there has to be sufficient time and place
for the hydrogen to degas in case that it has dissolved into the pool.
Hydrogen gets into the pool e.g. through the formation of hydroxide layers in
case of oxidized surfaces in a wet atmosphere or by dirt or remainders from
greases and oils. Another possibility is the direct entering through the humidity in
the air in case of insufficient gas-shielding or sputtering wire feeding.
Measures to prevent pores are the use of inert shielding gases, dry stocking of
work pieces and wires and a removal of the oxide layer by brushing, pickling or
milling of edges.
Besides the interaction of oxides with oxygen, oxides have another negative
characteristic: due to the considerably higher melting temperature compared to the
base metal (2050 ° C resp. 66 °C) notches can occur at the upper and the root bead
or oxides can sink into the pool [1].
Measures for degasification are especially an adapted welding speed to give the
gas pores time to rise as well as the creation of degasification areas (bevel opening
angles as large as possible) where ever the design allows. In additon, air gaps
between the sheets, e.g. through distance sheets can improve the degasification.
The problem of hot crack formation is especially occurring with alloys having a
large solidification interval where the elements Mg, Si and Cu are critical [2].
Reason for this is a too small quantity of intergranular eutectic during the
solidification process accompanied by shrinkage stress. Countermeasures are the
use of filler metal to bring the composition of the weld metal out of the crack-
critical area, a suitable design enabling a shrinkage and an adapted weld heat input
(not too high solidification speeds).
3 Process Stability of Welding of Aluminium Materials
3.1 Welding power source- and welding torch technology
The GMA-, TIG- and Plasma welding process are suitable for the gas metal arc
welding of aluminium.
GMA-Welding
While welding of thick plates is preferably done using spray arc, the pulsed-arc
welding is recommended for the welding of thin sheets. Fig. 2. is a current – time
diagram for a typical pulsed-arc process.
With the growing application of the inverter technology new possibilities are
arising for the inert-gas welding process. The GMA-process too, having so far
4 4 E. Schubert
only limited access into the industrial production of structural components out of
aluminium, is profiting from these developments.
pulsed current
non-short-circuiting metal transfer
medium current
current
background current
pulse time actual welding time
time
Fig. 2. Current-time-characteristic for impulsed welding of aluminium alloys
pulse
voltage
background
current neg.current time
negative
background
pulse time current
pulse period time (pulse frequency)
Fig. 3a. MIG-AC-Pulsating of voltage

pulse
voltage
background
current off-time
pulse time
pulse period time (pulse frequency)
Fig. 4b. MIG-DC-Pulsating of voltage
These process variants enable

a process-stable welding of
structural components out of
aluminium up to a thickness of
0,6 mm. Fig. 4. shows the cross-
section of a fillet weld with the
clearly visible minimum
penetration.
Fig. 5. Cross-section of a fillet weld of AIMgSi

welded by using the MIG-AC process
But also regarding conventional GMA welding of aluminium materials special

requirements on the torch technology are arising, especially when using pulsed
welding. A torch with a double-cooling circuit enables stable welding processes
even at high pulse currents due to an optimal cooling of the torch neck, contact tip
and gas nozzle. Fig. 5. shows the sectional drawing of such a torch with two
cooling circuits.
6 6 E. Schubert
Fig. 6. Sectional drawing and photo of a double cooling-circuit-GMA-welding torch for

impulse welding
electrode at negative pole electrode at positive pole
el el
ec ec
s
s
tro tro
n
on
io
ns ns
si
s
ga
ga
type of penetration
Fig. 7. TIG-welding with negative and positive polarity at the electrode

TIG Welding Process

The TIG-AC welding can be recommended for the TIG welding of aluminium[10].
Fig. 6. shows the TIG welding with different polarities at the electrode.
The electrode is cooling down during the minus phase enabling a deep
penetration while during the plus phase a cleaning effect occurs as the oxide layer
is torn open by gas ions. The penetration during this phase is more flat and wider
and the thermal load of the electrode is very high. In practice therefore, AC-
welding is applied in order to combine the advantages.
Plasma Welding
The Plasma welding is offering possibilities similar to the TIG welding with the
difference of a stronger arc-constriction by the so-called plasma gas [9]. Additional
advantages are the low heat input, low quantity of spatter and the very smooth
seams.
Fig. 7. is a diagram showing an industrial plasma torch used for aluminium
welding. The plasma plus-pole torch with a rating above 200 A is under
development [5] and promises a large application potential.
Plasma gas
back cap
upper cooling circuit
cooling body
tungsten electrode
gas diffusor shielding gas
lower cooling circuit
ceramic shielding gas nozzle
Plasma nozzle
Fig. 8. Industrial plasma welding torch
The conventional inert-gas welding process stands in competition to the laser-

beam welding and the hybrid processes. While the laser processes are superior in
8 8 E. Schubert
12
P a ra m e te rs :
10 b u t t j o in t , 1 m m
L a s e r: 2 ,5 k W
welding speed m/min
8 c o n v e n t io n a l w e ld in g :
s ta n d a rd p a ra m e te rs
6
4 s ta in le s s s te e l
s tru c tu ra l s te e l
2 A lu m in iu m
0
L a s e r (C O 2 , Y A G ) HLDL P la s m a M IG /M A G W I IG
T G
P a ra m e te rs :
s t a i n le s s s t e e l P a r a m e t e r : S tu m p fs to ß , 1 m m
b u t t jo in t, 1 m m
s tru c tu ra l s te e l L a s e r: 2 ,5 k W
L a s e r, 2 ,5 k W
2 ,5 k co on nv v. eSncthiowneai lß wv ee rl df ai nh gr e: n :
A lu m in iu m
g sä tna gn idg ae r d
A rpbaeriat smp a
e rt ea rms e t e r
2
energy per unit length kJ/cm
1 ,5
0 ,5
0
L a s e r (C O 2 , Y A G ) HLDL P la s m a M IG /M A G W
T I IGG
Positional tolerance ofder

Positionstoleranz power source gap dim. tolerance
Spaltmaßtoleranz
S
1,25
zul
äs 1,00
admissible tolerance
sig
e
0,75
To
ler
an 0,50
z
in 0,25
m
0,00
Laser (CO2, YAG) HLDL Plasma WIG
TIG MIG/MAG
Fig. 9. Comparison of GMA-, TIG, Plasma- and Laser welding processes

welding speed and energy input, the inert-gas welding processes offer enormous
advantages on the cost side. The fig. 8 a) to c) show a comparison of the possible
welding speeds, energy inputs per unit of length and the gap bridging abilities.
The diagrams show the especially large potential of the plasma welding
technology as a bridge between GMA- and TIG-welding on one side and the laser
technology on the other side.
3.2 Wire Feeding
The feeding of soft, easy deformable wires with low stability and low resistance to
wear and the risk of oxide formation and humidity absorption is a great challenge
for the wire-feeding systems.
One solution, already proven in industrial use, is a system consisting of a
heated wire-spool housing, a wire spool with drive motor and a digitally
controlled 4-roller drive inside the housing, a cable assembly and a second
digitally controlled 4-roller drive directly mounted to the torch, as shown in fig. 9.
Fig. 10. Wire feeding equipment for Al-wires (Aluminium Power Drive: APD)
Both 4-roller drives are running absolutely synchronous as push-push drive and
a certain wire quantity is always available in the cable assembly as reservoir. The
rollers are preset and only require very small pressing forces as the wire roll is
driven separately so that only minimum wear or deformation will occur at the
wire. In addition, this is also minimizing the tensile forces acting on the wire. The
heated housing keeps the temperature above dew point guaranteeing a dry wire
with uniform oxide layer without humidity so that a welding with constant
processing parameters and without the danger of pore formation due to oxide or
hydrogen from the wire.
10 10 E. Schubert
3.3 System Aspects
The integration into complete production cells plays an important role for the
extensive use of the single components. The power source, the welding torch, the
wire-feeding unit and the welding robot should produce trouble-free in a
production cell with highest-possible equipment availability and stability. As
especially at the welding torch and during the wire feeding wear and dirt
accumulation is occurring – e.g. by weld spatter and worn products - which cannot
be avoided, peripheral devices have to enable an automatic cleaning or an
exchange of the components in questions, if possible. To avoid damages at the
welding torch in case of collision, emergency shut-down devices have to be used.
Modular systems are especially suitable for frequently changing welding jobs.
Fig. 10. shows a concept of a wire-feeding system for different inert-gas welding
processes, the so-called Master-Feeder-System. The user is selecting the suitable
wire bunch (small coil, large coil, barrel, etc.) and welding process (GMA, TIG,
Plasma, Laser or Hybrid) depending on the welding job. The required filler metal
is fed by the first 4-roller drive after the separately driven wire bunch and from
there transported through a cable assembly to the second 4-roller drive situated
close to the process. The wire is then led into a GMA torch or is made available
for other processes as cold or hot wire. Fig. 11. shows a processing head for Laser-
GMA welding.
Process -depending Adjustment of wire

wire feeding : packages :
barrel spool
MIG torch
Cold-/hot wire: large spool
TIG small spool
Plasma
Laser
Fig. 11. Modular concept for wire feeding with different welding processes (MIG, TIG,
Plasma, Laser)
11
Fig. 12. Laser-GMA processing head
4 Examples for Application
Since 1999, the automatic inert-gas welding of structural components out of

aluminium in combination with the wire-feeding concept (APD) described is used
in the production of the AUDI A 8. Different seams at the wheel arch have been
welded using in this process and equipment. The use of the APD system requires
minimum intervention by the operator (change of spool once a week) and thus
enables an economic production.
The proven APD system could also be used for the GMA welding in the
production of the AUDI A2. Here too, this system is used for welding jobs at the
wheel arch with 17 welding units in total, see fig. 12.
12 12 E. Schubert
Original
component
Process picture
GMA weld seam
Fig. 13. Al-wheel arch of the Audi A2 welded with APD
Besides this application, the wire feeding technology described above is success-
fully implemented as Laser cold-wire feeder at the successors of A2 and A8.
Fig. 13. is showing the cold-wire feeding in detail.
To observe the narrow process tolerances an
additional positioning possibility is existing in x-, y-
and z direction. The cold-wire feeding can be adjusted
to any focus length and interfering contours of laser
processing heads.
Fig. 14. Laser cold-wire feeding system

13
5 Conclusions
The material aluminium is making special demands to the welding technology.

Due to new developments in the field of power sources, adapted welding torch
designs and new wire-feeding units these demands can be met economically as
well as regarding process stability.The integration of these components as a
system into the production cells allows to carry out welding jobs with high process
stability and equipment availability. The modular concept presented for wire-
feeding has certainly a large future potential for feeding wires difficult to handle
for the whole range of inert-gas joining processes. This does not only apply to the
material Aluminium but also to the welding and soldering of other light metals as
well as the proven steel materials and their developments, the high strength and
super-high strength steels. In addition, applications in the field of Laser as well as
hybrid processes become possible so that here too, improvements in process
stability can be expected.
References
[1] Baum L, Fichter V (1999) Der Schutzgasschweißer. DVS-Verlag, Düsseldorf

[2] Dilthey U (1995) Schweißtechnische Fertigungsverfahren. VDI-Verlag, Düsseldorf
[3] Dilthey U, Lüder F, Wieschemann A (1999) Erweiterte Möglichkeiten beim
Schweißen von Aluminiumlegierungen durch den Laser-MIG-Hybridprozeß.
Aluminium; Bd. 75, Nr. 1/2, pp 64-75
[4] Goecke SF, Dorn L (2000) MAG-Chop-Arc-Schweißen für Feinstblech > 0,2 mm.
Große Schweißtechnische Tagung, 2000, Nürnberg, DVS-Verlag, Düsseldorf, pp
163-1682.
[5] Kabatnik L (2000) Plasma-Pluspolschweißen. Vortragsmanuskript Große
Schweißtechnische Tagung 2000, Nürnberg,DVS-Verlag, Düsseldorf
[6] Klock H, Schoer H (1977) Schweißen und Löten von Aluminiumwerkstoffen.
DVS-Verlag, Düsseldorf
[7] Nickel HG (2000) MIG-MAG DC-O und AC-0-Spannungspulsen. Pers. Mitteilung,
Nisterberg
[8] N.N. (2000) Werkstoffverteilung beim PKW in Deutschland. VDI-Nachrichten,
Düsseldorf, pg 41
[9] Radscheit C (1999) Plasmalöten - Grundlagen und Einsatz; Neue Löttechnologien
für den Dünnblechbereich. SLV München
[10] Zahariev S, Herold H (1996) Besonderheiten der Anwendung des
WIG-Wechselstromschweißens von Aluminium unter Nutzung
modifizierter Halbwellen. DVS-Berichte 176, pp 34-37
14
Nd:YAG – Laser – GMA – Hybrid Welding

in Shipbuilding and Steel Construction
Ulf Jasnau, Jan Hoffmann, and Peter Seyffarth
Welding Training and Research Institute Mecklenburg – Western Pomerania Ltd.,

Rostock, Germany
jasnau@slv-rostock.de
Abstract. The paper deals with the Nd:YAG-laser – GMA – hybrid welding
process for the use in shipbuilding and steel construction. The advantages of a la-
ser – arc – hybrid welding process in comparison with a laser alone or an arc alone
welding process are shown. The current possibilities of the Nd:YAG-laser – GMA
– hybrid welding process are shown by some results from hybrid welding of steel
and aluminium alloys. Furthermore the advantages of the Nd:YAG-laser technol-
ogy, especially for a simple and flexible automation are described. The Nd:YAG-
laser – GMA – hybrid welding is a process which demands a higher degree of
automation in shipbuilding. Regarding to the advantages and related to the avail-
able Nd:YAG-laser sources a perspective for the use of the Nd:YAG-laser tech-
nology in shipbuilding and steel construction is pointed out. A current German re-
search project under participation of the authors is presented in the paper. One aim
of this project is to develop a system for the automatically programming of robots
and other welding systems in shipbuilding. Therefore the new programming tool
should use the real geometrical data of the component before welding, resulting
from a image processing tool. Concluding, the first manufacturing system in ship-
building (a welding gantry for 12 meter T-joints) outfitted with Nd:YAG-laser –
GMA – hybrid welding equipment at a German shipyard is presented. The retrofit-
ting of this welding gantry has been planned and accompanied by the company the
authors are with.
1 Introduction
The process of laser beam welding is today a well established joining process in
industrial applications. The advantages, e.g. deep and narrow welds with high
speed and low heat input are accompanied from some disadvantages like the nec-
essary precise preparation of the edges of parts to be joined and the precise clamp-
ing of the work pieces.
Especially in shipbuilding and steel construction the advantages of the laser
technology are to be used only with high costs and efforts due to the boundary
conditions in the production line. With the coupling of the laser beam and the arc
Nd:YAG - Laser - GMA - Hybrid Welding in Shipbuilding and Steel Construction 15
15
it is possible to extrude the range of application of the laser technology and their
advantages.
Since the fundamentals of a laser-arc-hybrid process have been described in the
70th [1] different options of this technology were studied, like the laser-GMA-, the
laser-TIG- or the laser-plasma-process. In the most cases of this variations a CO2-
laser was used as a laser source because of the higher laser power and better beam
quality. With the now available higher laser power up to 6 kW in case of
Nd:YAG-laser this laser source will be more interesting for the laser-arc-hybrid
welding in shipbuilding and steel construction. The gas-metal-arc welding is today
the preferred welding process in shipbuilding and steel construction in Germany.
Therefore and because of some important advantages of the Nd:YAG-Laser the
coupling of this type of laser with the GMA-welding process will be contem-
plated.
2 Process Fundamentals
The fundamentals of the coupled process are nearly the same for both, the CO2-
laser and the Nd:YAG-laser. In Fig. 1 the general principle of the process is
shown. Laser process and arc process have a common process zone and weld pool.
Only one shielding gas flow is necessary, mostly the gas flow is realized via the
GMA-torch.
laser beam welding electrode
laser induced metal vapor arc
protective gas cover

fusion zone
keyhole
welding direction
workpiece
Fig. 1. Principle of the laser-GMA-hybrid welding process
The superposition of the arc and the laser leads to different interaction between
the two processes. The stabilization of the arc under the influence of laser radia-
tion or a laser induced plasma is one of the main advantages. The wandering of the
arc is prevented, because the laser beam evaporates material and ionizing the at-
16 16 U. Jasnau, J. Hoffmann, and P. Seyffarth
mosphere, producing good conditions for ignition and burning of the arc. So the
transfer of the energy of the electric arc is more efficient and the thermal effect of
the coupled process is in excess of a simple sum of the effects of the laser and arc
heat sources taken separately [2].
Depending of the energy ratio of the two energy sources the character of the
coupled process may be either more arc-like or more laser-like. The welding
depths increases with the increase of laser energy whereas the weld width in-
creases with the arc energy. For a given sheet thickness to be joined the proportion
between the “laser part” and the “arc part” of the hybrid weld pool give us an es-
timation about the process character. The transverse sections in Fig. 2. are show-
ing an example for such different process characters.
Fig. 2. Comparison between a more laser-like (left) and a more arc-like (right) laser-GMA-
hybrid weld (sheet thickness 5 mm, I-groove)
Depending on the welding task the one or the other process character may be
preferable. For the control of the process character and the utilisation of the advan-
tages of the coupled process (Fig. 3) the effects of a great amount of welding pa-
rameters have to be known.
The effect of the laser beam in a hybrid weld is well illustrated by comparing
transverse sections of GMA welds and hybrid welds with the same welding speed
(Fig. 4. ). The welding depths increases from 1,5 mm to 5,1 mm using a 4 kW
Nd:YAG-laser beam. Despite the high welding speed the quality of the top bead
shape is good in case of hybrid welding, resulting from the laser-stabilized arc.
The effect of the arc, resulting in a better gap bridging ability and a better weld
geometry then in a pure laser weld is illustrated in Fig. 5. The addition of the arc
power and the process stabilisation in hybrid welding leads to a significant higher
welding speed. So the heat input per unit of length remains almost constant in
comparison with the laser alone welding process.
17
arc laser
• low cost energy source • deep weld penetration
• gap bridging ability • high welding speed
• filler material • low heat input
• influence of structure
hybrid process
• better gap bridging ability, compared with laser welding

• higher welding speeds, compared with arc and laser
• deeper weld penetration, compared with arc and laser
• lower heat input compared with arc-welding
Fig. 3. Coupling of the advantages of arc and laser welding in a hybrid process
sheet thickness: 8 mm
laser type: Nd:YAG
laser power: ~ 4 kW
welding speed: 2 m/min
without laser with laser
shielding gas: 82% Ar / 18% CO2
Fig. 4. Transverse sections of a GMA-weld and a laser-GMA-hybrid weld
sheet thickness: 5 mm
I-groove
laser type: Nd:YAG
laser power: ~ 4 kW
shielding gas: 90% Ar / 10% CO2
laser weld, laser-GMA-hybrid weld,

welding speed: 1,1 m/min welding speed: 2,5 m/min
Fig. 5. Transverse sections of a laser weld and a laser-GMA-hybrid weld

Besides the parameters laser power and arc power (arc voltage and arc current)
and welding speed the geometrical formation of the laser beam and the torch and
also the type of shielding gas are very important.
The arbitrary geometrical formation of the laser beam and the torch (torch be-
fore or behind or besides the beam in welding direction) and the possibility to
couple more than one GMA-torch with the laser beam offers a great variety of
geometrical formations for the solution of welding tasks.
The usable shielding gas is one of the biggest difference between the hybrid
welding with CO2-laser and Nd:YAG-laser. For the work with the CO2-laser gas
mixtures with a high Helium content are necessary. In case of Nd:YAG-laser also
Argon and active gases, e.g. mixtures from Ar, CO2 and O2 are usable. So it is
possible to influence the metallurgy of the weld pool like in case of pure GMA
welding. Pure Argon and active shielding gases are not usable for hybrid welding
with a CO2-laser, because of the risk of ignition of an protection plasma above the
weld pool. This plasma interrupt the absorption of the laser radiation into the work
piece and the hybrid welding process discontinues. This disadvantage of the CO2-
laser is caused in the different wave lengths of the radiation of the two laser types
mentioned above. The CO2-laser with a wave length of 10,6 µm offers other opti-
cal conditions then the Nd:YAG-laser with a wave length of 1,06 µm.
The practical qualification of the Nd:YAG-laser-GMA-hybrid welding for dif-
ferent shipbuilding applications is the main content of two research projects the
authors company is involved. Besides the welding of constructional and shipbuild-
ing steel also the welding of aluminium alloys for shipbuilding has to be done in
one of the projects. In a lot of welding tests the hybrid welding of square butt
joints and T-joints of the mentioned materials is carried out under laboratory con-
ditions. Welding tests at a real welding gantry at the shipyard “Kvaerner Warnow
Werft” in Rostock, Germany, for the qualification of the hybrid welding process
also have been started.
Despite the high welding speeds reachable with the coupled process the techni-
cal and economical effects of the new welding process are not only resulting from
the higher welding speed compared with GMA-welding. The main aspect of the
positive effects is the lower energy per unit of length, leading to lower distortions
and less postweld straightening.
Some results of the welding tests are shown in Fig. 6 and Fig. 7.
For aluminium alloys the laser-GMA-hybrid welding offers an additional ad-
vantage over the pure laser welding compared with the welding of steel. During
laser welding of aluminium alloys the keyhole is very instable due to the lower
viscosity of the aluminium melt versus steel melt. This leads to numerous col-
lapses of the keyhole and to a lot of big pores, so called process pores. The out-
gassing of these pores are very difficult due to the very small weld pool and short
cycle time in case of laser welding. The phenomenon of process pores is limiting
the welding depths in case of laser welding of aluminium. To solve this problem
some strategies were developed, e.g. the welding with two laser spots for the
enlargement of the weld pool. In principle all these strategies leads to a higher
necessary laser power at constant welding depths.
19
The laser-GMA-hybrid welding is a good possibility to realise the higher power

at the working piece. With the coupling of the laser beam and the arc high welding
depths and high welding speeds are also possible on aluminium alloys without the
danger of process pores. The bigger weld pool geometry and the modified flow
conditions in the melt allows the outgassing of pores. Furthermore the override of
the metallurgy of the weld with the melting filler wire is much better compared
with pure laser welding.
root formation
transverse section
Parameters:
sheet thickness: 5 mm current type/ polarity /arc type: DC / + / pulsed arc
weld type: I-groove arc current / voltage: 290 A / 29 V
welding position: PA wire feed rate: 12500 mm / min
laser type: Nd:YAG arc power: ~ 8,4 kW
laser focal length: 200 mm welding speed: 3000 mm / min
laser power: ~ 4 kW shielding gas: 12 l /min (Ar / O2)
Fig. 6. Laser-GMA-hybrid weld on constructional steel

1 cm
typical top bead shape
transverse section
Parameters of the weld (transverse section):

sheet thickness: 5 mm current type/ polarity /arc type: DC / + / pulsed arc
weld type: I-groove arc current / voltage: 162 A / 24 V
welding position: PA wire feed rate: 12.000 mm / min
laser type: Nd:YAG arc power: ~ 4,0 kW
laser focal length: 200 mm welding speed: 2500 mm / min
laser power: ~ 4 kW shielding gas: 45 l /min (Ar / He)
Fig. 7. Laser-GMA-welds on aluminium alloy 5083
3 The Equipments of Production Lines
3.1 Theoretical Aspects
The possibility for transferring the Nd:YAG-laser beam via a flexible laser light
cable is a second big advantage of the Nd:YAG-laser caused in the lower wave
length. In contrary, the CO2-laser beam must be transferred via a mirror system to
the working point.
Laser light cables and laser working heads are manageable by industrial robots,
a lot of such solutions in automobile manufacturing are making these solutions to
state of the art.
21
But the flexible beam transfer in connection with the higher available laser
power of modern Nd:YAG-laser systems up to 6 kW makes the use of this laser
type also very interesting for production lines in shipbuilding and steel construc-
tion. In case of using a Nd:YAG-laser only the working head has to fulfil the high
requirements on the dynamical behaviour and not the whole beam transfer way,
the complete machine tool respectively, as in case of using a CO2-laser. A com-
mon welding gantry or another welding machine, respectively the point of the
working head, should be fulfil the requirements according to [3] as a supposition
for refitting with a Nd:YAG-laser source.
In connection with the laser-GMA-hybrid welding process the advantages of
laser technology are usable for conventional production lines. Nevertheless the
Nd:YAG-laser-GMA-hybrid welding demands a higher degree of automation in
the production line. A precise seam tracking system including measuring of the
gap size and the control of laser- and / or arc-power is an elementary equipment.
The control system of the welding gantry must be able to control higher welding
speeds than in case of pure GMA-welding and a lot of new functions, connected
with the use and control of the laser system. But the effort for refitting a conven-
tional welding gantry with Nd:YAG-laser equipment is much lower than the pur-
chase of a new special designed laser welding gantry.
Another advantage of the Nd:YAG-laser sources is the possibility to connect
one laser source with up to 6 handling systems and to add the radiation of two or
three laser sources into one laser light cable. So also with solid state lasers a power
of more than 12 kW at the work piece is possible. As already realised in the auto-
mobile industry so called “lasernetworks” with several laser sources and several
handling systems are also thinkable for production lines in shipbuilding and steel
construction. Such a “lasernetwork” can be installed “step by step”, regarding to
the actual demands of the shipyard. Also a occasionally insert of laser technology
for special manufacturing tasks is possible. This flexibility is only caused in the
type of laser source, the Nd:YAG-laser and the possibility to transfer the laser
beam via laser light cable.
In the two already mentioned German research projects the predicting problems
with the use of Nd:YAG-lasers in shipbuilding, e.g. the occupational safety and
health and the necessary higher level of control systems and automation should be
solved. One of these projects is going on in a regional network of the authors
company and two other research companies, attended by several German ship-
yards and related firms.
Besides the qualification of Nd:YAG-laser-GMA-hybrid welding and problems
regarding to safety and control of the laser system the aim of this project is to de-
velop a system for the automatically programming of robots and other welding
systems in shipbuilding. Therefore the new programming tool should use the real
geometrical data of the component before welding, resulting from a image proc-
essing tool. Thus the expensive manual work for the programming of welding ma-
chines and the dependence from not up-to-date CAD-data should be replaced in
the future. Further on with such a automatically programming tool it is possible to
react without delay on short-term changes in the manufacturing tasks, the flexibil-
ity and reactivity of the production line increases.
For the automatically programming the data resulting from the image process-
ing tool are independent from any kind of machine control system and should be
therefore usable for all programmable welding machines in a production line. The
further processing of the data should be realized with a general software solution,
usable for the different programmable robots and welding machines in a produc-
tion line. Despite this great variability of the planned system an adaptation to ex-
isting machines and manufacturing lines will be necessary.
Additionally to the specific adaptation of programming tools and laser-GMA-
hybrid welding equipment the theoretical and practical training of the staff is a fo-
cus point in the project of the three incorporated research companies.
3.2 Practical Applications
The first industrial application of the laser-GMA-hybrid welding process in a

shipyard was realized with a CO2-laser (Fig. 8) at the German shipyard “Meyer-
Werft”. This production line was designed especially for the hybrid welding of
plates and stiffeners with a CO2-laser. Panels with a geometry of 20 m *20 m are
welded completely at this line.
“Meyer-Werft” is specialised in great cruise liners for the international market.
Therefore a lot of panels with lower sheet thickness are used for their products.
Fig. 8. CO2-Laser–GMA–hybrid welding from panels and stiffeners at shipyard

“Meyer-Werft” in Papenburg, Germany [4]
At the German shipyard “Kvaerner Warnow Werft” in Rostock (Fig. 9) a com-

mon welding gantry was refitted with equipment for the Nd:YAG-laser-GMA-
hybrid welding. During the above mentioned regional research project the prob-
23
lems regarding with such a refitting should are solved. The refitting was done after
measuring the dynamical behaviour at the point, were the future hybrid working
head shall be located. The results correlated well with the requirements regarding
to [3]. The first test welds made with the refitted conventional welding gantry also
verified the suitability of the gantry for use with laser technique.
In the meantime also other German shipyards are interested in such a refitting
solution.
Fig. 9. Welding gantry (right) and Nd:YAG-laser-GMA-hybrid welding head (left) at the
shipyard “Kvaerner Warnow Werft” in Rostock, Germany
4 Conclusions
The Nd:YAG-laser in connection with the gas metal arc welding offers great pos-
sibilities for the use of laser technology in shipbuilding and steel construction.
The lower energy per unit of length resulting from laser based joining processes
leads to lower distortions and less postweld straightening. Besides the higher
welding speed compared with conventional welding methods the laser-GMA-
hybrid welding offers a better gap bridging ability compared with pure laser weld-
ing and makes the advantages of laser technology usable for manufacturing under
the conditions in shipbuilding and steel construction.
The high flexibility of the Nd:YAG-laser equipment, resulting from the wave-
length of the Nd:YAG-laser radiation and their optical properties enabled an im-
plementation of laser technology according to the actual demands of a company.
In principle the refitting of conventional welding machines with Nd:YAG-laser
equipment, such as robots and welding gantries is possible.
The implementation of laser equipment and technology into shipbuilding and

steel construction leads to a higher level of automation in this branches, connected
with a higher level of product quality and reproducibility. Besides the technical
requirements for the introduction of laser technology in the mentioned industrial
areas the qualification level of the staff, workers as well as technicians and engi-
neers must be increased.
References
1. Eboo, Steen, Clarke: Arc Augmented Laser Processing of Materials; Conf.

Proc. Of Advances in Welding Processes, Harrogate, UK, 1978, p. 257
2. Gvosdetzky, Krivtsun, Chzhenko, Yarinich; Laser-Arc Discharge: Theorie
and Applications; Harwood Academic Publishers, 1995
3. N.N.:IIW-Document IV-789-01: Acceptance test for Nd:YAG-laser beam
welding machines, part 1: machines with optical fiber delivery; 2001
4. N.N.: Schiffe aus Licht; in “INFORM-Das Magazin für Umformtechnik”,
01/2002, Schuler AG, Göppingen
Computer Vision Sensing and Intelligent Control
of Welding Pool Dynamics
S.B. Chen,1 D.B. Zhao,2 Y.J. Lou,3 and L. Wu3
1. S. B. Chen is with Institute of Welding Technology, Shanghai Jiao Tong University,

Shanghai, 200030, P. R. China.
2. D. B. Zhao is with Mechanical Depart., Qinghua Univ., Beijing, 100080, P. R. China.
3. Y. J. Lou and L. Wu are with and he was with Welding Division, HIT , Harbin,
150001, P. R. China.
sbchen@sjtu.edu.cn
Abstract. This paper presents a successful application of intelligent technologies,

such as computer vision, image processing fuzzy-neural networks, for sensing,
characterization, modeling and control of the welding pool dynamic process in the
pulsed Gas Tungsten Arc Welding (GTAW). The image processing algorithm is
discussed in detail. The results of the 2-dimesion and 3-dimesion geometric
characterization of the weld pool from the single-item pool images are presented.
Neural models and self-learning neural controller for the weld pool dynamic
process have been developed. The experiment results on the process indicate that
the designed vision sensing and control systems are able to emulate a skilled
welder’s intelligent behaviors: observing, estimating, decision-making and
operating. The work in the paper shows great potential and promising prospect of
artificial intelligent technologies in the welding manufacturing.
1 Introduction
Artificial intelligence (AI) is an applied science and technology across multiple

disciplines. Industrial requirements and promotion of AI techniques have resulted
in considerable development in AI. Developing effective AI methodology for
practical applications is all along attracted research efforts in scientific and tech-
nical fields.
In practical engineering applications of AI, a challenging problem is how to
control a complex object with uncertainty (Willis, et al,1992). In general, the means for
26 26 S.B. Chen et al.
resolving this problem includes the sensing techniques for simulating human
sensing organ functions, modeling techniques for cognizing, control techniques for
reasoning and decision-making, and administering of motion and operation.
Computer vision technology has become a powerful tool for a wide variety of
engineering applications (Hoffman ,1991; Marr,1982). Its use in the industry has ac-
celerated advancement in quality control and automation systems. AI provides
powerful approaches to control complex systems through synthesis of intelligent
techniques.
The non-linear behaviour and uncertainties of many practical systems make it
difficult for conventional methods to model analytically and control these systems.
Many real industrial process plants fall into this category, and hence intelligent
techniques are required to model and control such systems. Among many different
“intelligent” approaches, neural network and fuzzy methodologies have emerged as
very powerful tools for designing intelligent control systems, **owing to their ca-
pabilities of emulating human learning processes (Willis et al ,1992). Artificial neural
networks (ANN) shows that the method has been used in a modest scale to develop
nonlinear models and control systems. For example, recent studies (Andersen
et.al,1990; Lim, et al, 1993; Pham et.al, 1999) and their cited references have demonstrated
the utility and flexibility of the concept within the domain of process engineering.
Artificial neural network applications have great prospect and potential in various
fields of industrial engineering.
As is well known, arc welding processes are extremely nonlinear and multi-
variable-coupled. They involve many uncertainties, such as, influences of metal-
lurgy, heat transfer, chemical reaction, arc physics, and magnetization (Saedi, et al,
1988). As a result, it is very difficult to obtain a practical and useful controllable
model of an arc welding process through classical modeling approaches. Until now,
control of weld quality, such as welding penetration, and fine formation of welding
bead (i.e., control of the size and changes of the welding pool), is still a perplexed
faced by control engineers or for weld technologists (Vilkas, 1966).
The weld seam accuracy is difficult to control due to the non-linearity and un-
certainties of the process. Intelligent control systems should be developed for
modeling and control of the welding process, as they derive the control performance
based on human experience, knowledge, and logic techniques, instead of mathe-
matical process models. Andersen K (1990), Lim T.G. et al (1993) have studied neural
network and fuzzy techniques for arc welding processes.
Another critical difficulty of dynamic control of an arc welding process is how to
detect weld pool geometrical features, such as weld bead width and penetration,
especially from the topside of weld pool, conveniently and in real-time. Though
various efforts have been made to sense weld pool sizes in real-time from the top-
side, such as ultrasonic detection, infrared sensing, pool image processing, and ra-
diographic sensing (Brzakovic et al ,1991; Lee et al, 1996; Pietrzak el al, 1994; Richardson et
al 1994), results of weld quality control have been achieved with limited successes. In
order to achieve better control of weld quality, we should not only improve reliable
measurement means, but also introduce advanced real-time control methodologies.
The two technical fronts should be paid equal attentions, and developed simulta-
neously.
Computer Vision Sensing and Intelligent Control of Welding Pool Dynamics 27
27
In this study, a computer vision system combining video camera and image
processing, was used in the closed loop system to control a pulse GTAW in
real-time. A novel sensing system, which captures the topside and backside vision
images of a weld pool in the same frame simultaneously, was employed for moni-
toring and modeling of the welding process. The real-time image processing algo-
rithm was developed to acquire topside and backside sizes of weld pool. Further-
more, neural network models of pulsed GTAW process were established to predict
weld pool features from measured data. Based on the ANN models, experiments on
intelligent control schemes for the weld pool dynamic process were conducted.
2 The Pulsed GTAW Process and Monitoring Systems
2.1 Description of the pulsed GTAW process
The pulse GTAW is one of arc weld dominant technologies. It is widely used in the
high-quality weld manufacturing, especially in high-precise thin joining. The major
quality index in pulsed GTAW is precise penetration and fine formation of the weld
seam. In order to obtain good quality of welding products, dynamics of the welding
pool should be exactly dominated in real time, i.e., regulating accurate size of weld
pool. Main influences on weld pool in welding process involve weld current, such
as pulse duty ratio, peak current, base current, arc voltage, and welding speed; ex-
periment conditions such as the root opening or geometry of the groove, material,
thickness, work piece size, electrode tip angle, and rate of shielding gas flow;
welding conditions such as heat transfer condition, arc emission and so on. In de-
signing control systems, we should deal with following problems: sensing of weld
pool features, modeling of weld pool dynamic process, and designing effective
controllers for the process.
2.2 Monitoring systems of the pulsed GTAW
The structure diagram of monitoring system for the pulse GTAW welding is shown
in Fig.2-1.
The experiment system includes a double-side visual sensing subsystem, which

consists of CCD camera , recorder, frame grabber, and monitor, image processing
sections, welding quality controller completing by the computer , welding power
supply, and welding current setup was regulated by a computer. Single chip mi-
crocomputer (SCM) was used for control of travel speed. A wire feeder was applied
for filling wire metal into the weld pool during pulsed GTAW.
SCM motion Personal

control board computer
Interface
M anipulator Frame
Recorder
grabber
Power
Wire supply M onitor
feeder
CCD
camera
Torch Backside Topside
Work piece Composed

Work plate filter system
Travel direction
Fig.2-1 The structure diagram of experimental system for pulsed GTAW
2.3 Double-side visual sensing system for weld pool dynamics
In many practical cases, accessing to the backside of the weld piece is not possible.
According to the experience of skilled welder, the geometry of weld pool can pro-
vide instantaneous information about welding penetration. Topside and backside
images of weld pool should be captured and the geometry should be extracted for
modeling and control of the dynamic process. The backside width and penetration
could be predicted by the dynamic model and controlled by intelligent controller.
Therefore, a double-side visual sensor was designed to detect the face and back of
the weld pool for modeling welding penetration and formation process. The sensing
system consists of topside and backside light path and composite filters. Both top-
side and backside images concentrate on a same target of the CCD camera through
the above double-side imaging light path system.
3 Image Processing of Weld Pool in Pulsed GTAW
3 .1 Two dimensional character processing of the weld pool image

without wire filler
Bead-on-plate experiment was conducted on low carbon steel during pulsed GTAW
using the double-side imaging system, the experiment conditions are omitted here.
A complete weld pool image in a frame is shown in Fig.3-1, in which the left is
backside image and the right is the topside image. In Fig.3-1, the nozzle, arc center,
topside molten portion, and topside solidified portion could be clearly distinguished
from the topside image of weld pool.
29
Backside image Topside image

of weld pool of weld pool
Nozzle
Arc centre
Backside molten partition
partition
Topside molten
Backside solidified partition
partition Topside solidified
partition
Fig.3-1 A frame complete weld pool image of pulsed GTAW
Topside
Backside
(a) (b) (c) (d) (e) (f) (g)
Fig.3-2 The visual images of different time's weld pool in a pulse cycle
Fig 3-2 is top/backside pool serial images in a pulsed cycle. Fig.3-2 (b),(c),(d)
are pool images in pulsed peak current and (e),(f),(g) in the pulsed based current.
Based on the above analysis, the time sequence of pulsed current for collecting
images were determined. The taking image current was selected as 60A, taking
image time as 80ms for a frame complete weld pool image.
Used of image contrast between arc mirror reflect of the pool liquid surface and
arc diffuse reflect of frozen metal on work-piece surface, the arc disturbance could
be changed into an advantage for taking a clear image of weld pool. Based on arc
spectral analysis, an appropriate filtering light system was designed in wavelength
600nm 700nm and the stable top/backside images was obtained as Fig 3-3.
Nozzle
Backside of Arc centre

weld pool partition
Topside of
weld pool
Unmolten
Solidified
material
material
(a) Backside visual image (b) Topside visual image

Fig.3-3 Canonical topside and backside visual image of weld pool
3.1.1 Exponential base filter processing of weld pool images

The first step of weld pool image processing was filter and wiping off noise dis-
turbance. In order to reduce calculation, the recursion exponential base filter was
used to smooth images. The response property of exponential base filter is similar to
the Guass filter(Marr,1982).
One dimensional recursion exponential base filter

The unit sample response of Exponential Base Smoothing (EBS) was defined as:
S ( n) = k (α n + 1) e−α n 3-1
In Eq.(3-1) n is a discrete variable, α is a constant of the filter for

one-dimesional space range, k is a proportion factor defined as :

(1 − e− α )
k= 3-2
1 + 2 α e − α − e −2 α

Decomposing S ( n ) as cause-effect and non-cause-effect parts and supposing
filter input as x ( n ) yn
( ) is the output response of the EBS filtering S ( n ) we

have
y ( n ) = yc ( n ) + ya ( n ) 3-3
−α
yc ( n) = k x ( n) + e ( α − 1) x ( n − 1) + 2 e y c ( n − 1) − e
−α −2 α
y c (n − 2)
y a (n) = k [e−α (α + 1) x(n + 1) − e−2α x(n + 2)] + 2 e−α y a (n + 1)

− e−2α y a (n + 2)
Initial conditions as: x( 0) =0 y c ( 0) = y c ( −1) = 0 n = 1, 2,"" M
x ( M + 1) = x ( M + 2 ) = 0 y a ( M + 1) = y a ( M + 2) = 0

n = M ,""2,1
In Eq.(3-3), yc ( n ) is the response for the filt cause-effect part, ya ( n ) is the
response for the filt non-cause-effect part. One dimensional recursion exponential
base filtering algorithms could be realized by Eq.(3-3).
Two dimensional recursion exponential base filtering algorithms

Based on one-dimensional exponential base filtering function, the separable
two-dimensional exponential base filter was developed. For two-dimensional input
x ( m, n ) , one-dimensional recursion filtering algorithms along one direction could
be first completed due to separability of the filter, and then taking its output as input
for next one-dimensional filtering algorithms. The top/back side images of the weld
pool were smoothed by the above algorithms, i.e. Fig 3-4. The results shown that
31
noises in the topside image was filtered and the image margin was fully maintained
while α =1.41( σ = 1. 0 ). Taking α =0.94( σ = 1.5 ), the noises in backside image
was ideally sieved by the exponential base filter. Since the two-dimensional ex-
ponential base filter could be separated two one-dimensional recursion filters in two
directions, the computing time and space costs of the algorithms was greatly re-
duced, e.g., the algorithm time cost on a PC-486 computer with main frequency
100MHz was not more than 8ms for an image window with 160×190 pixel. The
time cost processing the same size image by the Guass filter was about 50ms. It is
obvious that exponential base recursion filter is suitable to process images in
real-time.
(a) (b) (c) (d)

Fig.3-4 The smoothed image of weld pool with EBS algorithm
(a) Original topside image (b) Topside image smoothed with EBS algorithm
(c) Original backside image (d) Backside image smoothed with EBS algorithm
3.1.2 Contrast Enhancement algorithms for the weld pool
Based on the contrast enhancement algorithms presented by Gordon et al. (1984)

and Beghdadi et.al.(1986), for supposed center pixel point ( x ,
y ) with greyscale
Gxy and window Wm and the each pixel point with greyscale Gij the following
contrast enhancement algorithms, named as CE, was adopted to enhance the con-
trast of the weld pool. The CE algorithms follows as:
Step1 For the pixel point( x ,

y ) calculating average greyscale G in its

adjacent region:
G = ¦ Gij / ( m × m)

(3-4
(i , j )∈Wm
Step2 For all pixel points in the window

Wm calculating the edge value ∆ ij

of the pool:
∆ ij = Gij − G
(3-5
Step3 For the window

Wm , calculating weighted average greyscale Exy for

the pool edge:
E xy = ¦ ∆ ij ∗ Gij ¦ ∆ ij

(3-6
(i , j ) ∈Wm (i , j ) ∈Wm
Step4

Calculating the contrast degree Cxy of the pixel point ( x , y ):

Cxy = Gxy − E xy Gxy + E xy
(3-7
Step5

Changing ′ = F (C xy )
Cxy to contrast transfer function C xy
Cxy′ = F ( Cxy ) = ( Cxy ) a b , b = 2 p , a < b

(3-8
Step6

Defining greyscale of the pixel point ( x , y G ′ and calculating

) as xy
functions as following:
R xy = (1 − Cxy′ ) (1 + Cxy′ ) if Gxy ≤ E xy
(3-9
= (1 + Cxy′ ) (1 − Cxy′ ) if Gxy > E xy
Gxy′ = E xy ⋅ R xy

(3-10
Step7

Repeat algorithms Step1 to Step6 for each pixel point.
Using the above CE algorithm for the topside images, the contrast enhancement
processing results are shown as Fig. 3-5. Let β=a/b window size m, for
β = 0. 5 to β = 0. 25 and m=5 to m=13, the edges and background of the weld
pool are learly distinguished in Fig. 3-5. Trading off the enhancement effect and
calculating complexity, β = 0. 5 and m=9 were determined in processing and
calculating time was not more than 16ms.
(a) (b) (c) (d) (e) (f) (g)

Fig.3-5 Contrast enhancement of the topside image of weld pool
(a) EBS smoothed image (b) CE( β = 0.5 , m=5) (c) CE(β = 0.5 , m=9)
(d) CE(β = 0.5 ,m=13)(e) CE(β = 0. 25 , m=5) (f) CE( β = 0. 25 , m=9)
(g) CE(β = 0. 25 , m=13)
33
3.1.3 Extracting the geometry characters of the weld pool
Defined the topside pool characters as: maximum width W f max , maximum half
length L f max , and the backside pool characters as: maximum width
Wb max maximum length Lb max and area S b , shown as Fig. 3-6, these
parameters could be used to discribe the pool characters in full penetration.
y
y
Lf m ax
Lbm ax
a b
o x c x
o
Wf max
Wbmax
Sb
(a) Topside of weld pool (b) Backside of weld pool

Fig.3-6 The definition of feature size parameters of fully penetrated weld pool
Algorithm for extracting the topside characters of the weld pool

Based on the greyscale distribution of the image in Fig.3-7, the algorithm ETG for
extracting top geometry of the weld pool was developed as the following:
The algorithm ETG: extract-top-geometry (image window topw)
{ Step1: get-centre-point C (image window topw);
Step2: for (i=-5; i<=5; i=+2)
{get-line -grey (along weld pool width direction fkv, through point
C, line-i)
find-edge-point (line-i) ( find point A and B from greyscale dis-
tribution);
calculate-width (line-i);
}
Step3: find-max-width (W f max from 11 lines);
Step4: find-max-centre( point C’);
Step5: get-line-grey (along weld pool length direction fkw, from C’);
Step6: find-peak-point((along fkw);
Step7: find-max-peak(point D);
Step8: calculate-mid-length( L f max from C’ to D);
}
o x
fkv
B
C
C'
A
fkw
y
Fig.3-7 Characteristic points of the topside image of weld pool

o x
bkw
Ep
Lp
Cp
Rp
Sp
bkv
y
Fig.3-8 Characteristic points of the backside image of weld pool
Algorithm for extracting the backside characters of the weld pool

The binaryzation of the back weld pool images , as Fig. 3-8, was obtained by the
EBS filter and the threshold division. From Fig.3-8, the pool length was determined
by the edge point Sp and Ep along bkw, and the width was determined by the edge
point Lp and Rp along bkv. Based on weld pool vertical-square picture distribution
by the EBS, the algorithm extracting back geometry (EBG) was developed as the
following:
The algorithm EBG extract-back-geometry(image window backw)
{ Step1: calculate-histm ( histm-histogram, for back pool image);
Step2: histm-moving-smooth( averaging-moving algorithm for histogram);
Step3: find-threshold
Step4: binaryzation( for back pool image);
Step5: get-centre-point(Cp in Fig.3-8);
Step6: for(i=-5; i<=5; i=+2)
{ get-line-grey (line-i grey from Cp, along bkw,)
35
find-edge-point(Sp and Ep, along bkw);

calculate-length (along line-i);
}
Step7: find-max-length ( Lb max from 11 lines);
Step8: get-interval (determining 21 lines from Lb max along bkv);
Step9: for(i=0; i<=20;i++)
{ get-line-grey (line-i along bkv)
find-edge-point (Lp and Rp along bkv);
calculate-width (along line-i);
}
Step10: find-max-width ( Wb max from 21 lines );
Step11: calculate-area( S b from 21 lines and Lbmax );
}
The signal flowchart of processing images of weld pool by the above algorithms
is shown as Fig.3-9, EBS, CE , ETG, EBG as the above, TD is the binary algorithm
for finding threshold from the histogram distribution of the backside weld pool
image.
Fig.3-9 Signal flowchart of processing images of weld pool
The above real-time processing algorithms for the top/backside weld pool im-
ages was running on the PC-486 computer, the calculating time for a frame image
was not more than 50ms for the top image and 30ms for back image. Ii was suffi-
cient for real-time closed loop control of welding dynamic process.
3.2 Three-dimensional character processing of the weld pool image

with wire filler
Although many achievements of sensing, modeling and control of GTAW pool

were proposed, studies on GTAW with wire filler was seldom carried out and re-
ported in present. Since two-dimensional image processing extracted the weld pool
edges, width, length and so on, which only reflected the plane shape variety, if there
is a convex and concave changes on weld pool surface and a high precise re-
quirements for weld seam height figuration, extracting three-dimensional charac-
ters of the weld pool would be necessary. In our study, three-dimension surface
parameters of weld pool has been extracted from single-item image of the weld
pool. One of evident advantages using single-item image is to made sensing in-
strument more simple and practical.
3.2.1 The topside images of the weld pool with wire filler
Under the supposed experiment conditions, welding parameters were set as follows:
pulse peak current 120A, base current 60A, pulse duty ratio 40%, and travel speed
2.5mm/s. According to imaging principle, image was determined by light source,
camera and object shape. In this experiment, imaging current was set as 30A, and
the light source of arc could be supposed as a point light source. The shutter of CCD
camera was set as 1/1000s, and the iris diaphragm and filter ratio were mounted.
The topside image of the weld pool with wire filler was taken as Fig.3-10.
In Fig.3-10, when the weld pool was partially penetrating, the top shape of weld
pool was approximate to a ellipse ,i.e., a convex model image; while on full pene-
trating, the pool image was similar to a peach shape, i.e. a concave model. The
length and stem shape varieties of the weld pool were most obvious.
Fig.3-10 The shape variation of topside weld pool
3.2.2 The character definition of weld pool with wire filler

1) Defining the top shape characters of the weld pool with wire filler as follows:
2) Defining the top height characters of the weld pool with wire filler as follows:
Due to the impact of arc plasma, the surface of weld pool was depressed in full
penetration, while the surface of weld pool could be convex in partial penetration or
with wire filler. The rear part of weld pool showed the different concave or convex
shape distinguishably, as shown in Fig.3-12. Where Sw was defined as
37
yr y
L th L tt
Lt x
0
R hl 1 xr
Wt
Fig.3-11 The definition of shape parameters of topside weld pool
concave area in the pool width y axis, Sl as concave area in the pool length x axis,
Wt as the top width, Lt as the top length, then the concave parameters Dl and Dw
Along top length and width of the weld pool were defined as:
Dl = S l / Lt , Dw = S w / Wt (3-11)
The surface height Ht of the weld pool was defined as maximum height of the
rear part, and it is similar to the seam surplus due to the frozen metal attached on the
rear part of weld pool.
3) The character definitions of the back weld pool is similar to that of weld pool
without wire filler in Fig.3-6(b)
Welding
Welding torch
direction
Weld pool Solid pool

Arc
Ht
Workpiece
a) Concave type
Welding Welding torch
direction
Weld pool Solid pool

Wire Filler
Ht
Arc
Workpiece
b) convex type.
Fig.3-12 The definition of height parameters of topside weld pool
3.2.3 The image processing of the filling wire weld pool

For short, calibration of the vision sensing systems is omitted and the processing
algorithms for the top characters of weld pool images are briefly described as the
following.
1) Image processing of the top weld pool

The different surface heights of the filling wire weld pool lead to obvious dif-
ference between greyscale distributions of concave and convex type images, so that
the different processing algorithms should be designed for them. The threshold di-
vision method is more suitable to convex type object images, and the edge extrac-
tion method is suitable to concave type images. In this paper, the image processing
algorithm TSE(Topside Shape Extraction) was developed to extract top characters
of the filling wire weld pool. It is consist of TI (Type Identification) algorithm for
distinguishing concave or convex type images, EE(Edges Extraction) for extracting
top edge points of the weld pool, and ER(Edges Regression) for imitating top edge
curve of the weld pool.

i) Type identification of concave or convex images TI

In this paper, Hough transformation [10] was used to distinguish the rear pool
shapes and judge concave or convex type images. The calculating results are shown
in Fig.3-13.
ii) Edges Extraction of top edge points of the weld pool EE
For the convex type image, usually, the image histogram indicates that the image
greyscale distribution has typical twin apex property, binary threshold value could
be determined by finding the vale point of twin apices, and the convex type binary
image could be obtained in Fig.3-14(a). The edge points of the weld pool could be
precisely extracted by the edge tracking method, as shown in Fig.3-14(b).
O X
fkw
{E'}
fkv {E}
Y
a) Convex type
39
O X
fkw
C
{E'}
{E}
fkv
Y
b) concave type
Fig.3-13 Type identification of topside image
a) Thresholding of convex image b) Edge tracing of the thresholding image
fkw
U2
U1
c
D2
fkv D1
c) Edge extraction of concave image

Fig.3-14 Extracting edge points of topside image
a) Convex type b) concave type

Fig. 3-15 The results of edges regression for topside pool
Lt
Topside
TI EE ER Wt
image
R hl
Camera
Fig.3-16 Signal flowchart of image processing for topside pool image
The signal flowchart of processing images of weld pool by the above algorithms
is shown as Fig.3-16, the algorithms TI, EE and ER are as the above. The whole
processing algorithms for the top/backside weld pool images was running on the
PC-486 computer with main frequency 100MHz, the calculating time for a frame
image was not more than 50ms. It was sufficient for real-time sensing and control of
welding dynamic process.
2) The backside image processing of the weld pool with filling wire is similar to that
of no-filling wire pool, here is omitted.
3.2.4 The extraction of the surface height of the weld pool with wire
filler
In the image of the weld pool with wire filler, the arc inverted reflection in the weld
pool image could be distinguished due to mirror reflection on molten metal surface
of the weld pool. So a simple method for extracting the surface height of weld pool
was presented from the reflection phenomenon: surface weight of weld pool could
be indirectly calculated by the extracted distance between the tungsten tip and its
inverted reflection in weld pool image. The pool images with different surface
height are shown as Fig. 3-17. The tungsten pole nozzle and terminal (i.e., the arc
initial terminal), arc shape in the images can be clearly distinguished in Fig3-17.
The algorithm calculating top height from the weld pool images follows as:
41
Fig.3-17 Weld pool image with different surface height of weld pool
Step1: Image filtering (Guass filtering)
Step2: Binaryzation of images
Processing results shown as Fig.3-18(b) by threshold division of pool image hist-
gram,
Step3: Determining the position of tungsten tip, the result calculated by binaryza-
tion images and Hough transformation is shown as Fig.19 and the length L1 in
Fig.17, the processed image as 3-18(d).
Step4: Calculating the surface height of the weld pool
Because the tungsten pole was warded by welding torch nozzle in welding, the
distance between the tungsten tip and its inverted reflection in weld pool image
couldn’t be directly calculated. So we obtained distance between the tungsten pole
and nozzle, L0 as Fig. 3-18(e), and let arc length as La, then the top surface height of
the weld pool, Ht was calculated by
H t = La − fcal.z * ( L0 + L1 ) / 2.0 (3-11)
a) Initial image b) binary image c) width calculation d) height extraction e) distance

from the tip to nozzle
Fig.3-18 The height result from image processing topside weld pool
16
W , P ix e ls
12
4 Arc Tungsten
0
0 2 4 6 8 10 12
Num be r
Fig.3-19 Determination of the position of tungsten tip

The concave and convex of the weld pool surface could be obtained by the above
image processing and calculating. Comparing the calculated and measured results,
shown as Fig.3-20, the average error of the concave and convex height was about
0.1mm. The errors were brought due to such reasons: a) the mirror assumption on
the metal surface of weld pool; b) the height changes as solidifying metal in weld
pool; c) the image quality and processing precision.
1 0.2
0.8 0
Ht, mm
Ht, mm
0.6 -0.2
0.4 Calculated -0.4 Calculated
0.2 Measured -0.6 Measured
0 -0.8
5 7 9 11 13 15 17 19 5 7 9 11 13 15 17 19
Pulse number Pulse number
a) Convex surface b) Concave surface

Fig.3-20 Comparing the calculated height of topside weld pool and the measured rein-
forcement
4 Modeling and Control of Weld Pool Characters Based on

Two-Dimensional Image Processing
Based on extracted two-dimensional characters of weld pool images, the relation-

ship between the top shape parameters and back width of weld pool without wire
filler could be established for predicting and control of the back width of weld pool.
4.1 The neural network modeling of weld pool dynamics
The traditional modeling method is usually unsuitable to complex welding dy-

namics. Since artificial neural networks (ANN) is able to preferably approach
complex non-linearity and uncertainty of unknown object and non supposition on
the object is necessary for modeling, it was used to weld pool dynamics.
4.1.1 Neural network modeling principle of the welding pool dynamics

Generally, neural network modeling is directly learning the input/output datum
from the system experiment, the index model convergence is so-called cost function
to be minimum. The identification modeling principle by neural network for the
welding pool dynamics is shown as Fig.4-1, where TDL (Tapped Delay Line) is
time delay unit.
43
Fig.4-1 The principle diagram of dynamic process identification with neural network
As is well known, identification modeling process by neural network usually

includes to design exciting persist signal determine model structure, choose iden-
tifying algorithms, train the model and verify model precise, here the details are
omitted.
4.1.2 Neural network model for predicting the backside size of the
weld pool
In order to exactly control full penetration and weld seam forming, the neural
network model for predicting the backside size of the weld pool, BNNM(Backside
Neural Network Model), was established as Fig.4-2, the model includes 17 inputs,
24 hide layer units, the output of the model, Wb max , for the back maximum width of
the weld pool. The examined results of the model is shown as Fig.4-3. The average
relative error of the model output Wb max was 3.32%, mean-square error was 4.01%.
The net model was converged and the model precise was desired.
4.2 The neuron self-learning control of the based-on weld pool in

pulsed GTAW
As to the multivariable and time-delay non-linearity in welding process, the con-

ventional PID closed-loop control was limited because its coefficients were hardly
regulated on-line, and the simple fuzzy control didn’t contain the adaptive function
with the variation of welding conditions. Aiming at the characteristics of changing
structure and coefficients, a self-learning controller based on single neuron was
proposed in this study. Only the desired output and the real output detected in
on-line were needed to form the self-learning closed-loop control system, without
the on-line identification of the process coefficients. The weight of neuron was
corrected on-line with the improved BP algorithm, and the performance of error
was minimized to optimize the output of the control system.
Hidden layer
18
19
Input layer 20
21
Ip(t)
1 22
Ib(t)
2 23
Vw(t-2)
3 24 Output layer
Vw(t-1)
4 25
Vw(t)
5 26
Ua(t-2)
6 27
Ua(t-1)
7 28
Ua(t)
8 29
δ (t-2) W bmax(t)
9 30 42
δ (t-1)
10 31
δ (t)
11 32
W fmax(t-2)
12 33
W fmax(t-1)
13 34
W fmax(t)
14 35
Lfmax(t-2)
15 36
Lfmax(t-1)
16 37
Lfmax(t)
17 38
39
40
41
Fig. 4-2 The structure of BNNM neural network dynamic model
7
BNNM output
6
W bmax, mm
4
actual output
3
0 30 60 90 120 150 180
Sample number
Fig.4-3 The result of detecting BNNM model
4.2.1 Neuron self-learning control system for pulsed GTAW

The schematic diagram of the neuron self-learning control system is shown in
Fig.4-4, where the effect of signal convert is to combine the desired and predicted
backside width, for generating the output variables X = {x1,x2,x3} as the input of
neuron self-learning controller, which are summed and transferred nonlinear to
derive the variation of pulse duty ratio (∆ δ ). The actual outputs are measured by
MS and input in BNNM to attain the predicted backside width Wmb max , with which
the set value R input in Signal Converter. At the same time, the input weights of
neuron is adjusted in on-line with BP algorithm according to the error of backside
width.
45
x1
R w1 z −1
Converter
Wbmax
∆δ + δ
Signal
x 2 w2 Pulsed GTAW
¦ F ( s)
+ Process
x w3
3
WP
Wfmax
Wmbmax
BNNM MS Lfmax
Fig.4-4 The schematic diagram of single neuron self-learning control syste mduring pulsed
GTAW
4.2.2 Designs of self-learning neuron controller for based-on plate in

pulsed GTAW
In Fig.4-4, the input variables X = {x1,x2,x3} of the controller for based-on plate in
pulsed GTAW are the error, error with first order, error with second order between
desired backside width and BNNM output, shown as follows.
x1 = α1e( t ) = α1 ( R − Wmb max ( t )) 4-1
x2 = α 2 ∆e( t ) = α 2 ( e( t ) − e( t − 1)) 4-2
x3 = α 3 ∆2 e( t ) = α 3 ( e( t ) − 2e( t − 1) + e( t − 2 )) 4-3
Where α 1 , α 2 and α 3 are constant indicating the emphasis degree in the

controller, selected as α 1 is 1.0, α 2 is 0.3, α 3 is 0.1. Weight normalization is to
avoid the saturation of weight during learning process, shown as follows:
3
wi′ = wi ¦w
j =1
2
i 4-4
The sum of weighted neuron is:

3
s = ¦ wi′ ⋅ xi 4-5
i =1
F ( s) is nonlinear transfer function, selected as hyperbolic tangent function.
∆δ = F ( s ) = γ (1 − e−2ξ S ) (1 + e−2ξ S ) 4-6
Where γ and ξ are two constants, γ sets the saturated value of control
variable, and ξ sets the linear degree of control variable. The larger is γ , the
more possible is to attain the desired value. The smaller is ξ , the wider is the linear
work region to restrain the fluctuation of stable status. γ is selected as 300 and ξ
is 0.135.
Object function is minimized by the follow equation.
1
E( w) = ¦
2 k
( R − Wmb max ( k ))2 (4-7
The derived corrected formula of weight is.

∆wi = ηϕ x i (4-8
wi ( k + 1) = wi ( k ) + ∆wi(4-9
Where i = 1, 2, 3, η is learning rate, equals to 1.0. ϕ is the equalized out-

put error of neuron, resembled with the actual output error.
ϕ = ( R − Wmb max ( k )) ⋅ (1 − ∆δ ( k ) γ ) ⋅ (1 + ∆δ ( k ) γ ) (4-10
4.2.3 Experiments on self-learning neuron controller for based-on

plate in pulsed GTAW
To test the feasibility of the self-learning neuron control schematic for practical use,
the experiment during bead-on-plate pulsed GTAW on the dumbbell specimen,
2mm thickness mild steel plate, was conducted, the desired seam width is 5.00 mm.
The controlled and responsing curves are shown in Fig.4-5. The maximum error
between BNNM output and desired value was 0.26mm. Tested data was compared
with the desired value, the maximum error was 0.30mm, the average error was
0.10mm, and the root-mean-square deviation was 0.08mm. The perfect effect of
topside and backside photographs are also shown as Fig.4-6. The top/backside
images during welding process are shown as Fig.4-7, one could see that the welding
pool images were maintained stable and controlled effect was fine.
Moreover, PID and Fuzzy controller were also implemented for pulse GTAW
process control. Each control scheme was limited to ensure each backside width.
Fuzzy control could simulate the worker’s experience, but without the adaptive
regulation with the varied conditions. While the neuron self-learning control could
attain perfect control effect with different set values and conditions, and was suit-
able for the varied structure and coefficients of welding process specially.
8 50
40
6
W bmax, mm
30
δ, %
4
20
pulse duty ratio
2 Vw=2.5mm/s; Ip =120A; Ib=60A; actual output 10
l=3.0mm; L=8l/min; BNNM output
0 0
0 10 20 30 40 50 60 70 80 90 100
Time, s
Fig.4-5 The neuron self-learning closed-loop control curves of dummy bell workpiece
during pulsed GTAW
(a) Topside
47
(b) Backside
Fig.4-6 A photograph of dummy bell workpiece with neuron self-learning control
Welding time, s
10 15 20 25 30 35 40 45 50 55
backside topside
backside topside
60 65 70 75 80 85 90 95 100
Welding time, s
Fig.4-7 The topside and backside images of weld pool of dummy bell workpiece with
neuron self-learning closed-loop control
5 Modeling and Control of Weld Pool Characters Based on

Three-Dimensional Image Processing
Based on extracting three-dimensional characters of the weld pool images, the re-
lationship among the top shape and height parameters and back width of the weld
pool with wire filler was modeled by the ANN, and predicting and control of
characters of weld pool could be realized.
5.1 The neural network model for the top height and back width of the
weld pool with filler
In order to reflect relationships between the top height and back width of the weld
pool with filler, the neural network dynamic model for the top height and back
width, BHDNNM(Backside and Height Dynamic Neural Network Model), was

established as Fig. 5-1. The model includes the current values of the top pool shape
and welding parameters, such as welding current or pulse duty ratio , filling rate,
travel speed, and their former moment values, 21 input variables, 5 hide nodes.
Input layer Hidden layer
P(1)
1 22 P(1) =Ip (t) P(13)=Lt(t)
P(2) P(2) =δ(t) P(14)=Wt(t)
2 23
P(3)
Output layer P(3) =Vw (t) P(15)=Rhl(t)
#
3
# #
P(4) =Vf(t) P(16)=Lt(t-1)
P(5) =Ip(t-1) P(17)=Wt(t-1)
# # # W b(t) P(6) =δ(t-1)

27 P(18)=Rhl(t-1)
P(7) =Vw(t-1) P(19)=Lt(t-2)
28 Ht(t)
P(8) =Vf(t-1) P(20)=Wt(t-2)
# # # P(9) =Ip(t-2)
P(10)=δ(t-2)
P(21)=Rhl(t-2)
P(20) P(11)=Vw(t-2)
20 25 P(12)=Vf(t-2)
P(21)
21 26
Fig.5-1 Structure of BHDNNM
With the trained BHDNNM and inputs of the model, backside width and topside
height could be predicted, and the predicted and measured values were shown in
Fig.5-2. The results showed that mean errors of backside width and topside height
were 0.004mm and 0.042mm respectively, and the relative mean square error were
5.54% and 7.83% respectively. The statistic results verified the validity of
BHDNNM.
Under the variation of heat-sinking conditions, the backside width and topside
height varied irregularly, but the backside width and topside height could be pre-
dicted accurately with the developed model BHDNNM, then these predicted values
could be input to the controller as feedback for control of bead shape.
5.2. Double-inputs and Double-outputs (DIDO) Self-learning

Fuzzy-neural network Control of the backside width and topside
height of the weld pool
From the step response of the weld pool process with wire filler, one can see that the
pulsed duty ratio is a main impacting variable to the back pool width, and the filling
rate to the top pool height. The backside width and topside height could be regulated
respectively by pulse duty ratio and filling rate. While the both shape parameters
could not be controlled at the desired value with the single-variable controller. The
pulse duty ratio and filling rate should be regulated simultaneously for controlling
the backside width and topside height of the filling wire pool.,i.e, The double-input
and double-output (DIDO) controller should be developed.
49
8.0
Measured Estimated
6.0
Wb,mm
4.0
2.0
0.0
1800 1820 1840 1860 1880 1900
Sampling Number
1.2
Measured Estimated
0.6
Ht,mm
0.0
-0.6
-1.2
1800 1820 1840 1860 1880 1900
Sampling Number
Fig. 5-2 Testing results of BHDNNM
5.2.1 Design of the DIDO Self-learning Fuzzy-neural network

Controller
Fuzzy controller was capable of humanoid inferable function. On designing fuzzy

controller, the decision of membership function and fuzzy rules was the hindrance.
An improved DIDO self-learning fuzzy controller based on single layer neural
network for real-time perfecting fuzzy rules was designed as Fig.5-3. Where the
MS was the measuring system for detecting the topside shape parameters TSP
Learning
Algorithm
Co
∆δ
z-1
W bg + Si e 1 ce 1 Multi-variable + δ
Pulsed GTAW Wb
nv Self-learning Fuzzy
e 2 ce 2 + Process with
Htg gn + Ht
ert Controller Wire Filler
+ ∆ Vf + Vf
- z-1
Wbp WP & TSP
BHNNM MS
Htp
Fig.5-3 Double variable self-learning fuzzy controller for butt pulsed GTAW with wire
filler
(Lt ,Wt and Rhl) and the welding parameters WP, such as welding current, travel
speed, etc. The predicted backside width (Wbp) and topside height (Htp) were
generated by BHDNNM. The given backside width (Wbg) and topside height (Htg),
Wbp and Htp were input to signal converter for calculating the errors ei(i=1,2) and
change in errors ce i(i=1,2).
Fuzzy Rules and Inference

Takagi-Sugeno rules (Takagi, Sugeno ,1985) were adopted as the fuzzy rules.
Rule1: if e1 is E11 and ce1 is CE11 ande 2 is E 21 andce2 is CE21 ,
then∆δ = u11 and ∆Vf = u 21
Rule2 : if e1 is E12 andce1 is CE12 ande 2 is E 22 andce2 is CE22 ,
then∆δ = u12 and ∆Vf = u 22
"" Where
Rulen : if e1 is E1n andce1 is CE1n ande 2 is E 2n andce2 is CE2n ,
then∆δ = u1n and ∆Vf = u 2n
Case : e1 is E10 andce1 is CE10 and e 2 is E 20 andce2 is CE20
Conclusion: ∆δ = u10 and ∆Vf = u 20
Eij and CEij are the fuzzy set of ei and cei, uij is the control variable, ei0 and
cei0 are the actual sampling values, i=1,2 , j=1,2,…,n
The matching degree of the fact in the rules is:
h j = µ E1 j (e10 ) ∧ µ CE1 j (ce10 ) ∧ µ E2 j (e 20 ) ∧ µ CE2 j (ce 20 ) , j = 1,2,..., n (5-1)
Where µ Eij (ei 0 ) and µ CEij (cei 0 ) are the membership of ei0 cei0 to fuzzy sets
Eij CEij, and “∧” is for minimum calculation.
The conclusion ui0 was derived as follows:
n
¦h u j ij n hj n
¦ ¦ h′ u
j =1
ui0 = n
= n
u ij = j ij , i = 1,2 (5-2)
¦j =1
hj j =1
¦
j =1
hj j =1
Where h′j represents the normalized value of the membership hj.

In the fuzzy control rules, the premise variables were described as fuzzy set and
constant. The conclusion variable was described as constant, adapted according to
control performance, and derived without de-fuzzier calculation.
Realization of Single-layer Fuzzy Neural Network

The memberships of the premise variable in the rules were set as isosceles tri angle,
51
1 1
0.8 0.8
0.6 0.6
MF
MF
0.4 0.4
0.2 0.2
0 0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 -0.5 -0.25 0 0.25 0.5
Zone of E Zone of CE
Fig.5-4 Fuzzy sets of premise variables
shown in Fig.5-4. The sets of e1 e2 were [-1,1], and 11 fuzzy sets were defined.
The sets of ce1 ce2 were [-0.5,0.5], and 5 fuzzy sets were defined. The set of error
and change in error were [-2,2] and [-1,1] for the backside width, and [-1,1] and
[-0.5,0.5] for the topside height. The overlap of the neighboring fuzzy sets was
50%, so a known value of any premise variable could activate two fuzzy sets.
There were four variables in the premise, so a known fact could activate 16 (24=16)
fuzzy rules. There were maximum 16 non-zero values in the n normalized matching
degrees, and they were numbered renewably as h′m, m=1,2,…,16.
The above inference process could be realized with a single-layer neural net-
work, shown in Fig.5-5. The single-layer neural network had 16 inputs, corre-
sponding to the 16 normalized matching degree h′m. The network had two outputs,
corresponding to the two control outputs of the fuzzy controller. The transferring
function of the neuron was sigmoid function.
1 − e − oi
16
u i = u i ,max
1 + e −oi ,
oi = ¦ h′ u
m =1
m im , i = 1, 2 (5-3)
Where ui,max was the largest scope of output variable, u1,max=∆δmax=10%,
u2,max=∆Vf=5mm/s. uim denoted the weight m connecting to neuron i, as well as the
conclusion constant to be regulated in the control rules.
hm h'm
e1 u1 (∆δ)
1 1
ΣS u1,max
ce1 Matching 2 2
Normali-
e2 degree
...
...
zation
ce2 calculation ∆ Vf )
u2 (∆
16 16
ΣS u2,max
Single layer neural network

Fig.5-5 Double variable fuzzy controller constructed by single-layer neural network
Learning Algorithm of the controller

The performance cost of the neural network learning was the output errors of
welding process.
2
1
E (t ) =
2
¦ [r (t ) − y (t )]
l =1
l l
2
(5-4)
Where rl(t) represented the given backside width and topside height Wbg and Htg,
and yl(t) represented the predicted Wbp and Htp. Then the weights of the neural
network was corrected as follows:
∂E (t )
u im (t + 1) = u im (t ) + ∆u im (t ) = u im (t ) − β
∂u im (t )
2
∂y l (t ) ∂u i (t ) ∂o i (t ) (5-5)
= u im (t ) + β ¦ l =1
[rl (t ) − y l (t )] ⋅ ⋅ ⋅
∂u i (t ) ∂o i (t ) ∂u im (t )
Where β was learning coefficient, β=1.0.
∂y l (t )
was the partial differential of the output l to the input i at time t.
∂u i (t )
∂y l (t ) y l (t ) − y l (t − 1) ∆y l (t )
≈ = (5-6)
∂u i (t ) u i (t ) − u i (t − 1) ∆u i (t )
and,
∂u i (t ) 1 u (t )
= u i ,max ⋅ [1 − ( i ) 2 ] (5-7)
∂oi (t ) 2 u i ,max
∂oi (t )
= hm′ (t ) (5-8)
∂u im (t )
then the final formula of weight correction was derived.
2
∆y (t ) 1 u (t )
¦
u im (t + 1) = u im (t ) + β el (t ) ⋅ l ⋅ u i ,max ⋅ [1 − ( i ) 2 ] ⋅ hm′ (t ) (5-9)
l =1 ∆u i (t ) 2 u i ,max
5.2.2 Experiments on DIDO self-learning Control systems for weld

pool with wire filler
To verify the performance of the double-variable self-adaptive fuzzy controller,
butt welding experiments with wire filler were conducted on dumbbell-shape
specimens. The minimum regulating unit was 1% for pulse duty ratio, and 0.1mm/s
for Vf. The backside width and topside height were given as 4.0mm and 0.2mm.
The errors and changes in errors were calculated with the given and predicted value
by the BHDNNM, and were looked up in the fuzzy control rules. If matched, the
change of the control variable could be derived directly. If not, the self-learning
was carried out. The control impacting was added at initial 7th pulse. The weld
pool images were shown in Fig.5-6, and it was concluded that the double-sided
shape of the weld pool controlled reliably.
The variation of the shape parameters was shown in Fig.5-7. This indicated that
the backside width and topside height accessed to the given value quickly when
control affected. The controlled parameters were maintained around the given
53
values even in hard heat-sinking conditions. Statistic results showed that the
maximum errors of the backside width and topside height were -0.44mm and
0.06mm, the mean errors were -0.09mm and 0.006mm. The results testified the
feasibility and accuracy of the double-variable self-adaptive fuzzy controller.
Photographs in Fig.5-8 also showed that backside width and topside height of bead
were stable.
Time, s
5 10 15 20 25 30 35 40 45 50
Topside
Backide
Topside
Backside
55 60 65 70 75 80 85 90 95 100
Time, s
Fig.5-6 Double-side images of butt welding with wire filler by double variable fuzzy
control during pulsed GTAW
5 0.5 60 8
4 0.4
6
Wb, mm
50
Ht, mm
Vf, mm/s
3 0.3
,%
4
2 0.2
40
1 0.1 2
Wb Ht δ Vf
0 0.0 30 0
0 20 40 60 80 100 0 20 40 60 80 100
Time, s Time, s
a) Double-side shape parameters of weld pool b) pulse duty ratio and filler rate
Fig.5-7 Curves of double variable fuzzy control
6 Conclusion Marks
The works in this paper has shown that the artificial intelligence methodology, such
as computer vision sensing, image processing, fuzzy logic, artificial neural net-
works, intelligent control scheme etc., would be greatly suitable to be applied in
sensing, modeling and control of the welding pool dynamics in pulsed GTAW.
Developments of AI technology applied in welding manufacturing would inaugu-
rate an inspired approach of solving difficult problems for many years. We believe
that the many puzzle problems in welding automation and robotic welding need to
be processed by AI. And AI technology would tend towards to practicality in
welding and manufacturing.
a) Topside
b) Backside
Fig.5-8 Photographs of dumbbell-shaped workpiece with double variable fuzzy control
during pulsed gas tungsten arc butt welding with wire filler
Acknowledgment
This work was supported by the National Natural Science Foundation of China and
Science and Technology Committee of Shanghai, China, No.021111116.
References
1. Andersen, K., Cook, G.E., 1990. Artificial Neural Networks Applied to Arc
Welding Process Modeling and Control. IEEE transaction Industry Application
26 (9), 824-830.
2. Beghdadi, A., Negrate, A. L., 1986. Contrast Enhancement Technique based on
Local Detection of Edges. 1Comput. Vision Graphics Image Process, 46 (9),
162-174.
3. Brzakovic, D., Khani, D.T., 1991. Weld pool edge detection for automated control
of welding. IEEE Transactions on Robotics and Automation 7 (3), 397-403.
4. Canny, A., 1986. Computational Approach to Edge Detection. IEEE Trans. on
Image Processing 28(2), 679-698.
55
5. Gordon, R., Rangayan, R. M., 1984. Feature Enhancement of Film Mammograms

using Fixed and Adaptive Neighborhood Appl. Opt. 23(5), 560-564.
6. Hoffman, T., 1991. Real-time imaging for process control. Advanced Materials &
Processes 140 (3), 37-40.
7. Kim, E.W., Allem, C., .Eagar T.W., 1987. Visible Light Emissions during Gas
Tungsten Arc Welding and Its Application to Weld Image Improvement. Welding
Journal 66 (12), 369s - 377s.
8. Lee, C.W., Na S.J., 1996. A Study on the Influence of Reflected Arc Light on Vi-
sion Sensors for Welding Automation. Welding Journal. 75 (12), 379s-387s.
9. Lim, T.G., Cho, H. S., 1993. Estimation of Weld Pool Sizes in Gma Welding
Process Using Neural Networks. Journal of Systems and Control Engineering
207(1), 15-26.
10. Marr, D., Vision, A., 1982. Computational Investigation into the Human Repre-
sentation and Processing of Visual Information. W. H. Freeman and Company, San
Francisco.
11. Pham, D.T., Karaboga, D., 1999. Self-tuning fuzzy controller design using genetic
optimization and neural network modeling, Artificial Intelligence in Eng. 13(2),
119-130.
12. Pietrzak, K.A., Packer, S.M., 1994. Vision-based weld pool width control. ASME
Journal of Engineering for Industry 116 (2), 86-92.
13. Richardson, R.W., Gutow, D.A., 1984. Coaxial Arc Weld Pool Viewing for
Process Monitoring and Control. Welding Journal 63 (3), 43-50.
14. Saedi, H.R., Unkel, W., 1988 Arc and weld pool behavior for pulsed current
GTAW. Welding Journal 67(11), 247s - 255s.
15. Salter, S., Deam, R.T., 1986. A Practical Front Face Penetration Control System
for TIG Welding. Proceedings of an International Conference on Trends in
Welding Research, Gatlinburg, Tennessee, USA, May 18-22, 381-392.
16. Takagi,T., Sugeno, M., 1985. Fuzzy identification of systems and its applications
to modeling and control. IEEE Trans. on Systems, Man and Cybernetics 15(1),
116-132 .
17. Vilkas, E. P., 1966. Automation of gas tungsten arc welding process. Welding
Journal 45(5), 410s - 416s.
18. Willis, M. J., et al., 1992. Artificial neural networks in process estimation and
control. Automatica 28(.6), 1181-1187.
Surface Shape Reconstruction of Weld Pool
during Pulsed GTAW from Its Single Image
D.B. Zhao1, J.Q. Yi1, S.B. Chen2, and L. Wu3
1
Lab of Complex System, Institute of Automation, Chinese Academy of Sciences,
Beijing 100080, P.R. China
2
Institute of Welding Technology, Shanghai Jiao Tong University, Shanghai
200030, P.R. China
3
National Key Laboratory of Advanced Welding Production Technology, Harbin
Institute of Technology, Harbin 150001, P.R. China.
dongbin.zhao@mail.ia.ac.cn
Abstract. This paper addresses an improved algorithm of shape from shading

with a single real image, and its application on three-dimensional surface
reconstruction. The improved algorithm consists of smoothing constraint, boundary
height constraint, and weighted light error. For consideration of actual visual
sensing systems, the feasibility of the algorithm is verified with a single synthetic
image. Finally, the surface shape of weld pool during pulsed GTAW is
reconstructed successfully from its real single image.
1 Introduction
The algorithm of shape from shading is based on the principle of imaging geometry
reflection. Several algorithms were developed by Horn [1], etc, to extract the surface
shape of an object, aiming at ideal reflection model, such as Lambertian surface,
orthographic projection, and a distant light source. The reflection model becomes
more complex in actual imaging conditions, such as non-Lambertian surface,
prospective projection, and a nearby point light source. A generalized reflection
model was established by Lee [2] to recover the object surface shape from multiple
images taken by different illumination directions. But the multiple images are
impossible in most cases, so shape from a single image under actual imaging
conditions is crucial for many applications, such as aerial image, human face
photos, and industrial vision applications.
Surface Shape Reconstruction of Weld Pool during Pulsed GTAW 57
57
Visual imaging is a key method to sense pulsed gas tungsten arc welding process
(GTAW). Zhang [3] developed a laser structured light system to achieve the specular
reflection of a pulsed laser projected on weld pool, to calculate the
three-dimensional weld pool surface shape. The shape of weld pool is crucial for
weld quality. Therefore, the more accuracy of the three-dimensional weld pool
surface shape is measured, the higher quality of the weld can be achieved.
This paper presents an improved algorithm of shape from shading with a single
image and its application on weld pool surface reconstruction.
2 Basic Algorithm of Shape from Shading
The generalized imaging reflection geometry model is shown in Fig.2.1. In the

figure p is a point on object surface, and n is its normal line. S is a point light source,
r is the distance from the light source S to the point p. i is the unit vector toward the
light source S, and v is the unit vector toward camera C. The specula direction h is
the bisector of i and v. θi, θv and α represent the angles between n and i, n and v, n
and h respectively.
Considering a nearby point light source, diffuse and specula reflection of object
surface, and perspective project of a camera, a generalized reflection model (image
intensity E) is derived
* I0 T exp{−k r [cos −1 (hT n)]2 } T
- (v n z ) 4 ( β d (i T n) + β s ), (i n) ≥ 0 (2.1)
E = R ( z i , z j , z l ) R (i , n , v , r ) = + r 2
!
(i T n)(v T n)
-0, otherwise
,
Where kr is the surface roughness parameter, βd and βs are the diffuse and specula
reflective coefficients. A square image domain can be divided into a set of Mt
non-overlapping triangles Tk, k=1, 2, …, Mt, with Mn nodal points Pk, k=1, 2,…, Mn.
The smooth object surface can be approximated with the corresponding triangle
surface patches Ti with nodal points pk (zi, zj, zl). The variable vectors i, n, and v can
be described with the three vertices pk (zi, zj, zl). With Taylor series expansion, the
generalized reflection model is linearized.
Mn
Rk ≈ # wkm z m + ξ k (2.2)
m =1
Point light
s ource C
n
S θi α Camera
i h θv
v
d ωi d ωv
dA p
Fig. 2.1. Generalize reflection geometry model

58 58 D.B. Zhao et al.
Where
Mn
ξ k = Rk ( zin −1,z nj −1,zln −1 ) - # wkm z mn −1 ,
m=1
* ∂Rk ( z i , z j , z l )
- , if m ∈ Vk = {i, j , l}
wkm = + ∂z m ( zin−1 , z nj−1 , zln−1 ) , Vk is the set of each
-0 , otherwise
,
triangle vertices.
Light error is taken as the cost function.
2
Mt ' 2$ Mn Mt .1
eb = # ( E k − R k ) = # ( E k − %% # wkm z m + ξ k //2 (2.3)
k =1 k =1 ) & m =1 03
To minimize the cost quadratic function, a linear formula is derived.
Mt Mt
Az = d , [A ]m,n = 2 # wkm wkn , [d ]m,n = 2 # ( E k − ξ k ) wkm , 1 ≤ m, n ≤ M n (2.4)
k =1 k =1
Where A is a sparse and symmetric coefficient matrix, and d is a load vector. The
height matrix of the vertices z can be solved by iterative method.
Synthetic image is generated with the reflection model, and is often used to
verify the feasibility of reconstruction algorithms. The object is supposed as a
most half sphere embedded in a plane zp=-100 mm. The sphere diameter is 6.0
mm, and the diameter of the intersecting circle is 4.0 mm. The position of the
camera is xc=0, yc=0, zc=0, the lens focal length is f=2.5 mm, and the target
dimensions are t=4.8×4.8 mm2, corresponding to the maximum image sizes
512×512. The parameters of the object surface are βd=0.5, βs=0.5, and kr=5.0.
The light intensity is supposed as rs2. Two synthetic images are formed with
different illumination directions and a fixed camera, shown in Fig.2.2. The image
sizes are 32×32.
With the above basic algorithm, the surface height can be reconstructed
accurately with the two images, shown in Fig.2.3 (a). But the surface height results
from a single image have a large error, shown in Fig.2.3 (b).
(a) (b)
Fig.2.2. Synthetic images with different illumination
directions: (a) xs=60 mm, ys=0 mm, zs=0 mm; (b)
xs=0 mm, ys=60 mm, zs=0 mm.
59
3 Improved Algorithm of Shape from a Single Image
To improve the accuracy of the algorithm, smoothing constraint, some prior

knowledge about the surface height, and weighted cost function are proposed.
The object surface is supposed as smooth, then the cost function are renewed.
e = eb + λe s , Cz = b , C = A + λB (3.1)
Where es is the cost function of the smoothing constraint. λ is a smoothing
factor.
The algorithm of shape from shading is

-98
effective specially to reconstruct smooth
surface, which can be considered as a curved -99
surface embedded in a known background
z, mm
plane. We take the background plane or -100
boundary as prior knowledge. Other
-101
information can also be considered, such as
24 24
the maximum or minimum height, the position y, 16
16 l
pix 8 8 i xe
of the occluding boundary, etc. el 0 x , p
The illumination direction influences the (a)

accuracy of the calculation results greatly. -98
Where the normal line of the surface patch is
similar to the illumination direction, the image -99
z, mm
intensity is the highest and the calculation

-100
error is the largest. So grayness-weighted
regulation is introduced -101
Mt J 24
j j 2 24
eb = # # w( E k ) ⋅ ( E k − R k ) (3.2) y, 16
p ix 8 8
16 el
pix
k =1 j =1 el 0 x,
Where ω(Ek) is the grayness-weighted factor, (b)

*-e − ( Ek −150) 2 / 5000 , E ≥ 150 -98
and w( E k ) = + k
-,1 , otherwise
-99
z, mm
When grayness is smaller than 150, the

weight is the maximum 1.0, which indicates -100
that image point with low grayness plays an -101

important role in the surface height 24
24
reconstruction. y, 16 8
pix 8
16 el
ix
p
With the proposed smoothing constraint, el 0 x,
boundary condition, and grayness-weighted (c)

regulation, the improved algorithm of shape Fig.2.3. Reconstructed surface
from a single image is formed. Surface height results: (a) from two images
with the basic algorithm; (b) from a
height is reconstructed accurately from a single image with the basic
single image, shown in Fig2.3(c). Statistic algorithm; (c) from a single image
results show that the average error is -0.04 with the improved algorithm.
mm, and the mean square error is 0.11 mm.
4 Surface Height from a Single Weld Pool Image
The visual sensing system for GTAW process is shown in Fig.4.1. The world
coordinate system o-xyz with the center of weld pool as the original, and the camera
coordinate system O-XYZ with the CCD camera as the original are both Cartesian
coordinate systems. Two mirrors, M1 and M2, are used to reflect weld pool. O1A
and O2B are the normal line of the mirrors, described by the angles to the axes. A
point p(x,y,z) in the world coordinate system is reflected by M1 and M2, and is
focused to a point Pf(Xf,Yf,-f) on the focus plane of the CCD camera. With the
visual sensing system, weld pool images are captured, shown in Fig.4.2. The left
corresponds to convex weld pool surface on local patches with large wire feed
speed, and the right corresponds to concave weld pool surface with small wire feed
speed. The image sizes are 128×128. After shape reconstruction from weld pool
images, surface height results in the camera coordinate system should be
transformed into the world coordinate system considering the real visual sensing
system characteristics.
The imaging conditions are:
Point light source: In the visual sensing process for GTAW, electric arc can be
thought as a point light source [4]. Its position is s(xs,ys,zs)=(0, 0, la)= (0, 0, 3), where
la is the electric arc length. The light intensity is supposed as I0 = rs2 .
Camera: The lens focal length f is 50 mm. The CCD target sizes are 4.8×3.6
mm2, corresponding to the maximum image sizes 500×592;
Surface reflectance property: The parameters of weld pool surface are selected
as βd=0.3, βs=0.7 and k=3.0. While the workpiece and solidified weld surface are
thought as Lambertian surface βd=1.0, βs=0.0.
Fig.4.1. Visual sensing system for GTAW process

61
With the improved algorithm of shape from a single image and the coordinate
system transformation, the surface height of weld pool is calculated from a single
weld pool image, shown in Fig.4.3. The surface heights of weld pool are
reconstructed successfully. The results indicate that low wire feed speed cause weld
pool surface to depress and surface height to decrease, resulting in full penetration.
0.5
z, mm
0
-0.5 6
3
-1
0
3
m
-1 -3
m
x, m
y,
-5 -6
m
-9
(a) (a)
0.5
z, mm
-0.5 6
3
-1
0
3
-3
mm
-1
x, m -5
y,
m -6
-9
(b) (b)
Fig.4.2. Weld pool images with different Fig.4.3. Surface height results
surface shape during GTAW with wire reconstructed from single weld pool
filler: (a) local convex surface; (b) images: (a) local convex surface; (b)
concave surface. concave surface.
5 Conclusions
The problems of shape from shading with a single image are analyzed. An
improved algorithm is proposed incorporating smoothing constraint, boundary
condition, and grayness-weighted light error. Finally, the surface height of weld
pool is successfully reconstructed from its single image considering visual sensing
coordinate transformation. The surface shape of weld pool provides abundant

information for modeling and control of pulsed GTAW process.
Acknowledgements
This work is funded by National Science Foundation of China No.59575057 and

No. 59635160.
References
1. Horn BKP (1990) Height and gradient form shading. International Journal of Compute
Vision 5(1):37-75.
2. Lee KM, Kuo CJ (1997) Shape from shading with a generalized reflectance map model.
Computer Vision and Image Understanding 67(2): 143-160.
3. Kovacevic R, Zhang YM (1995) Vision sensing of 3D weld pool surface. 4th
International Conference on Trends in Welding Research, pp5-8.
4. Zhao DB, Chen SB, Wu L, Chen Q. Extraction of three-dimensional parameters for
weld pool surface in pulsed GTAW with wire filler. (To be published in ASME Journal
of Manufacturing Science and Engineering)
63
A Novel Intelligent Monitor for Manufacturing

Processes
M. Ge* and Y. Xu
Department of Automation & Computer-Aided Engineering, The Chinese

University of Hong Kong, Hong Kong
mingge@acae.cuhk.edu.hk
Abstract. The development of condition monitoring in continuous-time mass

production is a traditional but critical topic owing to its role in increasing cost-
effectiveness. A new method for intelligent manufacturing processes monitor was
presented using the newly developed learning approach, Support Vector
Regression, and related noise density model for efficient and effective modeling.
A new similarity measurement method was introduced resulting in a real-time
system developed. The system was applied for certain manufacturing process
condition monitoring. The experimental results showed that the proposed method
is substantially more effective than others. In addition, the new monitor requires
only normal condition samples for implementation, an attractive feature for shop
floor applications. The novel system ensures the automatic and intelligent
operation, and it has potential applications on the other manufacturing processes.
1 Introduction
It is reported that more than half of the manufacturing cost is attributed to the
production process; the demand of the cost reduction in production becomes curial
[1]
. Aiming to increase the economic benefit, the monitoring systems are mainly
designed to increase the productivity and quality of production, resulting in the
broad application of process monitors; however, it is difficult for some
manufacturing operations, i.e., sheet metal stamping, laser welding.
The detection of faults in manufacturing processes is of great practical
significance. Most manufacturing processes involve a great number of process
variables, many of which interact with one another. When any of these process
variables deviate beyond specified limits, faults occur. The traditional approaches
to fault detection, hence, are based on threshold (limit) checking. The more
advanced methods involve the spectral analysis of signals emanating from the
processes[2].
*
Please address the corresponds to this author at mingge@acae.cuhk.edu.hk
64 64 M. Ge and Y. Xu
Modern methods use decision-making relying on machine learning (ML), and

statistics[3], and they have improved the performance of monitoring systems.
However, the requirement of collecting sample sets under different conditions for
analysis often impedes their effectiveness in many real-world applications. It is
worth noting that although sensor and computer technology have been greatly
advanced, decision-making is still a subject of research.
The statistical approach to condition monitoring is concerned with the analysis
and interpretation of data, while ML deals primarily with learning rules or
structures. The two approaches are mutually beneficial, and hence they are often
used in combination. From a mathematical point of view, ML can be considered
as an estimate of a function f using n sets of training samples, resulting in the
construction of a classifier or an estimate. Ideally, f shall correctly describe both
the training samples and the unseen examples from the population. Traditionally,
classical ML methods are based on the Empirical Risk Minimization (ERM)
principle[4]. Unfortunately, tests have shown that ERM cannot ensure the
minimization of actual risk; moreover, several factors impact the complexity of
the learning machine. For example, the cases of overfitting and the local minimum
of Artificial Neural Networks (ANN) reveal the failure of the ERM principle. The
unreasonable design structure might also lead to failure[4].
Statistical learning theory can precisely identify the factors that need to be
taken into account for successful learning. Arising from statistical learning theory,
the support vector (SV) technique develops theory into a tool for not only
theoretical analysis but also creating practical algorithms for estimating
multidimensional functions[5]. The SV technique has recently gained popularity
for two reasons. First, it is satisfactory from a theoretical point of view: support
vector learning is based on simple ideas that allow us to learn intuitively from
examples. Second, it works well in many practical examples. For this reason
alone, the method has been applied to many real-world problems, such as pattern
identification, regression, and density estimation[6,7,8].
When the SV learning technique was applied to the function estimation, the
support vector regression (SVR) approach emerged. Many approaches have been
utilized to solve the function estimation. For example, radial basis function (RBF)
is typically used in many applications[9,10], and demonstrates relatively better
performance in regression estimation among the traditional neural network
techniques. Since the format of the RBF network is similar to the SVR, there have
been many comparisons in theoretical analysis and real-world application[7,11].
Evaluations show that SVR is the superior method because it is based on a unique
idea and methodology.
This paper presents a new condition monitoring system using the SVR
approach and a noise density model. The remainder of the paper is structured as
follows. Section 2 outlines the principles of SVR and introduces a new similarity
measurement method. Section 3 proposes the new intelligent monitor, and Section
4 describes the experimental design and how to apply the newly developed system
for monitoring the sheet metal stamping operation. Discussion of the experimental
results is presented in the Section 5. Section 6 concludes the paper.
65
2 Modeling with SVR and Similarity Measurement
2.1 The principle of SVR estimation
Although the SVR technique is relatively new, its basic theory is well understood
[12]
. Briefly, given a set of independent and identically distributed (iid) training
samples Sf = {(x1, y1), (x2, y2), …, (xn, yn)}, where x ∈ RN denotes the input
vector, and y ∈ R represents the output value, the goal is to find a function f that
correlates input and output:
N (2.1)
y = f (x, w ) = w, x + b = % wi xi + b ,
i =1
where w ∈ RN and b ∈ R are the weighting factors, *,* denotes the dot products
in RN, and f : RN → R. Introducing a nonlinear function ϕ(x), a more generalized
form is as follows:
N
(2.2)
y = f (x, w ) = % wiϕ ( xi ) + b = w , ϕ (x) + b .
i =1
The nonlinear function ϕ maps the input vector x onto a feature space Ζ, ϕ(x) ∈
Ζ Note here, w ∈ Ζ is in a different feature space to the one in Eq. (2.1). The SVR
method finds a function in the family F (x, w) = {f | f : RN → R} that minimizes
the risk function:
R(w ) = ' c (x, y, f (x, w )) | dp(x, y ) , (2.3)
where c(x, y, f(x, w)) denotes the loss function, which determines how estimation
error based on the training samples will be penalized; p(x,y) is the probability
distribution of (x,y). Vapnik designed the ε-insensitive loss function[4] as:
c(x, y, f (x, w )) = y − f (x, w ) ε = max(0, y − f (x, w ) − ε ) . (2.4)
Due to the lack of information on p(x, y), it is common to use the empirical risk
function estimated from the training sample set Sf, This results in the following
empirical risk function:
1 n (2.5)
Remp (w ) = % c(x, y, f (x, w )) .
n i =1
In addition, to overcome overfitting and enhance the generalization, a capacity
control term is often added leading to the following [13],
λ 2 1 n λ 2 (2.6)
Rreg (w ) = Remp (w ) +
2
w = %
n i =1
c(x, y, f (x, w )) + w ;
2
where λ > 0 is a regularization constant. The last term controls the flatness of the
function. In this context the use flatness denotes that one seeks small w. This has a
significant effect on the generalization capability of the algorithm.
Intuitively, the SVR maps the input vectors onto a high dimensional feature
space, and subsequently locates an optimal function f that not only reduces the risk
but also controls the generalization capability. It can be shown by use of the loss
function as Eq. (2.4) that solving a SVR is equivalent to solving the following
optimization problem:
1 n 1 2 n (2.7)
min $w, w& + C%(ξi + ξi*) = w + C%(ξi +ξi* )
w,b, , *
2 2
! !
i =1 i =1
s.t. yi −$w,ϕ(xi )& −b ≥ ε +ξi

$w,ϕ(xi )& +b − yi ≥ ε +ξi*
ξi, ξi* ≥ 0, i = 1,2,…, n.
where slack variables ξi,ξi* cope with additive noise, and ε represents the expected
level of precision. With the slack variables ξi and ξi* equal to zero, Eqs. (2.7) form
the so-called ε-tube. C, a function of (λn), in Eq. (2.7) is a parameter
corresponding to the value ||w||2. It provides a trade-off between two terms: the
flatness of f and the amount to which deviations larger than ε. According to Eq.
(2.7), a Lagrange function can be constructed as:
1 2 n n
L= w + C% (ξi + ξi* ) − %αi (ε + ξi − yi + w,ϕ (xi ) + b)
2 i =1 i , j =1 (2.8)
n n
− %αi* (ε + ξi* + yi − w,ϕ (xi ) − b) − % (ηiξi + ηi*ξi* ),
i , j =1 i =1
where αi, αi*, ηi, ηi* ≥ 0 are the Lagrange multipliers.

A more generalized form of SVR involves the use of a kernel function, K. A
kernel is a function, K, such that for all x, z ∈ Ω.
K (x, z ) = $ϕ (x) ⋅ ϕ (z )& . (2.9)
[14]
The kernel function must satisfy the Mercer condition .
'' K (x, z)ϕ (x)ϕ (z)dxdz > 0 , (2.10)

where K is a symmetric function, ϕ(x) is not always equal to zero, and
' ϕ (x)dx < ∞ . Replacing the w ⋅ ϕ (x ) with K and by the partial derivatives of
2
L with respect to the variables (w, b, ξi, ξi*), it can be shown that solving the
optimization problem in Eq. (2.8) is equivalent to solving the quadratic
optimization problem:
n
1 n n
(2.11)
min* − % (α i − α i* ) yi + % (α i − α i* )(α j − α *j ) K (xi , x j ) + ε % (α i + α i* )
α ,α i =1 2 i, j =1 i =1
67
n
s.t. % (α i − α i* ) = 0
i =1
0 ≤ αi,αi* < C, i = 1, 2, …, n.
This problem can be solved using various methods such as projected conjugate
gradients and Quasi-Newton methods. The quadratic optimization process
guarantees the solution must be the global minima. After the unique solutions α,
α* are found, the weight factor can then be represented as:
n
(2.12)
w = %(αi − αi* )ϕ (xi ) .
i =1
Note that, according to Karush-Kuhn-Tucker (KKT) complementarity conditions

[15]
, the product of the dual variables and constraints at the optimal solution must
vanish; that is
α i ( y i − (α i − α i* ) K (x i , x j ) − b − ε − ξ i ) = 0 (2.13)
.
α i* ((α i − α i* ) K (x i , x j ) + b − ε − ξ i* − y i ) = 0
Thus, a number of input vectors lies on the boundary of the ε-tube; and they are
referred to as the support vector (SV). Thus, according to the SVs, ξi,ξi* = 0, b can
be calculated as follows:
n
(2.14)
b = y i − % (α i − α i* ) K (x i , x j ) − ε for α i ,α i* ∈ (0, C ) .
j =1
Finally, the SVR can be constructed for real-valued function estimation as

f ( x, ,
! !
*
)= % (α i − α i* ) K (x i , x) + b . (2.15)
i∈SV
2.2 Similarity measurement
Let us examine Eq. (2.6) again. Given a set of training samples with additive
noise, Sf, the goal of SVR is to find a function f that has at most ε deviation from
the actually measured targets for all the training samples, and at the same time is
as flat as possible. To do this, the definition of the loss function c(x, y, f(x, w)) is
very important. Ideally, it should have a simple structure to avoid difficult
optimization problems. In addition, it should be general enough so that the data
can be fit.
Assuming that the set Sf was generated by f plus additive noise ξ with density
p(ξ),
y i = f (x i ) + ξ i . (2.16)
Then, the likelihood of an estimate Ff = {(x1, f(x1,w)), (x2, f(x2,w)), …, (xn,

f(xn,w))} based on Sf is
n n
P(F f | F) = ∏ P( f (x i , w ) | (x i , y i )) = ∏ P ( f (x i , w ) | y i )
i =1 i =1
. (2.17)
n n
= ∏ p ( y i − f (x i , w )) = ∏ p(ξ i )
i =1 i =1
To maximize the P(Ff | F) is equivalent to maximize logP(Ff | F):

n n n
log P(F f | F) = log ∏ p(ξ i ) = % log p(ξ i ) = % log p ( y i − f (x i , w )) . (2.18)
i =1 i =1 i =1
Therefore, in a maximum likelihood sense, by setting

c(x, y, f (x, w )) = − log p( y − f (x, w )) = − log p(ξ ) , (2.19)
optimal loss function can be achieved. Therefore, the desired density minimizes
the risk function described by Eq. (2.3) with the loss function described by Eq.
(2.19). Eq. (2.18) is equivalent to
n n (2.20)
log P (F f | F) = % log( p (ξ i )) = −% c(x i , y i , f (x i , w )) .
i =1 i =1
The importance of this result is twofold. On the one hand, it presents a principle
that enables the construction of a loss function. Once the noise density of the
system is defined, its related loss function can be obtained according to Eq. (2.19).
As in many practical applications, the standard Normal density model, N(0,1), is
used to describe noise.
1 1 (2.21)
p( y − f (x, w )) = p (ξ ) = exp(− ξ 2 ) .
2π 2
This results in the following loss function:

1 1 (2.22)
c(x, y, f (x, w )) = ( y − f (x, w )) 2 = ξ 2 .
2 2
On the other hand, the result provides a new similarity measurement method –
a theoretical standard by which it can be determined whether two signals are
generated in the same condition or not. Under normal conditions, the process
demonstrates unique characteristics that are reflected in its related signals. The
signals from abnormal conditions would exhibit deviations, referred to here as
noise. The predicted values emerge when substituting the unknown signal as input
into the regression modeling. With a certain noise density model, or the related
loss function, the unknown input vectors’ likelihood of estimating the function can
be calculated according to Eq. (2.20). As a result, it is possible to implement
condition monitoring by comparing the likelihood of an unknown signal with the
likelihood distribution of normal ones.
Employing the loss function defined in Eq. (2.22), the related SVR estimation
can be defined as
69
1 n
2 2 1 2 n
2 2 (2.23)
min* $w, w& + C%(ξi + ξi* ) = w + C%(ξi +ξi* )
w,b, , 2 2
! !
i =1 i =1
s.t. yi −$w,ϕ(xi )& −b ≥ ε +ξi
$w,ϕ(xi )& +b − yi ≥ ε +ξi*
ξi, ξi* ≥ 0, i = 1,2,…, n.
The problem can be derived using the method introduced in Section 2 and taking
into account that ξiξi* = 0 and, therefore, that the same relation αiαi* = 0 holds for
the corresponding Lagrange factors[12].
n n (2.24)
min − % (α i − α i* ) yi + ε % (α i + α i* )
i =1 i =1
1 n 1
+ % (α i − α i* )(α j − α *j )(K (x i , x j ) + δ i, j )
2 i, j =1 C
n
s.t. % (α i − α i* ) = 0
i =1
αi≥0, αi*≥0, i = 1, 2, …, n,
where δ is an n by n identity matrix. The precision error ε plays an important role
in the SVR estimation. Aiming to model the signal for condition monitoring, ε
was set to zero. This corresponds to a standard linear least squares regression with
a weight decay factor controlled by the parameter C. As C → ∞, the problem tends
to an unconstrained least squares, which is equivalent to leaving the dot product
matrix diagonal unchanged. According to the KKT complementarity condition, b
can be calculated using Eq. (2.25).
n (α i − α i* ) (2.25)
b = y i − % (α i − α i* ) K (x i , x j ) − ε − for α i , α i* > 0 .
j =1 C
The final function estimation shares the same expression as Eq. (2.15).
3 Novel Condition Monitoring System
To implement condition monitoring systems requires the measurement of typical

quantities and is related to process and quality data. In practice, the data are
preferred to employ the sensor signals with high signal-to-noise ratio (SNR).
Nevertheless, despite the methods employed, extraneous noise still remains
amongst normal signals. Some are related to measurements, but the process causes
others. Current thinking suggests that abnormal operations contribute to the
deviations in signal patterns, which can also be regarded as a kind of noise in
addition to normal signal. Thus, a state-of-the-art condition monitoring system can
be developed based on the SVR and noise density models employing the
architecture depicted in Figure 1.
Fig. 1. The architecture of the condition monitoring system

: off-line training path
: on-line monitoring path
It consists of two paths: off-line path is used in training and on-line path is used
for real-time condition monitoring. In off-line training mode, SVR is carried out
based on the available training sample sets in normal condition. Consequently, the
mean and variance of the likelihood values P will be calculated according to the
loss function. Finally, this enables us to calculate the threshold of P. In on-line
monitoring mode, by assessing the likelihood of real data, the system can detect
faults or otherwise and issue appropriate and efficient commands to the actuator.
The SVR-based process monitor demonstrates a number of interesting features.
First, it has an exceptional ability to estimate the functions of signals due to the
regression estimation in the kernel-induced feature space. By replacing the dot
product $ϕ(x),ϕ(xi)& with the kernel K(x, xi), inputs are mapped onto a high
dimensional feature space non-linearly. The dimension in the feature space is
determined by the kernel function. For example, if the kernel function is a 2nd
order polynomial for a d dimension vector, the dimension in the feature space is
d(d+3)/2. This effectively captures more information to depict the input. In
practice, various kernel functions can be employed as long as they comply with
the Mercer condition defined in Eq. (2.10).
Second, the solution for SVR is the globe optimal. The quadratic optimization
process guarantees the globe minima solution. Moreover, as shown in[15], there are
two parts associated with a learning machine designed according to ERM
principles: the empirical risk of the training samples Remp(w) and the confidence
interval Φ
R(w ) ≤ Remp (w ) + Φ (h ) , (2.26)
where h is called the Vapnik-Chervonenkis (VC) dimension and Φ is inverse

proportionate to the 1/h. The structural risk minimization (SRM) principle[13]
provides a new way of designing learning machines by endeavoring to locate a
hyperplane that can minimize the sum of the Remp(w) and Φ, and achieve the
71
goal of R(w) minimization. As opposed to the ERM approach commonly

employed in machine learning, the formulation of the SVR embodies the SRM
principle[15]; therefore, SVR not only constructs an optimal hyperplane that
displays a strong generalization capacity, but also overcomes overfitting and other
difficulties in the design structure of the learning machine.
Third, the system does not require complicated calculation during monitoring
and hence can operate in real-time. Although the dimensions of the feature space
increases, the construction of the optimal hyperplane does not require extra
computational load. The reason for this is the introduction of the kernel function.
By replacing the dot product with an appropriately chosen kernel function, it is
implicit that performing a non-linear mapping onto a high dimensional feature
space does not increase the number of tunable parameters. This means that the
kernel function computes the dot product of the feature vectors corresponding to
the two input vectors. As a result, the exceptional ability does not bring
computational complexity and load.
Table 1. The procedure for training the SVR-based condition monitoring system
Step 1. Obtain training sample sets in normal condition
S i = (x ij , yij ) nj=1
i = 1, 2, …, M; where Si ∈ S = {Si | i : number of training sample sets}
Step 2. (Optional) Signal processing for the de-noising.
Step 3. Obtain a sample set derived from S using statistics to train the system.
Step 4. Define a kernel function K(x, z); C could be determined by cross-validation.
Step 5. Define the loss function as Eq. (2.22).
Step 6. Apply quadratic optimization, Eq. (24), to find out αi,αi* (αi,αi* ≠ 0) → xi
(corresponding support vectors according to Eq. (2.13)) → acquiring b* (Eq.
(2.25)).
Step 7. xi, αi, αi*, # Eq. (2.15) → f(x, α, α*).
Step 8. for i = 1:M
Eq. (20) → log (Pi);
end.
Step 9. Log (Pi) → N(m,σ) → the threshold of log (Pi) = m - 3σ.
n
Step 10. Monitoring on-line: obtain the real-time data set, S new = ( x j , y j ) j =1 , and
substitute them into the system to calculate log (Pnew);
if log (Pnew) < m - 3σ,
then condition = normal;
otherwise condition = fault;
end.
Note: x → y: y can be acquired from x; X # y: substitutes x into y.
Finally, the system requires only the training sample sets produced under
normal operating conditions thus making it more feasible than other techniques.
Given a number of training sample sets, the SVR technique will calculate
dependency between input vectors and outputs; moreover, its loss function defined
by the noise density model helps compute thresholds of the likelihood
automatically. Thus, the system has the intelligence to cope with different
processes.
All of these features make the proposed process monitor very attractive for
real-world applications. In summary, the procedures for training the monitor and
carrying out the on-line monitoring are in Table 1.
In practice, three major factors may affect the performance of the system. First,
an appropriate kernel can make the system more efficient. The kernel function
reflects the geometric relationship between the new input vector and a specific
support vector. The Gram matrix K = {Ki,j}, where Ki,j = K(xi, xj), of the kernel is
positive definite, which contains all necessary information for the learning
algorithm. In condition monitoring, K is used to measure the similarities between
normal features and faulty features. In this paper, the exponential RBF kernel
function is employed.
Second, the size of the input vector may pose a computational limitation. In
high-speed stamping, a stroke is completed in a fraction of a second. Meanwhile,
SVR requires significant amount of computation load. In fact, to protect precise
moulds and ensure quality, real-time factors and the accuracy of detection must
have first propriety in process monitoring. Therefore, the size of input vectors
should be as small as possible.
Third, with regard to the signal type, it should be noted that an appropriate
signal with high SNR not only effectively represents physical characteristics, but
can also make the monitoring system more robust.
4 Experimental Study
4.1 Experimental design
To verify the performance, the new monitor has been applied on a complicated
process. Sheet metal stamping operation is one of the most commonly used
manufacturing processes which involves the design and lubrication of the mould,
the transient elastic and plastic deformation of the sheet metals, as well as the
static and dynamic behavior of the press[16]. Nearly 40 process variables could
affect the operation and the product. It is quite difficult in monitoring its
processes.
The experiments have been carried out on a C-frame double column press
(SEYI: SN1-25), a typical machine used in practice. The maximum stamping
force is 25 tons and the maximum speed is 110spm. It is reported that strain
signals are most commonly used in the case of monitoring stamping operations
[16], and rely on gathering meaningful information and robust physical properties.
Here a strain signal was employed, with the result that signal processing it was not
required. The raw signals are acquired from a strain sensor (Kistler 9232A)
73
mounted on the middle of the press frame. The signals were sampled using a NI
DAQ (PCI 4452), installed in an Intel Pentium III 550MHz PC. Sheet metal
stamping is a transient process in high speed, in order to capture the signal every
time at the same instance, a trigger mechanism, a proximity sensor installed near
the main shaft, was developed. Every time when the key way of the shaft rotates
passing the sensor, a pulse is generated and sent to computer to trigger the start of
the sampling.
Three experiments were conducted covering a range of situations. The first and
second experiments were conducted in a laboratory setting. In the third
experiment, data from real-world application were acquired using a JG21-14 press
(manufactured by Jieyang Machine Tool Works).
Parts produced were manufactured under normal conditions. When working
conditions changed, so-called faults developed and deviations occurred. Table 2
provides descriptions of conditions that occurred frequently in practice.
Table 2. Fault types occurring frequently during stamping process

Type Description
-- No detection: under normal conditions.
A Misfeed: workpiece is not aligned with the dies.
B Slug: slug(s) left on the surface of sheet metal.
C Too thick: workpiece thickness is thicker than normal.
D Too thin: workpiece thickness is thinner than normal.
E Material missing: the material is used up.
F Crack: occurred in the drawing operation due to bad lubrication, materials
or design.
G Double stroke: part left in upper module.
4.2 Experiment conduction
The first experiment incorporated a blanking operation with a simple mould. The
parts produced in the experiment are simple buckets used for mounting I/O boards
in desktop computers, whose material is SPCC and the blank dimension is
approximately 32 cm. The normal working condition is 85spm with workpiece
1.1mm. The signals in Figure 2 show that except the last two conditions, the other
signals are all similar in appearance.
The second experiment used a multi-station mould, and included the three
stamping categories. It produces the covers of the rear part of a small type motor.
The material is SPCE and the dimension of the parts is approximately 35mm x
6.6mm (diameter x height). Its normal working condition is 80spm with
workpiece 1.0mm. Due to its quite long multi-station mould and operation
procedure, two strain sensors were mounted on the left and right sides of the press
to detect faults. The strain signals from both sensors merged and provided us with
boosted information. Figure 3 presents its typical strain under different conditions.
Fig. 2. Typical strain signals under different stamping conditions in the first
experiment
Fig. 3. Typical strain signals under different stamping conditions in the second
experiment
The third experiment also used a multi-station mould; its products are the
battery caps. The material is copper and the approximate dimension of the part is
12mm x 4.3mm (diameter x height), designed to work under 150spm with
workpiece 0.7mm. It is noted the normal stamping force of the press is 140KN.
There are numbers of presses working in the same area, which may cause
disruption to the process, and the press is timeworn. Figure 4 consists of
illustration strain signals under different conditions in the experiment.
Material specifications and the design of the mould caused the battery caps to
break easily. Moreover, due to the age and condition of the press, along with
adverse working conditions, the SNR of the strain signals in the third experiment
was low. As shown in Figure 4, only serious cracks were detected by traditional
condition monitoring systems, i.e., peak value, RMS, kurtosis and skewness. In all
75
cases, operators are informed of the results too late to ensure reductions in the
number of faulty parts manufactured. No obvious phenomenon can be detected to
suggest that cracks will presently occur, and it is almost impossible to diagnose
these faults.
Fig. 4. Typical strain signal under normal condition in the third experiment
It was for just these circumstances that a condition monitoring system based on
SVR was developed. In order to develop the system, three major factors need to be
determined: the value of ε, the type of kernel function, and the training sample
sets. In section 3, ε was set at 0, and the exponential RBF kernel function was
chosen; meanwhile, its σ was set at 0.4
As to the third factor, training sample sets, a range of data was acquired from
signal(s) according to the scan rate within each stroke. To eliminate meaningless
data, and to ensure effective coverage of the entire stroke, specific points were
drawn. In the first experiment, 512 points were selected for each stroke. In the
second and third experiments, 1024 points and 640 points were selected
respectively. Further, because the dimension of the input vector has an obvious
impact on the computational load of the system, the number of points was reduced
on the basis of maintaining crucial features of the processes whilst still aiming to
guarantee the speed of the system. To this end, one in two points were chosen for
each stroke from the selected data in the first experiment, and one in four from
both the second and third experiments. In the SVR model, the regression step is T
= 10. Hence, the model is as follows:
y i = f (x i ) + ξ i = f ( xi −1 , xi − 2 ,..., xi −T ) + ξ i , i =1, 2, 3, …, n. (2.27)
where, x is the point of the strain signal. A total of 130 sample sets were collected
under normal and faulty conditions in each experiment. 30 training sample sets
produced under normal conditions and selected randomly act as training sample
sets for each experiment. The algorithm for training SVR is Sequential Minimal
Optimization (SMO), which has been well defined by Platt[15]. The algorithm
ensures the solution of quadratic programming optimization problem satisfies the
time requirement of the system.
4.3 Experiment results
Three experiments were carried out in progression according to the procedure

defined above. The distributions of the likelihood for the three experiments are
illustrated in Figure 5 (a), (b) and (c), respectively, where the solid line is the
mean value of the likelihood, the broken line is the threshold, * represents the
normal samples, and “•”represent abnormal samples
5 Discussion of Results
In summary, the results of the three experiments are represented in Table 3. In the
first experiment, the number of the sample sets produced under normal conditions
numbered 60, and the remaining 70 sets were the result of misfeeds, slugs, and
material either too thick or too thin. The second experiment consisted of 90
normal condition sample sets, with 40 sets resulting from misfeeds, slugs and
double strokes. In the third experiment, suspending the flow of lubricating oil
created faulty condition sample sets. Parts produced under such conditions are
faulty and must to be discarded; however, there is no clear information with regard
to strain signals.
Table 3: Experimental results using an SVR-based system

Experiment Training sample Testing sample set Success
set Normal sets Abnormal sets rate
1st 30 30 70 99%
2nd 30 60 40 94%
3rd 30 70 30 89%
Figures 5 (a), (b) and (c) show that the likelihood of the samples under normal
conditions is quite centralized in the first and second experiments, and that
samples in abnormal conditions have a wide range of distribution. Due to the poor
SNR in the third experiment, signal patterns for different working conditions are
similar, which was reflected in the distribution of likelihood. However, the
experimental results show that the system has robust performance. The approach
extracts the invariant features from the strain signal for normal conditions.
Moreover, to compare the new approach, the authors also employed the
backpropagation (BP) approach, a traditional ANN technique; a Hidden Markov
Model approach, another modeling technique; and a support vector machine
(SVM) approach. In each experiment the same sample was used. Table 4 presents
the results[16, 17].
77
Fig. 5. Experimental Results

(a) Top left: results for Experiment 1
(b) Top left: results for Experiment 2
(c) Bottom: results for Experiment 3
Table 4: Experimental results for fault classification using other approaches

Source of training sample Approach Success rate
Samples in Experiment 1 BP network (two layers) 93%
Samples in Experiment 1 Hidden Markov Model 92%
Samples in Experiment 1 Support Vector Machine 99%
Samples in Experiment 2 BP network (two layers) 84%
Samples in Experiment 3 BP network (two layers) Failed (<50%)
In comparison, the SVM performed perfectly. The SVM and SVR techniques
share a common basic technical background. The former requires samples from all
conditions for training the system, which results in the acquisition of information
richer than the latter; however, unstable signals in the third experiment make it
difficult to choose the training samples. However, by modeling for normal
condition signals and considering the characteristics of noise density, the proposed
system shows a significant improvement in the third experiment. More appealing
though, the proposed system only requires samples from normal operating
conditions for training, which makes the approach more ideal for real-world
application.
6 Conclusions
A new intelligent manufacturing process monitor was developed based on the

SVR and noise density model. Based on the experimental results, the following
conclusions can be drawn:
(a) Derived from statistical learning theory, SVR realizes the SRM principle in its
design. Unlike other learning algorithms, it overcomes the drawbacks of local
minima and overfitting, and it has a greater capacity for generalization. This
makes the system intelligent when dealing with complicated processes.
(b) SVR nonlinearly maps input vectors onto a higher dimensional feature space
via simple kernel, where a linear regression is constructed so that accuracy is
increased significantly. Simultaneously, kernel overcomes the problems of
dimensionality in both computation and generalization. This guarantees the
system is able carry out tasks accurately.
(c) The perfect characteristics of sample sets derived under normal condition for
training, and the robust performance within selection parameters, makes the
condition monitoring system more practicable.
(d) Together with the appropriate noise density model, the new similarity
measurement according to the system could reveal the characteristics of the
manufacturing process in different conditions.
(e) A condition monitoring system using SVR and related density noise model
has the potential for application in other manufacturing processes.
Acknowledgement
The partial support of this work by the Innovation and Technology Commission of
Hong Kong, under grant ITF AF/79/99, is gratefully acknowledged.
References
1. S. G. Tzafestas Advances in Manufacturing: Decision, Control and

Information Technology. London: Springer, 1999.
79
2. J. J. Gertler, Fault Detection and Diagnosis in Engineering Systems. New

York: Marcel Dekker, 1998.
3. W. G. Fenton, T. M. McGinnity, and L. P. Maguire, “Fault diagnosis of
electronic systems using intelligent techniques: A review,” IEEE Trans.
Syst., Man, Cybern. C, vol. 31, no.3, pp. 269-279, 2001.
4. V. N. Vapnik, The Nature of Statistical Learning Theory. New York:
Springer Verlag, 1995.
5. V. N. Vapnik, “An Overview of Statistical Learning Theory,” IEEE Trans.
Neural Networks, vol. 10, no.5, pp. 988–1000, 1999.
6. S. Gutta, J. R. J. Huang, P. Jonathon, H. Wechsler, and L. R. Rabiner,
“Mixture of Experts for Classification of Gender, Ethnic Origin, and Pose of
Human Faces,” IEEE Trans. Neural Networks, vol. 11, pp. 948–960, 2000.
7. T. B. Trafalis and H. Ince, “Support Vector Machine for Regression and
Applications to Financial Forecasting, IJCNN 2000,” Proceedings of the
IEEE-INNS-ENNS International Joint Conference on, vol. 6, pp. 348–353,
2000.
8. D. J. Sebald and J. A. Bucklew, “Support Vector Machine Techniques for
Nonlinear Equalization,” IEEE Trans. Signal Processing, vol. 48, pp. 3217–
3226, 2000.
9. J. A. K. Suykens and J. Vandewalle, Nonlinear Modeling: Advanced Black-
box Techniques. Boston: Kluwer Academic Publishers, 1998.
10. R. J. Schilling, J. J. Carroll, and A. F. Al-Ajlouni, “Approximation of
Nonlinear Systems with Radial Basis Function Neural Networks,” IEEE
Trans. Neural Networks, vol. 12, pp. 1-15, Jan. 2001.
11. K.-R. Müller, A. J. Smola, G. Rätsch, B. Schölkopf, J. Kohlmorgen, and V.
N. Vapnik, “Predicting Time Series with Support Vector Machines,” in
Advances in Kernel Methods - Support Vector Learning, B. Schölkopf,
C.J.C. Burges, and A.J. Smola, Eds. Cambridge, MA: MIT Press, 1999, pp.
243–254.
12. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector
Machines: And Other Kernel-based Learning Methods. Cambridge, UK:
Cambridge University Press, 2000.
13. V. N. Vapnik, Estimation of Dependencies Based on Empirical Data. Berlin:
Springer Verlag, 1982.
14. J. Mercer, “Function of Positive and Negative Type and Their Connection
with Theory of Integral Equations,” Philos. Trans. Roy. Soc. A 209, pp.
415–446, 1909.
15. B. Schölkopf, C. J. C. Burges, A. J. Smola, Eds. Advances in Kernel
Methods: Support Vector Learning. Cambridge, MA: MIT Press, 1999.
16. M. Ge, G. C. Zhang, R. Du, and Y. Xu, “Application of Support Vector
Machine Based Fault Diagnosis,” in Proceedings of XV IFAC World
Congress, Barcelona, Spain, 2002.
17. M. Ge, R. Du, and Y. Xu, “Condition Monitoring Using Marginal Energy
and Hidden Markov Model,” to be appeared in Journal of Control and
Intelligent Systems.
3-D Grasp Analysis and Synthesis
Using the Ray-Shooting Technique
Yun-Hui Liu, Dan Ding and Miu-Ling Lam
Department of Automation and Computer Aided Engineering, Chinese University

of Hong Kong, Shatin, NT, Hong Kong
yhliu@acae.cuhk.edu.hk
Abstract. The ray-shooting technique is a powerful tool for handling geometric

information in Computational Geometry and Computer Graphics. This chapter
demonstrates that this technique can be effectively applied to stability analysis and
planning of 3-D form-closure grasps in continuous and discrete domains. In form-
closure grasps, any external wrench applied at the grasped object can be always
balanced by the grasping forces at the contact points. A ray-shooting based formu-
lation, which leads to several simple and efficient algorithms, is presented for
qualitative test and computation of 3-D form-closure grasps as well as for auto-
matic fixture layout design of workpieces.
1 Introduction
Multi-fingered robotic hands have been extensively studied in robotics and auto-
mation since the late 1980s for their great potential in performing dexterous and
fine manipulation tasks that are carried out by human hands. To develop a robot
hand that is as skillful as human hands, it is necessary to solve various technical
issues including mechanical design [3], [25], [27], kinematics [9], [16], grasping
analysis and control [6], [10], [21], [26], dexterous manipulation [33], etc. Grasp
analysis and planning addressed here are problems of analyzing and judging sta-
bility of a grasp and finding a stable grasp on a given object, being the most fun-
damental issues in the field. The lack of real-time solutions to those problems has
been widely recognized one of major factors that obstruct wide applications of
multi-fingered robot hands in industry and other related sectors.
The stability of a grasp is characterized by form-closure and force-closure
properties [1], [25]. Under a form-closure grasp, any external wrench applied at
the grasped object can be balanced by grasping forces of the robot hand. The
difference of force-closure from form-closure lies in that kinematics of robot
hands is taken into account in force-closure analysis [1], while the form-closure
means geometric property only. Many research efforts have been directed to
testing form-closure property of a given grasp in last two decades. Salisbury and
Roth [25] have shown that a necessary and sufficient condition for form-closure
grasp is that the primitive contact wrenches resulted by contact forces at the
81
the primitive contact wrenches resulted by contact forces at the contact points
positively span the entire wrench space. This condition is further proven to be
equivalent to that the origin of the wrench space lies strictly inside the convex hull
of the primitive contact wrenches[15], [18]. Nguyen [19] proposed a simple test
algorithm for 2-finger form-closure grasps. Ferrari and Canny [5] use the largest
sphere inside the convex hull for the qualitative test and optimization algorithm
for 2D grasps. Trinkle [26] provided a formulation of quantitative test using
Linear Programming. In [7], Han et. al. formulized the problem as a linear matrix
inequality problem. Liu [10] developed a qualitative test algorithm of form-
closure grasps by transforming the problem to a ray-shooting problem of a convex
hull.
Another important issue concerning grasp stability is the sufficient and neces-
sary conditions for the number of fingers needed to achieve form-closure grasp.
Reulaux [23] has shown that in frictionless case at least 4 and 7 fingers are re-
quired to achieve 2-D and 3-D form-closure grasps, respectively. Mishra et al.
[15] found that 6 (resp. 12) fingers are sufficient for 2-D (resp. 3-D) frictionless
form-closure grasps. Markenscoff et al. [14] further tightened the results by
proving that 4 and 7 fingers are sufficient to achieve 2-D and 3-D form-closure
grasps, respectively. When Coulomb friction is taken into account, 2 and 3 fingers
are, respectively, sufficient in 2-D and 3-D cases.
Grasp synthesis is another important research aspect, concerning the problem of
placing contacts on an object with given geometry so as to prevent object motions.
In [20], Nguyen presented a geometric approach to find maximal contact regions
where two fingers can be positioned independently while maintaining a stable grip
on a polygon. Liu [11] proposed an algorithm for calculating all planar form-
closure grasps of n fingers on polygonal objects based on a new sufficient and
necessary condition for form-closure. Ponce et al. [21] presented an approach in
computing 3-finger stable grasps on planar polygonal objects with a projective al-
gorithm based on linear programming and variable elimination among linear con-
straints. In [22], they further extended the approach to 3-D 4-finger grasps. The
authors [4], [8] have recently developed an algorithm for computing form-closure
grasps by iteratively moving the convex hull of the primitive contact wrenches
towards the origin of the wrench space. However, the algorithm is not complete as
the search may trap at a local minimum point of the heuristic function.
An importance application of grasping analysis and synthesis is automatic fix-
ture layout design, which is a problem of finding a set of fixturing points called
fixels firmly holding a workpiece. Since the fixturing points must form a form-
closure so as to reject any motion tendency caused by forces generated during a
fabrication process, automatic fixture design is essentially the same as the synthe-
sis problem of form-closure grasps. Wolter and Trinkle [31] proposed an algo-
rithm for selecting discrete fixture points for polygonal/polyhedral parts to achieve
stability against the gravity forces. The stability considered here is different from
the form-closure. Form-closure imposes more and stricter constraints so that the
difficulty of the problem is increased. Their algorithm does not yield the minimum
number of fixturing points and the number of contacts cannot be specified before-
hand. A complete algorithm has been proposed by Brost and Goldberg [24] for de-
82 82 Y.-H. Liu, D. Ding, and M.-L. Lam
signing planar fixtures using modular components in a discrete domain. Wallack

and Canny studied a similar problem in [28], [29]. However, those algorithms can
be hardly applied to 3-D grasp planning problems. Wang [30] presented an algo-
rithm for optimal design for 3-D fixture synthesis in a point-set domain, but the
algorithm is not complete. It should be noted that the works mentioned above did
not take frictions into account. Mattikalli et. al. [13] proposed an algorithm for
finding all stable orientation of assemblies with frictions under application of
gravitational force, but the algorithm is not suitable for computation of form-
closure grasps.
In this chapter, we present a new framework using the ray-shooting technique
for stability analysis and planning of 3-D form-closure grasps. The ray-shooting
technique is a powerful tool for extracting geometric information in Computa-
tional Geometry and Computer Graphics. Based on the duality between convex
hulls and convex polytopes, a ray-shooting problem can be transformed to a Lin-
ear Programming (LP) problem [17], which can be solved in real-time with ad-
vanced computer technology when the size of the problem is not huge. It will be
demonstrated that the qualitative test and computation of a form-closure grasp,
and automatic fixture layout design of workpieces can all be formulized as ray-
shooting problems. In detail, the qualitative test can be transformed to a problem
of checking the intersection of the convex hull of the primitive contact wrenches
with a ray from an interior point of the convex hull to the origin of the wrench
space. This transformation leads to the development of an efficient algorithm for
qualitative test of form-closure grasps. On the basis of this qualitative test algo-
rithm, a simple and efficient heuristic method can be developed for searching for a
form-closure grasp on the surface of a 3-D object. The heuristic search is effec-
tively guided by the distance between the convex hull and the origin of the wrench
space, which is obtained in a ray-shooting process. To overcome the local mini-
mum problem in the heuristic search, a recursive problem decomposition strategy
is introduced. The combination of the heuristic search with the recursive problem
decomposition strategy results in a complete and efficient algorithm for finding a
form-closure grasp in a discrete domain. Compared to other heuristic methods,
this algorithm can always find a solution or report no solution. Compared to the
exhaustive search, this algorithm is more efficient. Finally, this chapter addresses
automatic fixture layout design on polyhedral workpieces and demonstrates that
this fixture design problem can be also transformed to a ray-shooting problem. To
demonstrate the effectiveness of this new formulation and efficiency of the algo-
rithms, several numerical examples are presented in this chapter.
This chapter is organized as follows. Section II reviews the frictional and fric-
tionless grasp models and the definition of form-closure grasps. Section III pre-
sents the ray-shooting based qualitative test algorithm. In Section IV, we present
an efficient local search algorithm and a complete algorithm for 3-D grasp plan-
ning. Section V demonstrates the application of the ray-shooting based formuliza-
tion in automatic fixture layout design of workpieces. Finally, we conclude this
chapter in Section VI.
83
2 Grasp Model and Form-Closure
This section reviews the grasp models used in this chapter and the definition of
form-closure grasps.
2.1 Assumptions
The work presented here is based on the following assumptions on robot fingers
and contacts among the fingers and the object:
1) The fingers and the grasped object are rigid bodies.
2) All contacts are point-to-point contacts, i.e. the fingers are assumed to be
hard fingers.
3) The model of the object to be grasped is available. The object is repre-
sented by either a continuous surface or discrete points. The
corresponding surface normal of any point on the surface is well-defined
and obtainable.
Robot finger
Object frame ri
Fig. 1 The coordinate frame attached to the object

It should be noted that 3) in the assumptions only assumes the availability of the
model of the 3-D object. No assumption has been made on the geometric shape of
the object. The object under consideration could be a polyhedron or one with
complicated surface. Denote the contact point of a finger on the surface by rj = (xj,
yj, zj)T, j = 1, 2, …, n, where n is the number of fingers. The contact points rj are
defined with respect to the object coordinate frame with origin at the center of
mass (Fig. 1). Denote the surface normal of the object at point rj by nj.
2.2 Frictionless Grasps
It is well known that seven frictionless contacts are necessary [15] to hold a 3-D
object in form-closure and sufficient [14] for a 3-D object without rotational
symmetries. We assume that seven hard fingers are to grasp a 3-D rigid body
without rotational geometries in the absence of friction at the contact points. We
denote a grasp by G = { ri }, i = 1, 2, …, 7, where ri denotes the contact point on
the surface of the object.
In the absence of friction, grasping forces fi are in the normal directions of the
surface of the object at the contact points. The force and moment τi, corresponding
to grasping force fi, applied at the center of mass of the object is given by
§ fi · § fi · § ni · (2.1)
¨¨ ¸¸ = ¨¨ ¸¸ = α i ¨¨ ¸
© τ i ¹ © ri × f i ¹ ©ri × n i ¸¹

wi
where αi is a non-negative constant representing the magnitude of the grasping

force. The combination of the force and moment is called wrench. The vector wi is
called the primitive contact wrench. W=(w1, w2, …, w7) ∈ R6x7 is the wrench
matrix of a 7-finger frictionless grasp.
Fig. 2 Linearization of a friction cone
2.3 Frictional Grasps
We consider multi-fingered grasps with Coulomb friction. It is known that three

fingers are sufficient to hold an object in 3-D frictional cases. Suppose that n ≥ 3
hard fingers are to grasp a rigid object in a 3-D workspace. Assume that the Cou-
lomb friction with friction coefficient µ exists at the contact points. To ensure
non-slipping at the contact point, the grasp force fi must satisfy
f ix2 + f iy2 ≤ µ 2 f iz2 (2.2)
where (fix, fiy, fiz) denotes x, y and z components of the grasp force fi w.r.t. the con-
tact frame located at the contact point on the surface of the object. The nonlinear
85
constraints in (2.1) geometrically define a cone called friction cone (Fig. 2). The
grasp forces can be exerted in any direction within the friction cone. To simplify
the problem, the friction cone is linearized by a polyhedral convex cone with m
sides. Under this approximation, the grasp force fi can be represented as
m
fi = ¦α
j =1
ij s ij , α ij ≥ 0 (2.3)
where sij represents the j-th edge vector of the polyhedral convex cone for the
grasp force fi. Coefficients αij are nonnegative constants. The wrench, correspond-
ing to the grasp force fi, applied at the center of mass of the object is given by
§ Rf i ·
w i = ¨¨ ¸¸ (2.4)
© ri × ( Rf i ) ¹
where R denotes the rotation matrix of the contact frame with respect to the object
frame. Substituting (2.3) into (2.4) derives
m
wi = ¦α
j =1
ij w ij (2.5)
where
§ Rs ij ·
w ij = ¨¨ ¸
¸
(2.6)
© ri × ( Rs ij ) ¹
Vectors wij are called primitive contact wrenches. The net wrench applied at the
object by the n fingers is
n m
w net = ¦ ¦α
i =1 j =1
ij w ij = W α (2.7)
where W and α are given by

W = (w11 , w12 , " , w1m , " , w n1 , w n 2 , " , w nm )
α = (α 11 , α 12 , " , α 1m , " , α n1 , α n 2 , " , α nm ) (2.8)
W is a 6 × nm wrench matrix and its column vectors are the primitive contact
wrenches. For convenience, wi with a single subscript i, instead of wij, is used to
denote the i-th column vector of grasp matrix W, and αi is used to represent the i-
th component of vector α.
2.4 Form-Closure Grasps
Form-closure is a measure of the capability of a grasp to completely constrain mo-

tions of the grasped object through the contacts under the application of external
forces and moments. This century old concept is defined as follows:
Definition 1: Form-closure grasp: Suppose that a n-finger frictional/frictionless

grasp is given. If any external wrench wext applied at the object can be always bal-
anced by the grasp forces fi of the fingers, the grasp is said to be form-closure.
It should be noted that the form-closure grasp in Definition 1 is called force-

closure grasp in [9], [19], [21]. We here follow the definition given in [1], [25],
[26]. Form-closure can be also defined as follows using wrench matrix W:
Definition 2: Form-closure grasp: W is the wrench matrix of an n-finger

frictionless or frictional grasp. For any external wrench wext ∈ R6 applied at the
object, if it is always possible to find an α with all αi ≥ 0 such that
W α + w ext = 0 (2.9)
the grasp is said to be form-closure.
It is well-known that a form-closure grasp is equivalent to that the origin of the

wrench space R6 lies exactly inside the convex hull H(W) of the primitive contact
wrenches wi. Any point P strictly inside the convex hull H(W) can be represented
by
N N
P= ¦
i =1
α i wi , ¦α
i =1
i = 1, αi ≥ 0 (2.10)
Since in form-closure grasps the origin of the wrench space is strictly contained by
the convex hull of the primitive contact wrenches, there exist coefficients αi such
that
N N
¦ α i wi = 0
i =1
¦α
i =1
i =1 all α i > 0 (2.11)
Note that in eq. (2.11) the coefficients αi cannot be equal to zero so as to guaran-
tee that the origin is not on the facets (boundary) of the convex hull.
3 Qualitative Test of Form-Closure Grasps
Qualitative test is a problem of checking whether a grasp is form-closure. The

problem definition is given as follows:
Problem 1: Given the contact positions of the n fingers on the surface of the
grasped object and their corresponding surface normal vectors, check whether the
given grasp forms a form-closure.
One possible method for the qualitative test is to construct the convex hull of the
primitive contact wrenches. However, construction of a convex hull in six dimen-
sional wrench space is not a trivial task according to the results in Computational
Geometry. For sake of computational efficiency, it is necessary to develop an ap-
87
proach without constructing the convex hull. In this section, a simple and efficient
qualitative check algorithm is presented based on the following observation:
Theorem 1: Suppose that an n-finger frictional/frictionless grasp is given. Denote

the convex hull of the primitive contact wrenches by H(W). Assume that point P is
an interior point of H(W). The ray from point P to the origin O of the wrench
space intersects the convex hull H(W) in a point Q only. The grasp is form-
closure if and only if the distance between points P and Q is strictly larger than the
distance between points P and O.
Proof: For a convex hull H(W), the definition of its interior point P means that
there always exists a r such that the spherical neighborhood region of point
P: B( P, r ) = {x : x − P < r } lies exactly inside the convex hull. Since H(W) is a
convex region, any ray emanating from its interior point P intersects H(W) only
in a point Q on the boundary. Denote the intersection of H(W) with the ray PO by
Q. The point Q lies on a facet of the convex hull H(W), i.e., on the boundary.
Any point on segment PQ except for the endpoint Q is an interior point of the
convex hull. If the distance between point P and the origin O is strictly smaller
than that between points P and Q, the origin O must be an interior point. There-
fore, there always exists a spherical neighborhood B(O, r) of the origin O that is
strictly contained by H(W).
This theorem is illustrated in Fig. 3. In Fig. 3(a), the origin O is on the segment
PQ and hence the grasp is form-closure. In Fig. 3(b), the origin O does not lie on
segment PQ and thus the grasp is not a form-closure grasp. It should be empha-
sized that one interior point P is sufficient for the check. The point P cannot lie on
the boundary of H(W). Obviously, a strictly positive combination of the primitive
contact wrenches is an interior point of the convex hull H(W). Therefore, we can
choose points P as the centroid of the primitive contact wrenches:
N
1
P=
N
¦w
i =1
i
(2.12)
where N denotes the number of the primitive contact wrenches. To locate the
intersection point Q of H(W) with the ray from the interior point P to the origin O
of the wrench space R6, we first detect the facet E of H(W) intersected by the ray
PO, and then calculate the intersection of the facet with the ray PO . The problem
of detecting the facet is closely related to the ray-shooting problem of a convex
hull:
Definition 3: Ray-shooting problem: Let M be a given set of points in Rd. Assume

that the convex hull H(M) contains the origin. Given a ray emanating from the
origin, find the facet of H(M) intersected by this ray.
It is well-known in Computational Geometry that the ray-shooting problem can be

transformed to a linear programming problem based on the duality between con-
vex hull and convex polytope. Assume that the convex hull H(M) contains the
origin. According to well-known results in Computational Geometry [17], the

convex hull H(M) can be dually transformed to a convex polytope CT(M) de-
fined by
Fig. 3 A form-closure grasp (a) and a non-form-closure grasp (b)
vT x ≤ 1 ∀v ∈ M
and vice versa by virtue of a dual transformation T that maps a point
vT=(v1,v2,...,vd) in Rd to the hyperplane:
v1 x1 + v 2 x 2 + • • • + v d x d = 1 (2.13)
This mapping is illustrated in Fig. 4. By the transformation T, a point v of the
convex hull H(M) is transformed to a hyperplane T(v). The transformation T is
reversible and has the following important properties:
• A vertex of the convex hull H(M) is transformed to a facet of the convex
polytope CT(M) and vice versa. For instance, in Fig. 4 the vertex D is
transformed to the facet T(D).
• A facet of the convex hull H(M) is transformed to a vertex of the con-
vex polytope CT(M) and vice versa. For example, in Fig. 4 the facet DC
of the convex hull is transformed to the vertex corresponding to the inter-
section of the facets T(D) and T(C).
• Points exactly inside the convex hull H(M) are transformed to redundant
hyperplanes of the convex polytope CT(M) and vice versa. For example,
in Fig. 4 the interior point G of the convex hull is transformed to the re-
dundant hyperplane T(G).
Note that a ray-shooting problem assumes that the convex hull contains the origin.
By applying a coordinate translation -P of on all points in R6, we readily change
the origin point to point P, which lies exactly inside the convex hull. After the co-
ordinate translation, the convex hull H(W) is dual to the convex polytope defined
by
89
T(G)
C
Convex hull H(M)

D

T(B)
G

B T(C)

T(D)

Convex polytope CT(M)

T(A)
Fig. 4 The duality between a convex hull and a convex polytope.
(w i − P) x ≤ 1
T
i = 1, 2 , " , N . (2.14)
Denote the direction vector of the ray PO by t. Based on the duality between con-
vex hull and convex polytope, the ray-shooting problem is equivalent to a problem
of maximizing the objective function:
z = tT x (2.15)
subject to the constraints in (2.14). Suppose that the optimal solution of the linear
programming problem is eG = (e1 , e 2 , " , e 6 )T . According to the duality, the facet
E of H(W) intersected by the ray PO is
e1 x 1 + e 2 x 2 + " + e 6 x 6 = 1 (2.16)
Then, the intersection point Q of H(W) with the ray PO is the intersection of the
hyperplane defined by (2.16) with the ray PO.
To summarize the discussions above, the qualitative test of a form-closure grasp
can be performed by the following algorithm:
Algorithm 1
1) Calculate all the primitive contact wrenches wi corresponding to the

given grasp.
2) Calculate the interior point P using eq. (2.12).
3) By the linear programming, calculate the optimal point eT=(e1,e2,...,e6)
that maximizes function z in eq. (2.15) subject to the constraints in eq.
(14). The hyperplane T(e) corresponding to the optimal solution:

eT x = 1
is the facet of H(W) intersected by the ray PO.
4) Calculate the intersection Q of the ray PO with the hyperplane T(e). The
point Q is the intersection of H(W) with the ray PO.
5) If the distance between points P and Q is larger than that between points
P and O, the grasp is form-closure; otherwise, it is not.
6) The algorithm ends.
Fig. 5(b) plots computation time of the algorithm with respect to the number m of
segments linearizing a friction cone for a 4-finger frictional grasps on a rectangle
box shown in Fig. 5(a). Note that the time is obtained on a Sun Ultra I170 using C
programming language. The grasp is not form-closure when the frictional coeffi-
cient is 0.3; it becomes a form-closure grasp when the frictional coefficient is in-
creased to 0.5. From the results, the qualitative check is performed efficiently.
For example, when m is equal to 100, which implies a reasonably accurate lineari-
zation of the friction cone, the computation time was 0.45 seconds when the fric-
tional coefficient is 0.5.
Finger 3 12000
10000
Friction coefficient=0.3
8000
Finger 4
Time (ms)
6000
4000
Finger 2
Friction coefficient=0.5
2000
0
Finger 1 0 50 100 150 200
m
250 300 350 400
(a) (b)
Fig. 5 A 4-finger grasp (a) and the results of its qualitative test (b)
4 Grasp Synthesis in Discrete Domain
Compared to the qualitative test, a more challenging and important problem is

how to find a form-closure grasp on a given object. This problem cannot be
avoided in motion planning of dexterous tasks. In this section, we address the
grasp synthesis problem in a discrete domain, i.e. we will solve the problem of
finding an n-finger form-closure grasp on an object represented by a set of discrete
91
points. It is assumed that the number of discrete points is big enough to yield
specified accuracy of approximation. Following is the problem defined:
Problem 2: Given a set Ω of surface points representing a 3-D object of arbitrary

geometry and their corresponding surface normals, find an n-finger frictionless or
frictional form-closure grasp in the point set Ω.
A simple, efficient and complete algorithm will be presented for solving Prob-
lem 2 based on the ray-shooting technique. To better explain the algorithm, the
description will be focused on the problem of finding a 7-finger frictionless grasp.
The extension of the algorithm to frictional grasps is discussed in the last subsec-
tion of this section.
4.1 Algorithm Outline
The basic idea of the algorithm to be presented is to combine a local heuristic

search with a recursive problem decomposition strategy. First, randomly select
seven points from the points set and check whether the selected points form a
form-closure. If the selected points achieve a form-closure grasp, the algorithm
finishes; otherwise, a local heuristic search is executed until a form-closure grasp
is found or a local minimum is encountered. The local search algorithm searches
for a form-closure grasp in the direction of reducing the distance between the con-
vex hull of the primitive contact wrenches and the origin of the wrench space. In
case the distance reaches a local minimum during the search process, the point set
is divided into subsets by a separating hyperplane in the wrench space and a few
sub-problems defined in the subsets, which have less points than the original set,
are generated according to existing conditions of form-closure grasps. To effec-
tively handle the process, we adopt a search tree whose root represents the original
problem. The sub-problems are represented as child nodes of the root. This proc-
ess is recursively carried out on every node of the search tree until a form-closure
grasp is found or all the nodes have been processed.
An inevitable step in the proposed algorithm is qualitative test of a form-
closure grasp, for which the ray-shooting based algorithm presented in the previ-
ous section is used. In addition to the qualitative test, an important property is
that the qualitative test algorithm detects the closest facet of the convex hull to the
origin of the wrench space. The closest facet will used to guide the direction of the
local search algorithm.
4.2 Local Heuristic Search
The local search algorithm iteratively searches for finger positions that yield form-
closure under the guidance of a heuristics. The local search starts from a randomly
selection of an initial grasp. It searches for a form-closure grasp in the direction of
reducing the distance between the convex hull of the primitive wrench and the
origin of the wrench space. The key idea here is to move the convex hull gradually
closer and closer to the origin of the wrench space through a iterative process.
This idea is illustrated in
Fig. 6.
Fig. 6 The local heuristic search process
We consider that each of the given points in set Ω is connected with four
neighboring points on the surface of the object. The connection means four possi-
ble motions of a finger at an iteration of the search. To simplify the problem, we
here assume that at a iteration only one finger can be moved to its neighboring po-
sition. Consequently, for a frictionless grasp with 7 contacts there are totally 28
candidate grasps to be selected at every iteration. Observed from the qualitative
test, it is desirable to move the convex hull H(W) towards the origin O of the
wrench space R6 until the origin O completely lies inside the convex hull. There-
fore, we consider the distance ||QO|| or ||PO|| as the heuristics and thus select the
candidate grasp with the minimum distance. The local search algorithm is de-
scribed in detail as follows:
1) k = 1; arbitrarily choose an initial grasp Gk = { ri }, for i = 1, 2, …, 7,
where all ri ∈ Ω. Calculate the primitive contact wrenches wik.
2) Check whether Gk is form-closure using the qualitative test algorithm; if
so, return Gk.
3) For each of the grasping position ri, locate the four adjacent positions
ζ i ∈ Ω, for l = 1, 2, 3, 4; generate all the 28 combinations of candidate
l
grasps CG i .
l
4) For each candidate grasp CG i , calculate the corresponding ||QO( CG i )||.

l l
5) Find the candidate grasp CG* with the minimum value ||QO||. If
||QO(CG*)|| is not strictly smaller than ||QOk||, report that local minimum
93
is encountered; return Gk. Otherwise, k = k + 1; update Gk = CG*,

wik(CG*), ||QOk|| = ||QO(CG*)||; go back to 2).
Note that the local search terminates under two situations: when a form-closure
grasp is obtained in 2) or when a local minimum is encountered in 5). In case a lo-
cal minimum is encountered, the algorithm needs to select another initial grasp
and start another round of the local search. In the next subsection, we propose a
recursive problem decomposition strategy to divide the problem into sub-problems
with fewer points so as to overcome the local minimum problem and at the same
time to improve the computational efficiency.
Fig. 7 The separating hyperplane used in the problem decomposition
4.3 Recursive Problem Decomposition
To cope with the local minimum problem, a recursive problem decomposition

strategy is employed to divide the problem in the original point set Ω into sub-
problems in sub-sets. The division is based on an important observation that the
primitive contact wrenches corresponding to the seven contact points must not lie
one side of any hyperplane passing through O to achieve a form-closure grasp.
Having this observation, we can drastically reduce the number of eligible candi-
date grasps by neglecting those grasps with all the wrenches on one side of a spe-
cific hyperplane called separating hyperplane.
Recall, in the qualitative test, that the hyperplane detected is the facet E of
H(W) intersecting with the ray PO. Here, we select a hyperplane Y that is parallel
to the facet E obtained in the local minimum iteration as the separating hyperplane
(Fig. 7):
e1 x 1 + e 2 x 2 + " + e 6 x 6 = 0 (2.17)
The hyperplane divides the wrench space into two half-spaces, which are de-
noted by Y + and Y − , respectively. Y + is the half-space:
−
e1 x 1 + e 2 x 2 + " + e 6 x 6 ≥ 0 ; and Y denotes the half-space:
e1 x1 + e 2 x 2 + " + e 6 x 6 < 0 and As shown in Fig. 4, the separating hyperplane
Y divides the original point set Ω into two subsets:
Ω (Y − ) = {r j ∈ Ω | w j ∈ Y − } (2.18)
and
Ω (Y + ) = {r j ∈ Ω | w j ∈ Y + } (2.19)
Since in a form-closure grasp the seven points should not all belong to subset
Ω (Y − ) or subset Ω (Y + ) , points of form-closure grasps must be selected from
both of these subsets. This gives rise to a division of the problem in the original
set into problems in the subsets based on existence conditions of form-closure
grasps. Six problems in the subsets need to be considered, which are grasps con-
sisting of
• one grasp point from subset Ω (Y − ) and six grasp points from subset
Ω (Y + ) ,
• two grasp points from subset Ω (Y − ) and five points from subset
Ω (Y + ) ,
• three points from subsect Ω (Y − ) and four points from subset Ω (Y + ) ,
• four points from subset Ω (Y − ) and three points from subset Ω (Y + ) ,
• five points from subset Ω (Y − ) and two points from subset Ω (Y + ) , and
• six points from subset Ω (Y − ) and one point from subset Ω (Y + ) .
The sub-problems obtained are represented as child nodes of the search tree
whose root represents the problem in the original set (Fig. 8). The (i,j) in the node
denotes that i and j grasp points must be selected from subsets Ω (Y − ) and subset
Ω (Y + ) , respectively, in order to obtain a form-closure grasp in the corresponding
subsets of points.
From the six child nodes, we select one and carry out the local search of a
form-closure grasp within the subset of points. When a new local minimum is en-
countered, the separating hyperplane is employed to further divide the subset into
three or four subsets. Based on the existence condition, a new set of sub-problems
are generated and represented as child nodes of the node selected. This process is
performed recursively until a form-closure grasp is found or all the nodes have
been explored. It should be noted that no child will be generated for a node if none
of the combination of the subsets divided by the separating hyperplane satisfies
the existence condition of form-closure grasps.
95
To determine which child should be traversed first, we consider a node with

eligible contact points more evenly distributed over the point sets has a higher
chance to contain form-closure grasps. Therefore, we define a heuristic function as
follows:
Original Problem
(1,6) (2,5) (3,4) (4,3) (5,2) (6,1)
Fig. 8 The search tree generated
2
§7 s
·
f ( < a1 a 2 " a s > ) = ¦ ¨ − ai ¸
i =1 © s ¹
(2.20)
where s is the number of subsets obtained in the decomposition and ai denotes the
number of fingers (contact points) in the subset i of the points. For example, for
the left-most node in Fig. 8, the a1 and a2 are 1 and 6, respectively. A node with a
smaller f value will be traversed prior to the one with a larger f value.
4.4 Extension to Frictional Grasps
The difference in frictional case from frictionless case lies that a contact point re-
sults in m primitive contact wrenches, so the primitive contact wrenches corre-
sponding to a grasp point may lie on both sides of the separating hyperplane. This
requests us to divide the original point set Ω into three subsets:
Ω(Y − ) = {ri ∈ Ω | wij ∈ Y − , for all j = 1,2,", m} (2.21)
Ω(Y + ) = {ri ∈ Ω | wij ∈ Y + , for all j = 1,2,", m} (2.22)
−
and Ω (Y ) = {ri ∈ Ω | ri ∉ Ω (Y ), ri ∉ Ω (Y )}
0 + (2.23)
where wij denotes the primitive contact wrenches. Ω(Y − ) denotes the subset con-
taining contact points with the m primitive contact wrenches fully located in the
half-space Y − . Ω(Y + ) denotes the subset containing contact points with the cor-
responding m primitive contact wrenches fully located in the half-space Y + .
Ω(Y 0 ) denotes the subset containing contact points with the corresponding m
primitive contact wrenches located on the both sides of the separating hyper-
plane.
Since in a form-closure grasp, the n grasp points should not all belong to subset
Ω(Y − ) or subset Ω(Y + ) . This gives rise to a division of the problem in the origi-
nal set into problems in the subsets based on the existence conditions of form-
closure grasps. Grasp points of form-closure grasps must be selected from the sub-
set(s) as stated in any one of the following combinations:
• Ω(Y − ) , Ω(Y + ) and Ω(Y 0 ) ,
• Ω(Y − ) and Ω(Y + ) only,
• Ω(Y − ) and Ω(Y 0 ) only,
• Ω(Y + ) and Ω(Y 0 ) only, and
• Ω(Y 0 ) only.
To illustrate the idea, let’s consider a 3-finger frictional grasp. Eight problems
in the subsets need to be considered, which are grasps consisting of
• one grasp point from subset Ω(Y − ) , one grasp point from subset
Ω(Y + ) and one grasp point from subset Ω(Y 0 ) .
• two grasp points from subset Ω(Y − ) and one grasp point from subset
Ω(Y + ) ,
• one grasp point from subset Ω(Y − ) and two grasp points from subset
Ω(Y + ) ,
• two grasp points from subset Ω(Y − ) and one grasp point from subset
Ω(Y 0 ) ,
• one grasp point from subset Ω(Y − ) and two grasp points from subset
Ω(Y 0 ) ,
• two grasp points from subset Ω(Y + ) and one grasp point from subset
Ω(Y 0 ) ,
• one grasp point from subset Ω(Y + ) and two grasp points from subset
Ω(Y 0 ) , and
• all the three grasp points from subset Ω(Y 0 ) .
Fig. 9 The search tree generated for a 3-finger frictional grasp

The sub-problems obtained are represented as child nodes of the search tree
whose root represents the problem in the original set (Fig. 9). The (i,j,k) in the
97
node denotes that i, j and k grasp points must be selected from subsets Ω(Y − ) ,
Ω(Y + ) and Ω(Y 0 ) , respectively.
From the eight child nodes, we select one by a similar heuristics and carry out
the local search of a form-closure grasp within the subset of points. When a new
local minimum is encountered, the separating hyperplane is employed to further
divide the subset into subsets. Based on the existence condition, a new set of sub-
problems are generated and represented as child nodes of the node selected. This
process is performed recursively until a form-closure grasp is found or all the
nodes have been explored. It should be noted that no child would be generated for
a node if none of the combination of the subsets divided by the separating hyper-
plane satisfies the existence condition of form-closure grasps.
4.5 Completeness and Complexity
Although the algorithm employs a heuristics to guide the local search and selec-
tion of the sub-problems, it always finds a form-closure grasp or reports no solu-
tion after all the nodes in the search tree have been visited. This can be explained
in the following. The algorithm divides using the separating hyperplane the prob-
lem into problems in subsets of the points when the distance between the convex
hull and the origin of the wrench space reaches a local minimum. The division
makes the numbers of points in the subsets smaller than those in the original point
sets. The search reaches a leave node when the division results in zero subset or all
the points lie on one side of the separating hyperplane. In both cases, it is guaran-
teed that no form-closure grasp exists in the subsets. The algorithm will always
terminate with reporting no solution after all the leave nodes are visited. There-
fore, the proposed algorithm is complete as it always finds a solution if there is
and otherwise reports no solution. Then, consider the computational complexity
of the algorithm. The complexity depends on the number of nodes of the search
tree.
Proposition 1: Denote by n the number of fingers/contact points. In frictionless
cases, the number of child nodes of each node is not more than 2 n − 2 ; In fric-
tional case, the number of each node is not more than 3n-2.
This can be simply proven. The separating hyperplane divides the subset for
each finger into at most 2 subsets in frictionless case. According to the existence
conditions, there are at most 2 n − 2 sub-problems to be considered. Therefore,
there are not more than 2 n − 2 children for each node in the search tree. In fric-
tional case, the separating hyperplane divides a subset into at most 3 subsets, so
there are not more than 3n − 2 children for each node.
Since the search tree generates new nodes whenever a local minimum is de-
tected, the computational efficiency is strongly dependent of the number of local
minimum points. Denote by K the total number of the local minima in the configu-
ration space of grasps. Note that the number K is determined by the surface ge-
ometry of the object. Once the object is specified, the number K is a constant. For
simplicity, we assume that the local minima are uniformly distributed over the
configuration space. The average number of attraction points for each local mini-
Nn
mum is , where N is the total number of the discrete geometric points on the
K
object surface. The corresponding number of points in the subset for each finger is
N N
. In other words, if there are less than points in subset set of each fin-
nK n
K
ger, we can assume that there is only one minimum point (global minimum) in the
subset. When ||QO|| is used as the heuristics, the local search process will defi-
nitely find a solution if there are form-closure grasps in the subsets. This is be-
cause the distance will be convergent to the minimum value, which is negative.
Therefore,
Proposition 2: Assume that the K local minimum points are uniformly distrib-
uted in the configuration space. In case there is a form-closure grasp in the given
K ln K
discrete point set, the number of nodes generated in the search tree is O( )
n
if the search tree is expanded in the breadth-first strategy.
Proof: For simplicity, assume that at each decomposition the separating hy-
perplane divides points in each subset equally, i.e., the numbers of points in the
two new subsets obtained after the division are the same. In the frictionless case,
the level l of the search tree leading to a smaller number than the average number
N
of attraction points from each local minimum in the subsets is calculated by
nK
N N ln K
l
=n l = (2.24)
2 K n ln 2
If the breadth-first search is adopted, the total number L of nodes generated in
the search tree satisfies:
1
ln K n ln K ln K (2.25)
L≤ ( 2 − 2) l = ( K − K n ) = O( K)
n ln 2 n ln 2 n
A similar result can be obtained for frictional grasps.
It should be noted that the number of nodes is independent of the number of
points representing the object in case there exists a solution. The complexity of the
tree is mainly determined by the surface geometry of the object and the number of
fingers. The total computational time is equal to the product of the number of
nodes with the times of calling the local search process. Note that the local search
algorithm is efficient because it is essentially a gradient-based algorithm and the
ray-shooting based qualitative test algorithm takes a time linear to the number of
the primitive contact wrenches.
The problem is more complicated when there exists no form-closure grasp in
the given problem. Theoretically, if the corresponding grasp is not form-closure at
the global minimum point in the subsets, it is not necessary to carry out any fur-
ther investigation. However, since no method is available to check that the mini-
99
mum reached in the local search process is the global one in the subsets, it is nec-
essary to further divide the subsets. From the nature of the strategy adopted in the
algorithm, it is reasonable to consider that most of the points are on one side of the
separating hyperplane obtained at the global minimum. This implies that the algo-
rithm would be able to understand non-existence of a solution in the subsets with a
few times of further decomposition. Therefore, a weaker result can be obtained
that the number of nodes generated in the search tree when there is no solution in
the problem is similar to that when there exists a solution. When the distance ||PO||
is used as the heuristics, the complexity is similar. However, a rigorous proof is
difficult and necessary.
Although the discussion above is based on assumptions, this algorithm works
more efficiently than the exhaustive search algorithm whose time is O( N n ) .
Compared to other heuristic approaches, the algorithm is complete.
4.6 Implementation
We have implemented the proposed algorithm using MATLAB. Two numerical

examples are used to demonstrate the usefulness and efficiency of the algorithm.
The first example concerns computation a 7-finger frictionless grasps and the sec-
ond example addresses computation frictional grasps. In both examples, we used
the model of a turbine airfoil (Fig. 10) with 1546 predetermined candidate loca-
tions and surface normals. Turbine airfoils are good examples of 3-D workpieces
with complex geometry. The geometric shape of an airfoil is primarily defined by
its aerodynamics. However, the geometric representation in a CAD system is usu-
ally approximated by parametric surfaces such as B-splines. Only a dense set of
points of the surfaces are defined exactly as calculated in the aerodynamic analy-
sis. In order to minimize the effects of the geometric approximation, the airfoil is
required to be fixtured or grasped at some of these precise surface locations in its
manufacture and inspection. Therefore, the point-set constraint is imposed by the
practical conditions related to the functional and/or manufacturing requirements.
In the first example, seven frictionless fingers are to grasp the airfoil. The al-
gorithm takes 68 iterations in total to obtain a form-closure grasp. The initial
grasp, selected randomly, shown in Fig. 11(a) is non-form-closure. After 62 itera-
tions, the algorithm encounters a local minimum in the performance measure
||PO||. A separating hyperplane is defined to divide the candidate contact points
into two subsets and the algorithm performs the local search process in the state
< 3 4 >. Then, a form-closure final grasp is found after 6 more iterations (Fig.
11(b)). Fig. 12 plots the distance difference ||PO|| - ||PQ|| and the distance ||PO||
against the iteration number.
In second example, we consider the problem of finding a frictional grasp on the
airfoil Assume that the number of fingers is 4. The friction cone at each contact
point is linearized with a polyhedral convex cone with 20 segments (i.e. m = 20).
The algorithm is required to obtain a 4-finger form-closure grasp on the airfoil
model in the presence of friction. A friction coefficient µ = 0.2 exists between the
100100Y.-H. Liu, D. Ding, and M.-L. Lam
object and each finger. The initial grasp upon random selection is non-form-
closure (Fig. 13(a)). The algorithm performs the local search in obtaining trajecto-
ries of finger motions in the direction to decrease ||PO||. In the 62nd iteration, the
distance between P and O reaches a local minimum value (Fig. 14). The search
tree is then generated and the child (1,1,2) is adopted. Then, a new grasp is se-
lected from points in the subsets corresponding to node (1,1,2) and another round
of local search was executed. Finally, a form-closure grasp is obtained (Fig. 13(b))
at the 71st iteration. The results ascertained the efficiency of the proposed algo-
rithm.
Fig. 10 An airfoil approximated by 1546 discrete points
Fig. 11 Frictionless grasp: the initial grasp (a) and the final grasp (b)
101
Fig. 12 Frictionless grasp: plots of the distances ||PO||-||PQ|| and ||PO||
Fig. 13 Frictional grasp: The initial grasp (a) and the final grasp (b)
5 Automatic Fixture Layout Design
A closely related problem to mutli-fingered grasping is automatic fixture layout

design, which is to find a set of fixturing points (locators and clamp) firmly hold-
ing a workpiece. A firm holding implies that the fixturing points must form a
form-closure. For sake of safety during fabrication process, in fixture layout de-
sign all contacts are conservatively considered frictionless, and thus the problem is
similar to synthesis problem of frictionless form-closure grasps. Since seven fric-
tionless contact points are sufficient and necessary to form a form-closure, we
suppose that the number of fixturing points to be found is 7. This section will
show that the ray-shooting problem can be also applied to automatic fixture layout
design on polyhedral workpiece. The problem addressed here is defined as fol-
lows:
Problem 3: Given a geometric model of a polyhedral workpiece with well-
defined surface normal vectors and vertices information,
(1) find an eligible set of fixturing surfaces on which seven fixturing loca-
tions satisfying form-closure property exist;
(2) find seven fixturing points on the eligible set of surfaces which ensure
the total constraint and an accurate localization of the workpiece.
Fig. 14 Frictional Grasps: Plots of the distances ||PO|| and ||QO||
5.1 Eligibility Set of Fixturing Surfaces
Fixturing surfaces are the surfaces of a workpiece used to locate and clamp the
workpiece where functional fixturing elements including locators and clamps are
in contact with these surfaces. Besides some trivial requirements like non-
machining surfaces and surfaces with accessible normal directions, the primary
concern when selecting the fixturing surfaces is whether there exists possible fix-
turing points yielding form-closure on them. A set of fixturing surfaces are said
eligible if there exists fixturing points on them yielding form-closure.
103
On a set of eligible surfaces, it is always possible to find seven fixturing points

ensuring form-closure. A simple condition is presented here to check the eligibil-
ity of a set of surfaces. A polyhedral workpiece is comprised of convex as well as
concave faces. Note that a concave surface can be always decomposed into con-
vex ones. The position vector ri of a contact point on a convex surface can be rep-
resented as a convex combination of the vertices of the surface. Denote by vij the
coordinates of vertex j on the i-th surface and by cij the corresponding multiplier.
Then position ri of the i-th fixturing point on the surface is a convex combination
of vij, i.e.,
ti
ri = ¦c v
j =1
ij ij
(2.26)
where ti is the number of the vertices of the i-th convex surface. The positive mul-
tipliers cij satisfy
ti
(2.27)
¦c
j =1
ij =1 cij > 0
Note that cij are chosen to be strictly positive to ensure that the fixturing points
calculated are inside the surfaces and not at the vertices or on the edges. cij are the
variables of interest, as they define the location of each fixturing contact. Then the
wrench applied at the object by the i-th contact can be rewritten as:
§ ti ·
§ ni
¨
· ¨ i =1
¦ cij ni ¸
¸ ti
§ ni ·
wi = ¨¨ ¸¸ = ¨ t ¸= ¦c ¨
ij ¨
¸
¸
(2.28)
© ri × ni ¹ ¨ i ¸ © vij × ni ¹
¦
¨ cij vij × ni ¸
j =1
© j =1 ¹
Let
§ ni · (2.29)
bij ∆¨¨ ¸
¸
© vij × ni ¹
which is called the vertex contact wrench here. Then introduce a matrix B:
(
B = b11 , b12 ,..., b1t1 ,..., bk1 , bk 2 ,..., bkt k ) (2.30)
where k is the number of convex surfaces of a workpiece.
Proposition 3: Assume that k convex surfaces of a workpiece are given. The k

surfaces are eligible if and only if the convex hull of the corresponding vertex con-
tact wrenches bij strictly contains the origin of R6.
Proof: The existence of form-closure fixturing points on the surfaces implies

the existence of strictly positive α i such that
¦α wi =1
i i =0
k (2.31)
¦α i = 1
i =1
∀α i > 0
Substituting eq. (2.28) into eq. (2.31) results in

k ti
¦αi ¦ cijbij = 0
i =1 j =1
(2.32)
Let β ij = α i cij , thus

k ti
¦¦ β b
i =1 j =1
ij ij =0 (2.33)
Note that
k ti
¦¦ β
i =1 j =1
ij = 1, β ij > 0 (2.34)
This means the convex hull of the points bij contains strictly the origin of R6.
From this proposition, by testing whether the convex hull of points in B con-
tains the origin, we can readily check the eligibility of a set of surfaces. For con-
venience, in the following part of this section we use bi with a single subscript i,
instead of bij, to denote the i-th column vector of the wrench matrix B.
To find a set of eligible surfaces, we cam apply the grasp synthesis algorithm
presented in the previous section. The problem here is similar to the problem of
finding a frictional form-closure grasp. Denote the set of convex surfaces of the
object by Ω. First, we randomly select 7 surfaces and check whether the convex
hull of their vertex wrenches contains the origin of the wrench space. If the con-
vex hull contains strictly the origin, a set of eligible surfaces has been found; oth-
erwise, apply the local search procedure by reducing the distance between the
convex hull and the origin until finding a set of eligible surfaces or encountering a
local minimum. In case a local minimum is encountered, divide by the separating
hyperplane the surfaces in the set Ω into subsets and represent the sub-problems in
the subsets as child nodes of the root of the search tree. Note that the vertex
wrenches of a surface may spread on both sides of the separating hyperplane, so it
is necessary to divide the surface set Ω into three subsets like the case of frictional
grasp. The same procedure is applied to the child nodes recursively until a set of
eligible surface is detected or all the nodes of the search tree have been processed.
5.2 Optimal Fixturing Points
The next step is to find fixturing points on the set of eligible surface. Usually the
automated fixturing point determination needs to perform an exhaustive search to
105
reach an optimal design solution. We here present a method to calculate the opti-
mal fixturing points under a performance index.
A given system of wrenches achieves form-closure when the following equa-
tions can be satisfied:
7 7 ti
¦ w = ¦α ¦ c b
i =1
i
i =1
i
j =1
ij ij =0 (2.35)
using the following variables:

β l = α i cij i = 1,2,...,7 (2.36)
j = 1,2,..., ti l = j + (i − 1) ∗ ti
The form-closure constraints can be formulated as:
N
¦β b
l =1
l l =0 (2.37)
¦β
l =1
l =1
β ∈ RN > 0
7
It should be noted that all the constraints are linear and involve N = ¦t
i =1
i vari-
ables. Besides the stability constraint in determining fixturing points, another im-
portant factor to be considered is the positioning accuracy of the workpiece pro-
vided by fixturing points [30]. The positioning errors of fixturing points result in
the localization error of the workpiece. To ensure a minimum influence of the po-
sitioning errors of the fixturing points on the positioning accuracy of the work-
piece, the fixturing points should be located as far away as possible from the cen-
ter of mass of the workpiece. So we consider to maximize the Euclidean distance
from each fixturing point to the center of mass of the workpiece, i.e.,
7
Maximize : ¦α
i =1
2 T
i ri ri .
(2.38)
The above equation can be rewritten as:

7 B D 7 B D
¦αi ¦¦ c j ck vTj vk = ¦¦¦ β j β k vTj vk

i =1 A C i =1 A C
(2.39)
where
i −1 i i −1 i
A : j = 1+ ¦t ,
z =1
z B: ¦t ,
z =1
z C : k = 1+ ¦t ,
z =1
z D :: ¦t
z =1
z
Thus we obtain a Quadratic Programming (QP) problem subject to the con-

straints eq. (2.37). Maximizing the objective function has the effect of maximizing
ri and forcing α i to tend to be equal for all i. The former means minimization
of workpiece locating errors and the latter implies even distribution of the fixtur-
ing forces. Finally, the variables of interest, multiplier cij, can be obtained as fol-
lows:
βl
cij = D
(2.40)
¦β
C
k
It should be noted that the objective function employed here sometimes may
produce fixtures with bad sensitivities for certain workpiece, but it works in most
of the cases to produce good fixtures that reduce the localization errors of work-
pieces.
Fig. 15 shows two views of the fixturing points, found by this algorithm, on a
polyhedral part. The part has eight surfaces and two of them are concave. After
decomposing the two concave surfaces, there are fourteen convex surfaces for the
part. The vertices and normal vector of each surface are the input parameters. Note
that those surfaces of other purposes, e.g., machining features, supporting, should
be eliminated from the input set. In this case, the large top face shown in Fig.
15(a) is assumed to be the surface to be machined, so we remove it from the input
surface set. The algorithm first randomly picks up three surfaces as the initial set,
and then finds the final eligible surface set including seven surfaces is obtained in
0.33 seconds of the CPU time of a SPARC Station 4 using Matlab. Next, the quad-
ratic programming is employed to calculate proper positions for fixturing points
on the final surface set selected above. It takes 0.44 seconds of the CPU time to
calculate the positions for seven fixturing points. This example ascertained the ef-
fectiveness and efficiency of this algorithm.
1.5
0.5 4
2 4
2
0
1.5 3
3
−0.5
1
−1
0.5 7
1 6
−1.5
0
−2 −0.5 3
3
2
2 −1
5 5 1
1
−1.5 0
0
−1 −1
−2
−2 1 −1 −2
0 0
−3 −1 1 −3
(a) (b)
Fig. 15 Fixture locations found on a polyhedral part
107
6 Conclusion
In this chapter, we presented a ray-shooting based formulation for stability analy-

sis and synthesis of 3-D form-closure grasps. First, it has been shown that the
qualitative test of a 3-D form-closure grasp can be transformed to a ray-shooting
problem. Then, based on this transformation we proposed an effective heuristics
for guiding the local search for form-closure grasps. A complete and efficient al-
gorithm has been developed for 3-D form-closure grasp synthesis in discrete do-
main by combining the local search with a recursive problem decomposition strat-
egy. Furthermore, we extended the ray-shooting based formulation to automatic
fixture layout design on polyhedral workpieces. Examples have been given to
demonstrate the effectiveness and efficiency of the algorithms developed on the
basis of this new formulation. It is important to note that this formulation provides
a good example of linking techniques in Computational Geometry with mechanics
problems.
The work presented is based on point-point contacts among the object and the
robot fingers, so the results may not apply to soft fingers or other kind of finger
models. One of important future works is the generalization of the methods to
various finger models. Other future works include optimization of form-closure
grasps and planning for dexterous manipulation tasks.
References
[1] A. Bicchi, “On the closure properties of robotic grasping,” International

Journal of Robotics Research, vol. 14, no. 4, pp. 319–334, 1995.
[2] M. Buss, H. Hashimoto, and J. Moore, “Dexterous hand grasping force opti-
mization,” IEEE Transactions on Robotics and Automation, vol. 12, no. 3, pp.
406–418, 1996.
[3] M. R. Cutkosky, "On grasp choice, grasp model, and the design of hands for
manufacturing task," IEEE Trans. on Robotics and Automation, vol. 5, no.
3, pp. 269-279, 1989.
[4] D. Ding, Y. H. Liu, Y. T. Shen, and G. L. Xiang, “An efficient algorithm for
computing a 3D form-closure grasp,” in Proc. IEEE/RSJ Int. Conf. on Intelli-
gent Robotics and Systems, 2000, vol. 2, pp. 1223–1228.
[5] C. Ferrari and J. F. Canny, "Planning optimal grasps," in Proc. IEEE Int.
Conf. on Robotics and Automation, pp. 2290-2295, 1992.
[6] J. R. Kerr and B. Roth "Analysis of multi-fingered hands," Journal of Robot-
ics Research, vol. 4, no. 4, 1986.
[7] L. Han, J. C. Trinkle, and Z. Li, “Grasp analysis as linear matrix inequality
problems,” IEEE Transactions on Robotics and Automation, vol. 16, pp. 663–
674, Dec. 2000.
[8] M. L. Lam, D. Ding, and Y. H. Liu, "Grasp planning with kinematic
constraints", in Proc. IEEE/RSJ Int. Conf. On Intelligent Robots and Systems,
2001, vol. 2, pp. 943-948.
[9] Z. Li, P. Hsu and S. Sastry, "Grasping and coordinated manipulation by a

mutlifingered robot hand," Int. Journal of Robotics Research, pp. 33-50,
1989.
[10] Y. H. Liu, “Qualitative test and force optimization of 3-D frictional form-
closure grasps using linear programming,” IEEE Transactions on Robotics
and Automation, vol. 15, pp. 163–173, Feb. 1999.
[11] Y. H. Liu, “Computing n-finger form-closure grasps on polygonal objects,”
International Journal of Robotics Research, vol. 18, no. 2, pp. 149–158,
2000.
[12] H. Maekawa, K. Komoriya, and K. Tanie, "Manipulation of an unknown ob-
ject by multi-fingered hands with rolling contact using tactile feedback," in
Proc. IEEE Int. Conf. on Intelligent Robots and Systems, pp. 1877-1882,
1992.
[13] R. Mattikalli, D. Baraff and P. Khosla, Finding all stable orientation of as-
semblies with friction, IEEE Trans. On Robotics and Automation, vol. 12, no.
2, pp. 290-301, 1996.
[14] X. Markenscoff, L. Ni, and C. H. Papadimitriou, “The geometry of grasping,”
International Journal of Robotics Research, vol. 9, no. 1, pp. 61–74, 1990.
[15] B. Mishra, J. T. Schwartz, and M. Sharir, “On the existence and synthesis of
multifinger positive grips,” Algorithmica, Special Issue: Robotics, vol. 2, no.
4, pp. 541–558, 1987.
[16] D. J. Montana, “The condition for contact grasp stability,” in Proc. IEEE Int.
Conf. on Robotics and Automation, 1991, vol. 1, pp. 412–417.
[17] K. Mulmuley, Computational Geometry: an Introduction through Random-
ized Algorithms. Englewood Cliffs, NJ: Prentice Hall, 1994.
[18] R. M. Murray, Z. Li, and S. S. Sastry, A Mathematical Introduction to Ro-
botic Manipulation. Orlando, FL: CRC, 1994.
[19] V.-D. Nguyen, “The synthesis of stable force-closure grasps”, A. I. Labora-
tory, MIT, Cambridge, MA, Tech. Rep. No. 905, 1986.
[20] V.-D. Nguyen, “Constructing force-closure grasps,” International Journal of
Robotics Research, vol. 7, no. 3, pp. 3–16, 1988.
[21] J. Ponce and B. Faverjon, “On computing three-finger force-closure grasps of
polygonal objects,” IEEE Transactions on Robotics and Automation, vol. 11,
pp. 868–881, Dec. 1995.
[22] J. Ponce, S. Sullivan, A. Sudsang, J. -D. Boissonnat and J. -P. Merlet, “On
computing four-finger equilibrium and force-closure grasps of polyhedral ob-
jects,” International Journal of Robotics Research, vol. 16, no. 1, pp. 11–35,
1997.
[23] F. Reulaux, The Kinematics of Machinery. New York: Macmillan, 1976.
[24] R.C. Brost and K. Y. Goldberg, A complete algorithm for designing planar
fixtures using modular components, IEEE Trans. on Robotics and Automa-
tion, vol. 12, no. 1, pp. 31-46, 1996.
[25] J. K. Salisbury and B. Roth, “Kinematic and force analysis of articulated
hands,” ASME J. Mechanism, Transmissions, and Automation in Design, vol.
105, pp. 33–41, 1982.
109
[26] J. C. Trinkle, “On the stability and instantaneous velocity of grasped fric-
tionless objects,” IEEE Transactions on Robotics and Automation, vol. 8, pp.
560–572, Oct. 1992.
[27] M. Umetsu and K. Oniki, "Compliant motion control of arm-hand system,"
in Proc. JSME Int. Conf. on Advanced Mechanics, pp. 429-432, 1993.
[28] A. S. Wallack and J. F. Canny, Modular fixture design for generalized poly-
hedra, in Proc. IEEE Int. Conf. on Robotics and Automation, pp. 830-837,
1996.
[29] A. S. Wallack and J. F. Canny, Planning for modular hybrid fixtures, in Proc.
IEEE Int. Conf. on Robotics and Automation, pp. 520-527, 1994.
[30] M. Y. Wang, “An optimum design for 3-D fixtures synthesis in a point set
domain,” IEEE Transactions on Robotics and Automation, vol. 16, pp. 839–
846, Dec., 2000.
[31] J. D Wolter and J. C. Trinkle, Automatic selection of fixture points for fric-
tionless assemblies, in Proc. IEEE Int. Conf. on Robotics and Automation, pp.
528-534, May, 1994.
[32] T. Yoshikawa and K. Nagai, Evaluation and determination of grasping
forces for multifingered hands, in Proc. IEEE Int. Conf. on Robotics and
Automation, pp. 245-251, 1987.
[33] H. Zhang, K. Tanie, and H. Maekawa, Dextrous manipulation planning by
graph transformation, in Proc. IEEE Int. Conf. on Robotics and Automation,
pp. 3055-3060, 1996.
110
Laser Visual Sensing and Process Control

in Robotic Arc Welding of Titanium Alloys
H. Luo, and X.Q. Chen
Singapore Institute of Manufacturing Technology

71 Nanyang Drive, Singapore 638075
xqchen@SIMTech.a-star.edu.sg
Abstract. Sensor errors arise from detecting the weld seam profile of titanium
alloys with laser-based vision sensors because the surface of such a material is
highly reflective to a laser light. In this study, the correct weld seam profile is
established from the distorted raw data by signal processing. Furthermore, a fully
automatic seam tracking and parameter adjusting system is developed. This system
features training and saving of the template, auto-finding of the starting welding
point, auto-calibration, detection of seam position and groove dimensions, tracking
control and on-line welding parameter adjustment. The system is capable of
tracking V-groove, fillet, lap and butt joints of titanium alloys with high accuracy of
less than 0.4 mm.
1 Introduction
Titanium, the fourth most abundant metal in the earth’s crust, features low density,
high strength and excellent resistance to corrosion. Titanium alloys have a
fascinating range of applications and are becoming increasingly important in a
growing spectrum of industrial sectors. These applications play a key part in
modern life and there is huge potential for expanded use of them in fields such as
aerospace, automotive, construction and medicine. Obviously, high quality welding
of titanium alloys is one of the critical processing technologies that determine their
applicability.
In order to achieve good welding quality, seam tracking is needed to align the
torch along the welding seam in despite of distortion or poor jig fixing. Various
approaches for seam tracking during arc welding processes such as through-the-arc
sensing [5-7,10], ultrasonic [1] and vision-based sensing [2-4,8,9,11] have been
extensively explored. Welding parameters also need to be controlled and adjusted
according to weld groove profiles.
Among various seam-tracking methods, laser-based vision sensor is the most
promising and most widely used technology due to its high speed, high accuracy
and high robustness to welding environment. However, laser-based visual sensors
are very sensitive to material surface and titanium alloys are very high reflective
Laser Visual Sensing and Process Control in Robotic Arc Welding 111
111
metals. Problems arise in seam tracking and parameter control during arc welding
of titanium alloys.
In this study, two laser-based sensors are investigated to obtain weld seam point
and weld groove profile. A titanium alloy welding specification database is set up.
Weld seam point is used to move the robot to position the torch along the seam,
while weld groove profile is used to extract welding parameters from the database
to ensure welding quality.
2 System Design and Experimental Set-Up
The experimental set-up consisted of a 5-axis AC servo driven robot, a welding

torch, a pulsed power source (Fronius TT2000) and a laser-based vision sensor- a
Meta 3D camera or a Keyence position-sensitive-detector (PSD) displacement
sensor. The PC interface inside the power source enabled the PC to control and
adjust the welding parameters. The sensor head was mounted in front of the welding
torch, separated by a look-ahead-distance (LAD) between the sensor and the torch.
The computer contained a frame grabber board or a data acquisition board, an
8-axis MEI motion controller card. The host computer served as an image or data
acquisition and robotic control system.
Figure 1 is a block diagram showing the various components of the system and
their relationships. Figure 2 shows the set-up of the 5-axis AC servo driven welding
robot. It consists of three Cartesian axes (X, Y, Z), and two rotary axes to control the
torch orientation.
Power Welding process Torch

supply
Laser sensor/camera
Interface Robot
Groove Seam point
dimensions position
PC
Process Motion
parameters Database Seam control card
tracking
Fig. 1. Block diagram of the robotic welding system

112112H. Luo and X.Q. Chen
Fig. 2. Experimental Set-up
The CPU sends signals to the MEI card, instructing it to move the welding torch
and the camera/sensor. The camera/sensor attached to the welding torch captures
images/profiles of the weld groove, and sends them to the frame grabber/data
acquisition card. Subsequently, the images/profiles are processed to detect the seam
point and the width and length of the weld groove. The seam point is passed to the
seam-tracking controller to align the torch in relation to the weld seam, while the
weld groove dimensions are passed to the welding specification database to extract
the welding parameters.
3 Sensing Algorithms for Seam Point and Weld Groove

Dimensions
Since the surface of the titanium alloys is very reflective to laser light, sensing
errors will arise when a laser-based vision system is used.
3.1 Laser Scanning Sensing
There are two PSD detectors (Y-direction and Z-direction) inside the Keyence PSD
sensor. As the red light laser scans across the work-piece by 30~40 mm in 20 ms,
the coordinate values (Y in points, Z in voltage) of 1000 points across the
work-piece can be acquired. Then the laser returns to its starting point in 10 ms,
which ends a measurement cycle.
For the Keyence PSD laser displacement sensor, precise weld groove profile was
achieved when the titanium plates were sandblasted. The V-groove profile is shown
in Figure 3.
113
Fig. 3. V-groove profile of sandblasted titanium plates
However, when this sensor was used for the very shining surface of titanium
plates, distorted signals were captured in the profile data. Figure 4 and Figure 5
demonstrate the V-groove profile without a gap between the titanium plates and that
with a gap respectively.
Fig. 4. V-groove profile without a gap
It was found that even though the sensed profile was distorted by the reflective
surface, points A and B are correct. The distance between point A and C or that
between point B and D is exactly the height of the weld groove. The position of
point E is the center of the groove. Therefore, the weld groove profile can be
constructed from the distorted data. Figure 6 showed the results. Seam point and
groove dimensions were available form the profile.
3.2 Structured Laser Vision System
The Meta 3D camera was used for this development work. It consists of two laser
diodes and a CCD camera. The sensor operates on the principle of active camera at
a known angle. The image resulting form the intersection of this light sheet with the
work-piece comprises two curved or straight-line stripe segments corresponding to

the two pieces of metal to be joined. The camera is located immediately in front of
the welding torch. Each stripe produces the weld profile image.
A B
C D
E
Fig. 5. V-groove profile with a gap
Fig. 6. Weld profile constructed form corrupted data
For the Meta sensor, since the surface of titanium alloys is very reflective to the
laser light, two reflection lines were seen in the image shown in Figure 7.
Reflective
lines
Fig. 7. 768 by 572 pixel weld groove image with origin (0, 0) at top-left corner
115
It was seen that there existed a -type shape inside the image. Because of its
!
difference from the other part of the image, it was trained and saved as a template.
Then, a feature-finding tool using a normalized correlation engine was used to
locate the given feature within a new image. Once executed, it indicates whether it
successfully finds its model by setting the “found” property to true, and the X and Y
coordinates of the identical feature can be read in pixel. It should be noted that the X
coordinate represents the Z-axis (vertical axis) of the real world. Furthermore, four
lines in the image can be extracted through pixel intensity finding and curve fitting
techniques. Their intersection points are found by solving the four equations
representing the four lines. Therefore, the weld groove dimensions can be obtained.
4 Seam Tracking and Control through Structured-Laser

Vision
The implementation of seam tracking and welding parameter control consists of the
following procedures:
• Training and saving of the feature template;
• Auto-finding of the starting welding point;
• Auto-calibrations;
• Detection of seam position and groove dimensions;
• Tracking control and parameter adjusting.
The detailed flow chart is shown in Figure 8. Some key steps in seam tracking
and parameter control process are explained in detail.
4.1 Region of Interest (ROI) Auto Placement
After loading the trained feature, the feature-finding tool uses it to locate a feature
within an image. If the feature was found, a result appeared as a point indicating
where the feature was located in the inspection image. Consequently, a training and
an inspection region of interest (ROI) are placed with their center at the point. After
placing the ROIs, the feature-finding tool re-trains the feature specified by the
training ROI as the new feature for the following processes such as calibration
process and seam tracking process. In these processes, only the inspection ROI will
be searched for the feature, hence significantly reducing image-processing time.
4.2 Auto Calibrations and Start Point Finding
In the feature-finding algorithm, the software will return the result in pixel
coordinates of the trained feature in the acquired image. Hence, it is necessary to
find out the actual size with respect to the real world. The purpose of calibration is
to synchronize the movement of the 3-axis robot to the work piece, and it is also
used to correlate the pixel size of the image to the real world units in millimeters. In
S tart Load job inform ation and feature
• M ove the robot along X direction

• G rab an im age
No
Feature Found
Y es
• C am era at the start of the w eld (X )
• Auto R O I p lacem ent, re-train the featu re
• Axis Y & Z calibration
• S tart process tim er w ith interval of 0.5s
Increase
m issing point G rab an im age
counter & set
m issing flag No
Feature F ound
Y es Y es C om pute interm ediate
M issing flag set m issing point using
line fit m ethod
No
• R eset m issing point counter & m issing flag
• C om pute the deviation betw een the
previous seam point and the current one
• C om pute the w eld groo ve dim ensions and
extract param eters fro m database
No
D eviation is
w ithin threshold
Y es
• Add the point (Y ,Z ) into the list box
• Increase seam point counter
S eam point counter>= Look No

A head Fram es
Y es
S tart w elding & m ake an y correction to the
seam path & set w elding param eter
Y es
N o of m issing point
> Look A head Fram es
No
No
S top the R obot idle?
w elding
Y es
process
E nd of w eld reached
• S ave the seam point into the file E nd

• M ove the robot to the start position
Fig. 8. Flow chart of seam tracking and process control

117
order to achieve high reliability, it is always necessary to calibrate every job to

obtain the calibration factor.
The calibration factor (CF) is the numbers of pixels in one-millimeter drift
(pixels per mm). The calibration factor of Y-axis and Z-axis are obtained by finding
the pixel coordinates of the trained feature in the acquired image as the reference
point (ref_pixel). The torch will then be shifted to the left and right by 1 mm from its
current position to capture the respective pixel coordinates of the acquired image
(left_pixel and right_pixel). The calibration factor for Y-axis is calculated by the
following formulae:
CF for Y-axis = abs [(left_pixel-ref_pixel) + (ref_Pixel-right_pixel)]/2
The Z-axis calibration is similar except that shifting the torch up and down by
1mm to capture the pixels coordinates (up_pixel and down_pixel). The calibration
factor for Z-axis is calculated by the following formulae:
CF for Z-axis = abs [(up_pixel-ref_pixel) + (ref_Pixel-down_pixel)]/2
As the torch is moving along X-axis, the first feature-finding point is where the
sensor is at the starting point of the weld seam ( -direction), while the torch is still
!
Look Ahead Distance (LAD) behind. If the sensor is at the center of the weld seam,
the Y-coordinate returned by the feature find tool should be 384, which is half of the
image length in pixels. If the sensor is not at the center of the weld seam, it is able to
move the robot Y-axis until the Y-coordinate reaches that value. Since the welding
torch is aligned with the sensor, after moving the -axis by LAD, the torch should
!
be in the center of the weld groove.

For Z-axis, the robot can be moved up or down until it reached the desired
Z-position where the X-coordinate should return the value, which equals to [desired
arc length * CF (Z) in pixels].
After these automatic adjustments, this found point should be the start of the
weld seam (X-direction), and the center of the weld groove (Y-direction) and high
above the work piece of the desired arc length. This point should be the very start
point of welding process. During the welding process, the robot will move in the
X-direction at the welding speed, arc length will be kept constant, and the torch will
be kept at the center of the groove despite any work piece distortion.
4.3 Missing Point Algorithm
Using feature-finding tool, there exist undetected points - missing points. In this
study, a two-point linear interpolation method was first used to find the undetected
seam point between the previous and the following detected seam point. This
algorithm is able to take care of any straight and titled path, and also able to
compute the coordinates of several consecutive missing points. Figure 9 shows
three consecutive missing points between two detected points.
The respective missing point’s Y-coordinate can be computed by the following
equation irrespective of whether the seam path has a positive or negative gradient.
Mn = P1+ n*(P2-P1) / (No. Of missing point+1)]
However, if P1 and P2 are in a curvature path, as shown is Figure 10, the
computed missing point is M1* based on the linear interpolation while the actual
seam point is M1. This tracking error is due to the fact that only two points are used
in the linear interpolation algorithm.
P2
M3
M2
M1
P1
D is t D is t D is t D is t
Fig. 9. Three consecutive missing points
M1 P2
M 1*
P1
D ist D ist
Fig. 10. Wrong computation of seam point (M1*)
The occurrence of this error is very rare, as the interval between two points is
very small. For instance, given the image acquisition and processing period is 0.5
ms and the welding speed is 4 mm/s, the interval between two points is 2 mm. In
such a short interval, two-point linear interpolation works very well. In a high speed
welding application, more points are needed for the interpolation, and curve-fitting
method can be adopted to reduce this computation error.
4.4 Seam Path Conversion Algorithm
One of the essential portions of this study is to acquire the seam path and convert it
into the actual welding path. Once the welding process starts, the X-axis will begin
to move from the starting point to the ending position as specified by the welding
length. There is always a LAD between the laser beam and the welding torch, which
causes difficulty in controlling the correction of the Y and Z-axis movement.
Hence, a seam welding path conversion algorithm is used to convert the seam
path to the actual welding path, where we need to take into account the correct start
119
welding position and to synchronize the movement of the welding torch in the Y
and Z-axis with the LAD.
The main problem in this algorithm is associated with the LAD. As the tip of the
torch lags behind the laser beam by the LAD, the immediate seam points captured
by the sensor head cannot immediately be used to move the welding torch. Hence, a
delay time is required before moving the welding torch to the desired position so as
to compute by the immediate seam point. The delay time is known as look-ahead
time, which can be calculated by the following equation.
Look-ahead time (sec) =Look-ahead-distance (mm)/welding speed (mm/sec)
From the look-ahead time, we can determine the number of look-ahead frames
required to execute the control. It is much easier to use the look-ahead frame
because all the computation is calculated based on the frame rate (number of frames
captured per second). Hence, the number of look-ahead frames is computed as
follows:
Look-ahead frames (frames) =look-ahead time (sec) * frame rate (frames/sec)
The system tracks the number of frames captured upon commencing the welding
process. Once the number of frames captured is equal to the look-ahead frames,
then the welding machine will be switched on.
Next, we need to compute the deviation in Y and Z-axis for the seam point found
at the instant the image is acquired. Figure 11 shows the diagram of calculating the
actual correction distance.
Torch sensor
position position
1 2 3 4 5 6
Y
!
2
!
Y+
!
3
!
Sensor
4
!
path
Y+2
!
4
!
X Y+3
!
Y Y+4
!
Actual seam
Y+4
!
path
Fig. 11. Calculation diagram for actual correcting distance
Assume the number of look-ahead frames is 4. When the torch is at position 1,

the sensor is looking at position 5. At this position, we know that the system already
found the seam point at position 5, which is shifted 4 to the left of the current torch
!
position. Before the sensor reaches the position 5, it has already captured the seam
point at location 2, 3 and 4.
Hence, the system will shift the torch to the left by when it is moving forward
!
to position 2. At the same time, the sensor at position 6 is also shifted by to the
!
left. Therefore, the sensor is only able to detect a 4 shift of the seam path and the
!
system will misinterpret that there will not be any deviation at position 6 as the
detected seam point is the same as the previous one.
However, the torch should be shifted by when moving form position 5 to
!
position 6. This is because when the torch is at location2, the sensor had already
shifted by .
!
In order to eliminate this problem, any shift of the torch away from the reference
position (horizon line) in Y-axis will be accumulated. This accumulation will be
added to the deviation of the seam point found.
The actual deviation at the particular seam point can be computed by the
following equation:
Shift_dev [k] (pixels) = Result returned by feature find [k] (pixels) +
Accumulative Error [k-1] (pixels) – Shift_dev [k-1] (pixels)
The accumulative error can be computed by the next equation:
Accumulative Error [k] (pixels) = Accumulative Error [k-1] (pixels)
+ Shift_dev [k] (pixels)
As the result is in pixels, it is necessary to convert it into millimetre by dividing
the calibration factor. The speed of the Y-axis must also be computed before the
move command is executed. The speed of the Y-axis has to be calculated using the
following equation:
Speed of Y (mm/sec) = [distance in X direction (mm) / Distance in Y direction
(mm)]* Speed of X (mm/sec)
The same method applied to the tracking in Z-axis to maintain the correct arc
length.
5 Results
Extensive welding trials were carried out to study the overall performance of the
seam tracking and parameter controller. The seam-tracking controller produced
satisfactory results with good tracking accuracy of less than 0.4 mm. Figure 12
shows a zigzag V-groove path.
The laser vision sensor tracked the actual seam accurately without prior
knowledge of the weld preparation. A weld sample of straight line welding is shown
in Figure 13.
The laser vision sensor captured the weld seam, and extracted groove features.
Subsequently, correct welding parameters were inferred from the process database
accordingly. Desired welds can be formed even in the presence of groove
variations.
121
C " !
C ! !
>
8 " !
>
<
8 ! !
- " !
>
<
- ! !
" !
H I JH JI HK KI LH LI #H #I H I $H $I %H I% &H &I H I H I JH
H I
' '
H H I I H H
H H H H H
( ) * + , . / 0 1 2
Fig. 12. Zigzag V-groove seam profile
Fig. 13. Titanium weld sample
6 Conclusions
The following conclusions are drawn from the study:
1. High reflectiveness of the surface of the titanium alloys causes the distortion of
the range data acquired by laser-based sensors. Correct weld seam profile can be
successfully recovered from the distorted raw sensor data.
2. A laser-based camera system has been successfully deployed in a fully

automated welding seam tracking system, with a feature finding technique to
track the weld seam reliably. This system features training and saving of the
template, auto-calibration, detection of seam position and groove dimensions,
tracking control and on-line welding parameter adjusting. It is able to find the
starting welding point automatically, and recovers missing seam point through
two-point linear interpolation during the welding process.
3. The system is capable of tracking in V-groove, fillet, lap and butt joints of
titanium alloys with high accuracy of less than 0.4 mm.
References:
1. Ajay Mahajan, Fernando Figueroa (1997) Intelligent seam tracking using ultrasonic
sensors for robotic welding. Robotica, Vol. 15 (3): 275-281
2. Chen X Q, Huang S S (1993) An investigation into vision system for automatic seam
tracking of narrow gap welding process. Chinese Journal of Mechanical Engineering,
Vol. 29 (3): 8-12
3. Chen X Q, Lucas J (1989) Sensors for arrow gap welding. Proceedings of Welding
Production Procedures Being Technology – Arc Welding 1989, DVS Berichte No. 127,
pp 101-105
4. Chen X Q, J Lucas (1989) A fast vision system for control of narrow gap TIG welding.
International Conference on Advances in Joining and Cutting Processes, Harrogate, UK,
pp 8-1 to 8-9
5. Cook G E (1986) Computer-based analysis of arc welding signals for tracking and
process control. IEEE Transaction On Industrial Electronics, Vol. 34: 1512-1518
6. Dilthey U, Stein L (1994) Through-the-arc-sensing: A multipurpose low-cost sensor for
arc welding automation. Welding in the World, Vol.34: 165-171
7. Dilthey U, Stein L Stein, M Oster (1996) Through-the-arc-sensing—an universal and
multipurpose sensor for arc welding automation. International Journal for the Joining of
Materials, Vol.8 (1): 6-12
8. Kakizaki, Takao Arakawa, Ken-ichi, et al. (1998) Development of a high-speed visual
sensing system for robotic seam tracking in automobile body sealing. NTT R&D,
Vol.47 (7): 813-818
9. Karsten Haug, Guenter Pritschow (1998) Robust laser-stripe sensor for automated
weld-seam-tracking in the shipbuilding industry. IECON Proceedings of Industrial
Electronic Conference, Vol.2, IEEE Comp Soc, Los Alamitos, CA, USA, pp 1236-1241
10. Wang J Y, Mohanamurthy P H, Fong M K, Devanathan R, Chen X Q, Chan S P (2000)
Development of a closed-loop through-the-arc sensing controller for seam tracking in
gas metal arc welding. Gintic Technical Report AT/00/013/AMP
11. Yu, Je-Yong, Na, Suck-Joo. (1998) Study on vision sensors for seam tracking of
height-varying weldment Part 2: Applications. Mechatronics, Vol.8 (1): 21-36
Intelligent Technologies for Robotic Welding
S.B. Chen,1 T. Qiu,1 T. Lin,1 L. Wu,2 J.S. Tian,2 W.X. Lv,2 and Y. Zhang3
1. Institute of Welding Engineering, Shanghai Jiao Tong University, Shanghai

200030, PR China. Email: rwlab@sjtu.edu.cn
2. Welding department, Harbin Institute of Technology
3. Dept. of Mechanical Engineering, Qinghua University
sbchen@sjtu.edu.cn
Abstract. This paper discusses intelligent technologies for the robotic welding,
which contains computer vision sensing, automatic programming for weld path
and technical parameters, guiding and tracking seam, intelligent control of weld-
ing pool dynamics and quality. Such an welding robotic systems with some artifi-
cial intelligent functions could realize collision-free path planning for complex
curve seam, detecting welding surroundings by laser scanning technology, identi-
fying the start of weld seam and guiding weld torch, tracking the seam, complete
real-time control of pulsed GTAW pool dynamical process by computer vision
sensing and intelligent technology. The above intelligent separated units or sub-
systems were integrated into an intelligentized welding flexible manufacturing cell
(IWFMC). Based on discrete event control theory and Petri net modeling method,
the related domination strategies for the systems was designed to supervise the
IWFMC.
1 Introduction
Although using robotic welding is propitious to increase productivity, improve

welder laboring status, ensure stable welding quality and realize welding automa-
tion for a small batch of products [34,36,21,9], effective application of welding robots
in practical production is still obstructed due to complexity and uncertainty in
welding process [1,4,14]. Recently, developing intelligentized technology for weld-
ing robots has become an attractive research field and inevitable uptrend[7,6,35]
At present, welding robots serving in practical production still are teaching and
playback type, and can’t often meet quality and diversification requirements of
welding production because this type of robots don’t have adaptive ability to cir-
cumstance changes and uncertain disturbances during welding process. In practi-
cal production, welding conditions are often changing, such as the errors of
pre-machining and fitting work-piece would result in differences of gap size and
124124S.B. Chen et al.
position, the change of workpiece heat dispersion during welding process would
bring on weld distortion and penetration odds . In order to overcome or restrain
various uncertain influences on welding quality, it will be an effective approach to
develop and improve intelligent technologies for welding robots, such as vision
sensing, automatic programming, guiding and tracking, and real-time intelligent
control of welding process [2,10].
To obtain enough precise sensing information from welding process under a
mass of disturbances is one of key technologies for intelligent welding robots.
During robotic welding, intelligent sensing mainly focuses in tracking seam and
monitoring pool feature, and vision sensing technology has been paid a hot atten-
tion in recent years. Generally, the vision sensing technology can be divided into
active method and passive method in welding according to imaging light source.
The active vision sensing method requires laser or other assistant light source to
illuminate welding region and to restrain welding arc interference for imaging
weld pool. Ref[22,3,23,8]reported some designed laser sensor systems for tracking
seam and detecting weld pool. The main problem of active vision sensing system
is too expensive to apply it in practical welding process.
Without assistant light source, the passive vision sensing method only utilizes
self-arc to illuminate welding region. Using a filter to wipe off disturbance, the
charge coupled device (CCD) camera captures weld pool images in selective
narrow spectrum range for picking-up pool features , so that the passive vision
method has been considered as a practical technique for monitoring weld pool
process. Ref.[38,15,25] investigated the passive vision sensing systems for tracking
seam and monitoring weld pool.
As is well known, arc welding process contains many complex factors, such as
material metallurgy, heat conduction, physical chemistry reactions, etc., it is a
typical multi-variable coupling, nonlinear, time-varying, random and uncertain
properties, so that it is very difficult to model and control welding dynamics by
classical linear system theory.
In recent years, some intelligent methods were introduced to welding process,
e.g., Ref.[ 17,28,11 ] investigated fuzzy reasoning application in welding quality con-
trol, and Ref.[12,29,40,24,37] showed research results artificial neural net works for
modeling and control of welding process.
In former researches, although some significant results were obtained in vision
sensing, modeling and intelligent control of weld quality and pool dynamics[27,26],
these intelligent technologies are still not realized effectively in welding robot
systems due to complexity of welding process, real-time and opening limits of ro-
bot systems.
Therefore, developing intelligentized technology for improving current teach-
ing and playback welding robot is necessary and exigent to satisfy high quality
and flexible requirements for welding products and advanced manufacturing up-
trend.
125
2 Intelligentized Welding Robot Systems
The principle scheme of an intelligentized welding robot systems is shown as

Fig.1, it contains a 9-freedom welding flexible mechanical platform with
6-freedom manipulator and 3-freedom positioner; an automatic programming and
simulation subsystem for welding task; a welding seam guiding and tracking unit;
an intelligent real-time control unit with vision sensing, modeling and control of
weld pool dynamical features; and a central control unit for harmonizing each
separated unit.
In Fig.1, Wp is welding power, Sp is signal processing interface for laser scan-
ning, IP is image processing interface for weld pool; WPPC is control computer
for welding power, TPPC is control computer for tracking seam process, VPPC is
process control computer for vision sensing and intelligent control of weld pool
dynamics, MPPC is control computer for collaborating the welding flexible me-
chanical platform with 6-freedom manipulator and 3-freedom positioner; CCPC is
central control computer for whole intelligentized welding robot systems.
Based on structure functions of intelligent robot system [ 16,30], systems in Fig.1
could be divided into three functional levels: executive level, collaborative level
and intelligent level. The executive level contains 9-freedom flexible platform,
sensors, controllers, welding power. The collaborative level includes each func-
tional control computers. The intelligent level includes the index command input
of human-computer interface, welding task programming and simulation, it ad-
ministrates whole systems by means of CCPC. The system in Fig.1 is named as an
intelligentized welding flexible manufacturing cell (IWFMC).
Fig. 1 The system scheme of intelligentized welding flexible manufacturing cell (IWFMC)
The photo of the flexible platform with 6-freedom manipulator and 3-freedom
positioner is shown as Fig.2, which was designed by our-self, due to its opening
framework and maintainability, the platform become a usable equipment for real-
izing various intelligent technique experiments. The basic mechanical experiment
on the platform validated that the system could realize satisfactory collaborating
movement, repeating orientated precision, continuous locus and velocity precision
for robotic welding tasks.
Fig. 2 The photo of 9-frredom flexible platform
3 Automatic Programming and Simulation for Robotic

Welding Task
Off-line programming and simulation is a key technology for improving flexibility

of welding robots Ref.[ 13,5,19]. The automatic programming and simulation subsys-
tem in Fig.1 includes CAD techniques for features of welding robot and
work-piece, path and parameter planning for robotic welding process.
3.1 Robotic welding off-line programming and simulation system

based on dynamic graphics
The module framework of designed off-line planning system is shown as Fig.3,

beside basic functional modules, some advanced modules, such as work-piece
feature CAD modeling, welding parameter and path program, were integrated into
the system for improving program efficiency. Moreover, this system has added a
programming and simulation of arc welding robot with vision and tracking sensors
for robotic movement reach-ability.
In order to obtain features of various welding work-pieces, it is necessary to
establish a basic modeler for identifying geometry features of the object, such as
joint and intersecting between sides and plane. The AutoCAD was used in the
system to indicate welding workpiece features.
127
Advanced Model
character Parameter path
CAD planning planning
Autocad 2000
Geometry Motion Path Teaching Off-line Program Collision Calibration

modeling model model model programming transition checking model
Basic Model
Fig. 3 Framework of off-line programming and simulation
An identification of welding seam features passed through two pipes is shown

as Fig.4.
Fig. 4 An identified results of welding seam features passed through two pipes
3.2 Welding parameter planning for the robot systems
In order to improve intelligence of the programming system, our studies has com-
bined rule-based reasoning RBR , case-based reasoning CBR and artificial
! " ! "
neural networks ANN with welding parameter planning so that off-line plan-
! "
ning of robotic welding techniques can meet some special requirements in practi-
cal production.
The framework of welding parameter programmer for the robot systems is
shown as Fig.5.
[34]
3.3 Path planning technology for arc welding robot
Path planning technology for arc welding robot contains following two aspects:
Fig. 5 The framework of welding parameter programmer
collision-free path planning and redundancy robotic path planning. The colli-
sion-free path planning aims at weld workpiece and circumstance, searching an
optimal welding path in manipulator space for weld quality. Redundancy robotic
path planning focuses on self-movement in arc robot, searching an optimal kine-
matics and welding poses path in joint space.
The collision-free path planning of arc robot
1. The collision-free path planning was expressed as an optimization of welding

torch poses. An optimizing model was presented to escape collision path, and
checking collision algorithm was developed in the system;
2. The heuristic searching technique was used to resolve collision-free path plan-
ning of arc welding robot by the designed producing system and so-called qual-
ity-step searching method.
Based on the AutoCAD software, a simulation system for collision-free path
! "
planning was developed.

129
Redundancy robotic path planning

1. An index function with multi-performance synthesis was presented for redun-
dancy robotic path planning;
Using inheriting arithmetic, A harmonizing control algorithm was developed

! !
for the redundancy manipulator system with six-freedom robot and

three-freedom positioner;
3. Based on the AutoCAD and 3DS software, a simulation system for redundancy
robotic path planning was developed.
4 Laser Scanning Sensing Tracking and Real Time Control

of Welding Robot Moving Locus
During robot welding process, due to various factor influences on welding condi-
tions, such as the machining and assembling errors, heated and remained stress
distortion, it is difficult for the teaching and playback robot to satisfy welding
quality requirement under welding condition anomaly changes. The IWFMC can
realized real autonomous control and correcting robotic manipulations by the fol-
lowing laser sensor for improving robotic adaptive ability, even if without off-line
programming and teaching and playback process.
4.1 The laser scanning sensor for weld seam tracking
The main performance parameters of the designed laser scanning sensor for weld
seam tracking used in IWFMC follows as:
1. Volume: 145mm×120mm×60mm "
2. Weight: 600g "
3. Scanning angle scope: ±7.5° "
4. Scanning times/second: 11.6 "
5. Data capture ratio: 4000 point/s "
6. Across measuring precision: 0.15mm "
7. Portrait measuring precision #
0.1mm "
8. Suitable to various work-piece surfaces "
9. Strong resisting interference ability: no influence aparting from laser scanning

plane 30mm under 200 Amp GTAW arc.
This sensor can capture the relative position information of the torch to cross
section of weld seam in real time, and obtain high precision profile of weld seam
cross section.
Autonomous control of welding robotic moving locus is one of basic indexes of
intelligentized robots. Realizing of welding robot moving locus can relax consis-
tent requirements for work-piece and dependence on welding clamps. In the
IWFMC, the moving locus of welding robot can be regulated in real time by laser
scanning sensing information and independent on off-line path programming or
teaching and playback.
4.2 Laser scanning-based identification of some typical joints
Using the above three-dimension laser scanning system, the detected results for
several typical seams, such as V-type groove, lap joint, butt seam, are shown as
Fig.6. The reticles in Fig.6 indicate welding position determined by measured in-
formation.
( ) * *
! " ' '
(a) V-type groove joint with little gap (b) lap joint
+ ,
- . # #
$ % & &
(c) butt seam in 1.5mm gap (d) butt seam in 0.15mm gap
Fig. 6 Detected results typical seams
Since identification of butt seam is affected by different gaps, usually, the

characteristic differences between small gap and no-gap butt are not evident, it is
very difficult to detect and identify the position of these joints. In general, the butt
131
seam is so various that it is easily confused with fillet and corner joint due to their
similarity. Therefore, some expert rules was used to determined the seam features.
5 Welding Robot Systems with Vision Sensing and

Intelligent Control of Pulsed GTAW Pool Dynamics
Up till now, no welding robot has provided with function of real-time sensing and
control of weld pool dynamics. Therefore, it is significant to realize an intelligen-
tized robot systems with vision sensing and real-time control of welding pool
process..
CCD
camera
Motor
Composed
filter system
Travel
direction
Reflecter
Torch Reflecter
Weldingdirection
Workpiece
(a). photograph of the visual sensor (b). Structure diagram of the visual sensor
(c) Installation of visual sensor and tracking sensor
Fig. 7 The systems of weld pool visual sensing and laser tracking sensors
132 S.B. Chen et al.
5.1 The vision sensing of the pool dynamical process
The photograph and the structure diagram of the visual sensor are shown in Fig.7
(a), (b). The visual sensor at the right and the laser scanning sensor at the left were
fixed with the end-effector of welding robot. In order to decrease the visual sensor
influences on pool images during movement of the torch, a two-step reflecting
light path system was adopted. Through a composed filter system , which con-
sisted of a neutral density filter and a narrow band filter, weld pool images and arc
light entered the CCD camera.
The vision sensor, image collecting board, the computer for image processing
and control of weld penetration and pool size, welding power and related inter-
faces composes the intelligent control subsystem for weld pool dynamic process.
5.2 Picking up dynamical features of pulsed GTAW pool images
Calibration of the image sensing system is omitted here for short.
Weld pool Character Parameters Definition

In order to investigate weld penetration and weld quality, two weld pool
parameters, topside maximum width Wfmax and topside half length Lfmax were
defined as Fig.8.
arc center
Lfmax
Wfmax
Fig. 8. Definition of characteristic parameters of weld pool during pulsed GTAW
Image Processing and Picking-up features of the weld pool images
Under the conditions in Table 1, welding experiment on the robotic system was conducted
and typical images are shown as Fig.9
Using image processing algorithms desigined in this paper, characteristic
parameters of weld pools were picked-up. Usually, robotic controller cannot
ensure that the angle between the direction of detecting and welding travel always
was accordant to the complicated seam. On the other hand, the angle was varing
during robot welding. Therefore, the algorithm for weld pool image processing
133
must be applicable for changing of observation direction during robotic arc

welding.
Table 1. The experiment conditions of pulsed GTAW
Pulse frequency f, Hz 1 Arc Length l, mm 3

Pulse duty ratio , %
! 45 T-pole dia. , mm
" 2.4
Peak current Ip, A 130 T-pole angle , ° # 30
Base current Ib, A 60 Flux of argon L, l/min 8.0
Traveling speed Vw, mm/s 2.0 Workpiece size mm×mm×mm 250×50×2
Fig.9(a) was an original image of weld pool captured at the base current time,
Fig.9(b) was an image processed through smoothing and filtering noise. The point
C in Fig.9(c) was determined as the arc center, The point T was determined as the
trail weld pool. Using edge picking-up algorithms, boundary points were deter-
mined as Fig.9(d), Fig.9(e) was a pool boundary approached by the quartic poly-
nomial. The pool character parameters were determined by Fig.9(f). The detail
image processing algorithms are omitted here.
(a) (b) (c) (d) (e) (f)
Fig. 9 Image processing and feature picking-cp of robotic pulsed GTAW pool
5.3 Controller for weld pool dynamic features during pulsed GTAW
Due to the complexity of weld pool dynamics during pulsed GTAW, we designed
a neuron self-learning Proportional-Summational-Differential (PSD) controller for
pulsed GTAW process, shown as Fig.10. Main merits of PSD controller are its
modeling-free of the complicated process and its adaptability. Using measured er-
rors between the actual outputs and the expected outputs, the PSD can realize
self-learning control of the process. The controlled variables of the system are
maximum width and half-length of the weld pool, the outputs of PSD controller
are pulse duty ratio and peak current respectively.
A neuron self-learning PSD controller’s output follows as
∆u (t ) = ω 1 (t ) x1 (t ) + ω 2 (t ) x 2 (t ) + ω 3 (t ) x3 (t ) (1)
Fig. 10 Diagram of a neuron self-learning PSD control system for pulsed GTAW
where u(t) is control input !
x i (t ) (i=1,2,3) are inputs of neuron, i.e.,

x1 (t ) = e(t ) !
x 2 ( t ) = ∆e(t ) and x 3 (t ) = ∆ e 2 (t ) . ω i (t ) (i=1,2,3) are weight-

ing coefficients corresponding to x i (t ) (i=1,2,3).
The weighting coefficients ω i (t ) (i=1,2,3) were modified during neuron
self-learning process, the learning rules follows as:
ω i (t + 1) = (1 − m)ω i (t ) + dri (t ) (2)
Here m, d>0 " d is learning ratio and ri (t ) is defined as
ri (t ) = z (t )u (t )[e(t ) + ∆e(t )] (3)

where z(t) is a teaching signal.
To ensure convergence and robustness of the learning algorithm, following

learning algorithms are adopted, and they compose controlling rules of single
neuron PSD, that is
3
∆ u ( t ) = K # ω ' ( t ) x i (t ) (4)
i=0
3
Where ω i' (t ) = # ω i (t )
i =1
$ω 1 (t + 1) = ω 1 (t ) + d I z (t )u (t )[e(t ) + ∆e(t )]
'
%ω 2 (t + 1) = ω 2 (t ) + d P z (t )u (t )[e(t ) + ∆e(t )]
'ω (t + 1) = ω (t ) + d z (t )u (t )[e(t ) + ∆e(t )]
& 3 3 D
The parameters dP, dI and dD are regulated according to the actual system.
135
5.4 Experiment of PSD control of weld pool dynamics during robotic

welding process
In order to verify effectiveness of PSD controller in robot systems, butt welding in

robotic pulsed GTAW process was conducted. The workpiece was dumb-
bell-shaped mild steel plate with 2mm thickness. The size and the shape of the
work-piece are shown in Fig.11. The varying curves of peak current and maxi-
mum width of the weld pool are shown as Fig.12. The peak value and duty ratio of
the pulse current were regulated by neuron self-learning PSD controller during
welding process. The Fig.13 is photos of controlled work-piece.
Fig. 11 Shape and the size of the work-piece
Fig. 12 Curves of neuron self-learning PSD control of

bead width during robotic pulse GTAW
(a) Topside (b) Backside
Fig. 13 Photographs of the PSD control of weld work-piece

6 Integration of Intelligentized Welding Robot Systems
The intelligentized welding robot systems has been integrated for uniformly ad-
ministering robotic collaborating movement, path and parameter planning and
simulation, guiding and tracking seam, visual sensing and intelligent control of
weld pool dynamic features. The integrated systems could be called as a welding
flexible manufacturing cell with some primary intelligent functions, or primary
intelligent welding flexible manufacturing cell (IWFMC).
6.1 The integrated structure designs of the IWFMC
Based on task characters of arc welding robot systems, the IWFMC was designed
in star-shape net framework, as Fig.14. It includes a robotic movement trajectory
control unit(RCU), a weld seam tracking unit(STU), welding pool penetration
control unit(PPU) and robotic welding power unit(WPU). The communication
and exchanging information among these units can be completed by the center
monitoring computer(CMC).
- - # # $ $
- - # # $ $
* * + + , ,
* * + + , ,
! ! " " ! !
! ! " " ! !
& & ' ' ( ( ) ) ) ) ( (
& & ' ' ( ( ) ) ) ) ( (
Fig. 14 The net topology structure of IWFMC
The information flow in IWFMC systems can be described as following:

1. Based on net communication techniques, robotic controller can down load and
run welding task program in real time from CMC, during welding process, the
robotic controller can regulate driving signals of joints by tracking sensing in-
formation at same time, the penetration unit will provide welding pool states
%
137
and welding current and speed information to weld power and robotic control-
ler through CMC.
2. The net system of IWFMC was designed by classified control mode and com-
bined with sensing system for realizing sharing control, fusing control and cen-
tralizing management by sensing information flow. The CMC can indicate, su-
pervise and correct welding process, path and parameter planning, multimedia
graph simulation, data from weld expert system and so on.
6.2 Petri net model for optimization of material flow and information
flow in IWFMC
The Petri net method in the discrete event dynamic system DEDS theory has
! "
very powerful token function for modeling, analysis, performance evaluation and
control of event elements and its logic sequence[18,31,39,20]. It is a significant re-
search direction to use Petri net method for optimizing and planning material
flow and information flow in welding flexible manufacturing cell/system[33,32] .
The discrete event and Petri net are very suitable to little batch and excessive
various welding products on Welding Flexible Manufacturing Cell(WFMC) and
can improve efficiency of WFMC and welding quality.
In order to assure welding product quality and satisfy intelligent requirements
for welding flexible manufacturing, a petri net model combined with sensing and
executing information during robotic welding process has been established by the
following rules:
Rule1: if input buffer is empty, then put workpiece to input buffer;
Rule2: if positioner is vacancy, then put workpiece to positioner from input
buffer;
Rule3: if tracking sensor sends regulating signal, then regulate robot running
locus;
Rule4: if pool feature is exceptional, then regulate weld current and speed;
Rule5: if welding is finished and output buffer is empty, then put workpiece
to output buffer;
Based on the above rules, an automatic Petri net (APN) model for IWFMC has
been established as Fig.15. There are two types of directed arc to be added in
common Petri net for APN model: inhibited arc, noting as , and enabling
! "
arc, noting as . The APN model is activated by exterior spring events,

! "
e.g., input information from sensors and action information from executors.
The APN model contains 18 places ps1, ps2, p1, p2, , p16 and 18 transi-
! "
tions t1, t2, , t18 , and its marks are explained as following:
! "
ps1/ps2 input/output sink, p1/p2: workpiece moving permit/prohibit (from sink to

$
buffer);p3/p4 positioner idle/busy; p5/p6 weld pool sensors (regulating sig-

$ $
nal)/( no regulating signal); p7/p8 tracking sensor (regulating signal)/ (no regu-
$
lating signal) ; p9/p10 workpiece moving permit/prohibit (from buffer to sink);

$
p11/p16 input/output buffer states; p12: workpiece ready p13/p14 welding

$
%
$
process start/end; p15 post- processing; w1/w2 weight value of inhibited arc.
$ $
A running process of the APN system can be described as following: In weld-

ing process starting, if input buffer p11 is empty, then put workpiece from the
input sink to input buffer; if the positioner is vacancy, then put workpiece to
positioner from input buffer p12 if workpiece is ready then start to weld
!
p13 during welding process, if weld pool sensor outputs regulating signals,
!
then to regulate welding parameters for control of weld penetration t14; if seam
tracking sensor outputs regulating signals, then to regulate manipulator to correct
moving locus t15 necessary processing after welding end p15 at the same
! !
time, if output buffer is empty p16, ; then down-load the welded workpiece to the
output buffer for delivering output sink. Thus whole welding process is com-
pleted.
Fig. 15 Automatic Petri net model of IWFMC
6.3 Functional Realization of composite IWFMC system
Based on IWFMC systems with exterior sensors, a running flow chart of the
IWFMC with model of welding automation Petri Net – WAPN is shown as
Fig.16.
139
The work-piece photos of visual tracking and coordinating control experiments

on IWFMC are shown as Fig.17.
System startup
Set-up welding state parameters
Download
off-line program
Guide to seam start TPPC
CCPC Get to the start?

monitor N MPPC
system Y
with
WAPN Run tracking system
WPPC
Run process control
VPPC
Get to the terminal? N
Y
Turnoff systems
Fig. 16 Running flow chart of WFMC systems with WAPN
Fig. 17 (a) Tracking experiment

Fig. 17 (b) Coordinating experiment
Fig. 17 Work-piece photos of the experiments on IWFMC
7 Conclusions
The current teaching and playback welding robots need to be improved by intelli-
gentized welding techniques for its effective application in practical production.
The intelligentized welding techniques for robot systems conclude the vision
sensing, guiding and tracking weld seam, programming path, pose and welding
parameters, knowledge modeling and intelligent real-time control of welding dy-
namics, optimizing supervision of welding flexible manufacturing cell/system, and
so on.
This paper has shown a primary intelligent welding robot systems with sepa-
rated sensing and control subsystems, so-called intelligentized welding flexible
manufacturing cell—IWFMC. The intelligent technologies contained in this sys-
tems will be a useful foundation for us to develop new-type intelligent welding
robot in the future.
141
Acknowledgments
This work was supported by National Natural Science foundation of China,

No.59635160 and Science and Technology Committee of Shanghai, China,
No.021111116.
References
1. B.F. Kuvin. Welding for Space. Welding Journal. 1992, 71(3): 41-44
2. Bob Irving. Sensor and Control Continue to Close the Loop in Arc Welding. Welding
Journal. 1999, 78(5): 31-36
3. C.G. Morgan, J.S. Bromley and P.G. Davey. Visual Guidance Techniques for Robot
Arc Welding. Proceedings of 3rd Conferences of Robotic Vision Sense Control, 1983:
pp615-619
4. E.J. Morgan-Warren. Remote Repair Welding in Nuclear Reactors. Welding & Mental
Fabrication. 1989, (8): 111-116
5. G. Grunwald, Modeling and Planning for Sensor Based Intelligent Robot Systems
(Task-Directed Planning of Cooperating Sensors: A Framework). Machine Perception
and Artificial Intelligence. 1995, 21: 143-159
6. Gerhard Teubel. Experience + Application Up-Date: Automation of A.W.-Operations
Using Robot-Technology. Welding in the World. 1994, 34: 75-84
7. J.D. Lane. Robotic Welding State of the Art. Robotic Welding—International Trends
in Manufacturing Technology. IFS (Publications) Ltd.UK. 1987: 1-10
8. J.E. Agapakis and J. Bolstad. Vision Sensing and Processing System for Monitoring
and Control of Welding and Other High Luminosity Processes. International Robots &
Vision Automation Conference, 1991: pp23-29
9. J. L. Pan, ” A survey of welding sciences in 21th century.” Proceeding of 9th Chinese
Welding Conference, Tianjun, China, Oct. 1999, (1): D-001-D-017
10. J.W. Barratt W.F. Clocksin P.G. Davey and A.R. Vialer. Visually Guided Robot
Arc-Welding of Thin Sheet Pressings. Proc. of 12th Int. Symp. on Industrial Ro-
bot.IFS(Publications) Ltd.UK.1987: 225-230
11. J.W. Kim and S.J. Na. A Self-Organizing Fuzzy Control Approach to Arc Sensor for
Weld Joint Tracking in Gas Metal Arc Welding of Butt Joints. Welding Journal. 1993,
Vol. 72(1): pp60s-66s
12. K. Andersen and G.E. Cook. Gas Tungsten Arc Welding Process Control Using Arti-
ficial Neural Networks. Proceedings of the 3rd International Conference on Trends in
Welding Research, Gatlinburg, Tennessee, USA, 1-5, June, 1992: pp135-142
13. Krister Brink, etc. Increased Autonomy in Industrial Robotic Systems: A Framework.
Journal of Robotic Systems. 1997, 19: 357-373
14. K. Masubuchi, .A.V. Gaudiano and .I.J. Reynolds Technologies and Practices of Un-
derwater Welding. Proceedings of The International Conference on Underwater Weld-
ing. Norway.1983, (6): 49-70
15. K. Oshima and M. Morita. Sensing and Digital Control of Weld Pool in Pulsed MIG
Welding. Transactions of the Japan Welding Society. 1992, Vol. 23(4): pp36-42
16. K. P. Valavanis, G. N., Saridis, Intelligent Robotics System: Theory, Design and
Applications. Boston, 1992: 12-18
17. L. Dimiter. Adaptive Robot Under Fuzzy Control. Fuzzy Sets and Systems. 1985, Vol.
17(1): pp23-28
18. L.E., Holloway, etc. Survey of Petri Net Methods for Controlled Discrete Event Sys-
tems. Discrete Event Dynamic Systems: Theory and Applications. 1997, 7(2): 151-190
19. L. Wu, K. Cui, S.B. Chen, Redundancy Coordination of Multiple Robotic Devices for
Welding through Genetic Algorithm, ROBOTICA, 2000,18(6), 669-676.
20. M.C. Zhou, Petri Net Synthesis and Analysis of A Flexible Manufacturing System
Cell. IEEE Transactions on Systems, Man and Cybernetics. 23(2): 523-531
21. Paul Drews., Gunther Starke. Welding in The Century of Information Technology.
Welding in the World. 1994, 34: 1-20
22. R.J. Beatlie, S.K. Cheng and P.S. Logue. The Use of Vision Sensors in Multipass
Welding Applications. Welding Journal. 1988, Vol. 67(11): pp28-33
23. R. Kovacevic, Y.M. Zhang and L.Li. Monitoring of Weld Joint Penetration Based on
Weld Pool Geometrical Appearance. Welding Journal. 1996, Vol. 75(10): pp317s-329s
24. R. Kovacevic and Y.M. Zhang. Neuro-fuzzy Model-Based Weld Fusion State Estima-
tion. IEEE Transactions on Control Systems Technology. 1997, Vol. 5(4): pp30-42
25. R.W. Richardson and D.A. Gutow. Coaxial Arc Weld Pool Viewing for Process
Monitoring and Control. Welding Journal. 1984. 63(3): 43-50
26. S.B. Chen, D. B. Zhao, L. Wu and Y. J. Lou, Intelligent Methodology for Sens-
ing , Modeling and Control of Pulsed GTAW :PART2—Butt Joint Welding,
Welding Journal, 2000, 79(6):164s-174s 21. S. Murakami. Weld-line Tracking
Control of Arc Welding Robot Using Fuzzy Logic Controller. Fuzzy Sets and Systems.
1989, Vol. 32(2): pp31-36
27. S. B. Chen, Y. J. Lou, L. Wu and D.B. Zhao , Intelligent Methodology for Sens-
ing , Modeling and Control of Pulsed GTAW :PART1—Band-on-Plate Welding,
Welding Journal, 2000, 79(6):151s-163s
28. S. Murakami. Weld-line Tracking Control of Arc Welding Robot Using Fuzzy Logic
Controller. Fuzzy Sets and Systems. 1989, Vol. 32(2): pp31-36
29. T.G. Lim and H.S. Cho. Estimation of Weld Pool Sizes in GMA Welding Process Us-
ing Neural Networks. Journal of Systems and Control Engineering. 1993, Vol. 207(1):
pp15-26
30. Timothy Hebert, etc. A Real-Time, Hierarchical, Sensor-Based Robotic System Ar-
chitecture. Journal of Intelligent and Robotic Systems. 1998, 21: 1-27
31. T.O., Boucher, etc. Petri Net Control on Automated Manufacturing System Cell. Ad-
vanced Manufacturing Engineering. 1990, 2(3): 323-331
32. T. Qiu, S. B. Chen, Y. T. Wang, L. Wu, Information Flow Analysis and Petri
Net-based Modeling for Welding Flexible Manufacturing Cell. SPIE Intl. Sympo-
sium on Intelligent Systems and Advanced Manufacturing. Vol.4192:449-456,
Nov. 6-8, 2000, Boston.
33. T. QIU, S. B. Chen, Y. T. Wang, L. Wu, Modeling and Control of Welding
Flexible Manufacturing Cell Using Petri Net, Journal of Materials Science &
Technology, 2001, 17(1): 181-182
34. Trailer, Manufacturer Depends on Robotic Welding to Boast Production. Welding
Journal. 1995, 74(7): 49-51
35. U. Dilthey, L. Stein. Robot System for Arc Welding—Current Position and Future
Trends. Welding & Cutting. 1992, (8): E150-E152
143
36. West Carrillton, Robot Assure Quality for Auto Seat Manufacturer. Welding Journal.
1995, 74(8): 57-59
37. Y. Kaneko. T. Iisaka and K. Oshima. Neuro-Fuzzy Control of the Weld Pool in Pulsed
MIG Welding. Quarterly Journal of the Japan Welding Society. 1994, Vol. 12(3):
pp374-378
38. Y. Li and J.E. Middle. Machine Vision Analysis of Welding Region and Its Applica-
tion to Seam Tracking in Arc Welding. Journal of Engineering Manufacture. 1993, No.
5: pp275-283
39. Y. Narahari, etc. A Petri Net Approach to the Modeling and Analysis of Flexible
Manufacturing System. Annals of Operation Research. 1985, (3): 449-472
40. Y. Suga and M. Naruse. Application of Neural Network to Visual Sensing of Weld
Line and Automatic Tracking in Robot Welding. Welding in the World. 1994, 34:
pp275-284
144
The Seam Tracking System

for Submerged Arc Welding
Hua Zhang Xuelei Ding Maohua Chen Benqi Da and Chunhua Zou
! ! ! !
Mechatronic Institute Nanchang University, Nanchang 330029, P.R. China

"
whqhzhang@public.nc.jx.cn
Abstract. The structure and the principle of seam tracking for submerged arc
welding were introduced in this paper. The method of seam recognition with com-
puter vision was mainly researched when the environmental light was changed. In-
telligent control system for seam tracking has been realized by using a self-
modulated fuzzy control set. The system was tested on the producing of spiral
submerged arc welded pipe for outside-welding and the primary results were ob-
tained.
1 Introduction
Since submerged arc welding has the advantages of high efficiency, well seam-
deform and without arc light interference, it’s widely applied in petrochemical in-
dustry and pressure vessel manufacture. Submerged arc welding is adopted in the
key engineering project “gas transportation from west to east” in China. Because
tracking accuracy is influenced by a number of factors such as changing of pipe
diameter, variation of image, flux leaking and so on, it’s not easy to control these
factors. So far an effective approach of seam tracking for outside welding is not
proposed yet and it is still in the stage of manual control. In practice, workers can
only judge seam position according the fact whether the light is just in the middle
position of seam. Seam tracking accuracy is influenced workers’ eye fatigue if
they watch the light point for a long time. How to get over above disadvantages,
increase tracking accuracy, reduce work intensity, replace manual control with
automatic control is the primary problem for researchers to solve.
2 Structure and Principle of Seam Tracking System
2.1 Framework of system
The standard parameters of this project are described as:

pipe : 1016×14.6, welding speed: 0.9 1.5m/min, welding voltage: 32 38v
!
" "
welding current: 390 1100A, pipe thickness: 7 10mm.

" "
145
The hardware of the seam tracking system consists of cross slide industrial con-
#
trol computer interface circuit image acquisition card(VIDEO-PCI-XR),

# #
CCD(Charge Coupled Devices) sensor, AC4064 I/O board and so on. The frame-
work of the system is illustrated in Fig.1.
I/O Interface Cross

$ %
Intelligent Control
Circuit Slide(Torch) Welding pipe
&
Vision
Image Processing Image Acquisition
Sensor(CCD)
Fig. 1. Framework of seam tracking system of submerged arc welding for outside-welding
2.2 Principle of system
The CCD sensor transfers seam images into video signals. And image acquisition
card transfers video signals into digital signals in the computer. Then all above
images are in the process of filtering, binarization, Hough transform according to a
software programmed with VC++6.0 seam error is obtained after these processing
in the end. Seam error is put into intelligent controller, digital quantity is reached
by a certain of algorithm ways. Cross slide is adjusted by the digital quantity
which is transmitted to analogue quantity through I/O interface circuit. The pur-
pose of precise tracking and error correction is obtained finally.
3 Method of Seam Recognition
3.1 Seam analyzing
First determine the processing region before image preprocessing. It’s not needed
that all the image region should be processed, because it has an impact on the real
146146H. Zhang et al.
time processing speed for occupying much more space of the memory and wasting
much processing time.
The purpose of the process is seam recognition. A region of 300×200(pixels) is
selected as the object of image processing to save memory space which contains
the seam information, enhance anti-interference capability and speed image proc-
essing. This region contains all the information, which is needed for seam recogni-
tion. The performance of system is enhanced according to this selection.
3.2 Process of filtering
To make it easy to differentiate feature line from background and reinforce feature
line, make the process of filtering. A template is used in the process of filtering;
this template is introduced as the following table.
Table 1. A template to reinforce vertical orientation
-1 2 -1
-1 2 -1
-1 2 -1
3.3 Process of binarization
Make the process of linear transformation similar to binarization, set gray level of
background zero, and gray level of feature line 255. f(x,y) represents the gray level
of original image. In term of some criterion get an appropriate threshold value ex-
pressed with T. g(x,y) representing gray level of processed image will be achieved
after segmentation ,it is described as following:
'255 f ( x, y ) ≥ T (1)
g ( x, y ) = (
) 0 f ( x, y ) < T
The threshold Selecting

To differentiate object and extract seam information from complex background,
the selection of threshold plays a key role in technique of threshold segmentation.
If the value of threshold is selected too big, it leads to that the object is judged as
background incorrectly. On the contrary, anti-results will be attained. So far there
are many methods of choosing threshold, the universal methods are histogram
threshold partitioning approach, two-dimension maximal entropy partitioning ap-
proach, fuzzy threshold partitioning approach and so on. Iterative threshold seg-
147
mentation approach is adopted in this paper. This approach is introduced as the

following:
(A) Determine minimal gray level expressed with ZL and maximal gray level
expressed with ZK, suppose the original value of gray level is
ZL + ZK (2)
TK =
2
(B) TK divides image into two parts ,one is object, the other is background. De-
termine the gray level of two parts ZO and ZB which are depicted respectively as
# Z (i, j ) × N (i, j )
z i, j < T K (3)
$ *
% +
% +
%% ++
Z = & ,
O # N (i, j )
z i, j < T K
$ *
% +
% +
% +
& ,
# Z (i , j ) × N (i , j )
z (i , j ) > T K (4)
ZB =
# N (i , j )
z i, j > T K
( )
Z(i,j) is the gray level of point (i,j), N(i,j) is the weight of the point (i,j), in
generally ,N(i,j)=1.0.
(C) Determine the new threshold TK+1, it is written as
(5)
Z +Z B
T K +1 = O
2
(D) If TK+1-TK then TK is the result. In this system

!
( ) *
+ , - # $ %
" & &
'
consider TK as the gray level of this image . Otherwise ,K ← K+1 &
go on carrying
out process (B).
3.4 Hough Transform
Seam information of the butt joint is extracted according to Hough transform after
binarization. The obvious advantage is high interference immunity. The essence of
Hough transform is to transfer many points to one point in two coordinate sys-
tems. Multiple points which represent a straight line or a well-regulated line in a
coordinate system is switched to a point in another coordinate system. This point
corresponds to parameters of original line. Make the process of Hough transform
is to classify some points as well as congregate all the points of a line. Then the
check of the feature of two-dimensional line can be converted to search for maxi-
mum in an array of one dimension. Perfect results can be achieved by means of
this approach even if there are just a little feature points of butt joint. It is proved
that this approach has high reliability and stability when the butt joint parameters
are changed greatly.
3.5 Result of seam recognition
It is complex in the producing spot of submerged arc welding for spiratron. There
are a great number of interferential factors and these factors are not stable ex-
tremely. So the primary task is to remove these disturbances.
(a) Original image (b) filtering result (c) binarization result (d) Hough transform result
Fig. 2. the process of image processing
Fig.2.(a) is the picture before image processing, which shows that there are a
lot of disturbances on the surface of work piece. According to above described
3×3 filtering template, the vertical orientation characteristic is emphasized and the
horizontal direction disturbances are removed effectively on the bottom of the im-
age, shown in Fig.2.(b). A method of selecting threshold automatically is adopted
in binarization, so seam feature changed constantly in the process of seam tracking
can be highlighted. In Fig.2.(c), background and seam object are distinguished ob-
viously after binarization which is the foundation of Hough transform. It is shown
in Fig.2.(c) that there is a little disturbance still. Fig.2.(d) shows that disturbance
has been removed and seam has been recognized through Hough transform.
Adjust torch and CCD before image processing. The center of image is exactly
oriented just before the torch. The center point represents the position of torch. At
this time the distance from this point to seam line is seam error, which is one of
the controller inputs. In this example the result indicates that the error is 12 pixels
after image processing, which is equal to 0.6mm as the resolution ratio of CCD is
0.05 mm/pixel. This time the real error is 0.8mm through accurate measurement.
So the processing error is 0.2 mm, the precision of recognition meets the require-
ment of seam tracking.
149
4 Fuzzy Controller of the Seam Tracking System
The welding process is subject to a great number of influencing factors, these fac-
tors are highly nonlinear and non-determinacy. It is so complex that classical con-
trol theory applied in automatic tracking system is not satisfactory. During recent
years fuzzy control has been improved increasingly in application of regulating
speed and tracking accuracy in seam tracking control system. But the application
range of simple fuzzy control is limited. Enough high regulating speed can be
achieved when error is smaller, however regulating speed can’t be amended ac-
cording to the changing of error when error is larger. A conclusion can be drawn
that simple fuzzy control is difficulty to meet the requirement of welding in reality
yet.
Fuzzy-p control theory is adopted in this seam tracking control system in this
paper. The concept of Fuzzy-p is expressed as following: Proportional control is
adopted to improve response speed and reduce regulating time when error is lar-
ger, Fuzzy control is adopted to improve the stability and control precision of re-
sponse and decrease system overshoot when error is smaller. Switch between two
control approaches is realized by designating threshold in advance. It is expressed
simply as
Proportional control if |E| EP !
Fuzzy control if 0 |E| EP is the absolute value of error

/
,
"
- . . 0
A great number of experiments approved that the effect of tracking are satisfy-
ing when partition threshold EP is 1mm,then select EP 1mm.
It is key to establish rules of fuzzy control in fuzzy control theory. In formula
method of fuzzy control, a universal mathematical model is
U 2
1 E+(1- )EC
1 (6)
Where E or e is the fuzzy quantity of error, EC or is the fuzzy quantity of er- 3 #
ror variance ratio, U is the output fuzzy quantity, is the adjustment factor. re- 1 1
flects the weight of e and it vary from 0 to 1. The primary contradiction of

3 # $
control system is to remove e when e is large relatively, this time should be in- 1
creased to enlarge the proportion of e weight in the control rules to improve dy-
namic characteristics of the system. On the contrary, when e is small the primary
contradiction is to restrain system overshoot to make system stable as soon as pos-
sible. So should be decreased, meanwhile ,1
1 is increased to enlarge the propor- % 1
tion of 3weight in the control rules. The rules of fuzzy control can be modified
#
through amending . Once is fixed, the rules of fuzzy control will be defined.
1 &
The situation of seam error and working state is varied rapidly due to the influence
of welding condition. The fixed fuzzy control can’t meet the demand of seam
tracking. In the practice, is obtained through table lookup. &
When is defined, control rules self modulated online can be realized accord-
&
ing to EC. Fig.3 describes the principle of self-modulated fuzzy

* +
/ )
' ( & - & 0
control seam tracking system.

checking
comparision
parameter
self-tune
! "
K1
'
fuzzifica inference defuzzifi

(
-tion Ec -cation K3 object

ec $ % &
engine
d / dt K2
Fig. 3. Structure of self- modulated fuzzy control seam tracking system
5 Conclusions
This system responding quickly can recognize seam for almost all kinds of distur-
bances, its accuracy rate is high. The resolution ratio of CCD is 0.05 mm/pixel.
The application of self- modulated fuzzy controller improves the control charac-
teristic. It meets the need of automatic positioning and automatic tracking as well
as easy to manipulate. The primary results have been obtained in the producing
spot. It is shown in the results that the tracking accuracy of this system can meet
the requirements of outside welding for spiratrons.
References
1. Gao Xiangdong et al. Application of Network Controller in the Network Controller in

the Seam Tracking of Arc-welding Robot. Control Theory and Application, 1999,
Vol.16, No.3
2. Kim J W, Na S J. A Self-Organizing Fuzzy Control Approach to Arc Sensor for Weld
Joint Tracking in Gas Metal Arc Welding of Butt Joint. Welding Journal. 1993.72
3. Zhang Hua, Xian An, Peng Shaopin et al. Inner Weld Auto Tracking and Fusion
Control System of Spiral Pipe. Welding Pipe.2002.1Vol.25, No.1
4. Huang Shisheng, Gao Xingdong, Yu Shiwei. Study On An intelligent Weld Seam
Tracking System, Chinese Journal of Mechanical Engineering. 1999(12), Vol.35,
No.6
151
5. Hu Shengsun, Hou Wenkao, Qin Baozhong. Study on The Fuzzy-P Controller With
Self-modulation Level Factor in Seam Tracking System. Journal of Tianjin Univer-
sity. 1999(3), Vol.32, No2
6. Wang Xiuyuan Hunag Shisheng, Xue Jiaxiang, et al. Application of Image Process-
!
ing on The Image Segmentation in Weld Detection. Electric Welding Apparatus.

2000(10), Vol.30, 32~34
152
Visual Measurement of Path Accuracy

for Laser Welding Robots
Dong Du, Bo Sui, Y. F. He, Qiang Chen, and Hua Zhang
Department of Mechanical Engineering. Tsinghua University. Beijing 100084.

P.R. China
dudong@mail.tsinghua.edu.cn
Abstract. A sensing method based on binocular vision theory is presented to

measure the path accuracy of the robots for laser welding. This method uses two
cameras fixed on the end-effector of the industrial robot to capture the images of
two marked-lines in the workspace. Based on line matching technology and stereo
vision theory, a mathematic model has been set up to calculate the space coordi-
nates of the two cameras from the captured images. This method is used to meas-
ure the path accuracy of a 6-DOF industrial robot. The results indicated that this
method meet the requirement of continuous path measurement of industrial robots
and the detection can be accomplished easily with little reference datum needed.
1 Introduction
Industrial robot has been widely used in the application of laser welding recently.
In these fields, the path accuracy of the robots is the key problem. But few types
of equipment can be used to measure the 3-D location of the end-effector when the
robot is running, and those available equipments are very expensive. An instru-
mentation that can test the variation of path accuracy for industrial robots in a
brief way must be developed.
Two kinds of measuring methods, which are contact and non-contact method,
have been used to measure the position accuracy of industrial robots. However,
the contact method cannot be performed continuous path measurement of dynamic
trajectory accuracy, as the measuring speed and measuring scope of this kind of
methods are very small. In non-contact method, there are two kinds of modes,
which are initiative mode and passive mode. The initiative measuring system send
signal such as infrared, laser or ultrasonic to the object, and then cipher-out the lo-
cation of the object by detecting the returned signal from the object [1,2]. To work
with this system a very complex and expensive device, difficult to operate, must
be set up. So the passive method is considered recently [3-5].
In this article, a binocular vision system using two cameras fixed on the end-
effector of the industrial robot has been setup to measure the path accuracy of in-
dustrial robots. To work on this system, the stereovision theory and the analytic
153
geometry equation in 3-D space have been studied. Then the projective plane
equation of a spatial line with a CCD (Charge Coupled Device) camera has been
introduced to get the mathematic model of this system, which is based on line-
match technology. The path accuracy of an industrial robot has been measured
simply and conveniently using this system, and the measuring results match the
demands of continuous path measurement for the laser-welding robot.
2 Camera Projection Model
The working process of a CCD camera is usually described by pinhole model (Fig.
1). Three coordinate systems have been used in this model. One is the world coor-
dinate system represented by (XW, YW, ZW), one is the camera coordinate system
(xc,yc,zc), the other is the image pixel coordinate system (u, v). The relationship be-
tween the retinal image coordinate and the world-coordinate can be written as
Eq.2.1.
$XW +
$u + %Y ,
% ,
zc v == [ L ⋅ F 0 ] M1 %
W ,
= MX W (2.1)
% , % ZW ,
&%1 -, % ,
& 1 -
$f 0 0+
$ R t+ %
Where , M 1 = % T , , M = [ L ⋅ F 0] ⋅ M1 , F = 0 f 0, ,
&0 1- % ,
%& 0 0 1 ,-
$1 +
%d 0 u0 ,
% x ,
% 1 ,
L=% 0 v0 ,
dy
% ,
%0 0 1,
% ,
& -
dx and dy is the physical distance between two adjacent pixels. u0 and v0 is the
lower case of the pixel corresponding with optical axis. R and t is the rotation and
translation matrix mapping from the world coordinate to the camera coordinate.
M1 is a 4×4 matrix, mapping from the world coordinate to the camera coordinate
system. M is the perspective projection matrix that accounts for both the extrinsic
parameters of the camera and the intrinsic parameters of the camera.[3]
154154D. Du et al.
YC
YW
XC
ZC XW
ZW
Fig. 1. Pin hole model
3 Visual Measurement Model for Path Accuracy
Two cameras, both focused on the lines marked in the world coordinate system,
are used in this measuring instrumentation. When the robot is moving, the relative
position of the marked-lines and the cameras will be changed, which can affect the
images captured by the CCD in a definite sampling rate. Consequently, the estima-
tion of the path accuracy is to calculate the relative position, which changes in one
sampling cycle time.
3.1 Projection model of a spatial line
Fig. 2 shows the relationship between a spatial line and its projection in the image
plane.
YC
O
ZC XC
image plane
l
N
ZW
L
YW
XW
Fig. 2. A spatial line and its projection

155
Suppose the projection l of a spatial line L in the image plane is expressed as an

equation:
KU=0
Where K = ( k1 , k 2 , k3 ) , U = (u , v ,1)T
With reference to Eq.2.1, we can get the following equation by premultiplication
the vector K.
$ XW +
$u + % ,
$ R t + YW
[k k2 k3 ] %v , = [ K ⋅ L ⋅ F 0] % T , % , = 0 (3.1)
1
% , &0 1- % ZW ,
&%1-, % ,
&1 -
S is used to represent the production of K, L and F. thus Eq.3.1 can be written as
SRXW + St = 0 (3.2)
Eq.3.2 describes the characteristics of the projection plane W, which is con-
structed by the optical center of the camera lens O, space line L and its projection
line l. The normal vector N of the plane W can also be obtained from this equation.
This vector is a 3×1matrix whose directional number is the production of S and R.
This equation is the key to calibrate the camera.
3.2 End-effector position estimation
The projection of two space lines in two CCD cameras is shown in Fig. 3.
Fig. 3. Binocular visual measurement model

156156D. Du et al.
Assumption:
CCD1 and CCD2 are fixed on the end-effector of the robot by rigid tools. The ro-
tation and translation from the world coordinate to the two camera coordinates are
described as (R1, t1) and (R2, t2) respectively; the CCD1 coordinate to CCD2 coor-
dinate is described as (R3, t3). Then, because
' X CCD1 = R1 X W + t1
*
( X CCD2 = R2 X W + t 2
*X
) CCD2 = R3 X CCD1 + t3
The relationship between (R1, t1), (R2, t2) and (R3, t3) can be described as follows.
R2=R3R1, t2=R3t1+t3 (3.3)
There are two marked-lines L1 and L2 in the world coordinate, which are linearly
independent. The directional number of them are (m1, n1, p1) and (m2, n2, p2), re-
spectively.
The parameter of the first sample point (R0, t0) and world coordinates of A1 and
A2 have already been calculated out by the calibration procedure. The rotation and
translation of the end-effector from sampling point i to sampling point i+1 is de-
scribed as (∆Ri, ∆ti) in the world coordinate system. Then the parameter of the
number i+1 (Ri+1, ti+1) can be calculated from the following relationship.
CCD1:
Ri+1,1=∆RiRi,1, ti+1,1= ∆t i+ ti,1 (3.4)
CCD2:
Ri+1,2=R3Ri+1,1=R3∆RiRi,1, ti+1,2= R3ti+1,1+ t3= R3∆t i+ R3 ti,1+ t3 (3.5)
The estimation of the path accuracy of the robot is to calculate out the numerical
value of rotation and translation matrices (∆Ri, ∆ti). To achieve this estimation,
some geometry constraints should be considered.
1) Perpendicular constraint: The inner product of the normal vector N and projec-
$$$#
tion line OA is nil Showed in figure 4. For example, that the vector Ni+1,11 is per-
pendicular with vector L1 in figure 4, thus the inner product of Ni+1,11L1 is zero.
' N i +1,11 L1 = Si +1,11 Ri +1 ,1 L1 = 0 ' Si +1 ,11 ∆ Ri Ri L1 = 0

*N L = S R L = 0 *S R ∆ R R L = 0
* i +1 ,21 1 i +1 , 21 i +1 ,2 1 * i +1,21 3 i i ,1 1
( #( (3.6)
* N i +1 ,12 L2 = Si +1,12 Ri +1,1 L2 = 0 * Si +1,12 ∆ Ri Ri ,1 L2 = 0
*N *
) i +1 ,22 L2 = Si +1 ,22 Ri +1,2 L2 = 0 ) Si +1,22 R3 ∆ Ri Ri ,1 L2 = 0
2) In-plane constraint: The extreme points A1 (XWA1) and A2 (XWA2) must match
the equation of the projection plane, which is constructed with marked-line L and
its projection l.
157
' Si +1,11 Ri +1,1 X WA1 + Si +1,11 ti +1 ,1 = 0

*
* Si +1,12 Ri +1 ,1 X WA2 + Si +1,12 ti +1 ,1 = 0
( #
* Si +1,21 Ri +1 ,2 X WA1 + Si +1,21 ti +1,2 = 0
*S
) i +1,22 Ri +1 ,2 X WA2 + Si +1,22 t i +1,2 = 0
'Si+111
, ∆ Ri Ri ,1 XWA1 + Si+111
, ( ti ,1 +∆ ti ) = 0
*
, ∆ Ri Ri ,1 XWA2 + Si +112
*Si+112 , ( ti ,1 +∆ ti ) = 0
( (3.7)
*Si+1,21 R3∆ Ri Ri ,1 XWA1 + Si+1,21 [R3 ( ti ,1 +∆ ti ) + t3 ] = 0
*S
) i+1,22 R3∆ Ri Ri ,1 XWA2 + Si+1,22 [R3 ( ti ,1+∆ ti ) + t3 ] = 0
From the Eq.3.6 and Eq.3.7 we can get eight linearly independent equations,
where only the matrices ∆Ri and ∆ti are unknown and need to be figured out. In
these equations, Si+1is made up of matrix K and the intrinsic parameters of CCD;
the former can be obtained from the image-reorganization of lines; the latter can
be achieved from the calibration of the CCD.
If we use γ, β, θ to represent the rotation along X, Y, Z-axis, the ∆Ri, ∆ti can be
written as
$cγ ⋅ cβ cγ ⋅ sβ − sγ ⋅ cθ cγ ⋅ sβ ⋅ cθ + sγ ⋅ sθ + $ ∆ti , x +
% , % ,
∆Ri = %sγ ⋅ cβ sγ ⋅ sβ + cγ ⋅ cθ sγ ⋅ sβ ⋅ cθ − cγ ⋅ sθ , , ∆ti = % ∆ti , y , (3.8)
%& −sβ cβ ⋅ sθ cβ ⋅ cθ ,- %& ∆ti , z ,-
where s- and c- is the abbreviation of sine and cosine.

There are 6 unknown variables that must be calculated in these eight equations.
To solve this problem, least square method is used to get an optimized value.
Thus, the deviation of the sampling point i+1 to sampling point i is achieved.
4 Experiment and Results
An experiment has been carried out to prove the feasibility of this measuring
method. The experimental system is made up of a calibration base, two CCD cam-
eras (resolution 800×600), some fixing tools and an industrial robot, just like fig-
ure 4. Two lenses, whose focal lengths are about 50mm and object field angles are
about 5.5 degree, are used in the cameras. There are two unparallel lines marked
in the calibration base. These lines are very brightness in the black background
and easy to be imaged by the cameras. The precision of the system is tested to be
little than ±0.2mm. In a common laser-welding application, the deviation of the
welding path is not allowed to be above ±0.5mm. Thus, this system can be used to
158158D. Du et al.
measure the path accuracy of industrial robot, which is used in laser-welding ap-
plication. A 6-DOF (Degree of freedom) articulated robot, whose reach radius is
about 1300mm and maximum payload is about 6kg, is measured. The measuring
results are list in figure 5 and 6 (the figures of rotation deviation along y-axis and
z-axis are elided). From the results, the higher the moving speed, the worse the
path accuracy of the robot. The path position accuracy of this robot is more than
4.05mm. This measuring result is proved to be credible by a contact measurement
method.
Industrial Robot
CCD Camera #1
Fixing Tools
CCD Camera #2
Marked-lines
Fig. 4. System setup
Fig. 5. Measuring results of position error of industrial robot in different moving velocities
159
11.22m/min
45.00m/min
2.4 90.00m/min
α/°
2.2
Rotation deviation along x-axis

2.0
1.8
1.6
1.4
0 5 10 15 20 25 30
Number
Fig. 6. Measuring results of orientation error along x-axis in different moving velocities
5 Conclusion
The measurement method of path accuracy for laser welding robot has been pre-
sented. Using this method, the path accuracy of industrial robots can be achieved
easily and conveniently. The measurement of a robot moving along a path of
straight line is performed successfully. The experimental result demonstrates that
the measuring method for industrial robots is capable of measuring path accuracy
of laser-welding robot.
Acknowledgements
This work is Supported by the Teaching and Research Award Program for Out-
standing Young Teachers in Higher Education Institutions of MOE, P.R.C. and
the National Nature Sciences Foundation of China ( No. 59975050).
References
1. Shirnzadeh Bijan. (1998) Laser-interferometry-based tracking for dynamic measure-

ment. Industrial Robot, 25(1): 35–41.
2. Yuan Jing, Yu S L. (1999) End-effector position-Orientation Measurement. IEEE tran-
sition on Robotics and automation, 15(3): 592–595.
3. Ma Songde. (1996) A self-calibration technique for active vision systems. IEEE Trans-
actions on robotics and automation, 12(1): 114-120
160160D. Du et al.
4. Motta JMST, Carvalho GC, McMaster RS. (2001) Robot calibration using a 3D vision-
based measurement system with a single camera. Robotics and Computer Integrated
Manufacturing, 17(6):487-497
5. Gordon Wells, Carme Torras. (2001) Assessing image features for vision-based robot
positioning, Journal of Intelligent and Robotic Systems, 7(1):95-118
Map Building and Localization for Autonomous
Mobile Robots
Y.L. Ip, A.B Rad, and Y.K. Wong
Department of Electrical Engineering, The Hong Kong Polytechnic University,

Hung Hom, Kowloon, Hong Kong SAR, China.
eeabrad@polyu.edu.hk
Abstract. In this chapter, we present our recent works on map building and
localization algorithms for autonomous mobile robots. We start with the
segment-based map building via enhanced adaptive fuzzy clustering technique.
Based on this algorithm, we develop a dynamic segment-based algorithm in
Bayesian state framework. A localization algorithm will be developed in the second
part of the chapter. We propose a model-based localization algorithm based on
fuzzy-tuned extended Kalman filter. These algorithms will be tested in real-time
applications and scenarios.
1 Introduction
Many industrial applications require autonomous mobile robots to be able to

perform their tasks in partially known or unknown environments with little a priori
information. This requirement dictates that the mobile robot is “aware” of its
surroundings. It should be able to build an initial map, be able to maintain and
update the map and finally to be able to use this map in order to determine its
position. The purpose of this chapter is to discuss these issues and suggest some
algorithms for map building and localization. The problem of simultaneous map
building and localization (SLAM) will not be discussed here.
Both map building and location algorithms rely on the perception device. This
could be in the form of vision, radar, lasers and ultrasonic sensors. We limit our
discussion to ultrasonic perception device. The algorithms described here can be
expanded to cover these perception devices with not much difficulty.
There are several approaches to solve the map building problem. Among those
are occupancy grid or geometric maps. The occupancy grid maps are generated
from stochastic estimates of the occupancy state of an object in a given cell. It is
rather easy to construct and maintain them with reasonable computing power in
small workspace. However, in large environments the storage space and the
processing of the maps become matters of concern. This study focuses on geometric
methodology. Models based on geometric boundaries approximate the location,
orientation and size of the boundaries and are applicable to environments
162162Y.L. Ip, A.B. Rad, and Y.K. Wong
susceptible to inaccurate and/or noisy data. The object boundaries can be

approximated with straight lines or curves from additional processing of the sensor
measurements.
The rest of this chapter is organized as follows: Fuzzy clustering and its
associated algorithms are reviewed in Section 2. A segment-based map building
algorithm is reported in Section3. This algorithm is extended to dynamic case in
Section 4. In Section 5, we discuss a localization method based on fuzzy-tuned
extended Kalman filter. Experimental results are included in Section 6. The chapter
is concluded in Section 7.
2 Fuzzy Clustering Algorithms

The fuzzy clustering algorithms have been used in many applications involving data
segmentation. The basic fuzzy clustering algorithm is the fuzzy c-means (FCM)
method developed by Bezdek (Bezdek 1981). FCM is aimed at minimizing the sum
of the squared distances between the data points and the c cluster centers Vi. The
objective function to be minimized is
cp
J m : M × R → R+
fc
n c m 2 c
J m ({U ik }, {Vi }) = # # (U ik ) (d ik ) | # U ik = 1 , k = 1,2,...,n
k =1 i =1 i =1 (1)
where, Mfc is the fuzzy partition space; U∈Mfc is a fuzzy c-partition of X;

V={V1,V2,…,Vc} are the cluster center vectors, Vi is a cluster center of p features,
Vi∈ℜ p ; X is the matrix of feature vectors, where X={ X1,X2,…,Xn }, Xi∈ℜ p; and dik
is given by
2 2
( d ik ) = x k − vi (2)
Here, n is the total number of feature vectors (data points) and c is the number of
cluster centers. The index i denotes each clusters, index k denotes each of n number
of data points and p is the dimension of ℜ space. The exponent m∈[1,∞) is used to
adjust the weighting effect of the membership value. The cluster centers vi can be
obtained from the following:
n m
# (u ik ) x k
k =1
vi = n , i = 1,2,..., c (3)
m
# (u ik )
k =1
The fuzzy membership matrix U can be obtained from the following:
1
u ik = , i = 1,2,..., c ; k = 1,2,..., n
2
c d ik (4)
# ( ) −1
m
j =1 d
jk
if dik = 0 then uik = 1 and ujk = 0 for j ≠ i.
163
The cluster center matrix V is randomly initialized and the fuzzy partition matrix
U is created by using Eq. 4. The initialization of V is done by randomly selecting a
value for each feature of a cluster center vi within the range of values exhibited by
the feature in the data set. The stopping condition is set to ustop. The theoretical
details and the proofs of the above are well documented and can be found in
(Bezdek 1981). This FCM can not be used directly for line segment detection in a
data set. It is more suitable for feature detection of linear shapes. For our purpose, it
is best to directly detect the line segment within the data set space. There are several
variations of FCM that allows detection of line segments, such as: Fuzzy
c-elliptotypes (FCE) algorithm (Bezdek 1981; Runkler and Palm 1996), Adaptive
fuzzy clustering (AFC) algorithm (Dave 1989; Phansalkar and Dave 1997).
2.1 Noise Clustering
Noise is a common problem in many environments. The data set obtained by the
ultrasonic sensors includes arbitrary distribution of noise and creates numerous
problems for minimization of squared error type algorithms. Noise Clustering (NC)
was proposed in (Dave 1991) and aims to alleviate this problem. In NC algorithm,
the noise is considered as a separate cluster which contain data that cannot be
classified into other clusters. Noise prototype is defined as a universal entity such
that it is always at the same distance from every point in the data set. From (Dave
1991), the definitions of the noise distance δ is defined by the following equation:
' c#−1 #n
( d ik )
21
2 ( 2
δ = ζ ( i =1 k =1 2 (5)
( n(c − 1) 2
) 3
where ζ is the value of the multiplier used to obtain δ from the average of
distances. Based on Eq. 5, δ can be calculated at each iteration of the sequence.
The details for the EAFC algorithm along with the NC technique are discussed in
next section.
2.2 Enhanced Adaptive Fuzzy Clustering algorithm (EAFC)

integrated with NC technique
The main feature of our proposed approach is to divide the AFC algorithm into two
phases. Phase I is a standard FCM iterative process whereas phase II is an AFC with
the cluster centers initialized by the last cluster center values from phase I. The
function of phase I is to select the cluster centers within the data set by the standard
FCM algorithm. After obtaining the cluster centers in the data set, the phase II
(AFC) is used to calculate the line segments. The previously calculated cluster
centers are used for initialization of the fuzzy partition matrix of AFC to obtain the
cluster with linear prototypes (line segments in our case). In this algorithm, the type
of distance norms is Euclidean. Let ustop_I and ustop_II be the stopping condition in
phases I and II respectively. The outline of the proposed algorithm shown as
following:
Phase I
1. Fix c, 2 ≤ c < n; fix 1 ≤ m < ∞; initialize the cluster center matrix V by using a
randomly selected entry in the data set; initialize the fuzzy partition matrix
U(ρ = 0) with Eq. 4.
2. Increment ρ = ρ +1. Compute the cluster center vi by Eq. 3.
3. Update membership function U(ρ = ρ+1) by using Eq. 4, along with dik
calculation by Eq. 2.
4. If |U(ρ) - U(ρ-1)| < ustop_I then stop, else repeat steps 2-4.
Phase II
1. Use the same c as select in Phase I, fix 1 ≤ m < ∞, initialize the cluster center
matrix V and the fuzzy partition matrix U by using results from Phase I,
initialize mixing coefficient αi, 0 < αi <1.
2. Increment ρ=ρ+1. Compute cluster center vi by using modified Eq. 3, defined
below:
n m
# (u ik ) x k
k =1
vi = n , i = 1,2,..., c - 1 (6)
m
# (u ik )
k =1
3. Compute the fuzzy scatter matrix using Eq. 7 and then obtain their
eigenvalues to calculate theαi by using Eq. 8, defined below respectively.
n m Τ
S fi = # (u ik ) ( x k − vi )( x k − vi ) , i = 1,2,..., c - 1 (7)
k =1
τ
αi = 1 − ( i ) (8)
ξi
where τi = min(λ1i, λ2i) and ξi = max(λ1i, λ2i), i = 1,2,…,c-1.
4. Update membership function U(ρ = ρ+1) by Eq. 9, along with γik calculation by
Eq. 10 and with δ calculation by Eq. 5, defined below.
1
u ik = , i = 1,2,..., c - 1 ; k = 1,2,..., n
2
c γ ik m −1 (9)
# ( )
j =1 γ
jk
2 *-(1 − α ) d 2 + α D 2 , i = 1,2,..., c - 1
(γ ik ) = + i ik i ik (10)
-, 2
δ ,i = c
where Dik is the distance of the point from the line prototype, and is
given by
165
( Dik ) 2 = (d ik ) 2 − (< xi − vi , d i >) 2 (11)

(ρ) (ρ-1)
5. If |U -U | < ustop_II then stop, else repeat steps 2-5.
3 Segment-Based Map Building via EAFC with NC for

Static Environment
In the above section, we provide a brief outline of the Enhanced Adaptive Fuzzy
Clustering Algorithm with noise clustering approach (EAFC with NC). The main
contribution of this algorithm is to extract the line segment from the noise sonar
measurements. In this section, we will introduce how to obtain the environment
model during the robot traveling in real time. This algorithm is based on the
recursive updating of the line segments to extract the features in the map and is
based on the following sequence of steps, that analyzes the uncertainly in sensor
information with different levels:
• As the mobile robot moves in the environment, the consecutive observations
n
obtained by the sonar sensors S is given by
n n n n n n n
S = {( x1 , y1 ), ( x 2 , y 2 ), # , ( x k , y k )}, n = 1, 2, # ,16 (12)
Here, n is the number of sonar sensors and k is the number of measurements at
each sampling interval.
• The world environment is modeled into two-dimension and divided into
squared cells C ( X ,Y ) (local models) (see Fig. 1) to reduce the complexity of
data structure. During the robot sampling, the sonar measurements from the
environment will transform to global coordinate system. At the same time, the
sonar data points are grouped in the related cells C ( X ,Y ) . The cell C ( X ,Y ) is
defined by
C ( X ,Y ) = {( x1 , y1 ), ( x 2 , y 2 ), # , ( x % , y % ), B1 , B 2 ,..., Bi } (13)
where X and Y are the cell location in the global frame X-axis and Y-axis,
respectively. Index % denotes the number of sensor measurements within the
cell. The ith extracted line segments from the corresponding sonar data points
is given by
Bi = {φ , ( x S , y S ), ( x E , y E ), ( xV , yV )} (14)
where φ is the unit-eigenvector of the segment and the end-points of the
segment, ( x S , y S ) and ( x E , y E ) are obtained by the EAFC with NC
approach. ( xV , yV ) is cluster center of this segment. The cell C ( X ,Y )
contains a discrete sample of the outline of the feature within the cell.
(a)
(b)
Fig. 1. a) Grouping the sensor measurements in cell C ( X ,Y ) . b) Updating the sensor
measurements and line segments simultaneously.
• To extract the line segments in each cell C ( X ,Y ) by the EAFC with NC and
store in Bi.
• Finally, the similar line segments within a cell are combined together by the
compatible line segment merging technique (Ip et al. 2002). It is proposed by
the authors’ pervious work. The contribution of the “compatible line segment
merging technique” is to merge the similar basic segments together to form a
single line segment. The basic segments are discontinuous outlines of the
object boundaries in the vicinity of the mobile robot trajectory (Fig. 1). In
order to improve the quality of the map, the number of line segments
representing the same boundary is merged up by evaluates the relation of the
eigenvector in both segments within the cell. In the proposed map building
algorithm, the compatible line segment merging technique is implemented in
two routines: “local grouping” and “global grouping”. When two segments
(i.e. 2 clusters in each cell) are extracted, we apply the compatible line
segment merging technique to merge those segments. This routine is called
“local grouping”. In “global grouping” routine, the cells are combined by the
compatible line segment merging technique based on the pervious updated
cells as the center and merging the updated cells with the surrounding cells.
Two routines are shown in Fig. 2. The merged segments and the location of
the cells are stored in the “global buffer” for display and global map
information. However, the segments in the cells are still stored in the local cell
memory. This information is saved for the next updating since the pervious
167
Fig. 2. The operation of “Local Grouping” and “Global Grouping” routines.

cell information is used for initialization in EAFC with NC. Since the EAFC
with NC segment extraction scheme is proposed by the authors’ pervious
work. The theoretical details and discussion of the above are well documented
and can be found in (Ip et al. 2002).
3.1 Outline of the proposed On-line Segment-based Map

Building algorithm
Task 1: Initialization. Fix the number of clusters c, the stopping condition ustop_I
and ustop_II for each cell. The size of each cell is required to fix.
Task 2: To obtain the sonar sensors measurements S n from the environment
(global frame) and transform to global coordinate system.
Task 3: To group the sensors measurements into the related cells C( X ,Y ) .
Task 4: To extract the line segments in each cell by EAFC with NC approach
and then store it in form of Bi in Eq. 14. The information in Bi (i.e.
cluster centers ( xV , yV ) and mixing coefficient α ) is used for the
initialization of EAFC with NC.
Task 5: “Local grouping” routine: to combine the similar cluster segments in
each cell by compatible line segment merging technique. Store all the
update cells in the buffer for “global grouping”. Repeat Task 2 to 5 for
next sampling interval. We use Tlocal to denote the sonar sensors
sampling interval.
Task 6: “Global grouping” routine: We use Tglobal ( Tglobal > Tlocal ) to denote the
“global grouping” sampling interval. We use updated cells (the updated
cells are stored in the buffer in Task 5) as a center to merge the
surrounding cells. The merged segments and cells location are stored in
“global buffer”.
Once the mobile robot finishes the navigation in the workspace, the final map
information is selected from each cell and the “global buffer”. If the cells are
overlapped, we select the segments in “global buffer” instead of that in the cell.
4 Dynamic Segment-Based Map Building via Bayesian

State Estimation Framework
In the pervious section, our proposed segment-based map building via EAFC with
NC for static environment is introduced. In this section, we will extend the proposed
map building algorithm to handle a dynamic environment. The definition of
dynamic case in this paper is stated as: the object in the modeling environment is not
fix and it can be relocated by the human.
169
4.1 Estimating the state of segments
We use the segment-based map building with sonar sensors to extract the object in
the environment. We propose using Bayesian update rule to estimate and update the
state of segments. Each extracted segment is divided into j parts. Each part is
assigned a probability function of its state s ( Bij ) which is conditioned over r
observations (sonar sensors measurements) within Bayesian framework. The Bij
denotes the jth part at the ith extracted segment in the related cell C( X ,Y ) . The state
variable s ( Bij ) associated with a part of segment Bij is defined as a discrete
random variable with two states, present and absent, denoted PRESENT and
ABSENT. Since the states are exclusive and exhaustive,
P [ s ( B ij ) = PRESENT ] + P [ s ( B ij ) = ABSENT ] = 1 . To determine how a
sensor reading is used to estimate the state of the a part of a segment, we start by
applying Bayes’ theorem (Berger 1985) to a part of segment Bij :
p[r | s( Bij ) = PRESENT]P[ s( Bij ) = PRESENT]
P[s( Bij ) = PRESENT | r ] = (15)
#s( B ) p[r | s( Bij )]P[ s( Bij )]
ij
where P[ s ( Bij ) = PRESENT ] is prior presented segment probability of a part of
segment Bij . P[ s ( Bij ) = PRESENT | r ] is new presented segment probability of a
part of segment Bij based on r observations.
In Eq. 15, the term p[r | s ( Bij ) = PRESENT ] corresponds to sonar sensor pdf
that will be discussed in the following section. In order to estimate the state of
complete segment, the average value of the j parts of segment states is calculated as
shown below:
# j P[ s ( Bij ) = PRESENT | r ] (16)
P[ s ( Bi ) = PRESENT | r ] =
j
The value of j can be determined by
length of segment Bi (17)
j≈
σ
where σ is range variance of the sonar sensors. Here, we use 40mm. The
minimum value of j is fixed at 1.
4.2 Fuzzy tuned sonar sensor model
The sonar pdf is further tuned to accommodate a dynamic environment. A similar

idea is proposed in the authors’ pervious work (Chow et al. 2002) for grid-based
map building. Here we proposed an enhanced and modified version of tunable sonar
pdf model. We select a zero-mean Gaussian distribution to represent the sonar
sensor range readings. Only radial uncertainty is considered. The pdf model is
shown as follows:
* $ − (r − z) 2 .
-k ε , if x < r and k o exp % /≤k
- % σ 2 / ε
& r 0
-
- $ − (r − z ) 2 .
if k o exp% />k
-
- $ − (r − z ) 2 . % σ 2 / ε
% / & r 0
p ( r | z ) = +k o exp
% σ 2 /
,
$ − (r − z) 2 .
(18)
- & 0
-
r and k o exp % / > 0.5 , if x > r
% σ 2 /
- & r 0
-0.5 , otherwise
-
-,
where
x = The distance from the sensor
k ε = A parameter that corresponds to the empty space that is tuned by a fuzzy
model.
k o = A parameter that corresponds to the occupied space and is tuned by a fuzzy
model.
2
σ r = The variance of the radial probability.
r = The sensor range measurement of the sonar sensor.
z = The true parameter space range value.
To extend the derivation to two spatial dimensions, consider the example of a

range sensor characterized by Gaussian uncertainty in both radial and angular
directions. This can be represented as:
*
- $ −θ 2 . $ −θ 2 .
% / % / > 0.5
-1 − (1 − kε ) exp% 2 / , if x < r and p ( r | z ) = k ε and (1 − kε ) exp
%σ 2 /
- & σθ 0 & θ 0
-
- if x < r and kε < p ( r | z ) < 0.5
- $ − (r − z) 2 . $ 2.
% /) exp% − θ / $ − (r − z ) 2 . $ 2.
-1 − (1 − k o exp% 2 / %σ 2 /
,
and (1 − k o exp% /) exp% − θ / > 0.5
- & σ r 0 & θ 0 % σ 2 / %σ 2 /
- & r 0 & θ 0
p(r | z, θ ) = + (19)
- $ 2. $ 2. $ − (r − z ) 2 . $ − θ 2 .
-k exp% − ( r − z ) / exp% − θ / , if p ( r | z ) > 0.5 and k o exp% / exp% / > 0.5
- o % σ 2 / %σ 2 / % 2 / % 2/
- & r 0 & θ 0 & σr 0 & σθ 0
-
-
-0.5 , otherwise
-
,-
where
θ = The azimuth angle measured with respect to the beam central axis.
2
σ θ = The variance of the angular probability.
In this algorithm, k ε and k o are tuned by the fuzzy systems to be further
described (see Fig. 4 and 5). In both fuzzy systems, the fuzzy model includes two
inputs and a single output with singleton fuzzier, product inference engine and
center average defuzzifier (Wang 1997). The tuning strategy is based on the
following four variables and their geometric representation is shown in Fig. 3:
171
PmaxB : the highest probability of the extracted segment PmaxBi within the sensor
cone.
P*B : the probability of the nearest segment within the sonar cone .
dr : the distance between the range reading and the extracted segment with the
highest probability within the sensor cone. (i.e. range of sensor - location
of the extracted segment with the highest probability within the sensor
cone)
∆dr : the change of dr. (i.e. ∆ dr = drt − drt −1 , where t is sampling time)
(a) (b) (c)

Fig. 3. a) Basic condition for tuning kε and k o . b) Condition for tuning kε . c) Condition
for tuning k o .
The special criteria for tuning these two parameters will be discussed below:
For the first criteria to tune these two parameters, the acute angle θ oi between
the extracted line segment Bi and valid sonar beam should be located within 90° to
θ lim it , where θ lim it is the angle limit. Since the half of valid sonar beam angle is
12.5°. The θ lim it is selected as about 77.5°. Fig. 3(a) shows the situation that
satisfies the first criteria. If the angle θ oi exit these limit, these sonar sensor data
point will not be used to update the probability of these line (or cell).
The parameter k ε corresponds to the probability that a cell is empty (or the
segment is absent in our case). In the traditional approach, this parameter does not
exist and is implicitly set near to zero. Therefore, when the object detected range of
sensor is much larger than its actual location, the probability of the corresponding
extracted segment may be reduced to close the zero at once during Bayesian update.
In order to avoid this problem, it is suggested to increase k ε when the range of the
sensor measurement is much larger than the extracted segment location and vice
versa. The corresponding membership function and fuzzy rule-base are shown in
Fig. 4 and 5, respectively. This fuzzy system is only activated when the detected
range of the sensor measurement is larger than the distance between the
corresponding extracted segment and position of sensor (i.e. dr = positive value)
and the highest probability of the corresponding extracted segment is larger than Pth
(i.e. PmaxB > Pth ), otherwise k ε will be a constant.
VS S M L VL
VS S M L VL
Pth 1 + 3Pth 1 + Pth 3 + Pth 1

Plim it 0.5 + 3Plim it 0.5 + Plimit 1.5 + Plimit 0.5
4 2 4
4 2 4
Input Space of PmaxB
Input space of P*B
Z S M L VL
Z S L
0 0.125 0.25 0.375 0.5

0 0.4 0.8
Input space of ∆ dr Input Space of dr
VS MS M ML VL
VS MS M ML VL
0.05 0.2 0.3 0.4 0.5

0.5 0.65 0.75 0.85 0.9
Output consequence Output consequence

fuzzy sets for ko fuzzy sets for kε
6 (b)
Fig. 4. The membership functions and the output consequence fuzzy sets for tuning a)
k o and b) k ε .
The parameter k o corresponds to the probability that the cell is occupied (or the
segment is present in our case). In conventional approaches, this parameter is fixed.
Therefore, when the object is moved towards the sensor, the probability can not
reach up again. In our approach, k o is tuned by the fuzzy system. It is suggested to
increase k o when the object is moved towards the mobile robot (or sonar sensor).
The corresponding membership function and fuzzy rule base are shown in Fig. 4
and 5, respectively. This fuzzy system is only activated when the detected range of
sensor is within the range between the location of sensor and existing extracted
173
segment (i.e. dr = negative value) and P*B is lower than 0.5, otherwise k o will be
fixed.
(a) (b)
Fig. 5. a) Fuzzy rule table corresponding to k o . b) Fuzzy rule table corresponding to k ε .
4.3 Outline for updating the state of an extracted segment
Note that the input and output memberships in Fig. 4 and 5 are tuned by simulation
trial. All the new extracted segments are initialized with 0.5 probability state. A
step-by-step procedure for updating the state of an extracted segment is presented as
follows:
1. Obtain the sonar sensor measurements and use the kinematics transformation
to obtain global location and store it in the cell C( X ,Y ) .
2. Update and extract the segment in the corresponding cell by the EAFC with
NC for all new sonar measurements if the total numbers of data point in the
related cells more than 5 data points (designed by human).
3. Locate the highest probability of extracted segment within the new sensor
measurement cone, and obtain the dr, if b<dr<a, go to step 4, otherwise go to
step 5, where a is positive value and b is negative value which are selected as
the value of sensor range variance σ (i.e. σ = ±40mm).
4. Divide the extracted segment into j parts and estimate the state of each part by
sonar sensor pdf (Eq. 19). Update the state of each part of a segment by Eq. 16
and obtain the average value to update the state of complete segment by Eq.
17. Go to step 7.
5. Tune the k ε or k o by fuzzy systems with the following conditions:
If dr > a then tune the k ε .
If dr < b and the probability of the corresponding segment less than 0.5 then
tune the k o . Go to step 6.
6. Based on the new tuned value k ε or k o , calculate the probability of the
corresponding segment be within the sonar cone by sonar sensor pdf (Eq. 19).
Go to step 7.
7. After updating the state of segments, if the probability of state is lower than or
equal to Plimit, then the segment and the corresponding data points are
removed. Go to step 1 when new sonar measurement(s) is/are available.
where Plimit is the lower limit of the probability of the all extracted segments or
objects. It is determined by the experimental trial. The Plimit can control the
sensitivity of the proposed algorithm. If the value of Plimit is increased, the segment
Bi will be deleted after few samplings. Otherwise, a longer time is required to
update a true situation.
4.4 Assumptions and limitations
For this paper we will consider a mobile robot moving about in the well-constructed
indoor environment with dynamic objects, such as box, furniture or other moving
object. The proposed algorithm can enable the mobile robot to be aware of the
changes around them. The paper focuses on the segment extraction from the sonar
measurements. Therefore, the automatic navigation and the localization system are
outside the scope of this study.
The parameters in the proposed algorithm are restricted to use sonar sensor. If the
mobile robot equipped with laser-range finders, the sensors pdf and sensor variance
should be changed.
5 Model Based Localization via Fuzzy Tuned Extended

Kalman Filter FT-EKF
The capability to acquire the position and orientation of an autonomous mobile

robot is an important element for achieving specific tasks requiring autonomous
exploration of the workplace. Once we obtain a map (a set of line segment) of the
robot workplace by a mobile robot or hand-measured, this map can be provided to a
mobile robot for further usage. Here, the world environment is assumed modeled
into two-dimensions. Just as humans use two-dimensional floor plans, the robot
uses a map, which projects all features into two dimensions. The complexity of the
map is thereby greatly reduced. The simplification is often reasonable in practice,
especially in man-made environments.
5.1 Mobile robot and sensor modeling and background formulation

of extended Kalman filter EKF
We have applied our algorithm to Pioneer 2DX (Pioneer 2 Mobile Robot Operation
Manual 2001) mobile robot which is a differential drive type robot. According to a
two-dimensional world assumption, the robot configuration w.r.t. the world
Τ
coordinate can be defined as X = [x y θ ] containing its position ( x, y ) and
orientation θ . For the robot motion (see Fig. 6), we assume that the incremental
motion is in a straight line (small robot motion) (Konolige 2001). A small robot
motion is defined as traveling a short distance “s” from the origin at angle θ and a
175
turn angle α . These two can be integrated to give the global position of the robot.
According to this assumption, the updated robot position and orientation are given
by a simple function as:
' x + s cos θ 1
X ' = ( y + s sin θ 2 (20)
( θ +α 2
) 3
The system model describes how the robot pose X (k ) changes with time k in
response to the control input u (k ) = [ s (k ),α (k )]Τ and a noise source v(k ) . This
noise source v(k ) is assumed to be a zero-mean Gaussian with covariance Q (k ) .
The system model is expressed as:
X (k + 1) = F ( X (k ), u (k )) + v (k ), v(k ) ~ N (0, Q(k )) (21)
where F ( X (k ), u (k )) is the non-linear state transition function. This state
transition function is equal to the robot pose update function (Eq. 20) which is given
by:
' x ( k ) + s ( k ) cosθ ( k ) 1
F ( X ( k ), u ( k )) = ( y ( k ) + s ( k ) sin θ ( k ) 2 (22)
( θ (k ) + α (k )
2
) 3
Fig. 6. Control command and the assumed small robot motion.
Estimate position can generally be obtained by two methods: internal and

external (Konolige 2001). The internal method exploits the motion or forces
produced by the robot itself with no regards to the external environment. On the
other hand, the external method relies on the signal from the environment to register
the robot’s position. In our case, we fuse these two methods together to estimate the
robot’s pose. Our Pioneer 2DX mobile robot consists two types of sensor, wheel
encoder (for internal method), ultrasonic sensors (for external method). In the
following paragraphs, a brief overview of each sensor is introduced.
The wheel encoders are used to measure wheel rotation and steering orientation.
For short distance travel, it is sensible to assume that only a small error occurs. With
this assumption, the error can be eliminated at each localization sampling instant.
There are three types of movement errors: k R is range error factor (unit:
mm 2 / m ), kθ is turn error factor (unit: deg 2 / deg ) and k D is drift error factor
(unit: deg 2 / m ) (Konolige, 2001; Thrun et al., 2001). For convenience, these errors
are represented in variance form. For example: If the robot moves a distance s and
turns an angle α , then the variance will be: var(s ) = k R s and
var(θ ) = kθ α + k D s . Since the wheel encoders are installed in the motor shaft, we
can estimate the short distance s and turn angle α via the left and right wheel
encoders (a simple calculation is used at every sampling time (see (Konolige
2001)).
Pioneer 2DX supports both front and rear sonar arrays. Each is equipped with
eight Polaroid ultrasonic sonar sensors (Pioneer 2 Mobile Robot Operation Manual,
2001). The sonar positions in all arrays are fixed. One sensor on each side and six
facing outward at 20-degree spacing provide 360 degrees coverage of the
environment. The local position and orientation of ith sonar (i = 1,2, $ , n S ) is
denoted by ( xi , yi ) and θ i , respectively and nS denotes the total number of
sensors. Therefore, the global position and orientation of the sonar at time k is given
by:
i
xS ( k ) = x ( k ) + xi cos(θ ( k )) − yi sin(θ ( k ))
i
y S ( k ) = y ( k ) + xi sin(θ ( k )) + yi cos(θ ( k )) (23)
i
θ S (k ) = θ (k ) + θi
The actual reading of the sonar sensor (observation) can be denoted as set Z (k )
where:
Z ( k ) = {z i ( k ) | 1 ≤ i ≤ n s ) (24)
In the 2-D map, it is contain a jth ( j = 1,2,..., n p ) line segments only, n p
denote the total number of planes. The form of the line segment representing here is
shown as following:
plj ( x plj [1], y plj [1], x plj [2], y plj [2]) (25)
where ( x [1], y [1]) : line segment starting point global X-Y coordinate.
j
pl
j
pl
( x plj [2], y plj [2]) : line segment end point global X-Y coordinate.
Now we define two more symbols related to the line segment (or plane), a plj
where is the angle of the normal line of the related segment (count from positive
side of x-axis) and it is given by:
$ x j [1] − x j [ 2] .
j −1% pl pl /
a pl = tan
%− (26)
[1] − y pl [ 2] //
j j
% y pl
& 0
and dplj where distance form the origin to the segment and it is given by:
177
j j j j
x pl [1] y pl [2] − x pl [ 2] y pl [1]
j
d pl = (27)
j j 2 j j 2
( y pl [1] − y pl [ 2]) + ( x [1] − x pl [ 2])
pl
From the Eq. 23, 26 and 27, the measurement function of the line segment due to
the ith sonar sensor at time k can be calculated as follows:
hi (k , plj ) = ρ v (d plj − x Si (k ) cos a plj − y Si (k ) sin a plj ) (28)
where ρ v is 1 or –1 to determine the visible side,
' δ δ1
a plj ∈ (θ Si (k ) − ,θ Si (k ) + 2 , δ is sonar beam angle,
) 2 23
Only one jth plane is matched with the ith sonar sensor.
Fig. 7. Measurement Model in graphical representation and validated sonar condition.
The graphical relationship is shown in Fig. 7. Finally, the observation model

describes how the sonar reading z i (k ) at time k is related to the robot’s position
and is given by:
z i (k ) = hi ( X (k ), plj ) + wi (k ), wi (k ) ~ N (0, Ri (k )) (29)
The term wi (k ) represents measurement noise (Wang, 1988). It is also assumed
to be a zero-mean Gaussian with covariance Ri (k ) .
EKF is often used to combine all measurement data (e.g., for fusing data from
different sensors) to get an optimal estimate in a statistical sense. The overall goal of
the EKF cycle is to find a new (a posteriori) estimated position Xˆ (k + 1 k + 1) and

its covariance Pˆ (k + 1 k + 1) , given the old (a priori) estimated position Xˆ (k k )
and its covariance Pˆ (k k ) at time k. The meaning of Xˆ (k + 1 k + 1) stated is the
predicted position X̂ at time k+1 given all observations up to and including time
k+1. The formulation of the EKF is as follows:
1. Filter Initalization.
Initialize the mean square error covariance matrix P (0 | 0) , predict the position
X̂ (0 | 0) , state noise covariance model Q(0) and measurement noise covariance
model R(0) .
2. Filter Prediction.
Calculate the predicted state Xˆ (k + 1 | k ) by using Eq. 22 as:
Xˆ (k + 1 k ) = F ( Xˆ (k | k ), u (k ))
' f x (k ) 1 ' x(k ) + s (k ) cosθ (k )1
(30)
= (( f y (k )22 = (( y (k ) + s (k ) sin θ (k ) 22
() f θ (k )23 () θ (k ) + α (k ) 23
Partial differentiate Eq. 30 with respect to the pose ( x(k ), y (k ),θ (k )) and it is
given the Jacobian matrix are:
' ∂f x ( k ) ∂f x ( k ) ∂f x (k ) 1
( ∂x ∂y ∂θ 2
( 2 '1 0 − s( k ) sin θ (k ) 1
∂f y ( k ) ∂f y ( k ) ∂f y ( k )
∇F = ( 2 = (0 1 s ( k ) cos θ (k ) 2 (31)
( ∂x ∂y ∂θ 2 ( 2
( ∂f (k ) ∂f ( k ) ∂f ( k ) 2 ) 0 0 1 3
( θ θ θ 2
() ∂x ∂y ∂θ 23
The predicted error covariance matrix:
Pˆ (k + 1 | k ) = ∇F Pˆ (k | k ) ∇F Τ + Q(k ) (32)
where
' k R s (k ) cos 2 θ (k ) k R s( k ) sin θ ( k ) cos θ (k ) 0 1
( 2
Q (k ) = (k R s (k ) sin θ ( k ) cos θ ( k ) k R s (k ) sin 2 θ ( k ) 0 2 (33)
( 0 0 kθ α (k ) + k D s( k ) 23
)
3. Observation.
Obtain the next observation as given by:
Z (k + 1) = {z i (k + 1) | 1 ≤ i ≤ nS } (34)
4. Measurement Prediction.
Determine the predicted measurement by Eq. 28 with the predicated state
Xˆ (k + 1 | k ) in the form:
179
zî (k + 1)) = hi ( plj , Xˆ (k + 1 | k )) (35)

5. Innovation.
Calculate the difference between the observation and predicted measurement.
For the ith sensor, the innovation vii (k + 1) is computed as:
vii (k + 1) = z i (k + 1) − zî (k + 1) (36)

6. Variance of the Innovation.
Determine the variance given by the innovation. It is formed as:
S i (k + 1) = ∇hi Pˆ (k + 1 | k )∇hi + Ri (k + 1)
Τ
(37)
where Ri (k + 1) is variance of the sensor measurement,
Τ
' ∂hi 1
( ∂x 2 Τ
( ∂h 2 ' − cos acl 1
∇hi = ( i 2 = ρv
( − sin acl
2 (38)
( ∂y 2 ( 2
( ∂hi 2 )( xns sin(θ (k + 1 | k ) − acl ) − yns cos(θ (k + 1 | k ) − acl )32
( 2
) ∂θ 3
Jacobian matrix ∇hi is derived from Eq. 28 by partial differentiation with respect
to pose ( x(k + 1 | k ), y (k + 1 | k ),θ (k + 1 | k )) .
7. Validation Gate.
v 2 (k + 1)
vi (k + 1) S i−1 (k + 1)viΤ (k + 1) = i ≤ g2 (39)
S i (k + 1)
where g is acceptable different between the measurement and prediction.
8. Kalman Gain.
After matching (accepted by the validation gate checking process), a number of
measurements and observations are used to update the position of the robot.
Then, the filter gain Wi (k + 1) is updated for the every accepted ith sensor
measurement and observation:
W (k + 1) = Pˆ (k + 1 | k )∇h Τ S −1 (k + 1)
i i i (40)
9. State vector update
At time k+1, the estimated robot pose can be obtained as follow:
Xˆ ( k + 1 | k + 1) = Xˆ ( k + 1 | k ) + # Wi ( k + 1)vi ( k + 1) (41)
i
Covariance update:
Pˆ (k + 1 | k + 1) = Pˆ (k + 1 | k ) − # Wi ( k + 1) S i (k + 1)Wi Τ ( k + 1) (42)
i
10.Repeat step 2 with Xˆ (k + 1 | k + 1) .

5.2 Formulation of proposed FT-EKF
In the traditional adaptation schemes of EKF localization, it is suggested to adapt

the process noise covariance Q (k ) with scaling the given priori Q (0) (Jetto et al.
1999a, 1999b). That means, the adaptation scheme is based on changing the scaling
factor of Q(k ) to get the updated Q(k + 1) , i.e.
Q(k + 1) = βQ (k ), where β is the scaling factor. A problem may arise (degrade the
EKF performance) if poor estimated elements in matrix Q (0) (at start) are used.
This is a distinct disadvantage in cases where it is difficult to determine the
elements in process noise covariance matrix Q (k ) . A novel fuzzy rule based tuning
scheme allows us to integrate the heuristic knowledge and simplifies the design
knowledge of the matrix Q (k ) . In our proposed algorithm, a simple fuzzy model is
employed. The strategy for tuning the parameters (or elements) in Q (k ) will be
discussed in following section.
5.2.1 Assumption statement
In conventional approaches (Jetto et al. 1999a; Meng et al. 2000), a number of the
adaptation schemes of Kalman filter are designed according to covariance
matching. The main idea is to make the filter residual covariance,
E[vii (k + 1)vii (k + 1) Τ ] , consistent with its theoretical value (Mehra 1972). Note that
Kalman gain W (k ) is influenced by both of process noise Q (k ) and
measurement noise R (k ) . From Eq. 32, 37 and 40, it can be observed that the value
of W (k ) can be increased by either increasing Q (k ) or decreasing R (k ) . We
have used this fundamental concept in our algorithm.
Now let us go back to basic concept of the Kalman filter adaptation scheme.
When the innovation vii (k + 1) approaches zero, the value of Q (k ) (or Kalman
gain W (k ) ) is appropriately changed. On the other hand, if the innovation
vii (k + 1) is increased, it boosts up the value of Q (k ) in order to reduce the
innovation. The main objective is to maintain the innovation process as small as
possible.
In our approach, three noise factors k R (k ) , kθ (k ) , and k D (k ) are tuned by
fuzzy rules. The tuning strategy is based on the assumption of availability of a
sufficiently accurate robot pose. The mean of innovation process is similar to the
mean square root value of the variance of innovation at time k. It is stated as
following:
λ (k + 1) ≈ 1 (43)
181
vi (k + 1)
where λ (k + 1) = , for all matched values between observation and
S i (k + 1)
prediction measurement.
The details of formulation of the fuzzy tuned EKF will be discussed in the next
section.
5.2.2 Adaptive estimation of Q (k ) by fuzzy rule based tuning scheme
Before we discuss the tuning condition for k R (k ) , kθ (k ) , and k D (k ) , two

additional terms are introduced as follows:
δf xy ( k + 1) = (δf x ( k + 1)) 2 + (δf y ( k + 1)) 2 (44)
where δf x (k + 1) = #W (k + 1)v(k + 1) , δf (k + 1) = #W (k + 1)v(k + 1) , and

x y y
δ f ( k + 1) = # W ( k + 1) v ( k + 1)
θ θ (45)
It can be observed from the above that the elements in the Kalman gain
W (k )
'W x ( k ) 1
can be stated as W ( k ) = (W y (k )2 .
( 2
()Wθ ( k ) 23
The terms δf x (k + 1) , δf y (k + 1) and δf θ (k + 1) represent any significant
change on X-axis, Y-axis and heading of robot, respectively. Similarly, the term
δf xy (k + 1) denotes any significant change in distance traveled by the robot. The
signal flow diagram and overall structure of the FT-EKF adoption scheme is shown
in Fig. 8.
5.2.2.1 Fuzzy system for tuning k R (k )

For tuning k R (k ) , a three inputs and single output fuzzy system has been
employed. The three inputs are λ (k ) , δf xy (k ) and v(k + 1) . When the λ (k ) at
time k is small enough which implies the factor k R (k − 1) is good enough, we
suggest either no change or decrease the factor of k R (k ) a little bit. This depends
on the innovation process v(k + 1) at time k+1 and gain action on δf xy (k ) . If an
appropriate value of k R (k − 1) is used at time k, the value of v(k + 1) and
δf xy (k ) should be small. Therefore, we suggest no change in k R (k ) . On the other
hand, the k R (k ) should be reduced a little bit when a large gain action δf xy (k ) is
obtained. According to a similar argument, the rule table for tuning k R (k ) can be
formed as show in Table 1a. The output of this fuzzy system should be ∆k R (k ) ,
i.e. k R (k ) = k R (k − 1) + ∆k R (k ) . For connivance, the input and output parameters
are normalized into 0 to 1 and –1 to 1, respectively. The corresponding input and
output membership functions are shown in Fig. 9a and Fig. 9b, respectively.
Control command u (k ) = ( s, α ) given by

Navigation System or Human Operator
Mobile Robot Platform
Odometry Sonar sensor

readings
Z ( k + 1)
Prediction NO
Position Prediction Matchin
g Update with the
Xˆ ( k + 1 k ) = F ( Xˆ (k | k ), u ( k )
YES predicted position
Measurement
Prediction Update
zˆ i ( k + 1) = hi ( p , Xˆ ( k + 1 | k ))
l Innovation
v ii ( k + 1 ) = z i ( k + 1 ) − zˆ i ( k + 1 )
Covariance Prediction Kalman gain update

Τ
Pˆ(k + 1 | k) = ∇F Pˆ(k | k ) ∇F + Q(k ) T −1
W i ( k + 1) = Pˆ ( k + 1 | k ) ∇ h i S i ( k + 1
Variance of Position update

innovation Prediction Xˆ ( k + 1 | k + 1) = Xˆ ( k + 1 | k )
Si ( k + 1) = ∇hi P(k + 1 | k )∇hi

Τ
+ Ri (k + 1)
+ # W ( k + 1)v ( k + 1
i
i i
Covariance update
Pˆ(k +1 | k +1) = Pˆ(k +1 | k)
T
− #Wi (k +1)Si (k +1)Wi (k +1)
i
MAP
(priori information) Past information
v (k ), S ( k ) , W ( k ), u ( k − 1)
Fuzzy rule
based tuning
system
State noise model

update Q ( k + 1)
Noise model
estimation scheme
Fig. 8. Overall structure of FT-EKF.

183
5.2.2.2 Fuzzy system for tuning kθ (k ) and k D (k )

Based on a similar reasoning for tuning in k R (k ) , the fuzzy system for tuning
kθ (k ) and k D (k ) is designed. However, the input δf θ (k ) is used instead of
δf xy (k ) . Otherwise, the tuning strategy is the same as k R (k ) . Also note that the
output fuzzy subsets are more since the control gain related in heading should be
more precise. If a wrong kθ (k ) and k D (k ) are used, the EKF diverge. The same
rule base is used in both of factors kθ (k ) and k D (k ) tuning but different scaling
factors are applied. The rule table is shown in Table 1b. The output of this fuzzy
system should be ∆kθ (k ) and ∆k D (k ) , i.e. kθ (k ) = kθ (k − 1) + ∆kθ (k ) and
k D (k ) = k D (k − 1) + ∆k D (k ) .
Another important factor is the sequence to update kθ (k ) and k D (k ) . We
assign a weighting factor according to the control command
u (k − 1) = [ s (k − 1),α (k − 1)]Τ . We then design a two inputs and one output fuzzy
system. Since the factor kθ (k ) is related to turning α (k − 1) and the factor
k D (k ) is related to traveling s (k − 1) , the output of this fuzzy system is the
so-called weighting factor β , where 0 ≤ β ≤ 1 and the corresponding rule table is
shown on Table 1c. Now the weighted output is ∆kθ (k )′ and ∆k D (k )′ . The
formulation of those factors are shown as follow:
∆kθ (k )′ = (1 − β )∆kθ (k ) (46)
∆k D (k )′ = β∆k D (k ) (47)
The corresponding input and output membership functions are shown in Fig. 9a and
Fig. 9c and 9d, respectively.
Table 1. Fuzzy rule table corresponding to a) the output ∆k R ( k ) , b) the output ∆kθ ( k ) and
∆k D (k ) , c) the output β .
λ (k ) : Z / M / L δf xy (k )
Z M L
Z Z/Z/Z Z / Z / PS NS / Z / Z
v(k + 1) M NMS / PS / PS NS / Z / PMS NS / Z / Z
L NMS / PS / PS NMS / Z / PMS NS / NS / Z
(a)
λ (k ) : Z / M / L δf θ ( k )
Z M L
Z Z/Z/Z Z / NS1 / PS1 NS1 / NS1 / NS2
v(k + 1) M Z / PS1 / PS3 Z / NS1 / PS1 NS2 / NS2 / NS3
L Z / PS2 / PS3 NS2 / NS2 / PS2 NS3 / NS2 / NS3
(b)
s (k − 1)
Z M L
Z M L VL
α (k − 1) M S M L
L Z S M
(c)
(a)
(b)
(c) (d)
Fig. 9. a) Input membership functions corresponding to λ ( k ) , v( k + 1) , δf xy (k ) ,
δf θ (k ) , s (k − 1) and α (k − 1) . Output consequences corresponding to b) ∆k R (k ) ,
c) ∆kθ ( k ) and ∆k D ( k ) and d) β .
185
6 Experimental Study
In order to evaluate the performance of the proposed dynamic segment-based map

building algorithm and FT-EKF localization algorithm, we carried out two
experiments with a Pioneer 2-DX mobile robot in indoor campus environment.
Pioneer 2-DX mobile robot equipped with a ring of 16 Polaroid ultrasonic sonar
sensors and an on-board 20 MHz Siemens 88C166 microprocessor with integrated
32K flash-Rom. The driving mechanism of the robot consists of one castor wheel
and two DC reversible motors with wheel encoders.
6.1 Experiment 1
For the first experimental study, the corridor outside the Control Research
laboratory was modeled. During the experiment in the corridor, there was one
moving object and the doors of two of the offices were open. Figure 10 shows the
actual image of the corridor. The total trajectory of the robot was round 8m, lasting
about 3.5 minutes, with an average speed of 50mm/s. The on-line segment-based
map building result is shown in Figure 11.
In this experiment, the dynamic object moved in three different positions
(position 1 to position 2 to position 3) which is shown in Figure 10b. In Figure 11a
and Figure 11b, the object is placed at position 1. After that, we move the object
from position 1 to position 2 as shown in Figures 11b and 11c, respectively. Finally,
we moved the object from position 2 to position 3. The Pioneer 2DX was made to
perform a “U turn” at position 2. In Figure 11e, we found that the object in position
2 was disappearing when we compared the map with Figure 11d. In Figure 11f, the
Pioneer 2 returned to the starting point. It could be observed that the object in
position 1 was also disappearing. From this experiment, it was concluded that the
proposed on-line segment-based map-building algorithm could build a map in a
dynamic environment. Note, when the object is moving, the Pioneer 2DX is stopped
until the object is reached to its defined position.
(a) (b)
Fig. 10. a) Photo of experimental site (corridor), b) 2-D floor plan.

(a) (b) (c)
(d) (e) (f)
Fig. 11. Online mapping during exploration with Pioneer 2DX: a) to f)
6.2 Experiment 2
In the second experiment, the proposed localization algorithm was tested in an

indoor environment containing a room and a corridor. The size of area was about
(11m × 7.3m). The room was a research office that included some partitions,
rubbish bins, desks, and chairs, cabinet and access doors. The layout of the
navigation area and different images of the environment are shown in Figure 12. In
this experiment, only the office partitions, desks, walls and doors were modeled by
several line segments (see Figure 12b). In order to compare the proposed FT-EKF
localization algorithm, odometric measurement and conventional EKF localization
approach are performed simultaneously. The validation gate of both model-based
localization algorithm (convetional EKF and FT-EKF) is 5 (i.e. g = 5). The result
trajectories are shown in Figure 13. The results indicate that both EKF and FT-EKF
localization algorithms estimate the actual trajectory satisfactory in spite of the
failure of odometric measurement. Average of innovation of the EKF localization
and FT-EKF localization over all time were equal to 55.301mm and 43.83mm,
respectively. The final error is shown in Table 2. It can be deduced that FT-EKF
algorithm has smaller final error than that of EKF localization algorithm.
187
Box Cabinet
Open Close
Close Rubbish Bin
Distilled
Research Office Water
Partitions
Desk
Chair
Corridor
End Point
Close
Close
Close Close
4m
Box
2m
Approximation Route
Close
Starting Point
2m 4m
(a) (b)
(c)
Fig. 12. Experimental environment. a) Floor Plan. b) Modeled Segment. c) Actual

environment.
Table 2. Final errors of experiment with Pioneer 2DX (real robot)
EKF FT-EKF
∆x (m) -0.81 -0.285
∆y (m) -0.0183 -0.014
Fig. 13. Result trajectories of experiment with real robot.
7 Conclusions
In this chapter, the important problems of map building and localization were
discussed and some new algorithms were proposed. These ideas were tested with
simulation studies as well as experimental verification. The next phase of this work
include the SLAM problem. The algorithms reported here can be extended for other
perception devices.
189
References
1. Berger JO (1985) Statistical Decision Theory and Bayesian Analysis.

Springer-Verlag, Berlin. Second Edition.
2. Bezdek JC (1981) Pattern Recognition with Fuzzy Objective Function Algorithms.
Plenum, New York
3. Chow KM, Rad AB and Ip YL (2002) Enhancement of Probabilistic Grid-based
Map for Mobile Robot Applications. Journal of Intelligent and Robotic Systems
34: pp.155–174
4. Dave RN (1989) Use of adaptive fuzzy clustering algorithm to detect lines in
digital images. Intelligent Robots and Computer Vision VIII, Vol. 1192(2), pp
600-611
5. Dave RN (1991) Characterization and detection of noise in clustering. Pattern
Recognition Letters 12, pp 657-664
6. Ip YL, Chow KM, Rad AB and Wong YK (2002) Segment-based map building
using Enhanced Adaptive Fuzzy Clustering Algorithm for mobile robot
applications. Journal of Intelligent and Robotic Systems 35, pp.221–245
7. Jetto L, Longhi S and Venturini G (1999a) Development and experimental
validation of an adaptive extended Kalman filter for localization of mobile robots.
IEEE Transactions on Robotics and Automation, vol 15, No. 2, pp 219-229
8. Jetto L, Longhi S. and Venturini G. (1999b) Localization of a wheeled mobile
robot by sensor data fusion based on a fuzzy logic adapted Kalman filter. Control
Engineering Practice, 7, pp 763-771
9. Konolige K (2001) Robot Notes (Robot Motion: Probabilistics Model; Sampling
and Gaussian Implementations; Markov Localization). SRI International
(http://www.ai.sri.com/~konolige)
10. Mehra RK (1972) Approaches to adaptive filtering. IEEE Transaction Automatic
Control, vol 17, Issue: 5 pp 693-698
11. Meng QH, Sun YC and Cao ZL (2000) Adaptive extended Kalman filter
(AEKF)-based mobile robot localization using sonar. Robotica, vol 18, pp 450-473
12. Phansalkar G and Dave RN (1997) On Generating Solid Models of Mechanical
Parts through Fuzzy Clustering, Proceedings of the Sixth IEEE International
Conference on Fuzzy Systems, Barcelona, Spain, vol 1, pp 225 -230
13. Pioneer 2 Mobile Robot Operation Manual. May (2001) ActivMedia Robotics,
LLC, v7b
14. Runkler TA, Palm RH (1996) Identification of Nonlinear Systems Using regular
fuzzy c-Elliptotype Clustering. Proceedings of the Fifth IEEE International
Conference on Fuzzy Systems, New Orleans, Louisiana, vol 2, pp 1026 -1030
15. Thrun S, Fox D, Burgard W and Dellaert F (2001) Robust Monte Carlo localization
for mobile robots. Artificial Intelligence, 128, pp 99-141
16. Wang LX (1997) A course in fuzzy systems and control. Prentice-Hall
International, Inc.
190
Control and Stabilization of the Inverted

Pendulum via Vertical Forces
D. Maravall
Department of Artificial Intelligence

Faculty of Computer Science, Universidad Politécnica de Madrid
Campus de Montegancedo, 28660 Madrid, Spain
dmaravall@fi.upm.es
Abstract. In this chapter, we present a detailed analysis of the possibilities of con-

trolling and stabilizing the inverted pendulum (IP) by means of a vertical force.
First, we establish the dynamic equations of the IP under the action of a generic
vertical force and then we analyze its control and stabilization. The main conclu-
sion is that the vertical force has an excellent stabilization effect, although it re-
quires a permanent fall of the IP support base when it is the only applied force.
Therefore, we investigate the combination of the vertical force with the customary
horizontal force, arriving at the stabilization conditions for different formal repre-
sentations of the system: ordinary differential equations, state variable representa-
tion and Liapunov´s direct and indirect methods.
1 Introduction
The inverted pendulum (IP) is a widely studied dynamic system, which has re-
ceived considerable attention in many fields, such as physics, mechanics, applied
mathematics, control theory, and the emergent computational techniques known as
soft computing [1]-[3]. There are several reasons behind such interest, in particular
the importance and ubiquity of the IP in many mechanisms, including robots. Fur-
thermore, its intrinsic theoretical interest and the strong challenges posed by its
stabilization and control have made the IP a sort of benchmark, in particular for
the comparison of soft computing (artificial neural networks, fuzzy logic, genetic
algorithms) and hard computing (ordinary differential equations, input-output con-
trol, state variable techniques, like optimal control or Liapunov´s stability).
Apart from being a benchmark in control engineering, the IP problem has al-
ways been a testbed for computational intelligence theories and models, as it em-
braces the customary sense-reason-action cycle, typical of intelligent systems [4]
and [5]. Furthermore, controlling a pendulum in its unstable top position is not
only an interesting physico-mathematical problem with a difficult engineering im-
plementation, as it involves high nonlinearities and fast sensory information proc-
Control and Stabilization of the Inverted Pendulum 191
191
essing, but it also, and very importantly, poses a strong challenge to any intelligent
artificial agent, as it demands the coordination of perception and action and, even,
some degree of reasoning. Thus, it is no wonder that IP stabilization has been at-
tempted by means of robotic manipulators and specialized sensors, [6]- [9].
The bibliography on IP is literally overwhelming. However, it is rather surpris-

ing that virtually all the technical literature refers to the planar pendulum with one
degree of freedom. Only very recently have a few references dealing with the
spherical pendulum with two degrees of freedom appeared [9]-[11]. Due to the
complex control problems involved, [9] addresses the stabilization of the spherical
IP by simultaneously controlling two uncoupled planar pendula (the respective
projections on the two orthogonal planes of the intertial coordinate system). Refer-
ences [10] and [11] apply the method of controlled Lagrangians to get theoretical
stability conditions for the spherical IP.
In practice, the only control action used in the technical literature for IP stabili-
zation is a horizontal force, which is almost universally materialized by means of
an electrical cart, i.e., the popular cart-pole system shown in Figure 1.
Fig. 1. The customary cart-pole system.
Note, in Figure 1, that the pendulum is constrained to move within the XOY
plane. As mentioned above, the control action is based on the horizontal displace-
ments of the electrical cart.
Exceptionally, some authors have considered an alternative control action con-

sisting of an oscillatory vertical force applied to the pendulum pivot. The stabiliz-
ing effect of a fast vertical oscillation applied to the pendulum base is known from
the early work of Stephenson in 1908 [12] . The Russian physicist Kapitsa was the
first, in the fifties, to produce a rigorous demonstration of the stability conditions
192192D. Maravall
of the IP when its suspension base oscillates at a high frequency [13] and, there-
fore, some authors use the expression Kapitsa pendulum to refer to this stabiliza-
tion technique [14]. Another control alternative is based on the application of a ro-
tational torque to the pendulum base, as proposed by Furuta and co-workers [15].
In fact, this arrangement leads to a different kind of planar IP, known as the rota-
tional IP [16] or, simply, the Furuta pendulum [17].
With the exception of vibrational control –i.e., based on oscillatory control sig-
nals- which is a well-known technique for controlling mechanical systems [18]-
[20], including, as mentioned above, the IP [21]-[23], the only previous work, to
our knowledge, that considers the application of vertical forces to stabilize the IP
was recently performed by Wu et al. [24] and [25], who employ the IP as a basic
element to analyze the postural stability and locomotion of multi-link bipeds. In
particular, Wu et al. model the IP base point according to cartilage and ligament
behavior in natural joints and they apply horizontal and vertical forces and, also, a
rotational torque to the base pivot. Using a very simplified linear model, the result-
ing overactuated control system is designed by means of Liapunov´s direct method
to obtain a desired trajectory of the IP´s center of gravity.
In this chapter, we conduct a detailed analysis of the possibilities of controlling

and stabilizing the IP by means of a vertical force. The chapter is organized as fol-
lows. First, we establish the dynamic equations of the IP under the action of a ge-
neric vertical force and then we analyze its control and stabilization. The main
conclusion is that the vertical force has an excellent stabilization effect, although it
requires a permanent fall of the IP support base when it is the only applied force,
which is, obviously, an unfeasible control policy. Therefore, we investigate the
combination of the vertical force with the customary horizontal force, arriving at
the stabilization conditions for different formal representations of the system: ordi-
nary differential equations, state variable representation and Liapunov´s direct and
indirect methods. In particular, we obtain the stability conditions of the IP for a PD
control algorithm. An appendix discusses some experimental results, including no-
table improvements in IP stabilization achieved when combining the customary
horizontal force with a vertical force.
2 Stabilization of the Inverted Pendulum

with a Vertical Force
In Figure 2 we have substituted the customary electrical cart by a platform of mass
M, on which the pendulum pivot is mounted. The pendulum has a total mass m and
length 2l. Apart from the gravitational force, the only existing external force is
purely vertical, Fy
The dynamic equations of the system can be straightforwardly obtained by ap-

plying Lagrange´s equations
193
Fig. 2. Planar inverted pendulum supported by a platform subject to a pure vertical force Fy
d $ ∂L * − ∂L = F (1)
dt %& ∂q#i +, ∂qi i; i = 1,2
L=K−P
where L is the Lagrangian, K is the kinetic energy and P the potential energy of the
system. Fi stands for the generalized applied forces and qi are the generalized co-
ordinates, which in this case are y and , respectively. As mentioned above, the
!
only existing force is F1 = Fy.
The kinetic energy is
K = 1 m ( x# p 2 + y# p 2 ) + 1 I θ# 2 + 1 M y# 2 (2)
2 2 2
where I is the inertia of the pendulum, which we will assume to be negligible from
now on. The coordinates of the pendulum´s center of mass are
x p = l sinθ ; y p = y + l cosθ (3)
where y is the vertical coordinate of the platform’s center of gravity, which we as-
sume to coincide with the pendulum hinge. As a consequence, we can also assume
that the pendulum mass is virtually concentrated at its top. The potential energy is
P = M g y + m g y p = ( M + m) g y + m g l cosθ (4)
After some operation, the Lagrangian turns out to be
L = 1 m l 2 θ# 2 + 1 ( M + m ) y# 2 − m l sinθ y# θ# − ( M + m) g y − m g l cosθ (5)

2 2
194194D. Maravall
By substituting into the Lagrange’s equations, we finally get the equations of the
system dynamics
( M + m) ##y − m l sinθ θ## − m l cosθ θ# 2 = Fy − ( M + m) g (6)

y + m l θ## − m g l sinθ = 0
−m l sinθ ## 2
which can be expressed in the standard compact form
M (q) q## + C (q,q# ) q# + G (q ) = τ (7)
where M(q) is the symmetric, definite positive inertia matrix, C(q,q) is the Corio-
lis/centripetal matrix and G(q) is the gravity vector
' y- ' (M + m) − m l sinθ -

q=( . ; M (q ) = (
)θ / )− m l sinθ m l 2 /.
'0 − m l cosθ θ#- ' (M + m) g - (8)
C (q,q# ) = ( . ; G (q ) = ( .
)0 0 / )− m g l sinθ /
'F -
τ = ( y.
)0/
It is interesting to compare these equations with the equations for the usual case in
which a pure horizontal force, Fx , is applied, as illustrated in Figure 1
( M + m) ##x + m l cosθ θ## − m l sinθ θ# 2 = Fx (9)

x + m l 2 θ## − m g l sinθ = 0
m l cosθ ##
In spite of the apparent similarity of formulae (6) and (9), the stabilization of the
IP in each case is totally different, as shown in the sequel.
As a first step in the analysis of this nonlinear system, let us focus on the situa-
tion of practical interest, namely, small IP movements around the unstable open-
loop position θ = 0, where we can introduce the following approximations
sin θ ≈ θ ; cos θ ≈ 1 − θ 2 / 2 (10)
which, substituted into the Lagrangian function (4), yields

195
L = 1 m l 2 θ# 2 + 1 ( M + m ) y# 2 − m l θ θ# y# − ( M + m) g y − m g l (1 − θ 2 /2) (11)
2 2
Then, from the Lagrange’s equations, we get the equation of small movements
( M + m) ##y − m l θ θ## − m l cosθ θ# 2 = Fy − ( M + m) g (12)

y + l θ## − g θ = 0
−θ ##
Similarly, for the horizontal force, we get
( M + m) ##x + m l θ## = Fx (13)

x + l θ## − g θ = 0
##
Unlike the dynamic equations of the horizontal case, which are totally linear, the
equations of the small IP movements are still nonlinear in the vertical case, mak-
ing the stability analysis and the control comparatively more difficult.
By solving the simultaneous differential equations of the horizontal force in θ,

we get
F
θ## − M + m g θ = − x (14)
Ml Ml
The respective characteristic equation yields hyperbolic sines and cosines and,
therefore, the pendulum dynamics is unstable. However, the horizontal closed-loop
dynamics can be straightforwardly stabilized [26] by introducing a feedback con-
trol law, such as a conventional PD algorithm
Fx = k pθ + k d θ# (15)
Unfortunately, this is not the case for the vertical force, as we then have an
autonomous, unforced pendulum dynamics
l θ## − ( g + ##
y) θ = 0 (16)
unless we consider vertical acceleration as an external control action. Specifically,

we observe that this equation is unstable if the following condition holds
196196D. Maravall
(g + #y#) > 0 (17)
as unbounded hyperbolic solutions are obtained in . On the contrary, if (g+ÿ) < 0,

!
then stable oscillating solutions are obtained, as the characteristic equation has the
following roots
g + ##
y g + ##
y (18)
r2 + = 0 → r1,2 = ± j
l l
so that
θ (t) = A cos [ω(t) ⋅ t + ϕ ] (19)
where the angular frequency (t) is time dependent, due to the variable vertical
!
force. Therefore, when the stable oscillatory condition ( g + ÿ ) < 0 holds, the pen-
dulum oscillates with a constant amplitude and a slowly varying frequency given
by
θ# 2 (0) ; ' −θ#(0) - (20)

A = θ 2 (0) + ϕ = tan −1 ( .
ω 2 (0) ) ω(0) ⋅ θ (0) /
g + ##
y
f (t ) = 1
2π l
The greater the vertical acceleration, the higher the frequency of oscillation. As for
the oscillation amplitude, given the usual condition ’(0) / (0) << (0) , it is vir-
! !
tually equal to the initial deviation (0).

!
As a first conclusion regarding the pure vertical force, we can guarantee a stable
oscillating behavior of the IP, if and only if the applied vertical force, Fy, produces
a negative vertical acceleration such that ## y < − g . As the stability of the IP de-
pends on the vertical acceleration, we must proceed with the analysis of the verti-
cal dynamics, which for small deviation angles of the IP turns out to be
( M + m ) ##y − m l θ θ## = Fy − ( M + m ) g (21)
By substituting the vertical acceleration given by (21) into the pendulum dynam-
ics, one obtains
197
' Fy m θ θ##- θ = 0
θ## − ( + .
(22)
) ( M + m )l M +m /
Then, the vertical force that guarantees oscillatory stability of the IP is
Fy < − m l θ θ## (23)
Assuming that the deviation angle has been stabilized by a vertical force satisfying
condition (23), then and its second derivative are sinusoids of opposite sign, so
!
that the upper limit of the stabilizing vertical force is always positive
Fy < m l θ θ## (24)
Now, let us suppose, conservatively, that Fy = 0, i.e., the platform-pendulum pair

is left in free-fall. In this case, the stability condition holds
( g + ##
y) = m l θ θ## < 0 (25)
M +m
and the respective small oscillations of θ will be of very low frequency
f (t) = 1 θ θ## (26)

2π M +m
with an amplitude given by (20).
Summarizing, the IP can be stabilized by means of the free-fall of the platform-

pendulum pair, which is obviously an impractical control strategy. For such rea-
son, we need to investigate whether it is possible, at some point after the stabiliza-
tion of the IP, to stop the platform falling and, even, to raise it to its original posi-
tion. More specifically, let us introduce a positive vertical force, Fy = (M + m) g,
just to balance the force of gravity, in which case the platform stops falling. Thus,
the dynamics of the vertical force is now
##
y= m l θ θ## (27)
M +m
which, substituted into the pendulum dynamics given by (16), yields the following
condition for maintaining the small oscillations of θ around the vertical position
ml ## > g (28)
M +m θθ
198198D. Maravall
Condition (28) is obviously violated for realistic values of the parameters M, m

and l and provided that we are, precisely, hypothesizing small values of the pendu-
lum’s deviation angle and its successive derivatives,. Consequently, the platform-
pendulum pair would have to remain in free-fall in order to maintain the stable ini-
tial oscillations.
Our preceding discussion of the IP stability conditions has been based on rather
qualitative reasoning. So, let us now take a more rigorous approach by analyzing
the Mathieu equation that determines the pendulum dynamics –see expression
(16)-. Thus, once the inverted pendulum has been stabilized by applying a negative
vertical force, the deviation angle of the pendulum and its two first derivatives are
θ (t ) = A cos(ω t + ϕ ) ; θ#(t) = − ω A sin(ω t + ϕ ) (29)

θ##(t) = − ω A cos(ω t + ϕ )
2
which, substituted into the vertical dynamics, yields the vertical acceleration
##
y= m l ω 2 A2 'sin 2 (ω t + ϕ ) − cos2 (ω t + ϕ ) - + Fy − g (30)
M +m ) / M +m
whence the pendulum dynamics turns out to be
− Fy
θ## + '() ( -
+ m ω 2 A2 cos2(ω t + ϕ ) .θ = 0
/
(31)
M + m) l M + m
which is, as mentioned above, the well-known Mathieu equation
x + (δ + ε cos(ω t ) ) x = 0
## (32)
The Mathieu equation, in particular its stability conditions, has been extensively
analyzed [1] and [27]. The analytical stability conditions of the Mathieu equation
confirm the conclusions that we drew from our qualitative discussion, namely, that
the IP stabilization with a vertical force – i.e., Fy > 0 and < 0 – is impossible, as
!
virtually all the corresponding negative half-plane encompasses instability regions

of the Mathieu equation. Another interesting result derived from the analytical
study of equation (32), not directly observed in the qualitative discussion, is that,
even for negative vertical forces –i.e., Fy < 0 and > 0 -, there are instability re-
!
gions that depend on the physical parameters Fy, M, m, l, (t) and (0) that must
!
be carefully analyzed. Given the complex interactions of these parameters and

their influence on the instability regions, the most advisable design strategy is to
choose as high as possible a and, afterwards, check an that does not drive the
!
"
system to an instability region.

199
As a general conclusion, the application of a single, sustained vertical force to

stabilize and control the IP is unfeasible, although its excellent and, in particular,
its fast stabilization effect makes the combination of the vertical force with the cus-
tomary horizontal force looks very attractive. Therefore, our next and central topic
is the stabilization of the IP via the combination of the vertical force with the cus-
tomary horizontal force.
3 Combination of Horizontal and Vertical Forces
After having investigated the stabilization of the IP by means of a vertical force,

we are now going to explore its combination with the usual horizontal force. The
mechanism for implementing the vertical force, Fy, is a platform of mass m’,
mounted on the customary electrical cart, which, as usual, produces the horizontal
force, Fx.
The total kinetic energy of the cart-platform-pendulum ensemble is
K = 1 M x# 2 + 1 m′( x# 2 + y# 2 ) + 1 m ( x# p 2 + y# p 2 ) (33)
2 2 2
and the potential energy
P = m′ g y + m g y p (34)
By applying Lagrange´s equations
d $ ∂L * ∂L (35)
% +− = Fi
dt %& ∂q#i +, ∂qi
q1 = x, F1 = Fx ; q2 = y , F2 = Fy ; q3 = θ , F3 = 0
we get the global system dynamics
( M + m′ + m ) ##x + m l cosθ θ## − m l sinθ θ# 2 = Fx (36)

( m + m′) ##y − m l sinθ θ## − m l cosθ θ# 2 = Fy − ( m + m′) g
cosθ ## y + l θ## − g sinθ = 0
x − sinθ ##
To analyze this highly nonlinear system, let us first make the following qualitative
remarks.
200200D. Maravall
1. From the study of the pure vertical force case, we know that the only exoge-
nous control action is vertical acceleration, which leads to the unfeasible stabiliza-
tion strategy of maintaining the platform in free-fall.
2. On the contrary, the pure horizontal force can be used as a feedback control
action that straightforwardly stabilizes the IP.
3. Due to the equivalence of the joint (x, ) dynamics of the combined case
!
given by (36) and the pure horizontal case, equation (9), we can, in principle, ex-
ploit the feedback stabilization capacity of force Fx by focusing on the joint (x, )
!
dynamics of the horizontal plus the vertical case and considering vertical accelera-
tion as an external control action.
Thus, let us rewrite the joint (x, ) dynamics of the combined case
!
( M + m′ + m ) ##x + m l cosθ θ## − m l sinθ θ# 2 = Fx (37)
x + l θ## − ( g + ##
cosθ ## y ) sinθ = 0
which is equivalent to the pure horizontal case –see equations (9)-, except that the
gravity dynamics is perturbed by the term ÿ sin . Remember that this perturbation
!
is generated by the vertical dynamics, given by the second equation of (36). Thus,
by considering vertical acceleration as an exogenous element in the pendulum dy-
namics, the combined forces case turns out to have the same formal structure as the
horizontal force case. Therefore, we can tackle the combined case as an ordinary
differential equation (ODE) problem and stabilize the IP via the horizontal force
Fx using any standard control law, as in [26]. Alternatively, we can also approach
the control problem with the state variable representation and stabilize the IP with
a plethora of available techniques, including Liapunov’s direct and indirect meth-
ods. Let us begin our study with the ODE approach, which conveys a very intuitive
and direct physical interpretation.
3.1 Ordinary Differential Equations Analysis
As usual, we are interested in the neighborhood of the IP vertical position, so

that we linearize the system dynamics (38) by approximating sin θ ≈ θ ,
cos θ ≈ 1 − θ 2 / 2 , θ# 2 ≈ 0 . After solving the ODE system in , we get
!
Fx
θ## − M + m′ + m ( g + ##y ) θ = − (38)
( M + m′ ) l ( M + m′ ) l
which is virtually equivalent to the pure horizontal case
F
θ## − M + m g θ = − x (39)
Ml Ml
201
Thus, let us apply a PD control law of the error variable
Fx = −k p e − kd e# = k p θ + kd θ# (40)
Which, substituted into (38), yields
kd # ' k p M + m′ + m
θ## + θ + )( − ( g + ##y ) /.- θ = 0 (41)
Ml Ml Ml
Again, it is equivalent to the close-loop horizontal force case, whose ODE [26] is
kd # ' k p -
θ## + θ + )( − M + m g.θ = 0 (42)
Ml Ml Ml /
the only remarkable effect of the vertical acceleration being on the root locus of
the combined forces case. As is well-known, both coefficients must be positive to
guarantee stability of (41)
kd
> 0 → kd > 0 ; k p > ( M + m′ + m ( g + ##
y )) (43)
Ml
The first condition has a straightforward interpretation; namely, it implies that the
feedback control force, Fx, must have a component directly proportional to the
pendulum angular speed . To illustrate this fact, the four possible states of the IP
!
have been represented in Figure 5. Note that in cases (b) and (c) the pendulum is
returning to its vertical position, while in cases (a) and (d) it is moving away from
it. In all cases, the orientation of the corresponding control force has been indi-
cated.
(a) (b) (c) (d)

Fig. 3. The four possible IP states. Observe the orientation of the respective feedback force.
The second stabilization condition in (43) is even more intuitive, as the feed-
back force must always have the same orientation as the IP’s angular displacement.
202202D. Maravall
Furthermore, the gain kp should guarantee the positiveness of the respective coef-
ficient. Note that when the platform is falling, the vertical acceleration ÿ strength-
ens IP stabilization. Inversely, when the platform is ascending to recover its origi-
nal position, the respective positive vertical acceleration detracts from IP
stabilization. Therefore, the vertical force component –i.e., the generation of verti-
cal acceleration- must be carefully designed to tackle with this double-sided effect.
Roughly speaking, when the pendulum is moving away from the vertical position,
the vertical force should be immediately activated to produce a strong negative
vertical acceleration . Inversely, with the pendulum recovering its vertical position,
we can make Fy > 0 to bring the platform towards its original position. In short, the
vertical displacement of the platform must be synchronized with the IP move-
ments. This general control strategy can be succintly formalized as follows
If [sgn(θ ) = sgn(θ# ) ] then ') Fy < 0 → ##

y < 0 -/ else ') Fy > 0 → ##
y > 0 -/ (44)
Apart from controlling the IP’s deviation angle, which is obviously the main goal,
it is also of interest to minimize the platform displacement, which must be con-
strained to some specific range. To this end, let us distinguish the following three
states of the cart-platform-pendulum ensemble.
1. The IP is moving away from the vertical position – i.e., sgn (θ ) = sgn (θ# ) .
2. The IP is returning to the vertical position –i.e., sgn (θ ) ≠ sgn (θ# ) - but is still
far from it.
3. As in state 2, but near the vertical position.
Note the fuzzy linguistic qualifiers introduced to make a further distinction in

the basic IP state returning to the vertical position.
Accordingly, we introduce the following control action for each IP state.
1. The pendulum is leaving the vertical position
Fy ( t ) = − )' k pθ θ ( t ) + kdθ θ# ( t ) /- (45)
2. The pendulum is returning to and is far from the vertical position
Fy ( t ) = − p ') k pθ θ ( t ) + kdθ θ#( t ) -/ − (1 − p ) ') k py y( t ) + kdy y# ( t )-/ (46)
where 0 < p < 1 weights the importance of the two control objectives: the IP devia-
tion angle, , and the platform movement, y. Note that the latter action is aimed at
!
minimizing the vertical displacement.

203
3. The pendulum is returning to and is near to the vertical position
Fy ( t ) = − ') k py y ( t ) + kdy y# ( t ) -/ (47)
We must remember that in our preceding discussion the vertical acceleration of

the platform-pendulum couple is given by
( m′ + m ) ##y − m l sinθ θ## − m l cosθ θ# 2 = Fy − ( m′ + m ) g (48)
so that the high nonlinearity of the vertical dynamics must be taken into account in
the tuning of the control parameters appearing in formulae (45)-(47).
Summarizing the basic philosophy of our combined forces control, the dynam-
ics of the IP is directly controlled by the customary horizontal force, plus the indi-
rect action of a vertical acceleration, ÿ, which, in turn, is controlled by the vertical
force given by formulae (45)-(47).
Continuing with our qualitative and general discussion of the combined forces
control, we are now going to proceed with the stabilization of the IP under the
state variable representation.
3.2 State Variable Representation
Although basically similar to the ODE analyis, the state variable representation can
be used to extend the possibilities of IP stabilization. In particular, as shown in the
sequel, we shall refine the stability conditions by means of Liapunov’s direct
method.
Following our proposed stabilization strategy, which basically involves control-

ling the joint (x, ) dynamics by means of the customary horizontal force and, si-
!
multaneously, by the vertical acceleration of the platform, let us introduce the fol-
lowing variables change in the pendulum dynamics given by (41)
x1 = θ , x2 = θ# (49)
to get the state variable representation. Thus, after some operation, we obtain
x#1 = x2 (50)
204204D. Maravall
( M + m′ + m ) ( g + ##y ) sin x1 − m l sin x1 cos x1 x22 − Fx cos x1

x# 2 = θ## =
( M + m′ ) l + m l sin 2 x1
which is of the more compact form
x#1 = f1 ( x1 , x2 ) ; x#2 = f 2 ( x1 , x2 ) (51)
Liapunov’s indirect method, also known as the first method, approximates the
nonlinear dynamics by the first, linear terms of the Taylor development around a
certain equilibrium point ( x1eq , x2eq )
∂f1
x#1 ≈ (x1 − x1eq ) + ∂f1 (x 2 − x2 eq ) (52)
∂x1 eq ∂x2 eq
∂f 2 ∂f 2
x# 2 ≈ (x
1 − x1eq ) + (x 2 − x2 eq )
∂x1 eq
∂x 2 eq
In vector form
x# ≈ J eq ( x − xeq ) (53)
where Jeq is the system’s Jacobian particularized into the equilibrium point of in-
terest, in our case, ( x1, x2 ) = (θ ,θ# ) = ( 0,0 ) . After some operation, we get for this
equilibrium point
' 1 0 - (54)
J eq = ( ( M + m′ + m ) − ∂Fx / ∂x1 1 ∂F .
( − x
.
) ( M + m′) l ( M + m′) l ∂x2 /
The characteristic equation, det ( λ I − J eq ) = 0 , yields
1 '( M + m′ + m ) ( g + ## ∂F 1 ∂Fx
λ2 − y) − x - λ + =0 (55)
( M + m′) l () ∂x1 ./ ( M + m′) l ∂x2
To guarantee stability, both coefficients must be positive
∂Fx ∂Fx
≡ > 0 # Fx = kd θ# ; kd > 0 (56)
∂x2 ∂θ#
205
which coincides with the respective ODE stability condition (43). Additionally,
∂Fx ∂Fx
≡ > ( M + m′ + m ) ( g + ##
y) (57)
∂x1 ∂θ
also coinciding with the second ODE’s stability constraint, provided that we apply
a PD control law such that F = k p θ + kd θ# .
Both the linearized ODE and Liapunov’s indirect method only guarantee the lo-
cal stability of the system. Liapunov’s direct method, also known as the second
method, is stronger as it provides the global stability conditions.
Thus, as for any mechanical system, let us try a Liapunov function based on the
total energy of the system
1 1 2
V (x1 , x2 ) = θ# 2 + (1 − cos θ ) = x2 + (1 − cos x1 ) (58)
2 2
which is definite positive for − π < θ < π , that determines a region beyond the
practical interest of IP stabilization. Its first derivative is
V# ( x1, x2 ) = x2 x#2 + sin x1 x#1 = x#1 ( x#2 + sin x1 ) = θ# (θ## + sinθ ) (59)
In order to get the global stability of the equilibrium point of interest

( x1, x2 ) = (θ ,θ# ) = ( 0,0 ) , we must guarantee that V# ( x1 , x2 ) is definite negative in a
region comprising the equilibrium point, so that
If (θ# < 0 ) then ')θ## + sinθ > 0-/ else ')θ## + sinθ < 0-/ (60)
∀θ / − π < θ < π ; ∀θ ≠ 0
Thus, solving θ## in the global nonlinear system given by (37) yields
θ## = 1
(61)
( M + m′) l + m l sin 2θ
y ) sinθ − m l sinθ cosθ θ# 2 − Fx cosθ -/
')( M + m′ + m ) ( g + ##
which substituted into (59) gives

206206D. Maravall
V# ( x1, x2 ) = − θ# {F cosθ − [( M + m′ + m ) ( g + ##y ) (62)

( M + m′) l + m l sin 2θ x
+( M + m′) l + m l sin 2θ − m l cosθ θ# 2 ] sinθ }
Note that in step (61) we have, again, made use of the control strategy based on
considering the vertical acceleration as an exogenous variable in the pendulum dy-
namics, which is the most feasible procedure for controlling the system when si-
multaneously applying horizontal and vertical forces.
From (62), we propose the control law
Fx = [( M + m′ + m ) ( g + ##
y ) + ( M + m′ ) l
(63)
+ m l sin θ − m l cosθ θ# /- tgθ + kd θ#
2 2
that with k d > 0 completely guarantees the definite negativeness of the first deriva-
tive of the Liapunov function for − π < θ < π
− kd
V# ( x1, x2 ) = θ# 2 (64)
( M + m′) l + m l sin 2θ
Note that the control law (63) can be expressed for the values of interest of the
variable as a conventional PD control law
!
Fx = k p tgθ + kd θ# ≈ k p θ + kd θ# (65)
In fact, the control law (63) is a refinement of the stability conditions that we ob-
tained in the ODE analysis and with Liapunov’s indirect method –see expressions
(43) and (56)-(57), respectively-.
Thus, by applying Liapunov’s direct method, we have arrived at a more precise

and refined control law that guarantees the global IP stability. Consequently, the
first step in the actual stabilization of the IP is to design a quantitative control law
by introducing specific performance indices such as rise time, settling time, per-
cent overshoot, bandwith etc. Afterwards, the global stability of the IP is guaran-
teed by additionally constraining the designed control law to satisfy the condition
given by (63).
207
4 Concluding Remarks
Although the use of high-frequency oscillating vertical forces for IP stabilization is

a well-known technique, the application of generic vertical forces to stabilize the
IP has not been, theoretically and practically, fully developed to date. In this chap-
ter, the novel idea of controlling and stabilizing the IP via vertical forces has been
introduced and thoroughly analyzed. After having established the dynamic equa-
tions of the IP with a generic vertical force applied to its base, we studied IP con-
trol and stabilization. The final conclusion is that the vertical force has an excellent
and fast stabilization effect, although at the cost of maintaining the IP in free-fall.
After this preliminary analysis, the chapter approaches its main contribution,
namely, the combination of the customary horizontal force with the vertical force.
Roughly speaking, the horizontal force permits a direct stabilization of IP by
means of a feedback control action, while the vertical force significantly improves
IP stabilization, mainly due to its fast response to external perturbations of the IP
equilibrium state. The theoretical analysis of the combined forces has been devel-
oped for both the ODE and the variable state representations. In particular, the
necessary and sufficient conditions of the local stability of the IP controlled by a
PD algorithm have been obtained. Furthermore, by applying Lyapunov’s direct
method, the control law, which turns out to be a PD-like feedback action that guar-
antees the global stability of the IP, has also been obtained. As a general conclud-
ing remark, the chapter has demonstrated the excellent properties of the vertical
force as regards the stabilization of the inverted pendulum.
Acknowledgments
The idea of stabilizing the inverted pendulum via a vertical force originated from
endless discussions with my father, Prof. Dario Maravall-Casesnoves of the Royal
Academy of Sciences, Madrid. The control law defined by expressions (46) and
(47) was proposed by Javier Alonso-Ruiz. Special thanks are due to Prof. C. Zhou
of the Singapore Polytechnic for very fruitful discussions and insightful comments
and for his invitation to write this chapter.
References
1. Khalil HK (1996) Nonlinear systems. Second edition. Prentice-Hall, Upper

Saddle River, NJ
2. Nelson J, Kraft LG (1994) Real-time control of an inverted pendulum system
using complementary neural network and optimal techniques. Proc. American
Control Conference, 2553-2554
208208D. Maravall
3. Zhou C, Ruan D (2002) Fuzzy control rules extraction from perception-based

information using computing with words. Information Sciences 142: 275-290
4. Ghosh BK, Xi N, Tarn TJ (1999) Control in robotics and automation: sensor-
based integration. Academic Press, San Diego, CA
5. Maravall D, de Lope J (2002) A reinforcement learning method for dynamic
obstacle avoidance in robotic mechanisms. In: Ruan D, D´hondt P, Kerre EE
(eds) Computational intelligent systems for applied research. World Scientific,
Singapore, 485-494
6. Schaal S (1997) Learning from demonstration. In: Mozer MC, Jordan M, Pet-
sche T (eds) Advances in neural information processing systems 9. MIT Press,
Cambridge, MA, 1040-1046
7. Sprenger B, Kucera L, Mourad S (1998) Balancing of an inverted pendulum
with a SCARA robot. IEEE/ASME Trans. Mechatronics 3 (2): 91-97
8. Schreiber G, Ott C, Hirzinger G (2001) Interactive redundant robotics: control
of the inverted pendulum with nullspace motion. Proc. IROS 2001, 158-164
9. Chung CY (2002) Balancing of an inverted pendulum with a kinematically re-
dundant robot. Int. J General Systems 31 (1): 1-15
10. Bloch AM, Leonard NE, Marsden JE (2000) Controlled Lagrangians and the
stabilization of mechanical systems I: the first matching theorem. IEEE Trans.
Automatic Control 45 (12): 2253-2270
11. Bloch AM, Chang DE, Leonard NE, Marsden JE (2000) Controlled Lagran-
gians and the stabilization of mechanical systems II: potential shaping. IEEE
Trans. Automatic Control 46 (10): 1556-1570
12. Stephenson A (1908) On induced stability. Philosophical Magazine 15: 233-
236
13. Kapitsa PL (1951) Dynamic stability of a pendulum with a vibrating point of
suspension. Zh. Ehksp. Teor. Fiz. 21 (5): 588-598
14. Fliess M, Levine J, Martin P (1995) Flatness and defect of non-linear systems:
introductory theory and examples. Int. J Control 61 (6): 1327-1361
15. Furuta K, Yamamoto M, Kobayashi S (1992) Swing-up control of inverted
pendulum using pseudo-state feedback. J Systems Control Eng. 206 (6): 263-
269
16. Widjaja M, Yurkovich S (1995) Intelligent control for swing up and balancing
of an inverted pendulum system. Proc. IEEE Conf. Control Applications, 534-
542
17. Fantoni I, Lozano R (2001) Non-linear control for underactuated mechanical
systems. Springer, Berlin
18. Meerkov S (1980) Principle of vibrational control: theory and applications.
IEEE Trans. Automatic Control 25: 755-762
19. Bogaevski V, Povzner A (1991) Algebraic methods in nonlinear perturbation
theory. Springer-Verlag, New York
20. Hillsley KL, Yurkovich S (1993) Vibration control of a two link flexible ro-
bot arm. Dynamics and Control 3 (3): 261-280
21. Acheson D (1993) A pendulum theorem. Proc. Royal Society of London, se-
ries A 443: 239-245
209
22. Baillieul J, Lehman B (1996) Open-loop control using oscillatory inputs. In:
Levine WS (ed) The control Handbook. CRC Press & IEEE Press, Boca
Raton, FL, 967-980
23. Dimeo RM, Thomopoulos SCA (1994) Novel control of an inverted pendu-
lum. Proc. American Control Conference, 2185-2189
24. Wu Q, Thornton-Trump AB, Sepehri N (1998) Lyapunov stability control of
constrained inverted pendulums with general base point motion. Int. J Non-
linear Mechanics 33: 801-818
25. Wu Q (1999) Lyapunov´s stability of constrained inverted pendulums. Proc.
American Control Conference, 293-297
26. Raya A, Maravall D (2001) Contribution to the control and stabilization of the
pole-cart system. In: Moreno-Diaz R, Buchberger B, Freire JL (eds) Com-
puter aided ystems theory. LNCS 2178, Springer, Berlin, 437-449
27. Jose JV, Saletan EJ (1998) Classical dynamics: a contemporary approach.
Cambridge University Press, Cambridge
Appendix. Experimental Results
J. Alonso-Ruiz and D. Maravall

Department of Artificial Intelligence, Faculty of Computer Science
Universidad Politécnica de Madrid, Madrid 28660
SPAIN
We briefly present some of the experimental results obtained from computer simu-
lations, in which IP stabilization via the combination of horizontal and vertical
forces is investigated and compared with a single horizontal force. We have con-
sidered the values of the system parameters to be as follows: cart mass, 2 kg; plat-
form mass, 0.2 kg; pendulum mass and length, 0.1 kg and 0.5 m, respectively.
Unless otherwise indicated, distances are in meters (m), time in seconds (s), forces
in newtons (N) and angular displacements in radians in all figures.
In all the reported examples the respective PD algorithm gains have been ob-
tained to optimize the usual performance indices: rise time, overshoot peak and
settling time, either for the combined horizontal and vertical forces or for the sin-
gle horizontal force. Furthermore, for the combined case, the stability conditions
obtained in the theoretical analysis –see expressions (43), (56), (57) and (63)- have
been applied. Figure 4 shows the IP trajectories for initial deviations from 5º to 30º
in 5º steps.
210210D. Maravall
(a) (b)
Fig. 4. Instances of IP stabilization with (a) a combination of horizontal and vertical forces
and with (b) a single horizontal force.
Note the significant improvement achieved by the addition of the vertical force,
which makes the IP stabilize much faster. In particular, the greater the IP initial
deviation, the stronger the positive effect of the vertical component. It is also in-
teresting to compare the control efforts, so the respective horizontal forces profiles
are depicted in Figure 5. Note the + 20 N restriction over the range of the applied
forces. The same initial deviations as in Figure 4 have been considered.
(a) (b)
Fig. 5. Profiles of the horizontal forces of (a) the combined case and (b) the single horizon-
tal case. The greater the initial deviation, the stronger the applied force.
Note that the reductions in the control effort achieved with the combination of
horizontal and vertical forces are significant. Of course, in these cases there is an
additional control effort produced by the vertical force, although its respective en-
ergy cost is comparatively negligible, because the platform mass is small in com-
parison to the cart mass. Figure 6 shows the vertical forces profiles and the respec-
tive vertical displacements of the platform.
211
Fig. 6. Vertical forces profiles and platform vertical displacements for the same range of
initial deviations as above.
212
Vision Based Motion Planning

of Humanoid Robots
M. Fikret Ercan and Changjiu Zhou
School of Electrical and Electronic Engineering, Singapore Polytechnic,

500 Dover Road., Singapore 139651, Republic of Singapore
http://www.robo-erectus.org
Abstract. Vision is the most effective sensing form for humans in understanding
their environment, recognition of the objects around them and navigating from one
point to another. Humanoid robots mimic human behaviour and they are mainly
intended for applications where robots need to work in an environment designed
for humans. Naturally, computer vision serves as a dominating sensing unit for
them. Computer vision is a powerful tool. In addition to steering the robot, it can
also play a key role in achieving robot’s intelligence and its interaction with out-
side world. This paper discusses various aspects of exploiting computer vision in
humanoid robots.
1 Introduction
In the past, robots were predominately developed and utilized in factories for
manufacturing and transportation purpose. However, a new generation of robots
has recently begun to emerge. These are service robots that cooperate with people
and assist them in their everyday tasks. Many wheeled mobile robots, operating in
indoor or outdoor environments, have been studied extensively for this purpose.
Lately, researchers are interested in developing human-like robots named human-
oids. The main objective of realizing a human like robot is to develop robots that
can operate in an environment designed for humans. Ideally, it is expected that
such robot would be a replica of humans with motion, sensing, reasoning, and so-
cializing capabilities and even emotional responses. The research in this field
promoted by well-respected researchers [3], though, it was initially perceived as an
ambitious research challenge. Recent developments, how-ever, demonstrate that
emergence of humanoid robots in our daily life is not too far. A significant exam-
ple to this is P2 and P3 and Asimo robots developed by Honda [12]. With such as-
tounding demonstrations, research in humanoid robots gaining popularity and it is
attracting many research institutions [21, 18, 16]. For instance, humanoid SIG is de-
signed to study integration of perceptual information to control robot [18]. The pro-
totype H7 is developed by Kawada industries to provide an experimental research
213
platform for full-body integrated sensing and control [16]. Recently, Sony an-
nounced a new biped robot [27]. Sony’s the SDR-4X is 50cm tall, weighs 6.5kg,
and uses a pair of 64-bit processors. It has speech recognition, speech synthesis,
and stereo CCD vision sensors to provide face recognition and distance measuring
capabilities.
Earlier work in humanoid robots mainly concerned with elemental functions
such as bipedal walking of robot, control of algorithms for smooth movement, and
multi-sensor fusion. A significant amount of study reported in literature was on
the study of bipedal locomotion and many methods introduced for controlling ro-
bot walking (see for instance [15]). With the improvements made on the stable bi-
pedal walking of humanoids, attempts to make them autonomous become viable.
Integrating a vision based perception system can advance robot locomotion and
behaviour. Using computer vision in robotics is not new and there are many vision
guided mobile robots reported in the literature [20, 6]. These systems generally util-
ize a dedicated hardware especially to rapidly extract useful information, such as
depth information from stereovision [21], obstacles or land marks, to steer the robot
reliably. However, in the case of humanoids integrating a sophisticated computer
vision unit does not only assist to steer the robot but also allow it to interpret vis-
ual data to recognize its environment, plan a proper motion and interact (even so-
cialize [2]) with its environment autonomously.
The capability of a robot’s understanding of its environment is a key parameter
to achieve these goals. A well-known approach for this purpose is model-based
processing of image data [20, 28]. It is a goal oriented approach and requires knowl-
edge about the robot’s environment through partially modelling of unchanging
parts in that environment. Basically, the following objectives are intended to
achieve:
• Localizing the robot by analyzing the scene and finding a match of the current
scene with those stored in a database.
• Detect obstacles on a mobile robot’s path. The knowledge based vision system
is used to detect unexpected object features in a scene. This is achieved by find-
ing mismatches with the world model defined in the system.
• Plan robot navigation by referring to correspondence between image features
and expected landmark locations.
Perception is achieved by a hypothesis and test method where the model-based
system produces hypothesis about the objects in the scene and its relationship with
other objects and verifies them with sensory input.
Recently, there are other innovative approaches used in vision-guided systems
that significantly differ from classical sensory input, analysis and action type of
control scheme. Instead, a neural network, which acts as a controller, is used be-
tween input pixels and motor drivers (see for instance [1, 22, 3]). This method pro-
poses that a robot will develop a proper behaviour by progressively learning its
environment and training its neural network controller. Apparently, a major draw
back is the time consuming training process of the network. This is an active re-
search area though application to robots operating in an uncontrolled environment
is not immediate. Current research demonstrates solution to rather simple problem
cases.
214214M.F. Ercan and C. Zhou
This paper is mainly focused on application of vision in humanoid robots in

particular perception and motion planning aspects. In the following section, vari-
ous features of computer vision will be discussed. Integrating a computer vision
system to humanoid robot will be elaborated in section 3 and underlying research
issues will be presented. Section 4, describes a humanoid robot and its vision sys-
tem developed in our institution.
2 Challenges of Computer Vision
A governing human sensing is vision; therefore, harnessing an artificial one will

benefit attaining autonomy for robots. A truly autonomous system must posses the
decision-making ability (intelligence) to work in a non-deterministic world, like a
human being. A reliable decision-making is highly depended on the information
collected from sensors. As images from the surrounding environment provide the
most valuable information, many robots employ a vision based sensory unit
[3,20]. However, there are two main factors governing a robust vision system and
they can be broadly categorized as software and hardware issues.
2.1 Software issues
Unlike the human vision system, which is endowed with natural mechanisms to
filter out and reduce the burden of irrelevant information, a vision-guided robot
must be able to extract useful information from the input images. However, there
is no single or standard solution to achieve this goal. Usually a set of image-
processing algorithms is selected for the application at hand, and they are acquired
from a wide selection of fields such as signal processing, statistics, and artificial
intelligence. There is no unambiguous definition on how algorithms are selected,
though, they are usually organized in a hierarchical order and the type of image
processing can be broadly classified into three levels considering the complexity
and data type involved in the operations [28].
Algorithms at the lower level of image processing deal with extracting basic
object features, such as corners, size, shape, orientation, linear line segments etc.
Operations mainly deal with array of numerical information.
The objective of the intermediate level algorithms is to group features extracted
from the image. The output of intermediate level algorithms is no more pixels but
symbols that represent object features found in the image.
In the higher-level, hypothesis regarding the elements in the scene are gener-
ated and verified with detected features and finally information about the scene is
collected for interpretation. Operations here are model directed and usually utilize
spatial relations in object features. Planning and initiating an action, based on the
scene analysis, are also conducted at the high-level.
The controlling of the computer vision operations is in bottom-up, top-down or
hybrid manner. Bottom-up approach requires a complete set of correct features to
215
carry out interpretation and this is not possible in practice. On the other hand, top-
down analysis is model guided. The features being sought trigger the operations.
Hybrid approach combines these two approaches where a bottom up feature ex-
tracting is performed first and then followed by a top-down verification and
matching phase.
The selection of algorithms and the control of vision operations are user de-
pendant and they are not perfect. Furthermore, due to occlusions of objects, noise,
and shadows low level image processing may not produce a good result. Conse-
quently, more intelligent algorithms needed for the high level processing to be
able to reason with incomplete data. An additional pre-processing is needed in
humanoid robotics application to compensate the errors occur in camera position
at every step of the robot.
2.2 Hardware issues
Vision computation in robotics is highly demanding, mainly due to the amount of

data and various types of operations involved. For instance, computations in a
typical robotic application must be completed within a constant time frame, which
ranges from 25-100 msec. A well-known approach to overcome this hurdle is to
employ multi-processor systems. However, considering the computation structure
discussed earlier, it is expected that such system should be tailored to the require-
ments of computer vision. For instance, the system must be capable of manipulat-
ing various data types such as pixels, symbols and other complex structures. The
system must perform low, intermediate and high-level algorithms simultaneously
and it must have a fast data I/O mechanism. A good scalability of computational
performance, flexibility and modularity are also desired features. There is a vast
amount of research presented to meet computational requirements of computer vi-
sion. An extensive discussion on high performance computer vision architectures
can be found in the following references [7, 9].
It is difficult to address this great variety of requirements by a single architec-
ture and there are various approaches used in research community. For the pur-
pose of compact and feasible hardware realization, application specific systems
are developed. These are usually VLSI based compact systems but they are not
flexible [8, 23]. A large number of them are proposed to speed-up low-level image
processing such as convolution, median filtering etc. The other common approach
is to construct a programmable and flexible system to address variety of algo-
rithms used in computer vision. These systems employed either a heterogeneous
multi-layer or a partitionable/reconfigurable structure. Heterogeneous multi-layer
systems combine multiple layers of computing platforms and provide a suitable
one to each abstract computation level, such as SIMD arrays for low level and
MIMD platforms for intermediate and high level processing. When processing
continuous stream of images, executing three processing levels simultaneously
generates a pipelining effect. On the other hand, partitionable/reconfigurable sys-
tems provide a pool of processors that can be partitioned into clusters and config-
ured as SIMD, Systolic, or MIMD depending on the application that is being han-
dled by each cluster [7]. A major shortcoming of these systems is their huge hard-
ware complexity that resulted from efforts to achieve flexibility.
Recent systems favour dedicated image-processing units based on VLSI large
processor arrays [8, 23], or a network of DSPs [9,11]. Recently, FPGA arrays become
popular due to their easy hardware configurability [29].
Problem size for a humanoid robot operating at indoors can be regarded as me-
dium-scale considering the data size and the processing requirements. Basic archi-
tectural requirements from vision hardware are the same, but the system should be
feasible and hold a low hardware complexity so that it can be realized easily. The
other concern in humanoids is apparently the physical limitations. The hardware
included to the system has to be compact and power consumption should be
minimal. This implies that achieving a compact vision hardware using off the
shelve components is rather difficult task.
3 Employing Computer Vision in Humanoid Robots
We have reasoned that computer vision can provide significant information to a

humanoid about its environment. In order a humanoid to have visual sensing vari-
ous computer vision areas have to be integrated. We can categorize them as navi-
gation, interaction and coordination. In the following, we will elaborate these ar-
eas.
3.1 Navigation
Computer vision techniques in this category, deal with gathering information

about the outside world by recognition and interpretation of the environment, ex-
tracting range information and tracking objects or landmarks. This information is
then used to guide the robot such a way that it can negotiate outside world autono-
mously. There are many techniques fall into this category, though, we will point
out three well-known methods: model based scene interpretation (or analysis),
stereo depth analysis and active vision.
Model based scene analysis: The major goal of the vision is to detect and locate
landmarks and obstacles to plan navigation. Interpretation system generates a se-
mantic and symbolic description of a scene in terms of detected objects and their
spatial relations. Model based scene analysis would be the most suitable tool for
humanoids operating indoor environments. The world model predefined to the ro-
bot and its navigation is goal directed. Hence, in each step robot is able to localize
itself and how far from the goal by comparing the model and real scene. There are
many significant work has been done in this field (see for instance Sandakly et
al.[26] as an example). Model-based object recognition or, more generally, scene
interpretation is a two-part process. The first phase is to generate a sequence of
hypotheses on object identities and poses, and the second phase is to evaluate
them based on the object models. The overall structure of computations includes
217
algorithms from three image processing level as described earlier in Section 2.

Major difficulties involved in scene interpretation are in selecting and tuning im-
age-processing algorithms that deliver object features effectively, optimizing the
processing time by selecting relevant data from the scene, and uncertainties and
impression of models and data gathered from scene. Among them the most diffi-
cult issue to deal with is the uncertainties and many studies using Bayesian prob-
ability, evidence theory and fuzzy sets have been presented earlier in the literature
[4]
.
For visual analysis of natural scenes, which include incompletely visible ob-
jects in an uncontrolled context, it is a highly intricate task to optimize the match
of a model to the data obtained from the scene. Local optimization techniques will
usually get stuck in local optima while techniques similar to exhaustive search are
limited by time and memory constraints. Consequently, in practice a critical aspect
of many object recognition problems is to develop a clever search method.
Stereo Analysis and 3D modelling: Humanoids, akin to any autonomous robot,
have to interpret 3D structure of their environment. There are two main ap-
proaches used in obtaining range information of object or constructing a 3D map
of a robot’s environment. One method is using active sensors such as sonar, radar,
or laser range finders. Sensors generate the signals and the bouncing signals from
the objects provide range information. The second method is using stereovision.
Cameras are placed with a fixed baseline and depth can be obtained by computing
the disparity in each point in image pairs. Apparently a key problem in stereovi-
sion is to find corresponding points in stereo images. Corresponding points are the
projections of a single point in 3D world in the different image spaces. The differ-
ence in the position of corresponding points in their respective images is called
disparity. There are many algorithms presented for the solution. They are gener-
ally based on window based correlation technique. The performance of the algo-
rithms is depended on selecting a proper window size and it is not trivial. Adap-
tive techniques for the solution introduced by Kanade et. Al. [17].
Apparently most of the humanoids presented in the literature employ stereovi-
sion to identify robots environment. In some of the applications, stereovision was
employed to track a target and to detect mobile objects [18, 21]. An interesting appli-
cation of stereovision in humanoid robots is demonstrated in [21], where stereovi-
sion is used to estimate the grade of the ground. The purpose of the study is to cal-
culate the proper posture of the humanoid so that it can walk not only on flat
ground but also on the uneven terrains such as slops. Another particular applica-
tion of stereovision in humanoids would be to perform leaping over an obstacle.
This task is particular to humanoid robots and solution requires utterly different
approach than the conventional methods used for obstacle avoidance in wheeled
mobile robots.
Active Vision: The ability of actively controlling camera parameters is vital to
achieving robust, real-time perception for a robot interacting with a complex, dy-
namic world. Active vision systems have mechanisms that can actively control
camera parameters such as orientation, focus, zoom and aperture in response to the
requirements of the task and external stimuli. More broadly, active vision incorpo-
rates attention that is selectively sensing in space, resolution and time by modify-
ing physical camera parameters or the way image data is being processed. An ac-
tive vision system actively extracts the information it needs to solve a task. For in-
stance, a camera-equipped robot can change the direction of its camera is looking
at, or change the zoom. In other words, an active vision system always interacts
with the world with a tight coupling between perception and actions. The process-
ing is also tied closely with the activities it supports, such as navigation or
manipulation, allowing simplified control algorithms and scene representations. In
this sense, the concept of active vision is different from traditional method of find-
ing detailed reconstructions of the external world.
The benefits of active vision were primary interest in a variety of humanoids
projects. A well known application of active vision is Kismet humanoid robot
where the robot socializes with humans by analyzing their behaviours [2].
3.2 Gesture recognition
Many robots developed today do not interact with people. As humanoids start ap-
pearing in our daily life, such as health-care, nursing, and entertainment, human-
robot interaction will be vital. Non-expert users will operate these robots therefore
a natural interface that makes instructing these robots as simple as possible would
be very useful. In this respect, gesture recognition, face recognition, facial gesture
recognition come handy to use in humanoid robots.
Gesture recognition: Gestures are an important aspect of human interaction,
both interpersonally and in the context of man-machine interfaces. There are many
aspects to the modelling and recognition of human gesture as they can be ex-
pressed through hands, faces, or the entire body. Gesture recognition is valuable in
applications requiring human/robot interaction as it can provide a redundant form
of communication or it can help to clarify spoken commands.. For example, the
user may say “left” simultaneously pointing to left direction with a hand gesture.
The robot needs to recognize only one of the two commands. This is beneficial in
situations where speech may be distorted or suppressed by environmental noise.
Furthermore, iconic information can easily be given to the robot. For instance, in-
stead of giving coordinates of the target point that robot should move, the user can
simply point to that spot. Motion gestures provide additional freedom in the de-
sign of robot interaction with gestures. In addition, they reduce the chances of ac-
cidentally classifying arm poses as gestures hence they appear better suited for
human robot interaction than static pose gestures.
While the area of gesture recognition is relatively new, there has been a great
deal of activity in the last few years. Some of the vision-based interfaces that al-
low people to instruct mobile robots via arm gestures can be found in the follow-
ing references [10, 19].
Face Recognition: Over the years there is a vast amount of search on face rec-
ognition and facial expressions [30]. Performance of these algorithms, such as ei-
genfaces, is depended on accurate detection of faces as they assume that face im-
ages are normalized in terms of scale and orientation. In numerous studies, face
detection techniques demonstrated such as using motion detection or skin colour.
219
Colour provides a computationally efficient yet effective method that is robust un-
der rotations in depth and partial occlusions. A neural network based algorithm is
also demonstrated by Rowley et. al. [25], where a neural network examines small
windows of an image and decides whether each window contains a face. In a re-
cent study, a model-based approach, using Hausdorff distance has been demon-
strated where the face recognition is realized without a face detection step [14].
The detection, recognition and model-based coding of faces have a potential
application in humanoid robots as they are interacting with humans. Therefore, it
is necessary to detect faces and facial gestures and react accordingly. For example
the application presented in [8], demonstrates a robots interaction through facial
recognition. The robot interprets the head gesture nodding as “Yes” and shaking
as “No”. The face recognition also endow robot with the capability of identifying
people. It can be practical for security/surveillance applications or developing per-
sonalized robots.
3.3 Coordination
Human beings easily use eyes to guide their arms and hands to reach objects, and
to perform any particular pose. A robot that can imitate such hand-eye coordina-
tion needs the sophisticated skills of perception, spatial reasoning, and communi-
cation [13]. This task requires a combination of computer vision methods stereo and
active vision components in particular. Grasp reflex would be one of the main ob-
jectives. The relative positions of hand and target have to be observed and the mo-
tion is refined until the object is grasped. A major challenge is the dynamic nature
of the robots interaction with its environment and with human. It is impossible to
program the robot anticipating every possible situation; hence the common con-
sensus in achieving this goal is by robot learning. (see for instance, Cambron et
al[5] ).
3.4 Key research areas
The major areas of visual perception and intelligence envisioned for humanoid
studies discussed above are still open to further studies. Although, each topic is a
major research area by itself, a key research area is the integration of them. In
terms of a full-scale computer vision application, a few systems deployed in the
literature and so far fairly limited progress has been achieved. This is mainly due
to a lack of robustness and methods for integration of various techniques into full
systems. The construction of systems gradually provides the insight to formulate
basic methods for the design, analysis, implementation, and evaluation of opera-
tional and robust systems that have a wide domain of application.
4 Design of Humanoid Robo-Erectus
Currently, we are developing humanoid robots in our research lab [24]. The ultimate
goal of this project is to build humanoid robots that can play a soccer game akin to
human. The idea of soccer playing robots has been introduced in early 90’s. Ap-
parently, the basic idea behind this initiative was to advance artificial intelligence,
control, and robotic fields. Soccer playing is used as a test bed to represent dy-
namic real world. The response from the rest of the scientific world was strong,
and the Robot World Cup Initiative, RoboCup, was started. The first Robot World
Cup, RoboCup-97, was held in Nagoya, Japan, in August 1997, and included the
participation of more than 40 teams. Since then, RoboCup became an annual
event. The vision is to develop a humanoid team which will have the potential of
defeating a real world cup winning team of humans in fifty years. Apparently, this
challenge poses all the possible set of problems in artificial intelligence field to
tackle.
The first step of the project was to build a robust humanoid robot that can per-
form bipedal walk, for straight, curved motions and in-place turns. The second
stage of the project includes identifying the goal, objects, team mates and perform-
ing basic moves such as kicking, passing and shooting the ball. The third stage of
the project would be the acquiring of a team strategy. Currently, we have devel-
oped a number of robots with different specifications. These humanoids can per-
form a steady bi-pedal walk, turn, and ball-kicking type of motions. The first ver-
sion of the robot had 10 DOF while the latest version has 22 DOF.
4.1 Structure of the humanoid
The task of soccer playing requires a humanoid to perform various behaviours and
motion types and some of them fairly complex. For the successful realization of
the first stage of the project, we had to consider all these motions and provide a
high DOF to humanoid. The first stage of the project was completed successfully
and demonstrated in RoboCup humanoid walk league 2002. Fig 1 shows our hu-
manoid robot at work.
Three control systems developed for these robots. They are PC-based, micro-
controller based, and PDA based controllers. Main consensus behind the PDA
based controller was to reduce to the cost of humanoid robot and make it more
portable and compact. The latest version of the humanoid includes all the limbs
and neck joints. In Fig 2, we present the organization of the basic joints and the
sensor positions on the robot.
4.2 Humanoid vision unit and related tasks
The basic structure of humanoid robot is stable and flexible. Currently, we are
tackling with challenges at the second stage of the project that requires a vision-
221
Fig. 1. Robo Erectus performing a steady walk
based unit to be incorporated to the system. Our immediate goal is to perform the
tasks described in RoboCup humanoid league. These tasks are briefly:
1. Performing a steady walk and making a U turn at a designated landmark
2. Detecting a ball and kicking
3. Executing a penalty kick against a goal keeper.
These objectives can comfortably be accommodated via a vision system. Fig 3 de-
picts a block diagram of the vision unit’s place in robot’s control structure. As
shown in this figure, vision unit provides feedback both local and global action
planning components of the controller.
The first task is about navigation and algorithms discussed earlier in Section 3.1
need to be incorporated. Here, vision unit provides information to global motion
and path planning units; they are the scene interpretation and stereo depth infor-
mation. The scene interpretation reports the spatial relationship about the objects,
and between the robot and objects. Here, the whereabouts of landmark and robot’s
spatial relation to it is vital to make a proper walk and U-turn around the land-
mark. The stereo depth information is used to provide 3D-proximity data to robot.
We employ a knowledge based top-down approach for the algorithm as the robot
is operating in a predefined environment. Our method is similar to that discussed
in [20] though in recognition phase we are also utilizing color parameter.
The second and third tasks are more complex as they require detecting a ball,
approaching to it and kicking. For a penalty kick against a goalkeeper, robot needs
to develop a strategy and it should have a reasonable control on kicking the ball to
a certain direction. The techniques used here fall into coordination topic discussed
earlier in section 3.3. Here, challenges are more on the local motion-planning unit
and we are basically dealing with two situations static and dynamic. In the static
Accelerometers and z
gyros are mounted to yaw pitch
y
provide body posture
information
row x
Vision system
4 force sensing
sensors for each foot
Fig. 2. The humanoid structure and sensory structure
case, robot negotiates with static objects and landmarks. For instance robot needs
to detect boundaries of the soccer field, position of the ball etc. In order to detect
the ball we use color and shape analysis. The goal tracking function should be
considered as a higher level of the basic locomotion control system in order to im-
plement a hierarchical autonomous locomotion control system as shown in Fig. 4.
For the static case planning has two major steps. The first step is to determine the
path to the goal, and the second step is to calculate the width of each step and the
turning angle of Θ.
In the dynamic case, we deal with moving objects such as the goalkeeper.
However, in this problem both robot and the target are in motion. The moving
goal tracking function consists of the following tasks:
1. Detection of the 3D position measurement of the goal in camera coordinates.
As the humanoid and the goal are both moving, the goal’s color rather than its
shape is utilized to detect the goal.
2. Planning of footstep for one step: For the path and footstep planning, we use
the world coordinate system which is more convenient for acquiring the knowl-
edge of the goal motion and the map information.
3. Camera posture control with self motion compensation: In order to keep the
goal in the field of view, the camera posture is controlled by the pan and tilt
joints of the neck. This can be perceived as an application of active vision in
humanoids. That is small deviations in robot’s gazing point due to walking mo-
tion is compensated by using head and neck joints. Robot has to keep its dy-
namic stability. However, unlike wheeled robot, humanoid’s body doesn’t
move along with the moving direction. Another, practical issue is vertical posi-
tion of the camera. Currently, cameras are fixed on the robot’s shoulders, which
223
result in rotation of input images up to 12 degrees with the robots bipedal walk-
ing. To compensate this rotation, a camera self-motion compensation system is
needed. For this purpose we are experimenting two approaches, one of them is
using a feedback from the gyroscope integrated into humanoid’s body and us-
ing the tilt joint to correct the camera position. It gives a measure of rectifica-
tion needed to correct the error occurred in camera alignment due to the robot
body movement. The second approach is by fixing a landmark in the world
model in the scene and counter rotating the image pixels by computing the op-
tical flow between two consecutive image frames. This method is purely in
software and computationally demanding.
4. In addition to calculations made for path planning, step size and turning angle
in the dynamic case we also calculate the speed of the walk in order to cope
with the dynamics of the moving goal.
Humanoid
Path Planning Gait synthesis Joint controller
robot
Global motion Local motion Potentiometers/

planning planning encoders
Range sensors/gyros/
force sensors,...
Vision
Fig. 3. A block diagram of vision and other sensory input to robot control mechanism
4.3 Architecture of the vision unit
The humanoid project is open ended and the future tasks of the humanoid will be
more and more complex. Current vision system utilizes a single CPU and it will
be limited for them. Hence, we have to take hardware aspect of the vision system
into account and provide a dedicated architecture for it. Our goal is to design a
system with minimal hardware complexity while satisfying the basic architectural
requirements demanded in a computer vision system in general. For this purpose,
currently we are building a multi processor unit with latest parallel DSPs from
Analog Devices. This class of processors provides computation flexibility sought
for the system as they can efficiently perform both low-level number crunching
type algorithms as well as high-level algorithms that require manipulation of data
structures. The hardware configuration for data communication is also made flexi-
ble by latest FPGA devices. Processors of the system can be partitioned and pipe-
lining of vision operations can be realized easily since FPGAs used in data com-
munication frontline. Each DSP holds its own memory and four of them share a
common memory for rapid data transfer. The vision architecture has two separate
units; they are a pool of processors and a image capture unit. A block diagram of
this system is presented in Fig 5.
Fig. 4. Path and footstep planning for humanoid turning
The structure of the processing unit is modular and it is built by integrating ba-
sic building modules. In each module one of the four DSPs acts as a controller or
master processor to the others. In addition to parallel computing, this unit also
aimed for multiprogramming. That is modules carry out vision tasks independ-
ently, this leads to a two-way communication structure between the modules as
well as between the processors of each module.
Intra-Layer Communication: although vision system is developed with building
blocks available in the market, we organize these basic modules and their commu-
nication hierarchy. The first level of the communication is the interaction between
the processors of a module. Here, a hybrid communication platform is employed
for this purpose. That is in each module processors can directly communicate to
each other via a communication link or by using a shared memory among them.
When processing images in a parallel architecture, collective communication op-
erations, such as broadcast, scatter and gather, are needed very often and they are
time consuming operations. Shared memory is suitable for collective communica-
tion, whereas direct links for data exchange between sub-programs running at in-
dividual processors.
Inter-Layer Communication: This mechanism required for transferring results
from one module to another. Communication by using shared memory between
modules is most likely to be a bottleneck on the performance and will limit the
flexibility. Therefore, we directly connected processors at successive modules.
225
The inter-layer communication allows pipelining of vision tasks and it enhances

computation performance.
Fig. 5. The architecture of the latest vision unit of the humanoid robot
The scene interpretation algorithm utilizes these modules simultaneously. The

pipelining effect is achieved when we process multiple frames. The flow of opera-
tions is bottom-up at first and then top-down at the second phase. In the first
phase, mainly low- level image processing algorithms are performed for feature
extraction and features are grouped to form tokens or vectors to represent objects.
At the second phase, matching of objects with model base is performed and using
knowledge (world model) about the spatial relations of objects, a description of
scene is built. At this stage hypothesis about missing object features or scene pa-
rameters are also generated and verified with further processing the input image.
5 Conclusions
In this paper, we have discussed issues concerning humanoid robots vision. The
desired intelligence and human like behavior of humanoids is strongly dependent
on the superiority of vision systems embedded into their design. Whilst it may
seem over ambitious to incorporate so many different aspects of vision into a hu-
manoid robot, desired goal can only be achieved when various visual operations
are employed simultaneously. In this paper, we have also briefly introduced the
humanoid robot developed in our institution. The ultimate goal of this humanoid
robot is to play soccer game. This task is used as a test bed as it embodies all the
major techniques discussed above to be incorporated.
Acknowledgments
This project is supported by the Singapore Polytechnic R&D fund and Singapore
Tote Board fund. Authors wish to acknowledge the many members of the ARICC
Centre who have contributed to this effort.
References
[1] S. Baluja, “Evolution of an Artificial Neural Network Based Autonomous Land Vehi-
cle Controller,” IEEE Transactions on Systems, Man and Cybernetics-Part B, vol. 26,
pp.450-463, 1996.
[2] C. Breazeal, “Sociable Machines: Expressive Social Exchange Between Humans and
Robots,” Massachusetts Institute of Technology, Department of Electrical Engineering
and Computer Science, PhD Thesis, May 2000.
[3] R.A. Brooks, C. Breazeal, M. Marjanovic, B. Scassellati and M.M. William-son, “The
Cog Project: Building a Humanoid Robot,” IARP First International Workshop on
Humanoid and Human Friendly Robotics, 1998.
[4] T. Caelli and W. F. Bischof, Machine Learning and Image Interpretation, New York,
Plenum Press, 1997.
[5] M. E.Cambron, and R.A. Peters II, “Learning Sensory Motor Coordination for Grasp-
ing by a Humanoid Robot,” Proceedings of the 2000 IEEE International Conference
on Systems, Man and Cybernetics, vol.5, pp. 870-875, 2000.
[6] K. H. Chen and W. H. Tsai, “Vision-Based Autonomous Land Vehicle Guidance in
Outdoor Road Environments Using Combined Line and Road Following Techniques,”
Journal of Robotic Systems, vol. 14, pp.711-728, 1997.
[7] A.N. Choudhary, J.H. Patel and N. Ahuja, “NETRA: A Hierarchical and Partitionable
Architecture for Computer Vision Systems,” IEEE Trans. on Parallel and Distributed
Systems, vol. 4, pp. 1092-1104, 1993.
[8] M. Doi, M. Nakakita, Y. Aoki and S. Hashimoto, “Real-time Vision System for
Autonomous Mobile Robot,” IEEE Int’l Workshop on Robot and Human Interactive
Communication, pp.442-449, 2001.
[9] M.F. Ercan and Y.F. Fung, “The Design and Evaluation of a Multiprocessor System
for Computer Vision,” Microprocessors and Microsystems, vol. 24, pp. 365-377, 2000.
[10] F. J., Roger, E. Kahn, M. J. Swain, and P. N. Prokopowicz , “Real-Time Gesture Rec-
ognition Using the Perseus Architecture,” University of Chicago, Computer Science
Technical Report, No:96-04, March 1996
227
[11] D.M. Harvey, S. P. Kshirsagar, C.A. Hobson, “Low Cost Scaleable Parallel Image
Processing System,” Microprocessors and Microsystems, vol. 25, pp. 143-157, 2001.
[12] http://www.honda-p3.com/
[13] R. Jarvis, “Interactive Hand/Eye Coordination between a Human and a Humanoid-A
proposal,” International Conference on Intelligent Robots and Systems, pp.1741-1747,
2000.
[14] O. Jesorsky, K.J. Kirchberg, and R. W. Frischholz, “Robust Face Detection Using
Hausdorff Distance,” Lecture Notes in Computer Science, vol. 2091, pp. 90-95, 2001.
[15] S. Kajita and K. Tani, “Study of Dynamic Biped Locomotion Rugged Terrain: Theory
and Basic Experiment,” Int’l Conference on Advanced Robotics, pp. 741-746, 1991.
[16] http://www.dh.aist.go.jp/h6/H6_H7.html
[17] T. Kanade and M. Okutomi, “A Stereo Matching Algorithm with an Adaptive Win-
dow; Theory and Experiment,” IEEE Transactions in Pattern Analysis and Machine
Intelligence, vol. 16, pp. 920-932, 1994.
[18] H. Kitano, H. G. Okuno, K. Nakadai, T. Sabisch and T. Matsui, “Design and Architec-
ture of SIG the Humanoid: An Experimental Platform for Integrated Perception in Ro-
boCup Humanoid Challenge,” Proceedings of the International Conference on Intelli-
gent Robots and Systems, pp. 181-190, 2000.
[19] D. Kortenkamp, E. Huber and R. P. Bonasso, “Recognizing and Interpreting Gestures
on a Mobile Robot,” Proceedings of AAAI-96,1996.
[20] A. Kosaka and A.C. Kak, “Fast Vision-Guided Mobile Robot Navigation Using Model
based Reasoning and Prediction of Uncertainties,” Computer Vision and Image Proc-
essing: Image Understanding, vol.56, pp. 271-329, 1992.
[21] M. Kumagai and T. Emura, “Vision based Walking oh Human Type Biped Robot on
Undulating Ground,” Int’l Conference on Intelligent Robots and Systems, pp. 1352-
1357, 2000.
[22] S. Nolfi and D. Floreano, Evolutionary Robotics: The Biology, Intelligence, and Tech-
nology of Self-Organizing Machines, MIT press, London, 2000.
[23] S. Okazaki, Y. Fujita, and N. Yamashita, “A Compact Real-time Vision System Using
Integrated Memory Array Processor Architecture,” IEEE Transaction on Circuits and
Systems for Video Technology, vol. 5, pp. 446-452, 1995.
[24] http://www.robo-erectus.org
[25] H. A. Rowley, S. Baluja and T. Kanade, “Neural Network-Based Face Detection,”
IEEE Transactions in Pattern Analysis and Machine Intelligence, vol. 20, pp. 23-38,
1998.
[26] F. Sandakly and G. Giraudon, “3D Scene Interpretation for a Mobile Robot,” Robotics
and Autonomous Systems, vol.21, pp. 399-414, 1997.
[27] http://www.sony.net/SonyInfo/News/Press/200203/02-0319E/
[28] C.C. Weems, “Architectural Requirements of Image Understanding with Respect to
Parallel Processing,” Proceedings of the IEEE, vol. 79, pp. 537-547, 1991.
[29] J. Woodfill and B. V. Herzen, “Real-Time Stereo Vision on the PARTS Reconfigur-
able Computer,” IEEE Symposium on FPGAs for Custom Computing Machines, April
1997.
[30] W. Zhao, R. Chellappa, A. Rosenfeld, and J. Phillips, “Face Recognition: A Literature
Survey,” Technical Report, University of Maryland, No: CS-TR4167R, 2002.
228
Evolution of Locomotion Controllers

for Legged Robots
A. Boeing and T. Bräunl
Centre for Intelligent Information Processing, University of Western Australia,

Perth, Australia
braunl@ee.uwa.edu.au
Abstract. The construction of a locomotion controller for legged robots is a diffi-

cult and time consuming process. This task can be automated through the use of
evolutionary techniques and an appropriately chosen control system. The pre-
sented approach utilizes a genetic algorithm that evolves the parameters for a
spline controller. The controller outputs its control signals to a virtual robot in a
mechanical simulator, enabling the robot designer to preview the robots locomo-
tion patterns. This approach has shown to produce walking patterns for a number
of robots with differing morphology. Extensions to the basic spline controller al-
low various forms of sensory information to be encoded enabling the robot to ma-
neuver over irregular terrain. The resulting system has shown to be a simple and
efficient method for automated controller design.
1 Legged Robots
Legged robots exhibit a number of advantages for locomotion[1]. The mobility of-
fered by legged vehicles is far greater than that of wheeled or tracked vehicles, as
they are not limited to paved terrain. The increased mobility offered, allows for a
far larger range of applications to legged vehicles. Another incentive for exploring
the use of legs for locomotion is that it provides an insight to the systems respon-
sible for human and animal locomotion. Humans are capable of complex move-
ments, whilst maintaining orientation, balance and speed. Robots that could mimic
human movements could seamlessly integrate with the human world, enlarging
the number of possible applications for legged vehicles. This makes the study of
bipedal locomotion particularly attractive.
Although there are a multitude of existing locomotion control techniques and
well described design processes, the automated generation of these controllers for
robots provides significant advantages. Often, the design process is quite complex,
time consuming to perform, and requires the control system to be completely re-
designed for small alterations to the robot [2]. Furthermore, humans often have
difficulty in understanding which sensors to incorporate to provide the best possi-
ble feedback to the robot. Designers are often biased towards feedback sensors
229
that are present in humans, such as vision and touch. These senses are not neces-
sarily the best sensors for the desired application.
By evolving the locomotion controller the robot designer is alleviated from the
controller design process, and the control system becomes more flexible, as the
robot can improve its controller to cope with environmental or structural changes.
Allowing the control system to evolve enables the controller to utilize sensory in-
puts that would normally be disregarded by human designers. The resulting con-
trollers are more adaptive to the robot’s environment. They are also more robust,
flexible, and usually provide superior performance to human designed controllers [2].
2 Control Architectures
There are a number of control systems that are applicable to the objective of robot
locomotion. Possible control systems range from simple oscillators [3] to neural
networks [2] to simple assembly programs [4]. The simplest oscillators consist of a
set of sinusoidal function generators whose outputs are combined to generate the
control signal for an actuator. These systems can represent a range of outputs by
altering the phase and amplitude of the sinusoids [3]. However, these systems are
generally incapable of expressing the complexity required for sustained locomo-
tion [5]. Thus, more complicated forms of control are desirable.
2.1 Neural Networks
Neural Networks are a popular form of controller for robot locomotion [6,7,8,9,10].
These controllers complement the biological systems believed to be responsible
for walking movement in humans and other animals. This form of control allows
application of the knowledge learned from neuroscientists studying rhythmical
controllers in animals [8].
Neural Networks consist of a set of interconnected processing element nodes
whose functionality is based on the animal neuron [11]. Neurons process informa-
tion by summing the signals that appear at the node’s inputs. Each of the input
signals is multiplied by a weight to simulate different input strengths. The
weighted sum is then passed through an activation function, which will produce an
output if the transformed sum passes a calculated threshold [11]. Traditionally, arti-
ficial neurons have been idealized for the sake of mathematical tractability [7]. One
of the simplest, and most commonly implemented neuron models, is the sigmoidal
neuron [6]. This neuron is a simple extension of the binary neuron model to express
a continuous softened step-function. The equations governing the output of the
sigmoidal neuron are given below.
yi = # j =1 w ji S j
n
(2.1)
230230A. Boeing and T. Bräunl
1
Si =
1 + e − yi (2.2)
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-10 -8 -6 -4 -2 0 2 4 6 8 10
Fig. 1. Sigmoid Function
There are many extensions possible to this model, and many have been applied
to the area of robotic locomotion.
2.2 Splines
Splines are piecewise polynomial functions expressed by a set of control points

[12]
. There are many different forms of splines, each with their own attributes.
There are two desirable properties for a spline to posses.
Continuity, so that the generated control signal translates to smooth velocity
and acceleration changes.
Locality of the control points, to reduce the influence of alterations of one con-
trol point to the overall shape of the spline.
The Hermite spline is expressed by the equations given in (2.3). The curve gen-
erated from a Hermite spline passes through the control points that define the
spline. Thus, a set of predetermined points can be smoothly interpolated by simply
setting the predetermined points as the control points for the Hermite spline. Her-
mite splines feature both the properties of locality and continuity. However, the
disadvantage of the Hermite spline is that the control point tangent values must be
specified.
The function used to interpolate the control points, given starting point p1, end-
ing point p2, tangent values t1 and t2, and interpolation parameter s, is shown be-
low:
231
f ( s ) = h1 . p1 + h2 . p 2 + h3 .t1 + h4 .t 2 (2.3)
where
h1=2s3-3s2+1
h2=-2s3+3s2
h3=s3-2s2+s
h4=s3-s2
0 ≤ s ≤1
3 Evolution
Evolutionary algorithms are a form of search and optimization algorithms, which

make use of some of the principles of organic evolution [13]. These types of algo-
rithms typically specify what is to be done, rather than how the task should be ac-
complished [13]. Although evolutionary algorithms are only a basic approximation
to the biological evolutionary process in reality, they have been proven to provide
a powerful means of problem solving [6,7,3,9,14].
3.1 Evolving Controllers
Genetic algorithms were applied to the evolution of neural controllers for robot lo-
comotion by numerous researchers [6,7,3,9,14]. Lewis et al. [9] successfully generated
locomotion patterns for a hexapod robot using a simple traditional genetic algo-
rithm with one point crossover and mutate. Isjpreet [14] evolved a controller for a
simulated salamander using an enhanced genetic algorithm. The large number of
experiments in this area clearly indicates that genetic algorithms are capable of
evolving neural controls which can describe legged robot locomotion [6].
The genetic programming approach has been shown to successfully generate
locomotion patterns for various control strategies. Banzhaf et al. [15] demonstrated
the pure genetic programming approach to develop assembly programs for robot
control. The system was then expanded to control a hexapod robot [16] using a B-
Spline based approach. Lewis also applied genetic programming to his neural con-
troller [10]. This demonstrated that both the genetic programming approach and the
genetic algorithm approach should both be capable of evolving adequate control
systems for legged locomotion [10,14].
Parker et al. [17] explored the use of cyclic genetic algorithms for locomotion
control of a small hexapod robot. The system demonstrated that the cyclic nature
needed to generate the oscillatory motions necessary for legged robot locomotion
could be abstracted from the control system and transferred and encoded into the
genetic algorithms chromosomes.
3.2 Genetic Algorithm
A common implementation of an evolutionary algorithm is the Genetic Algorithm

(GA) [13]. As with most evolutionary algorithms the Genetic Algorithm is based on
Darwin’s theory of natural selection, ensuring the survival of the fittest. The ge-
netic algorithm operates on a set of encoded variables representing the parameters
for the potential solution to a problem. The parameters (or genes) are combined
together to form a string of values, referred to as a chromosome [18]. Each of these
possible solutions are then assigned a fitness value according to how well it solves
the problem. The better solutions are then selected to “reproduce” with other solu-
tions, generating a new set of chromosomes which have inherited features from
the chromosomes they were created from. The least fit (worst solutions) are less
likely to be selected for reproduction, and thus eventually are removed from the
set of chromosomes on which the algorithm operates.
The basic methodology for the genetic algorithm is presented below:
1. Randomly initialize a population of chromosomes
2. While the terminating criteria has not been satisfied
a) Evaluate the fitness of each chromosome
b) Remove the lower fitness individuals
c) Generate new individuals, determined by a certain selection scheme,
utilizing selected operations.
Each iteration of these steps creates a new population of chromosomes. The to-
tal set of chromosomes at one iteration of the algorithm is known as a generation.
As the algorithm progresses it searches through the solution space, refining the so-
lutions to find one which will fulfill (or come as close as possible to fulfilling) the
desired criteria, as described by the fitness function.
3.3 Operators
The operators determine the method in which one or more chromosomes are com-
bined to produce a new chromosome. Traditional schemes utilize only two opera-
tors: Mutate, and Crossover [18]. Crossover takes two individuals and divides the
string into two portions at a randomly selected point inside the encoded bit string.
This produces two “head” segments and two “tail” segments. The two tail seg-
ments for the chromosomes are then interchanged, resulting in two new chromo-
somes where the bit string preceding the selected bit position belongs to one par-
ent, and the remaining portion belongs to the other parent. The mutate operator
randomly selects one bit in the chromosome string, and inverts the value of the bit.
Extensions on these operators are possible by interpreting the chromosome as a
string of non-binary values (such as 8 bit integer values, or 32 bit floating point
233
values). Two operators commonly used that rely on this interpretation of the
chromosome are the Non-Binary Average, and the Non-Binary Creep operators
[19]
. Non-Binary Average interprets the chromosome as a string of higher cardinal-
ity symbols and calculates the arithmetic average of the two chromosomes to pro-
duce the new individual. Likewise Non-Binary Creep treats the chromosomes as
strings of higher cardinality symbols and increments or decrements the values of
these strings by a small randomly generated amount [19].
3.4 Selection Schemes
In the natural world the organisms which reproduce the most before dying will
have the greatest influence on the next generation. In order to simulate this effect
in the genetic algorithm a selection scheme is used. The selection scheme deter-
mines which individuals of a given population will contribute to form the new in-
dividuals for the next generation Roulette Wheel selection and Tournament selec-
tion are two commonly used selection schemes [18].
In roulette wheel selection the chance for a chromosome to reproduce is pro-
portional to the fitness of the entity. Thus, if the fitness value returned for one
chromosome is twice as high as the fitness value for another, it is then twice as
likely to reproduce. Although genetic algorithms will converge to a solution if all
of the chromosomes reproduce, it has been shown that by duplicating unchanged
copies of the chromosomes future generations will generally have a significant in-
crease in the convergence rate towards the solution. The implemented genetic al-
gorithms attributes are given in Table 1.
Table 1. Operators attributes

Operator Name Probability of use
Crossover 35%
Mutate 10%
Random Replacement 10%
Inversion 5%
Average 30%
Creep 10%
3.5 Staged Evolution
A number of possibilities exist in enhancing the performance of a genetic algo-

rithm. Staged evolution is based on the concept of behavioral memory, and in-
creases the convergence rate by introducing a staged set of manageable challenges
[10]
. Initially limiting the search to a subset of the full solution space enables an
approximate solution to be determined. Incrementally increasing the complexity of

the problem will increase the solution space, providing the possibility of increased
performance as further refinements of the solution are possible. Applying this
strategy to a particular problem task, requires that the tasks is capable of being
split into further smaller sub-tasks which can be solved in order to contribute to
the overall solution.
3.6 Fitness Functions
Each problem to be solved required a unique fitness function describing that prob-
lem. Given a particular chromosome a fitness function must return a numerical
value indicating the appropriateness of the solution with respect to the overall goal
[20]
. For some applications, such as function optimization problems, the fitness
function will simply return the value of the function itself. However, for many ap-
plications there is no straightforward performance measurement of the goal, and
thus it must be expressed as a combination of the desired factors.
The fitness function used to evaluate the robot during the initial phases of evo-
lution was evaluated purely from the distance the robot traveled forward, minus
the distance that the robot’s center of gravity lowered. During later phases of evo-
lution the average speed at which the robot moved, and the distance that the robot
wavered from the desired path were incorporated. Finally, the distance that the ro-
bot was at termination from its desired terminating point was taken into considera-
tion.
4 Simulation
There are a number of advantages offered when evolving a system under a simu-
lated environment. Firstly, the risk of damaging the robot hardware during evolu-
tion is removed. Secondly, simulation can be performed in far less time than that
which is required for a robot to physically perform the desired gait. Simulators
also provide the experimenter with the ability to easily modify the environment
and model the interaction between objects.
Dynamechs [21], the simulation library utilized is a free, extensible and portable
mechanical environment simulator. Dynamechs calculates all environment vari-
ables and all forces generated by the actuators, in order to describe the movement
of all bodies in the system. The robots are expressed by multiple chains linked to a
mobile base described with modified Denavit-Hartenberg parameters. The simula-
tion library allows configurable integration step sizes and offers a number of inte-
grators from first order Euler to fifth order Runge-Kutta. This allows configuration
of the accuracy and speed of the simulation. The fitness evaluation functions
needed to operate the genetic algorithm were added to the package, along with the
simulated spline control system and the necessary sensors.
235
5 Spline Control
Although a neural controller offers a number of advantages for control, a spline

controllers provides a simple and fast method for evolving control. The imple-
mented spline controller consists of a set of joined Hermite splines. One set con-
tains robot initialization information, to move the joints into the correct positions
and enable a smooth transition from the robot’s standing state to a travelling state.
The second set of splines contains the cyclic information for the robot’s gait. Each
spline can be defined by a variable number of control points, with variable degrees
of freedom. A pair of start and cyclic splines corresponds to the set of control sig-
nals required to drive one actuator within the robot.
Cubic Hermite splines were implemented in the controller as they offer a num-
ber of desirable characteristics over other splines. The major advantage of the
Hermite spline is that the curve passes through all the control points. As the posi-
tion and tangent are explicitly specified at the ends of each segment, it is easy to
create a continuous curve.
An example of a simple spline controller is illustrated in Figure 2. Each spline
indicates the controllers output value for one actuator.
Fig. 2. Evolved spline controller for a three-jointed limb
The number of control points required for this controller is given by equation
(5.1).
a.(i + c) (5.1)
where,
a is the number of actuators
i is the number of initialization control points
c is the number of cyclic control points
In order to incorporate sensor feedback information into the spline controller,

another dimension was added. This resulted in a set of control surfaces for each
actuators cyclic information. The initialization splines were not extended as it was
assumed that any robot would be set-up on flat terrain. However, the initialization
splines can be easily extended to contain information dependent on the terrain’s

gradient through a similar method to that used to extend the cyclic splines. The
number of control points required for the extended controller is given by equation
(5.2). Extending the controller in this form, significantly increase the number of
control points required.
a.(i + c.s ) (5.2)

Where, s is the number of control points used to describe the feedback
Fig. 3. Spline controller for a single actuator with feedback
The example controller used an inclinometer to provide the sensory feedback.

Thus the controller’s output at any stage is expressed in terms of the current cycle
time, and the angle read from the inclinometer. This allows the controller to cor-
rect for undesired tilting during normal walking, and also allows for movement
over rough terrain. Figure 3 illustrates a resulting control surface for one actuator.
5.1 Spline Controller Encoding
In order to evolve the spline controller with a genetic algorithm, the controller’s
parameters needed to be encoded into chromosome representations. To enable
support for staged evolution the controller needs to be specifically designed such
that the evolution can proceed in this manner. The encoded spline controller
treated each control point value as an 8 bit fixed point number. In the initial phase
237
of evolution the control point locations within the walking cycle time were equally
distributed. This provided each control point with only one degree of freedom, and
reduced the available solution space but also significantly reduced the complexity
of the chromosome required to describe the controller. In the following stage of
evolution, the equally distributed time constraint was removed, providing the con-
trol points with an additional dimension of freedom. Finally the tangent values for
the control points in the spline were added to the chromosome, allowing final re-
finement of the solution.
6 Results
A number of spline controllers have been evolved for the control of various legged
robots. An example of a spline controller which generated locomotion without
feedback is shown in Figure 4. The walk is maintained for 10 seconds, before the
robot collapsed.
Fig. 4. Walking without Feedback
Fitness vs Generation for Extended Spline Controller
120
100
80
Fitness
Top Fitness
60
Average Fitness
40
20
0
1
42
83
124
165
206
247
288
329
370
411
452
493
534
575
616
657
Generation
Fig. 5. Genetic Algorithm Fitness

The increasing fitness function for the control parameters evolved for the gait
depicted in Figure 6 is illustrated above. The best and average fitness values in-
crease linearly during the initial phases of evolution. As the algorithm begins to
converge towards an appropriate solution the fitness values for the population in-
crease rapidly. The algorithm terminates once it has refined the solution after 500
generations.
Incorporating sensory information to provide feedback to the control algorithm
overcame the difficulties in maintaining a robots walk. The addition of feedback
allowed for continues walking to be achieved. However, integrating feedback sig-
nificantly impacts on the number of parameters required in the controller. The
controller utilized for the gait generation in Figure 4 consists of four initialization
control points, and four cyclic control points, with six controlled actuators. Thus,
the total chromosome size to represent the complete controller is 48 bytes during
its first phase of evolution. The expansion of this controller to enable feedback
that samples the sensor with four control points results in an encoded controller
size of 120 bytes. This represents a significant increase in the solution space that
the genetic algorithm needs to search.
Seeding the genetic algorithm with the previously evolved splines resulted in
uniform spline surfaces that were capable of expressing the initial movements re-
quired in bipedal locomotion. Approaching the extended controller through the
staged evolutionary method allowed the controller to be developed in under one
hundred hours on an 800MHz Pentium 3 system running Windows NT.
The use of an inclinometer also allowed the robot to detect alterations in the
terrain. Thus the resulting controller was capable of expressing bipedal gaits
which allowed the robot to maneuver over non-uniform terrain.
Fig. 6. Walking over terrain
Gaits for non-bipedal legged robots were also successfully evolved utilizing the
spline control system. The most successful gait evolved for the tripod robot is il-
lustrated in Figure 6. The robot achieves forwards motion by thrusting its rear leg
towards the ground, and lifting its forelimbs. The robot then gallops with is fore
limbs to produce a dynamic gait.
Fig. 7. Tripod Gait

239
7 Conclusion and Outlook
The proposed evolutionary control system was shown to generate locomotion pat-
terns for a bipedal robot. A system for integrating sensory information into a
spline controller was presented. Also, the possible improvements to a robot’s con-
trol system when provided with sensory feedback was demonstrated.
Future extensions of the system could provide a method for directly download-
ing the control system to a physical biped robot. Further evolution of the control
system on the physical robot could then be used to overcome discrepancies be-
tween the simulated and physical world.
Another useful extension could be to evolve the robot’s morphology as well as
the controller. This would allow the system to suggest structural alterations to ro-
bots, which could assist the robot in achieving its desired purpose.
Providing control over the placement of sensors on the robot would allow the
evolutionary system to optimize the sensor locations such as to provide the opti-
mal feedback to the controller. This should result in further improvements to the
adaptability and robustness of the generated control system.
References
[1] M. H. Raibert. Legged Robots That Balance Cambridge, MA: The MIT Press, 1986
[2] G. S. Hornby, S. Takamura, J.Yokono, O. Hanagata, T. Yamamoto, M. Fujita, “Evolv-
ing Robust Gaits with AIBO” in IEEE International Conference on Robotics and
Automation. pp. 3040-3045, 2000.
[3] J.Ventrella,“Explorations in the Emergence of Morphology and Locomotion Behavior
in Animated Characters “ in R.A. Brooks and P. Maes (eds.) Artifical List IV, pp. 436-
441, (MIT Press, Cambridge, MA 1994).
[4] J. Busch, J. Ziegler, C. Aue, A. Ross, D. Sawitzki, W. Banzhaf, “Automatic generation
of Control Programs for Walking Robots Using Genetic Programming” in European
Conference on Genetic Programming (EURO GP), 2002.
[5] D. Arnold “Evolution of Legged Locomotion”, MSc. thesis, Simon Fraser University,
School of Computing Science, 1997.
[6] M. Krueger “Genetic Algorithms can Evolve Neural Network Controllers for Robot Lo-
comotion”, BE thesis, University of Western Australia, 2001.
[7] R. Reeve “Generating walking behaviors in legged robots”, Ph.D. thesis, University of
Edinburgh, 1999.
[8] P. Wallen, O Ekeberg, A Lansner, L Brodin, H Traven, and S Grillner. “A computer
based model for realistic simulations of neural networks. II. The segmental network
generating locomotor rhymicity in the lamprey.” Journal of Neurophysiology, pp.
1939-1950, 1992.
[9] M. Lewis, A. Fagg, G. Bekey. “Genetic Algorithms for Gait Synthesis in a Hexapod
Robot.” in Recent Trends in Mobile Robots (World Scientific, New Jersey), pp. 317-
331. 1994.
[10] M. Lewis, A. Fagg, A. Solidum, “Genetic Programming approach to the Construction

of a Neural Network for Control of a Walking Robot” in IEEE International Confer-
ence on Robotics and Automation (Nice, France) pp 2618-23, 1992.
[11] K. Gurney, “Neural Nets”, UCL Press Ltd, 2002.
[12] Bartels, R. H., Beatty, J. C. and Barsky, B. A. “An Introduction to Splines for use in
Computer Graphics and Geometric Models” Morgan Kaufmann, 1987.
[13] J.H. Holland, "Adaptation in Natural and Artificial Systems", MIT Press, 1975.
[14] A. J. Ijspeert, “Evolution of neural controllers for salamander-like locomotion” in Pro-
ceedings of Sensor Fusion and Decentralised Control in Robotics Systems II pp. 168-
179, 1999.
[15] P Nordin, W Banzhaf, “An On-Line Method to Evolve Behaviour and to Control a
Miniature Robot in Real Time with Genetic Programming” in Adaptive Behaviour, vol
5 no 2 pp 107-140, 1997.
[16] J. Ziegler, W. Banzhaf, “Evolution of Robot Leg Movements in a Physical
Simulation” in K. Berns and R. Dillmann, (Eds.), Proceedings of the Fourth Interna-
tional Conference on Climbing and Walking Robots, CLAWAR , (Professional Engi-
neering Publishing 2001).
[17] G.B. Parker, D.W. Braun, I. Cyliax, “Learning Gaits for the Stiquito” Proceedings of
the 8th International Conference on Advanced Robotics (ICAR'97). (pp. 285-290).
1997.
[18] D. Beasley, D. R. Bull, and R. R. Martin, "An Overview of Genetic Algorithms: Part I,
Fundamentals" University Computing, vol. 15, no. 2, pp. 58-69, 1993.
[19] D. Beasley, D. R. Bull, and R. R. Martin, "An Overview of Genetic Algorithms:Part 2,
Research Topics" University Computing, vol. 15, no. 4, pp. 170-181, 1993.
[20]F. Busetti, ”Genetic algorithms overview”. Available online:
http://citeseer.nj.nec.com/464346.html [Accessed: 28th October 2002]
[21]DynaMechs (Dynamics of Mechanisms): A Multibody Dynamic Simulation Library
[Online] http://dynamechs.sourceforge.net/ [Accessed: 7th March 2003]
241
Gait Planning for Soccer-Playing

Humanoid Robots
Zhe Tang1,2, Changjiu Zhou2, and Zengqi Sun1

1
Department of Computer Science and Technology, Tsinghua University, Beijing,
P.R. China
2
School of Electrical and Electronic Engineering, Singapore Polytechnic,
500 Dover Road, Singapore
http://www.robo-erectus.org
Abstract. Gait planning for humanoid walking and penalty kicking, the two
fundamental tasks for the current humanoid soccer competition, is addressed in
this paper. First, an overview of some basic research issues in the field of
humanoid robotics is presented. Humanoid walking gait planning in both joint
space and Cartesian space is then discussed in a detailed way so as to make a clear
comparison. By introducing some new concepts such as maximum kicking range
and effective kicking range, a novel gait planning method for humanoid penalty
kicking is also presented. All the proposed gait synthesizing methods have been
successfully implemented on a soccer-playing humanoid robot, named Robo-
Erectus, developed in Singapore Polytechnic.
1 Introduction
The desired goals of humanoid robotics research are to develop robots that are
able to coexist and collaborate with humans, to replace humans in some of their
roles in the interaction between humans and robots, and to extend human
capabilities for interacting with environments[1]. So the motion planning for
humanoid robot is not only limited to walking, but also include many other tasks,
such as kicking, running, jumping and so on. Gait synthesis has been a very
important part of work in humanoid robot motion planning. Aiming at
participating in robot soccer competitions[2, 3], we have developed a series of
humanoid robots named Robo-Erectus in the ARICC Centre of Singapore
Polytechnic [4]. Humanoid walking and penalty kicking (PK) are two basic tasks
for the current humanoid soccer playing games. We will focus on some basic
research issues in the field of gait synthesis for soccer-playing humanoids in this
paper.
Vukobratovic et al.[5,6] have considered locomotion directly. The fundamental
aspect of their approach is the adaptation of human walking data to prescribe
242
242 Z. Tang, C. Zhou, and Z. Sun
motion of the lower limbs and the application point of the resultant of the reaction
forces among lower limbs and the walking surface. Furthermore, repeatability
conditions are added to impose further symmetry conditions on the motion.
Vukobratovic and his colleagues adopted a two-stage control synthesis. In the first
stage, the control synthesis has to ensure the system’s motion in the absence of
any disturbance along the exact nominal trajectories calculated in advance. In the
second stage external perturbation is involved, the nominal should be realized with
assistance from an additional control force. In this paper, we will only focus on the
first stage, designing a desired trajectory for humanoid robot without considering
external perturbation.
Furusho and Sano[7] achieved smooth 3D walking through a sensor based
control on a biped with nine links. They considered the motion in the lateral plane
as a regulator problem with two equilibrium states. In the sagittal plane, where the
walking control was based on the body speed in forward movement, they made the
body movement close to the desired smooth speed by controlling the ankle torque
of the support leg. The sole and ankle driving actuators underwent a force/torque
feedback control based on the sensor information. An inverted pendulum mode
was adopted. Mita and et al.[8] used a linear optimal state feedback regulator to
control a seven-link biped. Their work proved the usefulness of typical modern
control theory for studies on biped locomotion. Miura and et al.[9] approximated
the biped motion of the single support phase as an inverted pendulum, and also
gave trajectory planning and stability analysis in their work. Furthermore, their
methods were implemented on their two biped robots. Zheng and Hemami[10] have
addressed the impact effects of a biped contact with the environment. They
showed that the impact may cause large internal impulsive forces in the body, and
that the biped can control its initial posture in order to reduce the impulsive
component of the internal constraint forces. They also proposed a linear velocity
and nonlinear position feedback control method to slow down the biped motion so
that it finally reaches a static posture after hitting the ground. In addition, Zheng
and Shen[11] developed a biped robot, SD-2, which could climb sloping surfaces
using static stabilization criteria. Using a force sensor underneath the foot, their
robot was able to detect the transition of the supporting terrain from a flat floor to
sloping surface of up to 10 deg. They were the first to propose a biped walking
algorithm on a slope surface. Shih[12,13] proposed a gait synthesis of ascending and
descending stairs for his 7 DOF biped. The efficiency of the gait synthesis was
supported by the biped implementation. However, his method is only applicable to
his specific biped: variable length legs and translatable balance weight in the
body. Furusho and Masubuchi[14] have proposed some novel approaches to
synthesize a five-element biped. They adopted a hierarchical control structure for
lower level control. A reduced order model of biped locomotion, which is based
on the concept of local feedback, was presented. The results were verified
experimentally by using a walking machine and periodicity was demonstrated for
several walking patterns. Miyazaki and Arimoto[15] divided biped locomotion into
two modes, slow mode and fast mode. The joint motion trajectories were obtained
in the two modes respectively. They used the inverted pendulum model to
approximate the biped global locomotion. Their control algorithm was verified by
243
computer simulation. Kajita and Yamaura[16] used the term “potential energy
conserving orbit” to describe a particular class of biped motion trajectories, which
was based on an ideal biped model. They proposed a potential energy conserving
orbit. Huang and et al.[17] presented a practical method for biped trajectory
planning using the third-order Spline interpolation. However, they did not
consider the energy consumption, and most of the interpolation positions were
decided by experience or trial and error.
The rest of this paper is organized as follows. In the following section, for the
gait synthesis of humanoid walking, we propose a novel approach to generate
trajectories in both Cartesian Space and Joint Space from a robot manipulator
perspective. We clarify the difference between the traditional robot manipulators
and humanoid robot manipulators. Furthermore, we will also compare the gait
planning in the two different spaces. In Section 3, we give a detailed gait synthesis
for humanoid penalty kick, including kicking cycles and some basic research
issues in penalty kicking. We also propose some new concepts such as max
kicking range and effective kicking range. Finally, some remarks on gait planning
for soccer-playing humanoid robots are given in Section 4.
2 Gait Synthesis for Humanoid Walking
2.1 Classification
Different gait planning methods have been developed for humanoid walking
pattern generation.
1. Control goal based method. Researchers have addressed a wide variety of
control goals for humanoid walking, such as stable walking on uneven terrain[17],
energy efficient walking[16], walking on ascending and descending stairs[12,13].
2. Control model based method. There are model-based and model-free
walking planning schemes. The model-based method generates a walking gait
from a mathematical model[5-17]. The model-free method usually makes use of
intelligent techniques, such as machine learning[18-24], genetic algorithms[25,26],
neural networks[22,23,27] and fuzzy control[19-21].
3. Stability criteria based method. There are two different kinds of walking,
namely dynamic walking and static walking for humanoid robots. Most current
dynamic walking schemes use zero moment point (ZMP) as stability
criterion[1,17,19,20,21,24,27]. Static walking[11,12,13,22,23] is implemented in slow
movement, and utilizes center of gravity (CG) as stability criterion.
4. Trajectories generation space based method. Humanoid walking gait can be
planned in either joint space or Cartesian space. Both Cartesian space walking gait
planning[12,13,17] and joint space walking gait planning[18,28] will be discussed in this
paper. We will also give a clear explanation and comparison of the gait planning
methods in both joint space and Cartesian space in this section.
244
2.2 Manipulator
Although trajectory generation, one of most important issues in humanoid robots,

is different from path planning in manipulator with a fixed base [5], they share
many vital attributes. Both the manipulator and the humanoid robot use the same
dynamic Newton-Euler equation as follows:
.. .
τ = D ( q ) q + C ( q, q ) + g ( q ) (2.1)
where q is the n × 1 vector of generalized joint displacements, τ is the
n × 1 vector of applied generalized forces, D(q ) is the n × n mass matrix,
.
C (q, q) is an n × 1 vector of centrifugal and Coriolis terms, and g (q ) is an
n × 1 vector of gravity terms. (For details of parameters please refer to [14])
A humanoid robot can be considered as one or several independent
manipulators under certain conditions. In this way, many theories in robotic
manipulator can also be used for humanoid robots. In this paper, we will
demonstrate how manipulator motion planning method can be used for humanoid
motion planning. There are two spaces in trajectory generation for manipulator
motion: Cartesian space and joint space[29,30,31]. We will present the details of
trajectory generation for humanoid robot walking in both spaces.
One of the two main differences between the two spaces is that the normal
robot manipulator assumes a static base frame (i.e. world coordinate system)
which will never move. In Cartesian space, however, the humanoid robot motion
involves the movement of every joint. We have to find or assume a base frame to
apply the manipulator methods. The other difference is that humanoid robots
introduce unpowered degree-of-freedom (DOF), which greatly affects the
kinematics of the humanoid robot[5].
2.3 Stability Criterion
There are two kinds of humanoid robot stability: static stability and dynamic
stability. Static stability neglects the dynamics of the robots and takes centre of
gravity (CG) as stability criterion, so this stability is applicable only if the robot
moves very slowly. Most researches have adopted dynamics stability. The most
common dynamic criteria are center of pressure (CP), zero moment point (ZMP)
and foot rotation indicator (FRI), which are briefly described as follows[32]:
1. The CP is a point on the foot/ground surface where the net ground reaction
force actually acts.
2. The ZMP is the point on the floor at which the moment generated by the
reaction force and the reaction torque are balanced.
3. The FRI is the point on the floor/ground surface, inside or outside the base of
support, where the net ground reaction force would have to act to keep the foot
stationary.
245
The ZMP is identical to CP in a different definition. The ZMP was originally

introduced for humanoid robot stability in 1969[6]. Using this criteria has yielded
many impressive results[1,19,20,21,27,28]. We adopted the ZMP control for our
humanoid walking gait synthesis. The ZMP can be calculated from the following
equations:
$$
#in=1mi ($$zi + g)xi − #in=1mi $$xi zi − #in=1 IiyΩ (2.2)
iy
xzmp =
#in=1mi ($$zi + g)
$$
#in=1mi ($$zi + g) yi − #in=1mi $$yi zi − #in=1IixΩ ix
yzmp = n
#i=1mi ($$zi + g)
where $$ and
mi is the mass of link i, I ix and I iy are the inertial component, Ω ix
$$ are absolute angular velocity components around the x-axis and the y-axis at
Ω iy
the center of the gravity of link i, g is the gravitational acceleration,
( xzmp , y zmp , 0 ,) is the coordinate of the ZMP, and ( xi , yi , zi ) is the coordinate of
the mass center of link i on an absolute Cartesian coordinate system.
The angular accelerations Ω $$ are small and have little effect on the
$$ and Ω
ix iy
ZMP position compared with the link acceleration, so the two angular acceleration
is normally neglected.
2.4 Walking Phases
Humanoid robot walking includes two walking phases, namely, double-support

phase and single-support phase[5]. But for each leg, the walking process is
composed of a stance phase and swing phase. The double-support phase is the
overlap period of two legs’ swing phase.
During the double-support phase, the CG or ZMP moves from one foot to
another foot, this motion is achieved mainly in lateral plane, to simplify our
analysis, our motion planning is restricted to the sagittal plane.
Fig.1 gives a walking cycle starting from kth step, where Tc is period for one
walking step, Td is duration for double-support phase, and kTc+ Tm corresponds
to the point when the swing foot reaches its highest point.
2.5 Trajectory Generation in Cartesian Space
Humanoid walking is a periodic motion which alternates between the double-

support phase and the single-support phase. During the double-support phase, both
of the humanoid robot feet are in contact with ground. During the single-support
phase, only one foot is in contact with ground to support the humanoid, the other
leg swings forward. Our previous work[33] shows that the humanoid robot is prone
246
to falling at the switching point between the double-support phase and the single-
support phase.
In order to control a humanoid robot’s movement, we should first generate the
trajectories of humanoid joints. The trajectories should have first-order and
second-order derivative continuity. First order derivative continuity guarantees
smoothness of joint velocity, while second order guarantees smoothness of
acceleration or torque of joint.
There are two ways to represent the joint position, Cartesian space or joint
space. The current joint trajectory generation falls into these two spaces.
Furthermore, it can be proved that the first-order and second-order derivative
continuity in Cartesian space can guarantee the first-order and second-order
derivative continuity and vice visa. Because the two spaces conversion is a one-to-
one mapping. In fact, any trajectory generation in Cartesian space has to convert
to joint space to control each joint.
Fig. 1. Walking phase.
The trajectory generation in Cartesian space for the humanoid robot can be
summarized as follows: (the humanoid robot model is shown in fig.2. This model
is a simplified one which neglects the foot and hip, and only considers one leg
because the other leg is symmetric.)
(1) Defining the position of the hip position (xh(t),yh(t)) at some given time
points, then generating the trajectory of the hip using interpolation by
polynomials.
(2) Defining the position of the ankle position (xa(t),ya(t)) at the switching
points and some other interpolation points to constrain the walking
pattern. From these positions, the trajectory is generated using
polynomial interpolation.
(3) Based on the above positions of hip and ankle, the knee
position(xk(t),yk(t)) is obtained using the following geometric formulas:
247
k a 2 {
x = x + l cos tan −1 $% ( y − y ) ( x − x ) +, −
& h a h a - (2.3)
cos −1 $(l 2 + ( x − x )2 + ( y − y )2 − l 2 ) /
%& 2 h a h a 1
+ .
2l ( x − x )2 + ( y − y )2 , ) /
2 h a h a - 0
k a 2 {
y = y + l sin tan −1 $ ( y − y ) ( x − x ) + −
&% h a h a -,
cos−1 $(l 2 + ( x − x ) 2 + ( y − y )2 − l 2 ) /
%& 2 h a h a 1
+ .
2l ( x − x )2 + ( y − y )2 , ) /
2 h a h a - 0
(4) From the positions of the joints, the joint angles can be obtained from the
following inverse kinematics equation.
θ h = − tan −1 [ ( yk − yh ) ( xk − xh )] ± π / 2 (2.4)
θ a = −(π / 2 ± tan −1 [ ( yk − ya ) ( xk − xa )])

θk = θa + θh
Fig. 2. Simplified model of the humanoid robot
For the humanoid robot with a variable-length leg[12,13,34,35], there is no need

generate knee joint trajectory. This kind of robot simplifies control architecture at
the expense of more complex mechanical structure.
248
2.6 Trajectory Generation in Joint Space
For the trajectory generation in joint space, we take the two legs as two
manipulators. Then the trajectory generation for humanoid motion can be
considered as manipulator trajectory generation. The first step for manipulator
planning is to define the base frame. Based on humanoid walking phases, we
define the two manipulations as follows:
(1) During the double-support phase and left-leg stance, the left foot is the
base frame of the left manipulator, and the hip is the base frame of the
right manipulator.
(2) During the right-leg stance phase, the right foot is the base frame of the
right manipulator and the hip is the left manipulator’s base frame.
We name the foot base frame as the first base frame (BF1), and the hip base
frame as the second base frame (BF2). The corresponding legs using the frame are
named as the first-leg and the second-leg respectively.
The joint angles are defined as following:
' qasi t = kTc (2.5)

*
θ ai (t ) = (qami t = kTc + Tm
*q t = (k + 1)* Tc
) aei
' qhsi t = kTc
*
θ hi (t ) = (qhmi t = kTc + Tm
*q t = (k + 1)* Tc
) hei
where i = 1, 2 .Based on the above analysis, θ as1 = θ ae 2 , θ hs1 = θ he 2 , θ as1 = θ ae 2 ,

and θ hs 2 = θ he1 . Furthermore, because the motion of the humanoid robot
proceeds to double-support phase in the lateral plane after the single-support
phase, the angle velocity in sagittal plane is supposed to be zero, at the start and
end of the double support-phase:
θ$ai (kTc) = θ$ai ((k + 1)Tc) = 0 (2.6)
θ$hi (kTc) = θ$hi ((k + 1)Tc) = 0

where i = 1, 2 .
From the above via points and start and end conditions, smooth trajectories of
the hip and two ankles in joint space can be generated using the third order Spline
interpolation. These trajectories can achieve first and second derivate continuity in
joint space.
249
2.7 Experiment Results
To synthesize a gait for controlling a real humanoid robot, we have developed a

gait generator platform see Fig.4
! "using Visual C++ under Windows 2000.
This generator decomposes the humanoid motion into three planes: sagittal,
frontal and transverse. Most of humanoid research assumes that the motions in the
three planes are independent, which greatly simplifies humanoid motion analysis.
The humanoid robot parameters and other values are shown in Fig. 5 and Table1.
Fig. 3. Base frame
2.7.1 Planning in Cartesian Space
Let the foot landing angle be Qf = 5 deg and leaving angle be Qb=5 deg
respectively. We set the walking step length Ds = 5cm, Hao = 2cm, Lao = 4 cm,
and walking cycle 2*Tc = 1.8s, a walking cycle consists of two steps, double-
support phase Td = 0.2s. The ankle position at the following via points are
defined,
After defining the via points position, the ankle trajectory is generated by the
third-order Spline interpolation. The hip trajectory can be obtained in a similar
way. Then the knee position can be obtained from Eq.(2.3).
Table 1. Robot parameters

Length Lhip Lthigh Lshank Lan Laf Lab
(cm)
10 15 15 0 1.5 1.5
250
Table 2. Walking parameters

Time (s) Td Tm Tc
0.2 0.6 1
Fig. 4. Simulator interface
Fig. 5 Robot parameters and coordinate

251
2.7.2 Planning in Joint Space

When the hip and ankle positions at the via points are given, then by Eq.(2.4) the
joint angles can be obtained as shown in Table 3. A smooth joint angle trajectory
in the joint space is generated using the third order Spline interpolation. However,
the definition of the via points is not a trivial task, we normally get the positions
through trial-error method and experience.
Table 3. Via points value (rad)

qas1 qam1 qae1 qhs1 qhm1 qhe1
0.559 0.906 -.02 0.038 0.446 0.643
2.7.3 Comparison between Two Spaces

The comparison of experiment results are shown in Figs.6 to 17. The motion
planning in the joint space is simpler and computationally less expensive than the
motion planning in the Cartesian space, because motion planning in the Cartesian
space requires solution of inverse kinematics equation Eq.(2.4). Therefore, the
motion planning in joint space is more suitable for on-line trajectory generation.
However, the joint space motion planning does not provide straight line motion of
the end-effector. Therefore, it is not applicable for humanoid kicking, because the
kicking trajectory must be a straight line.
From Figs.6 and 7, it can be seen that the density variance of the stick for
Cartesian walking planning is smaller than that in the joint space. This indicates
that the liner accelerations of every link are smaller. As a result, the trajectory of
ZMP is smoother. This is also demonstrated in Figs. 16 and 17. Although the
trajectory planning scheme in the joint space makes the joint angular velocity
smoother, the liner velocities of every robot link contribute much more to the
dynamic walking stability. From this point of view, the walking planning in the
Cartesian space is desirable. However, one of drawback for motion planning in the
Cartesian space is that it is difficult to predict the joint’s angular velocity during
on-line trajectory planning in the Cartesian space[30].
Fig. 6 Consecutive walking for planning in the Cartesian space

252
Fig. 7 Consecutive walking for planning in the joint space
1.3 1.1
Ankle Ankle
Knee Knee
1.2 Hip 1 Hip
1.1 0.9
1 0.8
0.9 0.7
Angle(rad)
Angle(rad)
0.8 0.6
0.7 0.5
0.6 0.4
0.5 0.3
0.4 0.2
0.3 0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
t(d) t(s)
Fig. 8. Joint angle for planning in Fig. 9. Joint angle for planning in
the Cartesian space the joint space
10 10
X-axis X-axis
Z-axis Z-axis
9 9
8 8
7 7
6 6
Ankle Position(cm)
Ankle Position(cm)
5 5
4 4
3 3
2 2
1 1
0 0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
t(s) t(s)
Fig. 10. Ankle position for planning Fig.11. Ankle position for planning
in the Cartesian space in the joint space
253
18 16
X-axis X-axis
Z-axis Z-axis
16 14
14 12
Knee Position(cm)
Knee Position(cm)
12 10
10 8
8 6
6 4
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
t(s) t(s)
Fig. 12. Knee position for planning Fig. 13. Knee position for planning
30 30
X-axis X-axis
Z-axis Z-axis
25 25
20 20
Hip Position(cm)
Hip Position(cm)
15 15
10 10
5 5
0 0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
t(s) t(s)
Fig. 14. Hip position for planning Fig. 15. Hip position for planning
2 2
ZMP ZMP
CG CG
0 0
-2 -2
-4 -4
Y(cm)
Y(cm)
-6 -6
-8 -8
-10 -10
-12 -12
5 6 7 8 9 10 11 12 13 14 15 2 4 6 8 10 12 14 16
X(cm) X(cm)
Fig. 16. Fig. X ZMP and CG Fig. 17. Fig. X ZMP and CG
for planning in the Cartesian space for planning in the joint space
254
3 Humanoid Penalty Kicking
For kicking planning in this paper, we assume that the supporting foot keeps static
while the kicking leg moves towards the ball. Furthermore, the velocity of the
kicking leg at the start and end points is zero, so the moments from reciprocating
links are negligible. In the following discussion, CG is used as criteria to evaluate
humanoid static stability. Furthermore, humanoid penalty kicking requires that the
kicking foot trajectory should be straight in the kicking range, which cannot
achieve in joint space planning. Therefore, humanoid penalty kicking planning is
conducted in the Cartesian space.
3.1 Basic Research Issues of Humanoid Kicking
3.1.1 Kicking a Ball

There are many different methods for kicking a ball. As shown in Fig.18, different
kicking directions and different kicking points on the ball may result in different
kicking effects. Kicking in a horizontal direction is not the optimal way. But
accurate kicking requires accurate information on both the ball and robot
localization. To simplify the problem, we only consider kicking in horizontal
direction in this paper.
Fig. 18. Kicking a ball
3.1.2 Max Kicking Range and Effective Kicking Range

To achieve a longer kicking range, the humanoid robot’s hip should be kept as low
as possible, because at the max kicking range point, the kicking leg stretches
straight (Fig. 19). The lower the Hip Height (HH) is, the wider the max kicking
range is. But the HH is subjected to the constraint that the CG should lie in the
255
supporting foot, because if HH is lowered, the CG moves forward. So, there exists
a threshold value for the Lowest Hip Height (LHH).
Based on the above analysis, the kicking leg speed is zero at the maximum
kicking range point. If kicking the ball at that point, there is no momentum for the
ball. We need to define another range, effective kicking range (see Fig. 20). To
kick the ball into the goal, the minimum ball movement distance is required, and
hence the minimum momentum for the foot can be defined. Then there exists a
minimum velocity (Vmin) for the kicking foot to kick the ball into the goal. As
indicated in Fig. 21, where Ts is the time when the kicking foot enters kicking
range. The kicking range, in which the kicking foot velocity is greater than Vmin,
is defined as effective kicking range. This effective kicking range is decided by
many factors, e.g. the minimum ball movement distance, the floor property
(friction) and the foot trajectory planning.
Fig. 19. Max kicking range and lowest hip height
3.2 Gait Planning for Penalty Kicking
3.2.1 Kicking Cycles

The proposed kicking action can be decomposed into three steps (Fig. 22): (1)
Leaning the body and bending the legs, from t = T0 to T1, (2) Lifting kicking leg, t
= T1 to T2, (3) Kicking, t = T2 to T3.
For the first step, leaning is to move the center of gravity (CG) to the
supporting leg, bending for lifting kicking leg. From the previous discussion, to
get a wider kicking range the robot needs to bend as low as possible. At the same
time, the robot should lean in the frontal plane in order to move the CG to the
supporting foot. During the second step, the kicking leg should lift as high as
possible at same time the CG must keep within the supporting area. This position
256
can be obtained using the similar method as described above. For the kicking
cycle, the ankle trajectory of kicking foot is first to be generated. At the start and
end of a cycle, the kicking leg velocity is zero. This condition can be used to
balance the robot.
Fig. 20. Kicking range
Fig. 22. Kicking cycle Fig. 21. Kicking range vs. velocity
3.2.2 Ankle Trajectory for Kicking

Spline interpolation is utilized to generate ankle trajectory of the kicking leg. In
the kicking range, the kicking foot position in z-axis should be equal to ball radius.
And we define the kicking range starts from zero point. And we assume Ts as the
time for the kicking leg swing from the highest point to kicking range (xa(t) = 0).
(xa(t), ya(t), za(t) ) is the position of ankle. During the kicking phase, the ankle
should meet the following constraints:
257
' Hx t = T2 (3.1)
*
x a ( t ) = ( Lx t = T2 + Ts
* Rx t = T3
)
' Hz t = T2 (3.2)
*
za (t ) = (Rball + Lan t = T2 + Ts
*R + Lan T2 + Ts < t ≤ T3
) ball
where ( Hx , Hy , Hz ) is the position of the kicking foot at highest point. Lx is

the ankle’s x value at the point the foot enters the kicking range, Rball is the radius
of the ball. Because we assume the hip keeps static in this paper, once the ankle
trajectory is defined, the knee trajectory is also defined. From Eq.(3.2), it can be
seen that za(t) is a constant during T2 + Ts < t ≤ T3 , this condition guarantees the
kicking foot in the horizental direction.
3.3 Experiment
The kicking cycle time for this experiment is shown in Table 4.
Table 4. Kicking cycle time

Time T0 T1 T2 T3 Ts
(s) 0 0.3 1.8 3.3 0.9
Let Hz = 15cm, Lx = 0, Rball = 5cm, and LHH=88cm. According to the constraints

(Eqs.(3.1) and (3.2)) and the parameters given in Table 1 and Table 4, an ankle
trajectory can be generated as shown in Fig. 23. The kicking foot velocity is given
in Fig. 24. Based on this velocity, the effective kicking range can be derived. Fig.
25 shows the three joints angular trajectories. The CG position is given in Fig. 26,
which always lies in the supporting foot area, i.e. Lab<CGx<Laf. Fig. 27 shows
the video clips of the robot kicking. The experiment was conducted in the
Advanced Robotics and Intelligent Control Centre (ARICC) of Singapore
Polytechnic. The humanoid robot used in this experiment is a 12 DOF Robo-
Erectus RE40I which won second place in Humanoid Walk competition at the
RoboCup 2002 and got third place in Humanoid Robot Soccer competition at the
FIRA 2002. The proposed kicking planning method has been also applied for a 22
DOF Robo-Erectus RE40II (www.robo-erectus.org).
258
10 1.8
9 1.6
8
1.4
7
1.2
Velocity(m/s)
1
Z(cm)
5
0.8
4
0.6
3
0.4
2
1 0.2
0 0
-20 -15 -10 -5 0 5 10 15 20 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8
X(cm) t(s)
Fig. 23. Ankle trajectory of Fig. 24. Velocity of

the kicking leg the kicking foot
1.2 10
Ankle
Hip
9
1 Knee
8
0.8
7
0.6
6
0.4
Angle(deg)
Z(cm)
0.2
4
0
3
-0.2 2
-0.4 1
-0.6 0
0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 -5 -4 -3 -2 -1 0 1 2 3 4 5
t(s) X(cm)
Fig. 25. Angular trajectories Fig. 26. COG Trajectory

of the kicking leg
Fig. 27. Humanoid kicking

259
4 Concluding Remarks
Gait planning for humanoid walking and penalty kicking in both joint space and
Cartesian space has been discussed in a detailed way in this paper. By introducing
some new concepts such as maximum kicking range and effective kicking range, a
novel gait planning method for humanoid penalty kicking is also presented. All
the proposed gait synthesizing methods have been successfully implemented on a
soccer-playing humanoid robot, named Robo-Erectus, developed in Singapore
Polytechnic.
Research on humanoid gait planning is still in its early stage. There are still a
lot of problems remain unsolved, e.g. how to plan humanoid kicking while
maintain the robot’s dynamic stability. It is also noticed that many parameters
need to be tuned for both walking planning and penalty kicking planning. How to
further develop our humanoid robot with an on-line learning capability to perceive
new behavior will be our new research line [19-21].
References
[1] K. Hirai, M. Hirose, Y. Haikawa, and T. Takenaka, “The development of

honda humanoid robot,” Proc. of the International Conference on Robotics
& Automation, Leuven, Belgium, May 1998, pp.1321-1326.
[2] http://www.fira.net/
[3] http://www.robocup.org/
[4] http://www.robo-erectus.org/
[5] M. Vukobratovic, M. Borovac, B. Surla, D. Stokic, Biped Locomotion:
Dynamics, Stability, Control and Application, Springer-Verlag, Berlin-
Heidelberg, 1990.
[6] M. Vukobratovic, D. Juricic, “Contributions to the synthesis of biped
locomotion,” IEEE Trans. on Bio&Eng, BME-16, 1969.
[7] J. Furusho, A. Sano, “Sensor-based control of a nine-link biped,” Int.
Journal of Robotics Research, Vol.9, No.2, April 1990, pp. 83-98.
[8] T. Mita, T. Yamaguchi, T. Kashiwase, and T. Kawase, “Realization of a
high speed biped using modern control theory,” Int. Journal of Control,
Vol. 40, 1984, pp. 107-119.
[9] H. Miura, I. Shimoyama, “Dynamic walk of a biped,” Int. Journal of
Robotics Research, Vol. 3, No 2, Summer 1984, pp.60-74.
[10] Y. F. Zheng, H. Hemami, “Impact effects of biped contact with the
environment,” IEEE Tans. Systems. Man. and Cybernetics, Vol. SMC-14,
No. 3, May/June 1984, pp.437-443.
[11] Y. Zheng, J. Shen, “Gait synthesis for the SD-2 biped robot to climb
sloping surface,” IEEE, Trans. Robotics and Automation, Vol. 6, No. 1,
Feb 1990, pp. 86-96.
260
[12] C.-L Shih, “Gait synthesis for a biped robot,” Robotica, Vol. 15, 1997,
pp.599-607.
[13] C.-L Shih, “Ascending and Descending Stairs for a Biped Robot,” IEEE.
Trans. On Systems, Man, and Cybernetics- Part A: Systems and Humans,
Vol. 29, May 1999, pp. 255-268.
[14] J. Furuso, M. Masubuchi, “Control of a dynamic biped locomotion system
for steady walking”, Journal of Dynamic System, Measurement, and
Control., Vol. 108, June 1986, pp.111-118.
[15] F. Miyazaki, S. Arimoto, “A control theoretic study on dynamical biped
locomotion,” Journal of Dynamic Systems, Measurement, and Control,
Vol. 102, Dec 1980, pp. 233-239.
[16] S. Kajita, “Dynamic walking control of a biped robot along a potential
energy conserving orbit,” IEEE Transactions on Robotics and Automation,
Vol. 8, No. 4, Aug 1992, pp.431-438.
[17] Q. Huang, K. Yokoi, S. Kajita, K. Kaneko, H. Arai, N. Koyachi, and K.
Tanie, “Planning walking patterns for a biped robot,” IEEE Trans. Robot.
Automat, vol. 17, June 2001, pp.280-289.
[18] H. Benbrahim, and J. A. Framklin, “Biped dynamic walking using
reinforcement learning,” Robotics and Autonomous Systems, Vol. 22, 1997,
pp.283-302.
[19] C. Zhou, “Nero-fuzzy gait synthesis with reinforcement learning for a
biped walking robot,” Soft Computing, Vol. 4, 2000, pp.238-250.
[20] C. Zhou, “Robot learning with GA-based fuzzy reinforcement learning
agents,” Information Sciences, vol.145, 2002, pp.45-68.
[21] C. Zhou, Q, Meng, “Dynamic balance of a biped robot using fuzzy
reinforcement learning agents,” Fuzzy Sets and Systems, Vol. 134, No. 1,
2003, pp.169-187.
[22] W. Salatian, Y. F. Zheng, “Gait synthesis for a biped robot climbing
sloping surface using neural networks, Part 1: static learning,” Proc. IEEE
Int. Conf. on R&A, Nice France, May 1992, pp.2601-2606.
[23] W. Salatian, Y. F. Zheng, “Gait synthesis for a biped robot climbing
sloping surface using neural networks, Part 2: dynamic learning,” Proc.
IEEE Int. Conf. on R&A, Nice France, May 1992, pp.2607-2608.
[24] C.-M. Chew, Dynamic Biped Walking Assisted by Learning, Ph.D Thesis,
Dept. of Mechanical Engineering, MIT, Sep 2000.
[25] G. Capi, S. Kaneko, K. Mitobe, L. Barolli, and Y. Nasu, “Optimal
trajectory generation for a prismatic join biped robot using genetic
algorithm,” Robotics and Autonomous Systems, Vol. 38, 2002, pp.119-128.
[26] M.-Y. Cheng, and C.-S. Lin, “Genetic algorithm for control design of biped
locomotion,” Journal of Robotic Systems, Vol. 14, 1997, pp. 365-373.
[27] A. L. Kun, and W. T. Miller, “Control of variable-speed gaits for a biped
robot,” IEEE Robotics and Automation Magazine, Sep 1999, pp. 19-29.
[28] T. Furuta, T. Tawara, Y. Okumura, M. Shimizu, K. Tomiyama, “Design
and construction of a series of compact humanoid robots and development
of biped walk control strategies,” Robotics and Autonomous Systems, Vol.
37, 2001, pp.81-100.
261
[29] J. Koivo, Fundamentals for Control of Robotics Manipulators, John Wiley

& Sons Inc, 1989.
[30] J.J. Craig, Introduction to Robotics: Mechanics and Control, 2nd Edition,
Addison-Wesley, 1989.
[31] M. Vukobratovic, M. Kircanski, Kinematics and Trajectory Synthesis of
Manipulation Robots, Springer-Verlag, Berlin-Heidelberg, 1986.
[32] A. Goswami, “Postural stability of biped robots and the foot-rotation
indicator point,” Int. Journal of Robotics Research, vol.18, No.6, 1999,
pp.523-533.
[33] Z. Tang, C. Zhou and Z. Sun , “Trajectory planning for smooth transition
of biped robots,” Proceedings of the IEEE Int. Conf. Robotics and
Automation (ICRA’2003), 2003, pp.2455-2460.
[34] R. Dunn, R. D. Howe, “Towards smooth bipedal walking,” Proceedings of
the 1994 IEEE International Conference on Robotics and Automation, San
Diego, CA, May 1994, pp. 2489-2494.
[35] R. Dunn, R. D. Howe, “Foot placement and velocity control in smooth
bipedal walking,” Proceedings of the IEEE International Conference on
Robotics and Automation, Minneapolis, April 22-28, 1996, pp.578-583.
Appendix A. Cubic Spline Interpolation
For n breakpoints x1 < x2 < # < xn , the cubic spline interpolation S ( xi ) = yi ,

i=1,2,…n, the third-order spline function S(t) is a cubic polynomial for each
'
(ti,ti+1), and the first derivative S (t ) and second derivative S ''(t ) are continuous
on (t1,tn),
S ( x j ) on the subinterval [xj,xj+1] is denoted by the following equation:
Si ( x) = yi + yi ,i +1 ( x − xi ) +
( x − xi )( x − xi +1 )
[2 K i + K i +1 + K i ,i +1 ( x − xi +1 )]
6
for each i=1,2,…n :
yi +1 − yi
yi ,i +1 =
xi +1 − xi
K − Ki
K i ,i +1 = i +1
xi +1 − xi
Let hi = xi +1 − xi , Ki is the solution of the following equations:
262
'2h1 K1 + h1 K 2 = 6( y1,2 − m1 )
*
(hi −1 K i −1 + 2(hi −1 + hi ) K i + hi K i +1 = 6( yi ,i +1 − yi −1,i ) i = 2, 3,...n − 1
*h K + 2h K = 6(m − y )
) n −1 n −1 n −1 n n n −1, n
If the initial constraint S(x1)=m1, S(x2)=m2, the following equations are
obtained:
'a1 = 1/ 2
*
( hi +1
*ai +1 = 2(h + h ) − h a i = 1, 2,...n − 2
) i i +1 i i
' 3
*d1 = h ( y1,2 − m1 )
* 1
* 6( yi +1,i + 2 − yi ,i +1 ) − hi di
(di +1 = i = 1, 2,...n − 2
* 2(hi + hi +1 ) − hi ai
* 3
*d n = (mn − yn−1,n )
*) hn −1
h
If an −1 ≠ 2 , ai ≠ 2(1 + i +1 ) (i = 1, 2,...n − 2) then
hi
' 6(mn − yn −1,n ) − hn −1d n −1
*Kn =
( hn −1 (2 − an−1 )
*K = d − a K i = n, n − 1,...3, 2
) i −1 i −1 i −1 i
Fuzzy-Neural Impedance Control for Robots
Zhong-Li Xu, and Gu Fang
School of Engineering and Industrial Design, University of Western Sydney,

NSW 1797, Australia.
g.fang@uws.edu.au
Abstract. In conventional impedance control, the difficulties encountered in

obtaining an exact system dynamic model and selecting its impedance parameters
have prevented it from being applied to many real world applications. The
integration of Fuzzy Logic Control (FLC) and Neural Networks (NNs) into
impedance control can not only simplify the design procedure but also improve
the controller’s performance. In this paper a fuzzy-neural impedance controller is
introduced to control a robot to follow complex spatial edges. To design such a
fuzzy-neural controller, firstly, an FLC is designed by trial and error based on the
human knowledge of the impedance control. This FLC is then used to train an
NN. After the training, the NN, called the fuzzy-neural controller in this paper,
can be used to control the robot. By using this method the fuzzy-neural impedance
controller can handle the system inexactness and uncertainties effectively.
Furthermore, the designing process of this controller is simple. The performance
of this fuzzy-neural impedance controller is compared with that of other types of
impedance control methods. Simulation results are used to show the effectiveness
of such a fuzzy-neural based impedance control.
1 Introduction
Robot force control strategies should be employed in robotic contact applications,
such as peg-in-hole type of assembly and complex spatial edge following tasks. In
general, robot force control can be classified into hybrid position/force control and
impedance control [2] [9] [16]. In hybrid position/force control [10] since the
force and position are controlled separately and their directions are orthogonal,
this control method may cause complexity in task planning due to the separation
of the two control subspaces and instability due to its need of switching between
the two subspaces [12]. In contrast, because impedance control [5] takes general
dynamic relationship that includes both position/force and velocity/force
relationship between position error and interacting force into consideration, the
control task specification and programming are simplified. Many research works
have been reported on implementing impedance control. A six degree-of-freedom
(DOF) impedance controlled robot [6] is used to perform contact tasks in
assembly, such as in screwing a threaded rod into a threaded hole. An adaptive
264
264 Z.-L. Xu and G. Fang
force tracking impedance control [7] is proposed for a robot to perform

deburring/milling tasks. In [3] a robot impedance control with non-diagonal
stiffness matrix is discussed.
However, conventional impedance control assumes that accurate system
parameters and the dynamic model are available. Its performance could be
severely degraded if these assumptions are not satisfied. Furthermore, impedance
parameters are fixed in conventional impedance control method. However, these
parameters in reality may vary according to the change of operation conditions. In
contrast, fuzzy logic is capable of dealing with inexactness and neural networks
(NNs) have the ability of nonlinear mapping and learning. Due to these reasons
fuzzy logic and neural network based impedance control strategies are introduced.
Fuzzy logic is used in [1] and [14] to derive the robot impedance adaptation for
assembly tasks and touching and pushing type of operations. An NN impedance
controller is introduced in [15] to regulate the desired impedance of the robot end-
effector by learning. In [8] two NN compensation schemes, one position based and
the other force based, were proposed to counteract the uncertainties in the robot
dynamics and disturbances during applications.
In this paper, a fuzzy-neural impedance control strategy, which takes advantage
of both the FLC and the NN, is introduced to perform the complex spatial edge
following tasks. To generate the training data for this fuzzy-neural controller, an
FLC is firstly designed by trial and error based on the knowledge of the
conventional impedance control. This FLC is then used to train the NN.
The main contribution of the paper is the application of a fuzzy-neural
impedance control, which utilizes the advantages of both the FLC and the NN, for
robot in following complex spatial contours. The effectiveness of this fuzzy-neural
impedance controller is demonstrated by computer simulations. Furthermore, the
performance comparison of the proposed fuzzy-neural controller against the
conventional impedance controller, an FLC designed by trial and error and an NN
based impedance controller has also shown that the fuzzy-neural impedance
controller is more robust.
This paper is organized as follows. In section 2, the design method of a fuzzy-
neural impedance controller is introduced. The simulation results of the proposed
controller are presented in section 3. The discussions and performance
comparison of the proposed controller against other types of impedance
controllers are also presented in section 3. Conclusions are then given in section 4.
265
2 Fuzzy-Neural Impedance Control
2.1 Impedance control
When a robot manipulator is in contact with a work-piece, the desired torque

between the robot end-effector and the environment can be controlled using the
following equations [13]:
τ = J T [ K∆X + L∆X# ] + h(q, q# ) + c (q ) + f (q)
Where τ is an n×1 vector of control torque/forces necessary to drive the robot
joint actuators, JT is the transpose of an n×n vector of the robot Jacobian matrix ,
∆X and ∆X# are n×1 task space position and velocity error vectors, respectively, K
and L are the n×n diagonal stiffness coefficient matrix and damping coefficient
matrix in task space, respectively, q and q# are joint position and velocity vectors
of the robot manipulator, h(q, q# ) is an n×1 vector of Coriolis and centrifugal
forces of the robot, c(q) and f(q) are n×1 vectors of the gravity forces and the
frictional forces of the robot, respectively.
To use the impedance controller for spatial edge following applications, such as
applying sealant to the surfaces of aircraft wings, the robot velocity is relatively
low (60 to100cm/min) and the edge surface is smooth. Hence the Coriolis,
centrifugal forces, and the frictional forces can be ignored. Therefore, the
impedance can be simplified as:
τ = J T [ K∆X + L∆X# ] + c(q) (1)
The diagonal stiffness matrix K can be determined based on the particular job
to be performed. The stability requirement defines that the diagonal damping
matrix L should be greater than 2 K .
2.2 Fuzzy-neural impedance control
From the impedance control Eq. 1, it can be seen that when the number of DOF of
the robot is six (6), eighteen (18) input variables are required to calculate the six
joint forces/torques τ. The inputs are six Cartesian positions P (PX, PY, PZ) and
orientations Ω (Ωx, Ωy, and Ωz), or the six joint variables qi, six Cartesian errors Pe
and Ωe; and six Cartesian velocity errors P#e , and # e . Theoretically, a fuzzy
!
logic controller (FLC) should be able to map the nonlinear relationship stated in
Eq. 1. However, there are two factors that make this fuzzy mapping difficult, if not
impossible.
Firstly, in general, the number of rules, Nr, required of a FLC can be calculated
as Nr = Pn , where P is the number of partitions or membership functions (MFs) of
266
each input variables, assume all the variables have the same number of MFs and n
is the number of the variables. Therefore, to cover the entire control region for 18
input variables, even if each variable is to have only 3 MFs, almost 390 million
rules are needed. Obviously, it is practically impossible to design a FLC manually
with such an enormous amount of rules. In addition, even when such an FLC
exists, the computational time required for this FLC will make the real-time
implementation impossible. Although the hierarchical fuzzy logic control (HFLC)
[11] can be used in some cases to alleviate this problem, the HFLC is not
applicable in this case due to the difficulty in distinguishing which variables are
more important than the others.
The second factor that hinders the fuzzy logic mapping of Eq. 1 is that the
relationship of τ and the input variables are not intuitively clear. This lack of
intuitive understanding makes the fuzzy rule design difficult.
To overcome the above two problems, the proposed fuzzy-neural impedance
controller only calculates the forces/moments in Cartesian space according to the
Cartesian errors Pe and Ωe; and Cartesian velocity errors P#e , and # e . In another
!
words, the FLC is to be used to performed the following calculation:

F = K∆X + L∆X# (2)
Since the stiffness matrix K and the damping matrix L are diagonal matrices,
the above force equation can be decoupled into six equations as:
& Fx = K x X e + Lx X# e
) #
) Fy = K yYe + LxYe
) Fz = K x Z e + Lx Z# e
) (3)
'M = K Ω + L Ω #
) x Ωx xe Ωx xe
)M = K Ω + L Ω #
) y Ωy ye Ωy ye
)(M z = K Ωz Ω ze + LΩz Ω #
ze
Therefore six individual fuzzy-neural controllers can be designed to calculate
the forces and moments along X, Y and Z directions. For each fuzzy-neural
controller, there are only two inputs. Hence, the rule space is very manageable. In
addition, because the proposed fuzzy-neural controller only maps the
force/moment in one direction, the relationship of the inputs and output is
intuitively clear. This makes the fuzzy rule design relatively simple.
Since the design procedure of each of the six controllers is the same, in this
paper the fuzzy-neural controller is only designed to control the three Cartesian
forces. In another words, the robot considered is regarded as having only 3 DOF.
This allows the simulation to be simplified without loosing generality of the
proposed method.
267
The FLC designed by trial and error

To obtain a fuzzy-neural impedance controller, first, a fuzzy impedance controller
is designed by trial and error based on the knowledge of the conventional
impedance control expressed in Eq. 3.
To design the fuzzy logic controller for each Cartesian direction by trial and
error, 5 trapezoidal MFs are used for each input variable. The MFs are positive big
(PB), positive small (PS), zero (ZE), negative small (NS) and negative big (NB).
On the other hand, 9 triangle MFs are used for each output force (Fx, Fy, Fz). They
are positive very big (PVB), positive big (PB), positive medium (PM), positive
small (PS), zero (ZE), negative small (NS), negative medium (NM), negative big
(NB) and negative very big (NVB).
The FLC rule base is in the form expressed below
IF Pei is Am and P#ei is Bm THEN Fi is Cn
Where i = 1, 2, 3 represents the variables in X, Y and Z directions respectively;
Pei and P#ei are the respective position errors and velocity errors at each direction,
Fi is the output Cartesian force, m=1, 2, …, 5 is the number of input variable MFs
and n=1, 2, …, 9 is the number of output variable MFs. The fuzzy rules are
displayed in Table 1.
Table 1. Table of Fuzzy Rules

Pej
Fi
NB NS ZE PS PB
NB NVB NVB NB NM ZE
NS NVB NB NM ZE PS
P#ej ZE NB NS ZE PS PB
PS NS ZE PM PB PVB
PB ZE PM PB PVB PVB
The fuzzy-neural impedance controller

The input and output data from the FLC designed in the previous section can then
be used to train a neural network that learns the fuzzy mapping of the inputs and
output as stated in Table 1. After the neural network is properly trained, this neural
network, called the fuzzy-neural controller in this paper, can be used to perform
the impedance control. The objective of this fuzzy-neural impedance controller is
to utilize the advantages of both the FLC and the NN to achieve better control
performance for robots.
The fuzzy-neural controller for each axis has two inputs, Pei and P#ei , same as
those in the FLC described in the previous section. One hidden layer consists of
20 neurons and one output that represents the control force F.
268
3 Simulation Results
To demonstrate the effectiveness of the proposed control method, the fuzzy-neural

impedance controller is used to control the first three DOF of a PUMA robot.
Furthermore its performances are compared with that of the conventional
impedance controller, a neural network based impedance controller and the FLC
introduced in the first part of section 2.2.
3.1 Simulation setup
The simulations are carried out using Simulink and relevant Matlab Toolboxes.
The robot manipulator is configured as Left and Below Arm [4] and the payload is
set to 0.5 kg.
For comparison purpose, a conventional impedance controller is designed using
Eq. 1. The following stiffness matrix is used in simulations:
#40000 0 0 *
K = $$ 0 40000 0 ++ (N/m)
$% 0 0 5000+,
This stiffness matrix implies that while the control along X and Y directions are
position based, the control along the Z direction is a compliant force controller
[13]. In order to achieve an overdamping, L should be greater than 2 K ,
therefore, the following damping matrix L is chosen:
#500 0 0 *
L = $ 0 500 0 ++
$
$% 0 0 175+,
Also for comparison, the data obtained from this conventional impedance
controller is then used to train a neural network. This neural network is structured
with six inputs representing Pe and P#e , two hidden layers with 15 and 20 neurons,
respectively, and three outputs representing Fx, Fy and Fz. After the training is
completed this neural network, called the neural network based impedance
controller, can be used to control the robot.
It is assumed that the reference signals of X, Y and Z directions are all sine
waves. In addition, the robustness of the proposed controller is also examined by
introducing random disturbances into the simulation model.
Simulation results of the robot using four different control methods are
presented in section 3.2. The performance of the impedance controller is denoted
as ‘Imp’ in the figures. The performance of the neural network based impedance
controller is denoted as ‘ImpNN’. The results obtained using the FLC designed
using the trial and error method introduced in 2.2 are denoted as ‘FLC’ in the
figures. The simulation results of the proposed fuzzy-neural controller are
denoted as ‘FuzNN’.
269
3.2 Simulation results
As mentioned in 3.1, the reference signals in X, Y and Z are set as sine waves,
shown in Fig. 1. From simulation, it is found that the performances of the four
controllers along X, Y directions are similar. The tracking errors of the four
controllers at X direction without noise, and at Y direction with noise, are shown
in Figs. 2 and 3, respectively. It can be seen from Fig. 2 that the fuzzy-neural
controller produced the best results, with smoother steady state error and smaller
overshoot.
Fig. 1. Reference sine wave inputs for X, Y, and Z directions.
The statistics of the tracking errors when there is no disturbance is also shown
in Table 2. The fuzzy-neural controller has the smallest standard deviation of
error.
Table 2. Statistical information of the X direction tracking error without disturbances

Min Max Median STD
(mm) (mm) (mm) (mm)
Imp 6E-4 16E-4 3.1E-6 0.0002
FLC 4E-4 9E-4 -2.4E-6 0.00027
ImpNN -5E-4 15E-4 -2.5E-5 0.00017
FuzNN -0.9E-4 3E-4 0.24E-5 0.00004
270
To further evaluate the performance of the controllers, disturbances were added

to position and velocity error feedback. The tracking errors of the four controllers
along Y direction, when the system is subjected to disturbances, are shown in Fig.
3. It is clear that the fuzzy-neural controller out performed the other three
controllers and has a better robustness when the system is subjected to noises.
Overall, it has smaller error range, smoother tracking error and less overshoot. The
statistics of the tracking errors when disturbances are considered are shown in
Table 3.
Fig. 2. The comparison of the X direction control errors.
Table 3. Statistical information of the Y direction tracking error subject to disturbances
Min Max Median STD

(mm) (mm) (mm) (mm)
Imp -15E-4 5.5E-4 4.4E-6 0.0003
FLC -5.6E-4 13E-4 -5.2E-6 0.00025
ImpNN -5.6E-4 16E-4 -1.9E-5 0.00021
FuzNN -4E-4 0.78E-4 0.9E-5 0.00017
271
Fig. 3. The comparison of the Y direction control errors (subject to noises)
Fig. 4. The tracking error along the Z direction using the impedance controller
272
Fig. 5. The tracking error along the Z direction using the FLC.
Fig. 6. The tracking error along the Z direction using NN impedance controller.
273
Fig. 7. The tracking error along the Z direction using the fuzzy-neural controller.
At Z direction, the fuzzy-neural controller is again the best performer when

compared to the other three controllers. The tracking errors of the four controllers,
with and without disturbances, are shown in Figs. 4 to 7. Their statistics are also
tabled in Tables 4 and 5.
It can be seen from Figs. 4 to 7 and Tables 4 and 5, the fuzzy-neural controller
has the best control performance in almost every aspects, such as smallest
overshoot and error range, smoothest steady state error. Other than that, all four
controllers were able to maintain Ze, which is the distance between virtual
position and actual position of the robot end-effector, at about 2 to 2.5mm
distance.
Table 4. Statistical information of the Z direction tracking error without disturbances

Min Max Median STD
(mm) (mm) (mm) (mm)
Imp 0 0.0029 0.0025 0.00026
FLC 0 0.0036 0.0024 0.00029
ImpNN 0 0.0042 0.0024 0.00033
FuzNN 0 0.0023 0.0020 0.00020
274
Table 5. Statistical information of the Z direction tracking error subject to disturbances

Min Max Median STD
(mm) (mm) (mm) (mm)
Imp 0 0.0031 0.0025 0.00036
FLC 0 0.0045 0.0025 0.00041
ImpNN 0 0.0041 0.0024 0.00043
FuzNN 0 0.0022 0.0020 0.00018
4 Conclusions
In this paper a fuzzy-neural impedance controller is introduced to control a robot

in complex spatial edge following. This controller is built using a neural network
that is trained based on the fuzzy relationship the input and output data described
in section 2.2. Because of the advantages of both the FLC and the NN are utilized,
this control method provides the best control performances in comparison with the
other types of impedance controller. Simulation results have shown that the
proposed fuzzy-neural impedance controller has performed satisfactorily. The
next stage of the research is to experimentally implement and verify the proposed
method in complex spatial edge following applications.
References
1. Babaci S, Amirat Y, Pontnau J, (1996) Fuzzy Adaptation Impedance of 6 DOF Parallel

Robot Application to Peg In Hole Insertion. In: Proceedings of the 5th IEEE
International Conference on Fuzzy Systems, LA, USA, vol 3, pp 1770-1776.
2. Bruyninckx H, Schutter JD (1996) Specification of Force Controlled Actions in the
‘Task Frame Formalism’ - A Synthesis. IEEE Transactions on Robotics and
Automation 12(4): 581-589.
3. Caccavale F, Siciliano B, Villani L (1999) Robot Impedance Control with
Nondiagonal Stiffness. IEEE Transactions on Automatic Control 44(10): 1943-1946.
4. Fu KSR, Gonzalez C, Lee CSG (1987) Robotics: Control, Sensing, Vision, and
Intelligence. McGraw-Hill, Singapore.
5. Hogan N (1985) Impedance Control: An Approach to Manipulation, Parts I – III.
ASME Journal of Dynamic Systems, Measurement and Control 107(1): 1-24.
6. Jossi D, Donath M (1995) Using Six Degree of Freedom Impedance Control Robots to
Perform Contact Tasks in a Workcell. In: Proceedings of the ASME Systems and
Control Division. pp 207-216.
7. Jung S, Hsia TC (1999) Adaptive Force Tracking Impedance Control of Robot for
Cutting Nonhomogeneous Workpiece. In: Proceedings of the 1999 IEEE International
Conference on Robotics and Automation, Detroit, Michigan. pp 1800-1805.
8. Jung S, Hsia TC (1998) Neural Network Impedance Force Control of Robot
Manipulator. IEEE Transactions on Industrial Electronics 45(3): 451-461.
275
9. Patarinski SP, Botev RG (1993) Robot Force Control: A Review. Mechatronics 3(4):
377-398.
10. Raibert MH, Craig JJ (1981) Hybrid Position/Force Control of Manipulators. ASME
Journal of Dynamic Systems, Measurment, and Control 102: 407-410.
11. Raju GV, Zhou SJ, Kisner RA (1991) Hierachical fuzzy control. International Journal
of Control 54(5): 1201-1216.
12. Seraji H, Colbaugh R (1997) Force tracking in impedance control. International
Journal of Robotics Research, 106(1): 97-117.
13. Schilling RJ (1990) Fundamentals of Robotics: Analysis & Control. Prentice-Hall,
New Jersey.
14. Shibata M, Murakami T, Ohnishi K (1996) A Unified Approach to Position and Force
Control by Fuzzy Logic. IEEE Transactions on Industrial Electronics 43(1): 81-87.
15. Tsuji T, Ito K, Morasso PG (1996) Neural Network Learning of Robot Arm impedance
in Operational Space. IEEE Transactions on Systems, Man, and Cybernetics 26(2):
290-298.
16, Zeng G, Hemami A (1997) An overview of robot force control. Robotica 15: 473-482.
276
On Singular Perturbation Based Inverse

Dynamics Control for a Two-Link Flexible
Manipulator
Xuefeng Dai1,2, Lining Sun1,2, and Hegao Cai1,2
1
Robotics Research Institute, Harbin Institute of Technology, Harbin 150001,
P.R. China
2
Robotics Laboratory, Shenyang Institute of Automation,
Chinese Academy of Sciences, P.R. China
Xuefengxfdai@263.net
Abstract. Based on singular perturbation technology, the flexible-link

manipulator system model was divided into two sub-systems, one for slow
sub-system describing equivalent rigid motion, and one for fast sub-system
describing vibration. A novel inverse dynamics control scheme based on the two
sub-system models was proposed. The algorithm has no specific demand on the
desired trajectory.
1 Introduction
The use of lightweight material in manipulator links can achieve higher motion
speed, better energy efficiency and improved mobility. Unfortunately, the flexible
links result in vibration during and after motion. The control of flexible link
manipulator not only need to deal with macro motion, but also need to deal with
vibration.
The model of flexible link manipulators is a set nonlinear time-varying
differential equations [1]. Its control related to almost each branch of control
theory [2]. Among these results, singulator perturbation technology is used for a
single link manipulator control by Siciliano and Book[3], two-link flexible
manipulator control by Wang and Unbehauen[4] and structure optimization of
flexible manipulators [5]. Additionally, inverse dynamics control has deterministic
physical meaning. In front of the manipulator's sophisticated model, the scheme is
hard to be implemented. A novel methedology of realizing inverse dynamics
control based on singular technology is proposed in this paper.
On Singular Perturbation Based Inverse Dynamics Control 277
277
2 Modeling Flexible Manipulator
It is assumed that the amplitude of the higher modes of the flexible links is very
small compared to that of the first mode. The dynamics of a two-link flexible
manipulator could be expressed by the following two nonlinear time varying
differential equations[1]
Mθ## + g + Mq## + f1 + f 2 = T (2.1)
Mq## + Cq# + Kq + Mθ## + g + f 2 = 0 (2.2)
where θ = [θ1 θ 2 ] T
are the joint angulars, q = [q1 q2 ] the local
coordinates of two links, T the control input, M the positive definite and
symmetrical inertia matrix and varying vs joint angular θ 2 , C the damping
matrix, K the system stiffness matrix, g , f1 and f 2 the vectors containing
gravitational, Coriolis, and centrifugal terms. M , C and K have the form
#a + a2 + 2a3c2 a2 + a3c2 '
M =$ 1 (2.3)
% a2 + a3c2 a2 ()
#C − 2a3θ#2 s2 − 2a3 (θ#1 + θ#2 '
C=$ 1 ( (2.4)
#
% 2a3θ1s2 C2 )
# K − a3[(2θ##1 + θ##2 ) s2 + (2θ#1θ#2 + θ#22 )c2 ]'
K=$ 1 ( (2.5)
%0 K 2 − a3 (θ##1s2 − θ#12 c2 ) )
where c2 = cos θ 2 , s2 = sin θ 2 , other variables refer to paper [1].
Considering kii >> −a3[(2θ##1 + θ##2 )s2 + (2θ#1θ#2 + θ#22 )c2 ](i = 1,2) and
K2 >> a3 (θ##1s2 −θ#12c2 ) , for the purpose of simplicity in the following, let
$
K = diag ( K1 , K 2 ) instead of K .
The end trajectory of manipulator can be expressed as a set of <y, >, where
!
y is the straight distance of end effector from its initial point. And
y = Φ 0θ + Φ f q (2.6)
where Φ 0 = [l1 l2 ] l1 and l2 are the length of two links, respectively.
"
Φ f = [φ11 (l ) φ 21 (l )] , in general, for the n-th modal

φin ( x) = l ( F1,n sin d n χ + F2,n sinh d n χ + F3,n cos d n χ + F4,n cosh d n χ ) ,
1
here χ = x/l d n ≈ (n + )π , F j ,n ≈ (−1) j +1 (8M t l 2 + ρl 3 ) −1/ 2 .
4
#
278278X. Dai, L. Sun, and H. Cai
3 Inverse Dynamics Control Based on Singular

Perturbation
Denoting q with q = µ ξ , here µ = Kˆ . From the principle of singular

2 2 −1
perturbation, Eqs. 2.1 and 2.2 can be rewritten as

Mθ## + g + Mµξ 2 + f1 ( µξ ) + f 2 (θ , µξ ) = T (3.1)
Mµ 2ξ## + Cµξ# + ξ + Mθ## + g + f 2 (θ , µξ ) = 0 (3.2)
Let µ = 0 and take into account both f1 and f 2 visibly involving q , Eqs.
(3.1) and (3.2) are reduced to
##
Mθ + g = T (3.3)
ξ + Mθ## + g = 0 (3.4)
ξ can be resolved from Eqs. (3.3) and (3.4). Furthermore q can be obtained.
From Eq. (2.6), the approximated value y of y can be gained, too. These are
given by
$
q ≈ µ 2ξ = − K −1T (3.5)
y = Φ θ − Φ Kˆ −1T
0 f (3.6)
Since the desired end trajectory of manipulator yd is known, substitute y
in Eq. (3.6) with yd . Hence, one equation containing T as variable is
Φ f Kˆ −1T = Φ 0θ − yd (3.7)
Considering the desired motion direction of the manipulator end effector, we
have LT = 0 , where L ∈ R 2×1 . Now, from the above
−1
#Φ f Kˆ −1 ' #Φ 0θ − yd '
T =$ ( $ ( (3.8)
% L ) % 0 )
Up to now, T is still not suitable for control, because there're a last unknown
variable θ in Eq. (3.8). Next, discuss how to replace θ with its approximated
value θˆ .
Substituting Eq. (3.8) in Eq. (3.3), we have
−1 −1
## #Φ f Kˆ −1 ' #Φ 0 ' #Φ f Kˆ −1 ' # yd '
Mθ = $ ( $ (θ − $ ( $ (−g (3.9)
% L ) %0) % L ) %0)
Because of M varying slowly, so replace it with a constant matrix M̂ .
ˆ || , so ignore it for brevity, Eq. 3.9 becomes
Notice that || g ||<<|| M
On Singular Perturbation Based Inverse Dynamics Control 279
279
##
θˆ = Γˆ θˆ + βˆyd (3.10)
−1 −1
ˆ −1 ˆ −1
where Γ ˆ = Mˆ −1 #$Φ f K '( #Φ 0 ' βˆ = − Mˆ −1 #$Φ f K '( . Let η1 = θˆ ,
$ (
!
% L ) %0) % L )
#̂
η2 = θ then Eq. (3.10) is converted into
η# = Aη + Byd (3.11)
#0 I ' #0'
where A = $$ ( , I is an identity matrix, B = $ $ ( . From Eq. (3.11)
%Γ 0 ) %β )
t
η (t ) = & e A( t −τ ) Byd (τ )dτ (3.12)
0
substituting θ with θˆ in Eq.(3.8), control law T is derived.
4 Conclusions
The algorithm proposed in this paper could be outlined as following,

Given yd L, M C and K , form M̂ and K̂ .
"
# #
Calculating Γ̂ β̂ A and B .
$
# #
Resolving η from Eq. (3.12), constructing θˆ by its first two raws

%
elements.
Replacing θ with θˆ , calculating T from Eq. (3.8).
&
In the realization of the algorithm proposed in this paper, there's no specific

demand for continuously differentiability to the desired trajectory. And the
algorithm could be rearlized easily. On the other hand, inverse dynamics is a kind
of open-loop control. Its shortage is evidance.
The testbed of single link flexible manipulator had been set up [6]. Now, the
two-link flexible manipulator system is under developing. We are perfecting the
control algorithm in theory aspect and conducting empirical research meanwhile.
References
1. Liu K and Kujath MR (1996) Trajectory optimization for a two-link flexible

manipulator. Int. J. Robotics and Automation, 11(2):56-61
280280X. Dai, L. Sun, and H. Cai
2. Dai XF, Qu DS, Sun LN et al (2002) State of the art for control of flexible link
manipulators. In: Proc. 2002 Int. Conf. Contr. and Automation, Xiamen, China, pp:
113-117
3. Siciliano B and Book WJ (1988) A singular perturbation approach to control of
lightweight flexible manipulators. Int. J. Robotics Research, 7(4): 79-90
4. Wang GL and Unbehauen H(2002) Note on the relative degree of a flexible
manipulator and implications to inverse dynamics for tracking control. Robotica,
20(1): 33-48
5. Moallem M, Khorasani K and Patel RV (1998) Optimum structure design for
improving the dynamics of flexible-link manipulators. Int. J. Robotics and
Automation, 13(4): 125-131
6. Zhao HW (2000) Macro/micro powered flexible manipulator system and control
scheme research(in Chinese). Ph. D. thesis, Harbin Institute of Technology
Wavelets and Biorthogonal Wavelets
for Image Compression
Don Hong, Martin Barrett, and Panrong Xiao
Department of Mathematics and Computer Sciences, East Tennessee State

University, Johnson City, TN 37614, USA
hong@etsu.edu
Abstract. Digital images are widely used in computer applications.

Uncompressed digital images require considerable storage capacity and
transmission bandwidth. Efficient image compression solutions are becoming
more critical with the recent growth of data intensive, multimedia-based web
applications. This paper studies image compression with wavelet transforms. The
mathematical properties of several types of wavelets, including Haar, Daubechies
orthogonal, and biorthogonal spline wavelets are covered and the Embedded
Zerotree Wavelet (EZW) coding algorithm, which is used to code the transformed
wavelet coefficients, is introduced. In order to compare wavelet methods, a
software tool called MinImage is used to analyze the results of its performance and
output to compare the wavelet types.
1 Introduction
Data compression is the process of converting data files into smaller files for
efficiency of storage and transmission. It plays a very important role to rapid
progress being made in information technology. Without compression, it would
not be practical to put images, audio, and video alone on websites.
It is known that JPEG (Joint Photographic Experts Group) and MPEG (Moving
Pictures Experts Group) are standards for representing images and video. Data
compression algorithms are used in those standards to reduce the number of bits
required to represent an image or a video sequence. Compression is the process of
representing information in a compact form. Data compression treats information
in digital form, that is, as binary numbers represented by bytes of data with very
large data sets. Fox example, a single small 4″ × 4″ size color picture, scanned at
300 dots per inch (dpi) with 24 bits/pixel of true color, will produce a file
containing more than 4 Megabytes of data. At least three floppy disks are required
to store such a picture. This picture requires more than one minute for transmission
by a typical transmission line (64k bit/second ISDN). That is why large image files
remain a major bottleneck in a distributed environment. Although increasing the
bandwidth is a possible solution, the relatively high cost makes this less attractive.
282282D. Hong, M. Barrett, and P. Xiao
Therefore, compression is a necessary and essential method for creating image

files with the manageable and transmittable sizes.
As we know, a compression algorithm has a corresponding decompression
algorithm that, given the compressed file, reproduces the original file. There have
been many types of compression algorithms developed. These algorithms fall into
two broad types, lossless algorithms and lossy algorithms. A lossless algorithm
reproduces the original exactly. A lossy algorithm, as its name implies, loses some
data. Data loss may be unacceptable in many applications. For example, text
compression must be lossless because a very small difference can result in
statements with totally different meanings. There are also many situations where
loss may be either unnoticeable or acceptable. In image compression, for example,
the exact reconstructed value of each sample of the image is not necessary.
Depending on the quality required of the reconstructed image, varying amounts of
loss of information can be accepted.
This paper is concerned with a certain type of compression that uses wavelets.
Wavelets, introduced by J. Morlet [8], are used to characterize a complex pattern
as a series of simple patterns and coefficients that, when multiplied and summed,
reproduce the original pattern. There are a variety of wavelets for use in
compression. Several methods are compared on their ability to compress standard
images and the fidelity of the reproduced image to the original image. In order to
compare wavelet methods, a software tool called MinImage is used. MinImage
was originally created to test Haar and Daubechies wavelets [5]. Additional
functionality is added to MinImage to support other wavelet types, such as
biorthogonal spline wavelests. The results of MinImage’s performance and output
are then analyzed to compare the wavelet types.
2 Wavelet Transform
The fundamental idea behind wavelets is to analyze the signal at different scales or
resolutions, which is called multiresolution. Wavelets are a class of functions used
to localize a given signal in both space and scaling domains. A family of wavelets
can be constructed from a mother wavelet. Compared to Windowed Fourier
analysis, a mother wavelet is stretched or compressed to change the size of the
window. In this way, big wavelets give an approximate image of the signal, while
smaller and smaller wavelets zoom in on details. Therefore, wavelets
automatically adapt to both the high-frequency and the low-frequency components
of a signal by different sizes of windows. Any small change in the wavelet
representation produces a correspondingly small change in the original signal,
which means local mistakes will not influence the entire transform. The wavelet
transform is suited for non-stationary signals, such as very brief signals and signals
with interesting components at different scales.
Usually, wavelets are functions generated from one single function ψ , which is
called mother wavelet, by dilations and translations
283
x−b ,
ψ a ,b ( x ) =| a | −1 / 2 ψ ( )
a
where ψ satisfies .ψ ( x)dx = 0 .
The basic idea of wavelet transform is to represent any arbitrary function f as a
decomposition of the wavelet basis or write f as an integral over a and b of ψ a,b .
Let a = a 0m , b = nb0 a 0m , with integers m, n , and real numbers

a 0 > 1, b0 > 0 fixed. Then the wavelet decomposition is
f = # cm ,n ( f )ψ m,n .
For some very special choices of ψ and a 0 > 1, b0 > 0 , the functions ψ m, n
2
constitute an orthonormal basis for L (R). In particular, if we choose a0 = 2 ,
b0 = 1 , then there exists ψ , with good time-frequency localization properties
2
such that ψ m,n becomes an orthonormal basis for L (R) and thus,
c m ,n ( f ) =< ψ m ,n , f >= .ψ m ,n ( x) f ( x)dx .

In image compression, we are dealing with sampled data that are discrete in
time. We would like to have discrete representations of time and frequency, which
is called the discrete wavelet transform (DWT).
2.1 Discrete Wavelet Transform
For wavelets generated by a scaling function ϕ (x ) using a multiresolution

analysis (see [3] for examples), we can write
.
c m , n ( f ) =< ψ m , n , f >= ψ m ,n ( x) f ( x )dx as the following algorithm.
c m ,n ( f ) = # g 2 n −k a m −1,k ( f )
k
a m ,n ( f ) = # h2 n− k a m −1,k ( f ) (2.1)
k
where g l = (−1) l h−l +1 and hn = 21 / 2 . ϕ (2 x − n)ϕ ( x)dx. When f is

given in sampled form, then we can assume those samples as the highest order
resolution approximation coefficient a 0 ,n . Equation (2.1) gives the coding
algorithm on these sampled values with low-pass filter h and high-pass filter g. For
orthonormal wavelet bases, these filters give exact reconstruction by the following
equation:
a m −1,n ( f ) = # [h2 n −l a m ,n ( f ) + g 2 n−l c m ,n ( f )].
n
Compact Supported Orthogonal Wavelets
In this section, we will determine the filter coefficients for compact supported
orthogonal wavelets. We begin with the scaling function
φ (t ) = # hk 2φ (2t − k ).
k
By integrating both sides, we have
∞ ∞
. −∞
φ (t )dt = .
−∞
#h
k
k 2φ (2t − k )dt .
Substituting x = 2k − t and dx = 2dt on the right hand side, we get

∞ ∞ 1 1 ∞
. φ (t )dt = # hk 2 . φ ( x) dx = # hk . φ ( x)dx .
−∞
k
−∞ 2 k 2 −∞
Then, divide both sides by the integral and we have

#h
k
k = 2. (2. 2)
We use the orthogonality condition on the scaling function to get another

condition on {hk }:
. | φ (t ) |
2
dt = . # hk 2φ ( 2t − k )# hm 2φ ( 2t − m) dt
k m
= ## hk hm 2 . φ ( 2t − k )φ ( 2t − m) dt
k m
= ## hk hm . φ ( x − k )φ ( x − m)dx ,
k m
where in the last equation, we substituted 2t by x .

The integral on the right hand side is zero except k = m . When k = m , we
obtain
#h
k
2
k = 1. (2. 3)
By using the orthogonality of the translates of scaling function we have
. φ (t )φ (t − m)dt = δ m .
Substituting ϕ (t ) by scaling function, we get
' 2' 2
. ()# hk
2φ ( 2t − k )3 (# hk 2φ (2t − 2m − l )3 dt
k
4) k 4
= ## hk hl 2 . φ ( 2t − k )φ (2t − 2m − l ) dt.
k l
By substituting x = 2t , we get
. φ (t )φ (t − m)dt = ## h h . φ ( x − k )φ ( x − 2m − l )dx
k l
k l
285
= ## hk hl δ k −( 2 m +l ) = # hk hk − 2 m .
k l k
Therefore, we have equation
#h h
k
k k −2 m = δm . (2. 4)
By using equations (2.2)-(2.4), we can generate filter coefficients for scaling

function.
For k = 2 , from equations (2.2) and (2.3), we have
h0 + h1 = 2
h02 + h12 = 1.
1
The only solution is h0 = h1 = , which is the Haar scaling function.
2
For k = 4 , from equations (2.2)-(2.4), we have
h0 + h1 + h2 = 2
h02 + h12 + h22 = 1
h0 h2 + h1 h3 = 0 .
The solutions to these equations include the 4-tap (filter length is 4) Daubechies
scaling function.
1+ 3 , 3+ 3 , 3− 3 , 1− 3
h0 = h1 = h2 = h3 = .
4 2 4 2 4 2 4 2
We know the wavelet function is

ψ (t ) = # wk 2φ (2t − k ).
k
If the wavelet is orthogonal to the scaling function at the same scale, we have
. φ (t − k )ψ (t − m)dt = 0.
Then, we can obtain the wavelet filter coefficients from the scaling filter
coefficients [1].
wk = ±(−1) k hN −k .
Haar Wavelet
The Haar scaling function is defined as:

* 1 0 ≤ x <1
φ ( x) = +
,0 otherwise
The scaling function is plotted in Figure 1.
1 X
Fig. 1. Haar scaling function
N
In the scaling equation ϕ ( x) = # C k ϕ (2 x − k ) , only c0 = c1 = 1 , all the
k =0
other coefficients are zeros. Haar wavelets are defined as:
ψ a ,b ( x ) = 2 a / 2 ψ (2 a x − b) , b = 0,1,...,2 a − 1,
* 1 0 ≤ x < 1/ 2
where ψ ( x) = -
+− 1 1 / 2 ≤ x < 1 is the Haar mother wavelet.
- 0 otherwise
,
The Haar mother wavelet function is plotted in Figure 2.
Y
1
1 X
-1
Fig. 2. Haar mother wavelet
Daubechies Orthogonal Wavelets

There are no explicit expressions for Daubechies compact support orthogonal
wavelets and corresponding scaling functions. Table 1 presents the wavelet filter
287
coefficients for Daubechies 4 taps wavelet [3]. Figure 3 is the scaling and wavelet
functions.
Table 1. Coefficients for the 4-tap Daubechies low-pass filter
H0 .4829629131445341
H1 .8365163037378077
H2 .2241438680420134
H3 -.1294095225512603
Fig. 3. Daubechies 4-tap scaling and wavelet functions
Biorthogonal and Spline Wavelets

As we know, most images are smooth. It is reasonable to use a smooth mother
wavelet for image analysis. On the other hand, it is also desirable that the mother
wavelet is symmetric so that the corresponding wavelet transform can be
implemented using mirror boundary conditions, which reduces boundary artifacts.
Unfortunately, except for the Haar wavelet (the trivial example), no wavelets are
both orthogonal and symmetric.
To achieve the symmetric property, we can relax the orthogonality requirements
by using a biorthonogal basis. In this case, we keep the same decomposition as in
equation (2.1). The reconstruction becomes
~
a m −1, f ( f ) = # [h2 n −l a m ,n ( f ) + g~2 n −l cm ,n ( f )]
n
~ ~
where h and g may be different from h, g to obtain exact reconstruction. The
relations between them are given by the following equations.
~ ~
g~n = (−1) n h− n +1 , g n = (−1) n h− n +1 , and #h h n n+2k = δ k ,0 . (2. 5)
n
Define
~ ~~
φ ( x ) = # hn φ ( 2 x − n ) and φ ( x ) = # hn φ ( 2 x − n ) .
n k
Also define
~
ψ ( x ) = # g nφ ( 2 x − n) and ψ~ ( x ) = # g~nφ (2 x − n) .
n k
Then we can rewrite a m ,n ( f ) and c m ,n ( f ) as:

a m , n ( f ) =< φ m ,n , f >= 2 − m / 2 . φ m , n ( x) f ( x) dx
c m ,n ( f ) =< ψ m ,n , f >= 2 − m / 2 .ψ m ,n ( x ) f ( x ) dx
and reconstruction is f = # < ψ m ,n , f > ψ~m ,n .
m,n
The following figure shows the relation between the filter structure and
wavelets functions.
H (n) ~
H (n)
X X
G (n) ~
G ( n)
Fig. 4. Filter structure and the associating wavelets
~
For symmetric filters, the exact reconstruction condition on h and h given in
equation (2.5) can be represented by
~ ~
H (ξ ) H (ξ ) + H (ξ + π ) H (ξ + π ) = 1 ,
~ ~
where H (ξ ) = 2 −1 / 2 #h e
n
n
− jn ξ
and H (ξ ) = 2 −1 / 2 #h en
n
− jnξ
.
~
Together with the divisibility of H and H , respectively, by (1 + e − jξ ) k and
~
(1 + e − jξ ) k
We have [4]
~ ' l −1 $ l − 1 + p/ 2
H (ξ ) H (ξ ) = cos(ξ / 2) 2l (# %% 00. sin(ξ / 2) 2 p + sin(ξ / 2) 2l R (ξ )3 , (2. 6)
) p =0 & p 1 4
~
where R (ξ ) is an odd polynomial in cos(ξ ) and 2l = k + k .
Spline Filters
~ ~
Let us choose R ≡ 0 with H (ξ ) = cos(ξ / 2) k e − jkξ / 2 where k = 0 if k is
~
even, k = 1 if k is odd. We have
289
~ ' l −1 $ l − 1 + p/ 2
H (ξ ) = cos(ξ / 2) 2l − k e jkξ / 2 (# %% 00. sin(ξ / 2) 2 p 3 .
) p =0 & p 1 4
~
Then, we get the corresponding function φ which is a B-spline function.
~
We show one examples from this family with k = 4, k = 2 . Table 2 list the
~
their coefficients H n and H n (see [3]).
~
Table 2. Filter Coefficients with k = 4, k = 2
n 0 ±1 ±2 ±3 ±4
Hn 45/64 19/64 -1/8 -3/64 3/128

~
Hn 1/2 1/4 0 0 0
~
Fig. 5. Spline examples with l=k=k =4
A Spline Variant with Less Dissimilar Lengths
We choose R ≡ 0 , and factor the right hand side of equation (2.6) to break the
polynomial into a product of two polynomials in sin(ξ / 2) . Allocate the
~ ~
polynomials to H and H respectively to make the length of h and h as close
as possible.
~
The following example is the shortest one in this family (shortest h and h ). It
corresponds to l = k = 4 [3].
Table 3. Filter coefficients with l=k=4
n 0 ±1 ±2 ±3 ±4
Hn 0.602949 0.266864 -0.078223 -0.016864 0.026749

~
Hn 0.557543 0.295636 -0.028772 -0.045636 0
2.2 Comparison of Wavelet Properties
The following Table 4 shows the property comparison of three kinds of wavelets.
Haar and Daubechies wavelets have orthogonality, which has some nice
features.
1) The scaling function and wavelet function are the same for both forward
and inverse transform.
2) The correlations in the signal between different subspaces are removed.
Among the three kinds of wavelets, the Haar wavelet transform is the simplest one
to implement, and it is the fastest. The major disadvantage of the Haar wavelet is
its discontinuity, which makes it difficult to simulate a continuous signal.
Daubechies found the first continuous orthogonal compact support wavelet. Note
that this family of wavelets is not symmetric. The advantage of symmetry is that
the corresponding wavelet transform can be implemented using mirror boundary
Table 4. Property comparison of three kinds of wavelets

Property Haar Dau. Dau. Biorthogonal
Orthogonal Spline
Explicit Yes No Yes
Function
Orthogonal Yes Yes No
Symmetric Yes No Yes
Continuous No Yes Yes
Compacted Yes Yes Yes
support
Maximum No No Yes
regularity for order
L
Shortest scaling Yes No Yes
function for order L
291
conditions to reduce boundary artifacts. That is why we introduce the biorthogonal

spline wavelet. For the biorthogonal spline wavelet, the scaling function is a B-
spline. The B-spline of degree N is the shortest possible scaling function of order
N-1 and B-splines are the smoothest scaling functions for a filter of a given length
[12]. Since splines are piecewise polynomial, they are easy to manipulate. For
example, it is simple to get spline derivatives and integrals.
Next, we discuss how to apply wavelet transform to image compression.
3 Wavelets Applied in Image Compression
In order to compare wavelet methods, a software tool coded in Visual C++ called
MinImage is used. MinImage was originally created to test Haar and Daubechies
wavelets [5]. We add some new functionality to it in order to support other wavelet
types. Also, the EZW coding algorithm is implemented to achieve better
compression results. In the following, we discuss some implementation details of
MinImage.
3.1 Baseline Schema
The wavelet image compressor, MinImage, is designed for compressing either 24-
bit true color or 8-bit gray scale digital images. It was originally created to test
Haar wavelet using subband coding. To compare different wavelet types, other
wavelet types including Daubechies orthogonal and birothogonal spline wavelets
were implemented. Also, the original subband coding was changed to EZW coding
to obtain better compression results.
A very useful property of MinImage is that different degrees of compression
and quality of the image can be obtained by adjusting the compression parameters
through the interface. The user can trade off between the compressed image file
size and the image quality. The user can also apply different wavelets to different
kind of images to achieve the best compression results.
Figure 6 is the baseline structure of MinImage compressing sechema.
Sampled Preprocessor DWT EZW Coding
Compressed Entropy
Decompression Image Coding
Fig. 6. The baseline schema of MinImage

For more details on the Preprocessor, see [5]. In the rest of the chapter, we will
focus on implementation of discrete wavelet transform (DWT) and EZW coding.
3.2 Discrete Wavelet Transform (DWT)
The discrete wavelet transform usually is implemented by using a hierarchical

filter structure. It is applied to image blocks generated by the preprocessor.
Figure 7 shows the 1-D wavelet decomposition (forward transform) and
composition (inverse transform) algorithms.
original signal x[0..n-1], n = 2k
normalize x[i ] = x[ i ]
n
, i =
h=n
nDecompositionStep = 0
n
h>1
y
nDecompositionStep ++
y
nDecompositionStep > nMaxDecompostionStep
n
DecompositionStep( x[0..h-1] )
h=h/2
wavelet coefficients x[0..n-1]
Fig. 7. The 1-D wavelet decomposition algorithm in MinImage

293
wavelet coefficients x[0..n-1],

n=2k
nStart =
n
i=0
n
i<nMaxDecompostionStep
i ++
y nStart = nStart >>
MaxDecompostionStep
nStart >> i = = 0
nStart =
2
h=
nStart
n
h <= n
CompositionStep( x[0..h-1]
)
h ++
denormalize x[i ] = x[i ] × n , i =
the original signal x[0..n-

1]
Fig. 8. The 1-D wavelet composition algorithm in MinImage

As we know, images are 2-D signals. A simple way to perform wavelet

decomposition on an image is to alternate between operations on the rows and
columns. First, wavelet decomposition is performed on the pixel values in each
row of the image. Then, wavelet decomposition is performed to each column of
the previous result. The process is repeated to perform the complete wavelet
decomposition.
The following is the 2-D wavelet decomposition algorithm.
PROCUDURE 2D-Decomposition( ImageData[ 0..n-1, 0..n-1 ] )

FOR row = 0 TO n-1 DO
FOR col = 0 TO n-1 DO
ImageData[ row, col ] / = n
h=n
WHILE h > 1 DO
FOR row = 0 TO h-1 DO
DecompositionStep( ImageData[ row, 0..h-1 ] )
END FOR
FOR col = 0 TO h-1 DO
DecompositionStep( ImageData[ 0..h-1, col ] )
END FOR
h=h/2
END WHILE
END FOR
0 END FOR
1END PROCEDURE
3.3 Embedded Zerotree Wavelet (EZW) Coding
After the 2-D wavelet decomposition, the wavelet transform blocks contain the
wavelet coefficients. Then, we use Embedded Zerotree Wavelet coding to code the
transformed wavelet coefficients. EZW coding was designed by Shapiro [10] to
use with wavelet transforms. In fact, EZW coding is more like a quantization
method. This method is based on progressive encoding to compress an image into
a bit stream. The coding process is implemented by passes. For each pass, a
threshold is chosen against which all the wavelet coefficients are measured. If a
wavelet coefficient is larger than the threshold, it is encoded and removed from the
image; if it is smaller, it is left for the next pass. When all the wavelet coefficients
have been visited, the threshold is lowered, and the image is scanned again to add
more detail to the already encoded image. This process is repeated until the
stopping threshold has been reached.
A zerotree is a quad-tree of which all nodes are equal to or smaller than the
root. The tree is coded with a single symbol and reconstructed by the decoder as a
quad-tree filled with zeroes. EZW coding is based on the observation that wavelet
295
coefficients decrease with scale. When the coefficients are scanned, if the root is
smaller than a threshold, it is highly probable that all the coefficients in the quad-
tree will be smaller than the threshold. If this is the case, then the whole tree can be
coded with a single zerotree symbol.
3.4 Entropy Coding
Entropy coding is the last stage of MinImage to compress the EZW coded data to a
final wavelet image file by using a lossless method. A wrapper class is written to
apply the Zlib compression engine to perform this task. Zlib is a free compression
library. It compresses the source data by LZ77 (Lempel-Ziv1977) and/or Huffman
entropy encoding. Its deflation algorithms are similar with those used by PKZIP
(an MSDOS based software by PKWARE, Inc.).
4 Experiments
In this section, we analyze the compression results produced by MinImage. These

include the subjective and objective qualities of reconstructed images for different
wavelet types, timing of composition and decomposition for different wavelet
types, and timing of EZW coding algorithm for different compression ratios.
PSNR (peak-signal-to-noise-ratio) is the most commonly used value to evaluate
the objective image compression quality. It is calculated by the following
equation:
x 2peak
PSNR( dB) = 10 log10 ,
δ d2
where δ d2 is called MSE (mean-squared-error) and is calculated by the

following equation:
1 N
δ d2 =
N
# (x
n =1
n − yn ) 2 .
4.1 Objective Quality
To compare the objective quality for different wavelet types, we use the standard
testing image (see Figure 9).
Table 5 illustrates one set of compression results.
Fig. 9. Lenna 256*256

Table 5. Comparison of compression results by using different wavelets
Wavelet Compression Ratio Compressed File size PSNR
Haar 9.177 % 18048 bytes 35.230 dB
D4 8.613 % 16938 bytes 36.515 dB
D6 8.469 % 16709 bytes 35.246 dB
D8 8.426 % 16570 bytes 36.946 dB
D 10 8.336 % 16393 bytes 35.643 dB
SP 2_2 8.321 % 16365 bytes 36.962 dB
SP 2_4 8.278 % 16280 bytes 37.152 dB
SP 4_4 8.235 % 16196 bytes 37.644 dB
From the above table, we observe that biorthogonal spline wavelets achieve
better compression results than orthogonal wavelets. Haar wavelet has the worst
result.
4.2 Subjective Quality
To compare the subjective quality for different wavelet types, we choose five
different standard testing images. All compressed images have the same
compression ratio (1%). For each image, we present the reconstructed images in
random order to some volunteers to evaluate the quality based on the original
image (graded as 10). The following table 6 shows the result, which is conformed
to the objective result.
Haar wavelet obtains the worse subjective quality because of its discontinuity,
which results in the block artifact in reconstructed images. It is hard to tell the
difference of reconstructed images compressed by different Daubechies orthogonal
wavelets. Birorthogonal spline wavelets achieve the best subjective quality
because those reconstructed images are smoother than the others.
297
Table 6. Comparison of subjective quality
Wavelet Average grade

Haar 3.2
D4 5.5
D6 5.4
D8 5.4
D 10 5.6
SP 2_2 5.9
SP 2_4 6.0
SP 4_4 6.2
Table 7. Comparison of compression results by using different quantization methods
Wavelet Quantization Compression Compressed PSNR

Ratio File size
Haar UQ 9.177 % 18048 bytes 35.230
dB
EZW 8.788 % 17282 bytes 35.710
dB
D4 UQ 8.613 % 16938 bytes 36.515
dB
EZW 8.443 % 16605 bytes 36.433
dB
D6 UQ 8.469 % 16709 bytes 35.246
dB
EZW 8.058 % 15874 bytes 36.580
dB
D8 UQ 8.426 % 16570 bytes 36.946
dB
EZW 8.396 % 16511 bytes 37.140
dB
D 10 UQ 8.336 % 16393 bytes 35.643
dB
EZW 7.942 % 15619 bytes 36.381
dB
SP 2_2 UQ 14.550 % 28615 bytes 40.650
dB
EZW 14.510 % 28536 bytes 41.190
dB
SP 4_4 UQ 12.917 % 25403 bytes 40.710
dB
EZW 12.756 % 25086 bytes 41.271
dB
4.3 Quantization Methods
Originally MinImage used Uniform Quantization to code the transformed wavelet

coefficients [5]. EZW coding algorithm is adopted to achieve better compression
result.
Table 7 illustrates the compression results for these two different methods.
From this table, we can verify that EZW coding obtains better objective
compression results than Uniform Quantization method for each of the wavelet
types.
4.4 JPEG Vs. MinImage
JPEG stands for Joint Photographic Experts Group, the original name of the
committee that wrote the standard. JPEG is a standardized image compression
standard which uses the DCT (Discrete Cosine Transform). As we discussed
above, wavelet based compression method should obtain better compression ratios
than the DCT based compression method. The following test confirms this result.
The standard 512*512 grayscale Lenna image (see Figure 11) is used for this
test. The MinImage files on the right side of Figure 10 are compressed by using
the SP4_4 wavelet and the EZW coding. The left side images of Figure 10 are
compressed by using the JPEG standard algorithm called CJPEG, version 6 (2-
Aug-95) by Thomas G. Lane (Copyright (C) 1995) [7]. Table 8 lists results for
four different compression ratios.
Table 8. Comparison of compression results between JPEG and MinImage
Compression Compression Ratio Compressed File PSNR

Method size
JPEG 1.60 % 4213 bytes 21.93
dB
MinImage 1.19 % 3129 bytes 28.20
dB
JPEG 3.07 % 8078 bytes 30.40
dB
MinImage 2.97 % 7812 bytes 32.05
dB
JPEG 6.36 % 16752 bytes 34.74
dB
MinImage 6.06 % 15944 bytes 35.24
dB
JPEG 12.40% 32640 bytes 37.81
dB
MinImage 12.39 % 32618 bytes 38.47
dB
299
CJPEG MinImage
Compressed File Size:4213 bytes Compressed File Size:3219 bytes
PSNR:21.93dB PSNR:28.20dB
(a) (b)
CJPEG MinImage
Compressed File Size:8078 bytes, PSNR:30.40dB Compressed File
Size:7812 bytes, PSNR:32.05dB
(c) (d)
Fig. 10. Comparison of JPEG and MinImage
From the Table 8, we can see that MinImage has better objective (PSNR)
quality for all different compression ratios. Since JPEG applies DCT on 8*8
blocks, the block artifact (see image (a) in Figure 10) can be seen from the
reconstructed image, especially for low quality image.
Fig. 11. Original 512*512 grayscale Lenna
4.5 Timing Analysis
In this section, we show the decomposition and composition time comparison (see
Figure 12), EZW coding and decoding time comparison, and EZW coding time
with different compression ratio (see Figure 13). All timing data were collected
from a Dell Pentium III desktop machine running on Windows NT 4.0.
Comparison of Decomposition and Composition

Time
1.2
1
Time (Secs)
0.8
Decomposition
0.6
Composition
0.4
0.2
0
a
8
4
10
4
H
D
D
4_
2_
2_
D
Sp
Sp
Sp
Wavelet Type
Fig. 12. Comparison of Decomposition and Composition Time

301
Since the wavelet decomposition and composition times are entirely determined
by the filter length, Haar wavelet (2 taps) takes the least time and Daubechies 10-
tap orthogonal wavelet takes the most time.
Comparison of EZW Coding and Decoding Time

(SP4_4)
100
80
Time (Secs)
60 EZW Coding Time

40 EZW Decoding Time
20
0
0.75 1.5 3 6 12
Compression Ratio (%)
Fig. 13. Comparison of EZW Coding and Decoding Time
Figure 13 shows that the EZW coding and decoding take about the same time
for same the compression ratio. However, when the compression ratio increases,
the EZW coding and decoding time will increase significantly.
To explain this, let us recall the EZW coding algorithm. When the stop
threshold is high, for every main loop, the transformed wavelet coefficients
maintain a good zerotree structure, which makes EZW coding algorithm efficient.
On the other hand, with a lower stop threshold, a lot of time will be wasted to
check the potential zerotree structures, which are not most of the time for a small
threshold.
Since the EZW coding takes so much time, especially for higher compression
ratios, the total compression time will be dominated by the compression ratio.
From Figure 14 we can see that there is not much difference in the EZW coding
time between different wavelet types for a same compression ratio.
5 Conclusion and Final Remarks
This paper introduced some orthogonal and biorthogonal wavelet filters and
introduced the EZW coding method for image compression. A software tool called
MinImage is used to compare different wavelet methods.
The Haar wavelet transform is the simplest one to implement, and it is the
fastest. However, because of its discontinuity, it is not optimal to simulate a
continuous signal. Based on our experiments, Haar wavelet obtained the worst
compression result, which proves the above statement.
Comparison of EZW Coding Time
100
Coding Time (Secs) 90
80 0.75%
70
60 1.50%
50 3.00%
40 6%
30
12%
20
10
0
a
D8
4
4
2
_4
10
H
D
D
4_
2_
D
_2
Sp
Sp
Sp
Wavelet Type
Fig. 14. Comparison of EZW Coding time with different wavelets and different
compression ratios
Daubechies found the first continuous orthogonal compactly supported wavelet.

Note that this family of wavelets is not symmetric. The advantage of symmetry is
that the corresponding wavelet transform can be implemented using a mirror
boundary condition, which reduces boundary artifacts.
It was proved that no wavelet could be compactly supported, symmetric,
continuous and orthogonal simultaneously. To get the symmetry property, we
introduced biorthogonal spline wavelet by relaxing the orthogonal condition. For
the biorthogonal spline wavelet, the scaling function is a B-spline. The B-spline of
degree N is the shortest possible scaling function of order N-1 and B-splines are
the smoothest scaling functions for a filter of a given length. Our experiment also
proved that birothogonal spline wavelet outperformed the Daubechies orthogonal
wavelet.
Compared with the current JPEG image compression standard, MinImage
obtains better compression results. However, the run time of MinImage increases
significantly to achieve a high quality compression ratio because of the EZW
coding algorithm.
As an application, MinImage gives the user an easier way to do a variety of
tests about wavelet image compressions. It could be improved in several ways.
We end the paper with the following remarks.
1. To get the symmetric wavelets, one solution is to relax the orthogonal
condition. We derived the biorthogonal spline wavelet, which gives better
compression results than orthogonal wavelets for symmetric images.
Another solution is using the called multiwavlets [11]. The basic idea is
to use a vector of scaling functions to generate multiwavelets with
orthogonality, symmetry, continuity, and compact support properties at
the same time. In [6], some properties of multiwavelets with multiplicity
303
four are discussed. MinImage could be more powerful if multiwavelets

transform methods are added.
2. The EZW coding method [13] is one of the more efficient quantization
and coding methods for wavelet coefficients. However, MinImage could
be more powerful if some more efficient quantization methods, such as
space-frequency quantization [14] or context-based entropy coding [2],
can be implemented.
3. The entropy coding library, Zlib, is imported to MinImage. Since it uses
traditional coding methods, it may not be the best coding method for
EZW coded wavelet coefficients.
4. A basic introduction on image compression from discrete cosine
transform (DCT) to wavelets can be found in [9].
References
1. Burrus CS, Gopinath RA, Guo H (1998) Introduction to Wavelets and Wavelet
Transfroms. Prentice Hall, Englewood Cliffs, NJ.
2. Chrysafis C, Ortega A (1997) Efficient context-based entropy coding for lossy
wavelet image compression. DCC, Data Compression Conference, Snowbird,
UT, March, pp. 25 - 27.
3. Daubechies I (1992) Ten Lectures on Wavelets. SIAM, Philadelphia,
Pennsylvania.
4. Dunn S (1999) Digital Color. http://davis.wpi.edu/~matt/courses/color.
5. Gu H, Hong D, Barrett M (2003) Image Compression Using the Haar Wavelet
Transform. J Computational Analysis and Applications 5: 45-75.
6. Hong D, Wu A (2001) Orthogonal Multivavelets of Multiplicity Four, Computer
and Mathematics with Applications 20:1153-1169.
7. McCrosky C (1999) Demo of wavelet compression and JPEG.
http://www.cs.usask.ca/faculty/mccrosky/dwt-demo/index.html.
8. Morlet J (1983) Sampling theory and wave propagation. NATO ASI Series, Vol.
1, Issues in Acoustic signal/Image processing and recognition, Berlin, pp. 233-
261.
9. Saha S (2000) Image Compression - from DCT to Wavelets.
http://www.acm.org/crossroads/xrds6-3/sahaimgcoding.html.
10. Shapiro JM (1993) Embedded Image Coding Using Zerotrees of Wavelet
Coefficients. IEEE Transactions on Signal Processing 41:3445-3462.
11. Strela V (1996) Multiwavelets: Theory and Application. Ph.D. thesis,
Massatchusets Institute of Technology.
12. Unser M (1999) Splines, A Perfect Fit For Signal/Image Processing. IEEE Signal
Processing Magazine 16:22-38.
13. Valens C (1999) EZW Coding. http://perso.wanadoo.fr/polyvalens/clemens/ezw.
14. Xiong Z, Ramchandran K, Orchard MT (1997) Wavelet packet image coding
using space-frequency quantization. IEEE Trans. Image Processing 7:892-898.
Adaptive Neuro-Fuzzy Control of Systems
with Unknown Time Delay
F.H. Ho, A.B Rad, Y.K. Wong, and W.L. Lo
Department of Electrical Engineering, The Hong Kong Polytechnic University,

Hung Hom, Kowloon, Hong Kong SAR, China.
eeabrad@polyu.edu.hk
Abstract. We present an adaptive fuzzy logic controller, which learns a lower-

order model of the system via an on-line Neural Network (NN) identification algo-
rithm. The identification is based on the estimation of parameters of a First-Order-
Plus-Dead-Time (FOPDT) model. The outputs of the NN are three parameters:
gain, apparent time delay and the dominant time constant. By combining this algo-
rithm with a fuzzy logic controller with rotating rule-table, an adaptive controller
is obtained which -with very little a priori knowledge- can compensate systems
with unknown time delay. The simplicity and feasibility of the scheme for real-
time control provides a new approach for a variety of real-time applications.
Simulation and experimental results are included to demonstrate the adaptive
property of the proposed scheme.
1 Introduction
Identification and control of time delay systems have attracted a lot of attention
due to their theoretical and practical significance [7]. Many processes exhibit de-
lay characteristics in their mathematical formulation. Also, systems that are de-
scribed by rational transfer functions are generally represented by lower-order
transfer functions for the purpose of controller design [13]. Model-based control
literature is abundant of studies regarding their modeling, identification and con-
trol [11]. Well-known solutions to time delay compensation are control method-
ologies such as the Smith predictor [16] and Internal Model Control (IMC) [6]. In
contrast, one may notice that despite their importance, Soft-Computing [20] re-
search has not paid much attention to such systems. This may be due to the fact
that the soft-computing methodologies are essentially model-free approaches and
the knowledge of the time delay is only implicitly required. This argument is only
valid if the system under control contains a time delay much less than the domi-
nant time constant of the system. However, for effective control of systems for
which the ratio of time delay to the time constant is equal to or greater than unity,
and in particular the dominant time delay systems, explicit knowledge of the delay
is required. In such instances, the control strategy should be able to predict the
305
time delay and hence adjust the control signal accordingly. Neural networks archi-
tecture using the Nonlinear Auto-Regressive with eXogenous (NARX) model has
been suggested to identify systems with time delay [23]. Another approach using a
PID Neural Network has also been suggested in [18]. The problem is also ad-
dressed by fuzzy neural network [3] and fuzzy control structure [10]. However,
these methods, except [23], require the actual knowledge of the system time delay.
In this paper, we present a novel adaptive controller for systems with unknown
time delay. As with standard adaptive controllers, an on-line parameter estimation
algorithm is integrated with a control design methodology [1]. In the proposed de-
sign, an algorithm via neural networks estimates the parameters of a First Order
Plus Dead Time (FOPDT) model of a higher-order system from input/output data.
Based on the information obtained from this model, in particular, the estimated
time delay- the rule-table of a fuzzy controller will be adjusted on-line.
The rest of this paper is organized as follow: Section 2 is devoted to the idea of
approximating a high-order system with a FOPDT model using neural networks.
In Section 3, two adaptive designs based on the PID and a novel Fuzzy Logic
Controller (FLC) are derived. Simulations of time responses for different systems
and the two controllers are given in Section 4. Stability analysis is performed in
Section 5.Experimental results will be included in Section 6. Finally, the paper is
concluded in Section 7.
2 Lower Order Approximation of Systems

with Neural Networks
The study will be restricted to single-input / single-output (SISO) systems. It is

well known that a FOPDT model, with the transfer function given by Eq. 1, can
approximate many higher order systems [13]:
Y (s) b s m + b1 s m −1 + ... + b m − Ts K ⋅ e −θ ⋅ s
= 0 n e ≈ (1)
U (s) s + a1 s n −1
+ ... + a n τs + 1
Here, K is the system gain, τ is the dominant time constant, θ is the apparent
dead-time and m≤ n. Y(s) and U(s) are the Laplace transformed output and input
signals respectively.
The parameters the parameters K, τ and θ can be determined by various meth-

ods [15,17,19]. Smith's method [15] is a popular approach whereby the parameters
are determined from the process reaction curve by injecting a step signal to the
system. Although, this method is convenient and accurate, however, it should be
carried out off-line and it is very sensitive to noise. Other methods also suffer
from the same drawback and hence cannot be used in the design of an adaptive
controller since their parameters cannot be updated on-line.
306306F.H. Ho et al.
2.1 Online Approximating Approach
The proposed approach is conceptually simple and is inspired by the process reac-
tion curve method. The block diagram of the on-line approximating approach is
shown in Figure 1. As shown in this figure, the on-line identification consists of a
standard neural network and a module called the model generator. The control
signal u(t) is applied to the high-order system, the neural network, and the FOPDT
model generator at the same time. The outputs of the neural network are the three
FOPDT parameters, namely, the gain ( K ), the time constant (τ ) and the time de-
lay (θ ). These three parameters are fed to the first-order plus dead time model
generator to get the output of the model. The error between the output of the plant
and the output of the model is used to train the weights of the neural networks.
The training process tends to force the output of the FOPDT model generator to
approximate the output of the system. Thus, the inputs of the FOPDT model gen-
erator are the approximating parameters of the first-order model for the high-order
system. The output of the FOPDT model is expected to match the output of the
high-order system after the neural network converges.
The transfer function of the FOPDT model generator is rewritten below:

Ym (s) K ⋅ e −θ ⋅ s
= (2)
U (s) τ
s +1
According to the convolution theorem, the output of the FOPDT model genera-
tor can be obtained in the time domain as:
K ⋅ e− θ ⋅ s (3)
y (t ) = L- 1[ ⋅ U ( s)]
m τs + 1
where L-1 stands for inverse Laplace transform. Therefore, we get:
dy (t )
y (t ) = K ⋅ u (t − θ ) − τ m (4)
m dt
where, u(x) is the input of the FOPDT model generator. The model output ym(t)
depends on the parameters K, τ and θ at each time instant.
The neural network architecture used for plant modeling is a three-layer feed-
forward network. The basic structure, having one hidden layer with sigmoid func-
tion, is known to be powerful enough to produce an arbitrary mapping among
variables [4]. Thus, a three layer network is usually used for control applications.
The activation function used here is the standard sigmoid function with range be-
tween 0 and 1.
To train the above neural network, a direct learning strategy is employed for
on-line training. As the desired output of the neural network is unknown, the
FOPDT model generator is considered as an additional but not modifiable layer of
the neural network. The Back-Propagation (BP) algorithm [5] is used to update the
weights of the neural network. The algorithm consists of two passes, forward pass
and backward pass. The calculation of the forward pass and updating the connec-
tion weights from the input layer to hidden layer are the same as those in the stan-
dard BP algorithm. To up date the connection weights from the hidden layer to the
307
output layer, the momentum technique [5] is employed. The weight adjustment for
each iteration is derived below. The error function E is define by:
1 r
E = $ ( y − ym )2 (5)
2 t =1
u y
System
error &
K '
Num
K
Neural τ ÷
Den τ
Network θ
⋅
ym ym
θ
" #
$ %
delay X +
'
Num
FOPDT Model ÷Den
Generator
Fig. 1. The architecture of the on-line lower-order modeling of high-order systems
where r is the number of input/output pairs available for training the network, y
and ym are the output of the plant and the output at the FOPDT model at any time
instant t.
Within each time interval from t to t+1, the BP algorithm is used to update the
connection weights according to the following relationship:
∂E (6)
W ( t + 1) = W (t ) − η ⋅ + α ⋅ ∆W (t )
ij ij ∂W (t ) ij
ij
where η and α are the learning rate and the momentum factor respectively. ∆Wij
is the amount of the previous weight change. Using the chain rule, one has
∂E ∂E ∂y (t ) ∂φ (t )
= ⋅ m ⋅ n
∂W (t ) ∂y (t ) ∂φ (t ) ∂W (t )
ij m n ij
= − ( y (t ) − y m ( t )) ⋅
∂ y m (t )
⋅ X i ( t ) ⋅ (1 − X i (t )) ⋅ X j ( t )
(7)
∂φ n ( t )
where Xi is the output of the ith node of the output layer; Xj is the input vector
of the nodes of the jth output layer; Yn(t) is a 3x1 input vector of the FOPDT model
(the output vector of the neural network):
T
∂y m ( t ) ( ∂y m ( t ) ∂y m ( t ) ∂y m ( t ) / (8)
= , ,
∂φ n (t ) )* ∂K ∂τ ∂θ 01
To find the partial derivatives of the output ym(t) of the model generator
(FOPDT) w.r.t. gain (K), dominant time constant (T) and apparent dead-time (τ),
respectively.
∂ y m (t ) e −θ ⋅s (9)
= L -1[ u ( s )]
∂K τ ⋅ s +1
∂y m (t ) − sKe −θ ⋅s (10)
= L -1[ u ( s )]
∂τ (τs + 1) 2
∂ y m (t ) − sKe − θ S (11)
= L -1 [ u ( s )]
∂θ τs + 1
3 Design of the Adaptive Controller
Three designs are derived and their performances are compared with each other.
The first one is based on the PID control algorithm. The second and third are im-
plementations of Fuzzy Logic Controllers (FLC). It should be noted that the iden-
tification part is the same in all these algorithms.
3.1 On Line PID Tuning Control Method Using NN
The control structure for the on-line PID tuning is shown in Figure 2. There are
two parts in the control structure of the on line PID tuning method. The first part,
which was described in the previous section, is the approximation of high order
systems with FOPDT using neural networks, and the second part is the design of
the PID controller. The parameters of the PID controller can be obtained from the
corresponding parameters of the estimated FOPDT by neural networks. We have
used the Ziegler-Nichols ultimate cycle tuning method [24] to compute the pa-
rameters of the PID controller:
Kp = 0.6Ku Ti = 0.5Tu Td = 0.125Tu (12)
Here, Kp, Ti, Td, Ku and Tu are the proportional gain, integral time constant, de-
rivative time constant, the ultimate gain and the ultimate period respectively. The
ultimate gain and the ultimate period are calculated from the FOPDT model of the
high order plant [22].
The output of the PID controller is in the form of
1 dy f (12)
u (t ) = K P ( e (t ) + + e ( t ) dt − T d ⋅ )
Ti dt
309
1 (13)
e ( t ) = r ( t ) − y ( t ), y (s) = y(s)
f
T
1+ d s
10
where u(t), y(t), r(t), yf(t) are the controller output, process output, set-point,
and filtered derivative respectively.
-
PID u
Intput Controller PROCESS Output
r y
y
+
adjust Kp,Ti & Td
e
u -
Tuning
Algorithm K
Based on Neural Τ Model ym
First Order Plus Network τ Generator
Dead-time Model
error for training
Fig. 2. Overall structure of the auto-tuning PID controller
3.2 Adaptive Fuzzy Logic Controller (AFLC) for Time Delay Systems
Tanaka and Sano [9] suggested that rotating the rule-base of a fuzzy controller, in
effect, reduces the number of fuzzy rules and has the property of time delay com-
pensation. Here, we have extended this concept to design an adaptive fuzzy con-
troller, which has the capability of time delay compensation. The complete algo-
rithm is included in the appendix. The control structure for the adaptive FLC is
shown in Figure 3. This structure is the same as the PID controller except that it is
replaced by a fuzzy controller. The control structure consists of two parts: the first
part is the approximation of the high-order system with FOPDT model using neu-
ral networks, and the second part is the design of the FLC.
The scaling factors of the FLC are also determined on-line based on gain, phase
margin and FOPDT model [21]. The parameters of the controller are in the form
of
ρ (14)
G = e
r
(15)
1
Gde = Ge
% 4ω 2θ ,
& 2ω p − p + 1 -∆t
& π τ -
' .
ω pτ (16)
Gu = ( 4 − 2α )
Am KG de
α ≅ max(Ge | e |, G de | ∆ |) (17)
-
Fuzzy u
Intput Controller PROCESS Output
r y
y
+
e
u -
Rotation K
of the Neural Τ Model ym
Rule-base via Network τ Generator
Dead-time Model
error for training
Fig. 3. Overall structure of the on line tuning parameters FLC
where θ is the dead time, τ is the time constant and K is the process gain of the
FOPDT model. r is the set-point change range. Ge , Gde and Gu are the scaling fac-
tors for error, change of error and the control signal respectively. The parameter ρ
is a coefficient for the criterion of the Integral Time Absolute Error (ITAE). In this
paper, ρ is assigned to 0.2, which gives the system possible compromise between
ITAE index to set-point change and load disturbance. The ωp is the phase cross-
over frequency and is computed from:
π
Amφm + Am ( Am − 1)
ωp = 2 (18)
( Am2 − 1)τ
where Am and φm are gain and phase margin respectively. The gain and phase
margin pair used in this paper is (3,450) because this pair of gain and phase mar-
gins gives good response for both set-point change and load disturbance [21]. Two
adaptive fuzzy controllers are implemented. The difference between the two is
how the scaling factors are set. In the first algorithm (AFLC1), the scaling factors
are set manually during the initialization. In the second algorithm (AFLC2), the
scaling factors are set on-line. The implementation of the two adaptive FLC is as
follows:
1. Estimate the First-Order-Plus-Dead-Time (FOPDT) model by NN.

2. Calculate the error and change of error between set-point and the system
output.
311
3. Calculate the input and output-scaling factor from equation (14-16) [only
for AFLC2*]
4. Calculate the Φ0 from equation (A4).
5. Calculate the time delay compensation parameter ∆Φ from equation
(A5).
6. Estimate the final angle of rotation of control rule from equation (A3).
7. Calculate z and dz from the coordinate transformation equation (A2).
8. Convert v and dv to fuzzy from.
9. Evaluate fuzzy control value.
10. Compute the crisp value by defuzzification.
11. Deterimine the process output by the control value.
4 Simulation Results
To show the adaptive behavior of the algorithm, let us consider two processes as:
Process I
Y ( s ) 1.5 e −2.5s
=
U (s) (1 + s) 2
Y (s) 1 − 1 .4 s
Process II =
U (s) (1 + s ) 3
The first process is a time delay system and the second process is a non-
minimum phase system. First, adaptive control of Process I was simulated for
t=150s after which the system was changed from Process I to Process II. Subse-
quently, the system was switched back to process I. We simulated three control-
lers: Adaptive PID (APID), Adaptive Fuzzy Controller with off-line setting of
scaling factors (AFLC1) and Adaptive Fuzzy Controller with on-line setting of the
scaling factors (AFLC2). In the simulation, the set-point was selected to be a
square wave with amplitude 0.5 and a period of 80s. Gaussian noise with mean
zero and variance of 0.01 was injected at the output of the system. We employed a
fourth order Runge Kutta numerical integration algorithm for all time responses
and the integration interval was selected to be 0.1s. The neural network also used
the same time interval for updating its parameters. The simulation proceeded as
follows: the PID controller was initialized with Kp = 1, Ti =1000, Td = 0.0. The
architecture of the NN was (1,8,3) and the fuzzy controller was using fixed scale
factor with Ge=0.43, Gde=0.6, Gu=0.6. The adaptive fuzzy controller was initial-
ized with Ge=0.0, Gde=0.0, Gu=0.0. The architecture of the NN was (1,16,3). For
all controllers a bias term of 0.5 was added to all hidden nodes, the number of hid-
den units was determined by a trial and error procedure. The connection weights
were randomly initialized. The learning rate and the momentum parameter were
set at 0.8 and 0.1 respectively. Figures 4-6 show the overall performances of the
three controllers. In these figures, the set point and the output, the controller signal
and the estimated parameters of gain, apparent time delay and the dominant time
constant are shown in top, middle and bottom curves respectively. It should be
noted that the gains in Process I and Process II are different (1.5 and 1), since
some adaptive controllers cannot cope with change in steady state gain of the con-
trolled system. However, the proposed method can successfully track the dynamic
change. These simulation studies demonstrate the adaptive property of the pro-
posed algorithm. In all these system changes, the neural networks converged and
the estimated parameters of the FOPDT also converged to their steady state val-
ues. All controllers are known to provide stable and robust control under various
conditions. Tables 1 and 2 show the parameters of FOPDT model approximated
by several other methods such as Smith's [14], minimized error [8] and Tsang and
Rad [19]. The corresponding ultimate gain and the ultimate period for processes I
and II respectively. It should be noted that the parameters from all other methods
except the proposed one were obtained off-line, from open loop excitation with
unit step and were noise free. Furthermore, the values quoted for the proposed al-
gorithm are based on the last measurement before each system change and not the
average value. Fig. 4: Simulation results of the PID controller
Table 1. Parameters obtained from different methods for Process I

Process I
K T τ Ku Tu
Smith Method [18] 1.50 1.65 3.00 1.06 8.38
Minimized-error [8] 1.50 1.46 3.11 0.98 8.44
Tsang-Rad [19] 1.50 1.50 3.09 1.00 8.45
Proposed method for PID 1.49 1.40 3.14 0.97 8.44
Proposed method for FLC 1.51 1.39 3.09 0.96 8.32
Process I - - - 1.04 8.44
Table 2. Parameters obtained from different methods for Process II

Process II
K T τ Ku Tu
Smith Method [15] 1.00 1.89 2.43 1.93 7.22
Minimized-error [8] 1.00 1.67 2.55 1.74 7.35
Tsang-Rad [19] 1.00 1.78 2.44 1.85 8.45
Proposed method for PID 0.99 1.28 2.72 1.49 7.38
Proposed method for FLC 1.05 1.25 2.71 1.40 7.33
Process II - - - 1.54 6.83
313
Set-Point and Process O utput

1
0.9
0.8
0.7 O utput
Set-point
0.6
0.5
0.4
0.3
0.2
0.1
0 t/s
0 50 100 150 200 250 300 350 400
C o n tro l S ig n a l
1
0 .9
0 .8
0 .7
0 .6
0 .5
0 .4
0 .3
0 .2
0 .1
0
0 50 100 1 5 0 t/s 2 0 0 250 300 350 400
First Order Model Parameters

4
3.5
θ
3
2.5
2 K
1.5
1
τ
0.5
0
t/s
0 50 100 150 200 250 300 350 400
Fig. 5a. Simulation results of the AFLC1

Set-Point andProcess Output

1
0.8 Set-point
0.6
0.4
0.2
FOPDT auto-tuning FLC
0
t(sec)
-0.2
0 50 100 150 200 250 300 350 400
Control Signal
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
t/s
0 50 100 150 200 250 300 350 400

4
3.5
3 θ
2.5
2
Κ
1.5
1
0.5 τ
0
0 50 100 150 t/s 200 250 300 350 400
Fig. 5b. Simulation results of the AFLC1

315
Set-Point and Process Output

1
0.9
0.8 Output
0.7 Set-point
0.6
0.5
0.4
0.3
0.2
0.1
0
t/s
0 50 100 150 200 250 300 350 400
Control Signal
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
t/s
0 50 100 150 200 250 300 350 400
F irs t O rd e r M o d e l P a ra m e te rs
4
3 .5
θ
3
2 .5
K
2
1 .5
1
0 .5
τ
0 t/s
0 50 100 150 200 250 300 350 400
Fig.6. Simulation results of the AFLC2
5 Experimental Studies
5.1 Plant Description
A laboratory-scale process control unit (PCU) from Bytronic [2] was used in this
experiment. The system rig consists of a sump, a pump, manual/computer control
diverter valve and drain valve. The sump water is pumped to the pipeline and the
manual flow control valve to the process tank. The water is fed back to the sump
via the drain valve, thus completing the cycle. The rig can be used for level, tem-
perature and flow control. The objective in our study was to control the water flow
by manipulating the pump voltage. Figure 7 shows the schematic diagram of the
process rig.
A first-order with time delay model was identified, off-line as:
y ( s ) 1.5e −0.25 s (16)
G(s) = =
u ( s) 0 .5 s + 1
where y(s) and u(s) are the Laplace transform of the flow rate and pump drive
voltage respectively.
SYSTEM RIG cooler

indicator
stirrer
tank full flow temp
indicator display
switch cooler
level
temp prt flow rate
display display
dirverter
indicator
heat heater
display
manual
manual solenoid
heat
manual diverter diverter
input solenoid
drain gate
drain
drain valve
indicator
priming
valve
sump
sump temp pump
display
Fig. 7. Bytronic Process control unit
5.2 Experimental Results
The purpose of the experiment was to demonstrate the performance of the algo-
rithm on a real system. The plant (Bytronic process control rig) can be represented
by a FOPDT model at nominal operating points. However, the plant is inherently
nonlinear due to the dead-zone of the pump and the flow meter. The set point was
chosen as a staircase signal with amplitude 0.4 l/min, 0.8 l/min and 1.2 l/min and
period of 40 sec. The sample time chosen to be 0.05 sec. The on-line process
computer sampled the water flow rate at every sample interval and the system pa-
rameters were identified by the NN from the pump voltage and the water flow
rate. Due to the dead zone in the motor dynamic, the pump drive voltage minimum
and maximum were 0.8V and 5V respectively. Three controllers were tested:
317
Adaptive PID (APID), Adaptive Fuzzy Controller with off-line setting of scaling
factors (AFLC1) and Adaptive Fuzzy Controller with on-line setting of the scaling
factors (AFLC2). The experiment proceeded as follows: the PID controller was
initialized with Kp = 1, Ti =1000, Td = 0.0 and the fuzzy controller was using fixed
scale factor with Ge=0.43, Gde=0.6, Gu=0.6. The adaptive fuzzy controller was ini-
tialized with Ge=0.0, Gde=0.0, Gu=0.0. The architecture of the NN was (1,8,3). For
all controllers a bias term of 0.5 was added to all hidden nodes. The connection
weights were randomly initialized. The learning rate and the momentum parame-
ter were set at 0.8 and 0.1 respectively. Figures 8-10 show the performance of the
proposed algorithm for these controllers on the real system. It can been seen that
the performance of the ZN-PID control system is not very satisfactory, since the
derivative action amplifies the inherently noisy flow control system. Such a sys-
tem can be controlled better by an adaptive PI controller. It was noticed that both
FLC algorithms perform better than conventional controller for set-point tracking
as well as noise rejection. Besides, although the model of the plant changed at dif-
ferent set-points, the control system maintained a very acceptable performance.
6 Conclusions
In this paper, a new adaptive fuzzy control algorithm is presented. The control al-
gorithm is based on the parameters of a FOPDT model, which are obtained using
neural networks. The time delay compensation is achieved by rotating the rule-
base based on the estimated parameters in particular, the time delay. The perform-
ance of the adaptive FLC is compared with a conventional ZN-PID controller. It is
noticed that the both simulation and experimental results show that the proposed
adaptive control system has superior performance in terms of set-point tracking
and noise rejection. The simplicity of the scheme for real-time control provides a
new approach for implementing neuro-fuzzy applications for a variety of on-line
industrial control problems. Results presented clearly demonstrate the adaptive
property of the proposed method.
Acknowledgements
The authors gratefully acknowledge the financial support of the Hong Kong Poly-
technic University
1.4
1.2
Set-point
1
0.8
0.6
0.4 Output
0.2
0
0 40 80 t/s 120 160 200
C ontroller O u tp ut
1.4
1.2
0.8
0.6
0.4
0.2
0 t/s
0 40 80 120 160 2 00
Firs t Order Model Param eters

2
1.8
1.6 K
1.4
1.2
1 τ
0.8
0.6
0.4
0.2
θ
0
0 40 80 t/s 120 160 200
Fig. 8. Experimental results of the PID controller

319

1.4
1.2
1
0.8
0.6
0.4
0.2
0
t/s
0 40 80 120 160 200
C ontrol S ign al
1.4
1.2
1
0.8
0.6
0.4
0.2
0
t/s
0 40 80 120 1 60 200

2
1.8
1.6 K
1.4
1.2
1
θ
0.8 τ
0.6
0.4
0.2
0
t/s
0 40 80 120 160 200
Fig. 9. Experimental results of the AFLC1


1.4
1.2
1
0.8
0.6
0.4
0.2
0
t/s
0 40 80 120 160 200
C o n tro l S ig n a l
1 .4
1 .2
0 .8
0 .6
0 .4
0 .2
0
t/s
0 40 80 120 160 200
F irs t O rd e r M o d e l P a ra m e te rs
2
1 .8
1 .6
K
1 .4
1 .2
1
0 .8 τ
0 .6
0 .4
0 .2
0
θ
t/s
0 40 80 120 160 200
Fig. 10. Experimental results of the AFLC2
References
[1] Astrom, K.J., Wittenmark, B., Adaptive Control, Addison-Wesley Publishing Company,
Inc., Second Edition, 1995.
321
[2] Bytronic (United Kingdom), User Manual for the Bytronic process control rig (By-
tronic), 1994.
[3] Chen, S.B., Wu, L., and Wang, Q.L., “Self-learning fuzzy neural networks for control of
uncertain systems with time delays,” IEEE Transactions on Systems, Man, and Cyber-
netics, Part B, vol. 27, no.1, pp. 142-148,1997.
[4] Cui, X., Shin, K.G., “Direct control and coordination using neural networks,” IEEE
Transactions on Systems, man, Cybernetics, vol. 23, no.3, pp. 686-697,1993.
[5] Freeman, J.A., Skapura, D.M., Neural Networks: Algorithms, Applications and Pro-
gramming Techniques, Addison-Wesley Publishing Company, Inc, 1991.
[6] Garcia, C.E., “Internal Model Control – 1.A unifying review and some new results,” In-
dustrial and Engineering Chemistry, Process Design and Development, vol. 21, pp.
308-323, 1982.
[7] Gawthrop, P.J., Nihtila, M.T., and Rad, A.B., “Recursive parameter estimation of con-
tinuous-time systems with unknown time delay,” Control–Theory and Advanced
Technology (C-TAT), vol.5, no.3, pp.227-248,1989.
[8] K.R.Sundaresan and P.R. Krishnaswamy, “Estimation of Time Delay Time Constant
Parameters in Time, Frequency and Laplace Domains,” The Chemical Journal of
Chemical Engineering, April, vol. 56, 1978.
[9] K. Tanaka, K., Sano, M., Parameter adjustment laws for fuzzy PI controllers for first-
order lags systems with dead-time. In R. Lowen and M. Roubens.(ED): Fuzzy logic :
state of the art, pp. 531-540, Kluwer Academic Press, 1993
[10] Li, H. X., and Tso, S.K., “Higher-order fuzzy control structure for higher-order or time
delay systems,” IEEE Transactions on Fuzzy Systems, vol. 7, no.5, pp.540-552, 1999.
[11] Marshall, J.E., Time delay systems: Stability and performance criteria with applica-
tions, Ellis Horwood, New York, 1992.
[12] Rad, A.B., “Self-tuning control of systems with unknown time delay – A continuous-
time approach,” Control – Theory and Advanced Technology, Transactions on neural
networks, vol. 1, no.1, March, pp. 479-497,1994.
[13] Rad, A.B., Lo, W.L., and Tsang, K.M., “Self-tuning PID controller using Newton-
Raphson search method,” IEEE Transactions on Industrial Electronics, vol.44, no.5,
pp. 717-725, 1997.
[14] Smith, C.L., Digital Computer Process Control, United States: International Textbook
Company, 1972.
[15] Smith, C. L., Corripio, A.B., Principles and Practice of Automatic Process Control,
John Wiley and Sons, Inc., Second Edition. 1997.
[16] Smith, O.J.M., “Closer control of loops with dead time,” Chemical Engineering,
vol.53, no.5, pp. 217-219,1957.
[17] Sundaresan, K.R., Krishnaswamy, P.R., “Estimation of Time Delay Time Constant
Parameters in Time, Frequency, and Laplace Domains,” The Canadian Journal of
Chemical Engineering, vol. 56, April, pp. 257-262, 1978.
[18] Shu, H.L., Pi, Y.G., “PID neural networks for time delay systems,” Computers and
Chemical Engineering, vol.24, pp.859-862, 2000.
[19] Tsang, K. M. and Rad, A. B., “A new approach to auto-tuning of PID controllers,”
International Journal of Systems Science, vol. 26, no.3, pp.639-658, 1995.
[20] Tzafetas, S.G., Soft-computing in Systems and Control Technology, World Scientific
Publishing, Singapore, 1999.
[21] Xu, J.X., Liu, C., Hang, C.C., “Tuning of fuzzy PI Controllers Based on gain/phase
margin specifications and ITAE index,” ISA Transactions, vol.35, pp.79-91, 1996.
[22] Yu, C.C., Autotuning of PID Controllers, Springer, London, 1999.

[23] Yu, D.L., Gomm, J.B. and Williams, D, “Neural model input selection for a MIMO
chemical process,” Engineering Applications of Artificial Intelligence, vol.13, pp.15-
23, 2000
[24] Ziegler J. G. and Nichols N. B., “Optimum Settings for Automatic controllers” Trans-
actions on ASME, Nov, 759-768, 1942
Appendix A
The controller will be restricted to Single-Input / Single-Output (SISO) sys-

tems. It is well known that a higher order system can be approximated by a first
order plus dead time (FOPDT) model in form of:
Y ( s ) b0 s m + b1 s m −1 + ... + b m − Ts K ⋅ e −θ ⋅s (A1)
= e ≈
U (s) s n + a1 s n −1 + ... + a n τs + 1
Here, K, τ, θ are the gain, dominant time constant and the apparent dead-time
respectively and m≤n. U(s) and Y(s) are the Laplace transformed input and output
signals respectively. T is the system time delay and ai and bj are the coefficients of
the system transfer function.
In this paper, we designed a time delay compensated controller that is specially
suited to FOPDT model. The compensated effect is effectively use linear operator
transformation of system error and error derivative. In fact, there are two effect
can be achieved, First the number of control rules can be reduced. Second, the
fuzzy logic controller can be possible to do the time delay compensation.
Consider a standard fuzzy rule table (A1) in a symmetric form, and initial z and
dz in both diagonal axes. A linear operator A: e→ z that take a vector e and rotates
both the fuzzy rule table and vector z in clockwise direction by an angle Φ. The
matrix of A is equal to Eq. A2.
e
z
NL NM NS ZO PS PM PL
PL ZO PS PS PM PM PL PL
PM NS ZO PS PS PM PM PL
PS NS NS ZO PS PS PM PM
e
ZO NM NS NS ZO PS PS PM
NS NM NM NS NS ZO PS PS
NM NL NM NM NS NS ZO PS Φ (to=45o)
NL NL NL NM NM NS NS ZO
Fig. A1. Illustration of Rotating Table

323
(− sin Φ cos Φ / (A2)

A=) 0, z = Ae
* cos Φ sin Φ 1
.
where e = [e, e ] ∈ ℜ 2 and z = [ z, dz] ∈ ℜ 2
After the initialization of the rule-table, one diagonal of the rule-table gives the
same control action for any dz. In other words, the output of the fuzzy controller
depends only on z, (Table A2) hence the number of rules are reduced.
Table A1. New rotated rule table

dz NL NM NS ZO PS PM PL
Any z NL NM NS ZO PS PM PL
The angle of rotation Φ which is the summation of the two angles Φ0 and ∆Φ
defined as
Φ = Φο + ∆Φ (A3)
where Φ0 and ∆Φ are the angle of rotation base on First Order model and the
angle for time delay compensation respectively.
If the higher order system is modelled by a first-order with delay, three
parameters gan, time constant and time delay are obtained. The angle in equation
4 for angle of rotation base on First Order model Φ 0 is obtained using the time
constant parameter from the first order model without delay case.
kτ
Φο = tan −1 (− 1
) (A4)
k 2τ - 1 - K ⋅ f (e)
where k1 and k2 are the positive constant parameter in order to make the close-
loop reference response stable. In this paper, k1 and k2 are equal to 2. The function
f(e) is the control value and K is the static gain of the process.
Proof: Given in appendix B
The parameter Φ 0 have effect of rotation without consider the time delay. The
other parameter ∆Φ in fact has effect for time delay compensation based on the
first order with time delay model. It is confirmed from simulation results, the first
order plus time delay model parameter τ and θ have influence ∆Φ. e.g ∆Φ=g(τ,θ).
The function g for time delay compensation is approximate by a fuzzy system.
Table 2 show the time delay compensation parameter ∆Φ as a consequent part for
32 pairs τ and θ obtain by computer simulation for quickest response with 20 per-
cent overshoot as a performance index. The i th rule of the fuzzy system is of the
following form:
Rule i:
If τ is Ai1 and θ is Ain Then ∆Φ is ci
where Aij, i=1,2,……,m, j=1,2,……n are the fuzzy sets.
Table A2. Rule table for time delay parameters ∆Φ
θ
∆Φ 1 3 5 7
τ 1 0.00 0.099 0.317 0.42
3 0.351 0.537 0.597 0.632
5 0.466 0.589 0.636 0.666
7 0.505 0.611 0.622 0.683
9 0.526 0.624 0.667 0.695
11 0.539 0.634 0.676 0.704
13 0.549 0.642 0.684 0.711
15 0.557 0.648 0.690 0.717
Given the input data τ and θ, by using a singleton fuzzifier, product fuzzy in-
ference and weighted average defuzzifier, the output of the fuzzy system is in-
ferred as follow:
m n
$c ∏u
i =1
i
j =1
Aij (x j )
(A5)
∆Φ = m n
$∏ u
i =1 j =1
Aij (x j )
The implementation of the FLC is as follows:

1. Initial the two axes z and dz in both diagonals for the rule table.
2. Calculate the error and error derivative between set-point and the system
output.
3. Calculate the Φ 0 from equation (A4).
4. Calculate the time delay compensation parameter from equation (A5).
5. Estimate the final angle or rotation of control rule from equation (A3).
6. Calculate z and dz from the coordinate transformation equation (A2).
7. Convert z and dz to fuzzy from.
8. Evaluate fuzzy control value.
9. Compute the crisp value by defuzzification.
10. Determine the process output by the control value.
Appendix B
Proof of Lemma1: Consider the First Order model without delay

K (B1)
τS + 1
and the closed-loop system governed by
⋅⋅ ⋅
e+ k1 e+ k 2 e = 0 (B2)
where k1 and k2 are positive constant, we have e(t)→0 as t→∝. The close-loop
transfer function
325
Y ( s) KF
=
R( s) τs + 1 + KF
where F=f(e,Φ )
τsY + Y + KFY = KFR, Y = R − e
τs ( R − e) + ( R − e) + KF ( R − e) = KFR
⋅ ⋅⋅ ⋅ .
− τ e+ R − e − KFe = 0 # −τ e − e− KF e = 0
⋅⋅ 1⋅ 1⋅ ⋅
e = − e(1 + KF ) # − e(1 + KF ) = − k1 e − k 2 e
τ τ
⋅
⋅ e kτ
e ( k 2τ − KF − 1) = − k τe, tan Φ = = − 1
1 0 e k τ − 1 − KF
2
Appendix C
.
X m (t ) = AX m (t ) + Bu (t − T ), y m (t ) = CX m (t ) + Du (t − T )
t
X m (t ) = (t − t 0 ) X m (t 0 ) + + (t − λ ) Bu (λ − T )dλ
! !
(C1)
t0
t
y m (t ) = C (t − t 0 ) X m (t 0 ) + C + (t − λ ) Bu (λ − T )dλ + Du (t − T ) ,
! !
t0
(t ) = e At
!
Assuming that all initial states are zeros and D=0, the output equation becomes:
t
ym (t ) = + h(t − λ , p)u (λ − T )dλ
t0
− sT (C2)
e Gm ( s) = e − sT L[h(t,p)]
b0 s m + b1s m −1 + ... + bm B (s)
= e − sT C(sI - A)-1B = e− sT = e− sT m
s n + a1s n −1 + ... + 1 Am ( s)
where θ is the model time delay and h(t, p) =L-1[Gm(s)] is the impulse func-
tion of Gm(s). The vector is defined as p = [a1 a2 …an b0 b1 …bm ]. The partial
derivatives of the model output with respect to the time delay and the model pa-
rameters are as follows.
t
∂ym ∂
=
∂T ∂T + h(t − λ , p)u (λ − T )dλ
t0
t
∂
= + ∂T [h(t − λ , p)u (λ − T )]dλ
t0
t
∂u (λ − T ) (C3)
= + h(t − λ , p) dλ
t0
∂T
t .
= + [−hm (t − λ , p) u (λ − T )]dλ
t0
= − L−1[ se − sT Gm ( s)U ( s)]

t
∂ym ∂
∂ai ∂ai t+0
= hm (t − λ , p)u (λ − T )dλ
(C4)
t t
∂ ∂h(t − λ , p)
= + ∂a [h
t0 i
m (t − λ , p)u (λ − T )]dλ = +
t0
∂ai
u (λ − T )dλ
∞ ∞
∂h ∂h − st ∂
L[ ]= + e dt = + h(t , p)e
− st
dt
∂a i 0 ∂a i ∂ai 0
∂ ∂G m ( s )
= L[h(t , p)] = (C5)
∂a i ∂a i
∂h ∂G ( s )
# = L−1 [ m ]
∂a i ∂a i
∂h ∂G ( s )
Similarly = L−1 [ m ]
∂bi ∂bi
∂y m t ∂h(t − λ , p)
∂a i t+0
= u (λ − T ) dλ
∂a i (C6)
n −i
∂G m ( s) s Bm (s)
= L−1 [e − sT U ( s )] = L−1 [ −e − sT U ( s)]
∂a i [ Am ( s)] 2
∂y m t ∂h(t − λ , p)
∂bi t+0
= u (λ − T )dλ
∂bi (C7)
∂G m ( s) s m −i
= L−1 [e − sT U ( s)] = L−1 [e − sT U ( s)]
∂bi Am ( s)
∂y m B ( s)
= L−1 [− se − sT m U ( s )]
∂T Am ( s) (C8)
The control signal u(t) is filtered by the filter function in eq. (C6-C8) to find the
partial derivatives of ym(t) with respect to various model parameters.
Adaptive Robust Fuzzy Tracking Control
for a Class of Nonlinear Systems
Yansheng Yang1 and Changjiu Zhou2

1
Navigation College of Dalian Maritime University, Dalian, 116026, P.R. China
2
School of Electrical and Electronic Engineering, Singapore polytechnic,
500 Dover Road, Singapore 139651, Republic of Singapore
ysyang@mail.dlptt.ln.cn
Abstract. The tracking control problem for a class of nonlinear systems with un-
certain system function and uncertain gain function, which are the unstructured
state-dependent (or non-repeatable) unknown nonlinear functions, is investigated
in this paper. A fuzzy logic system is used to approximate the lumped non-
repeatable state-dependent uncertain function, then the fuzzy system is used as the
upper bound of uncertainty, and a novel of adaptive robust fuzzy tracking control
(ARFTC), that can evaluate the bound parameter of uncertainty on line, is pre-
sented. For the proposed algorithm, there are two advantages: (1) the adaptive
mechanism with one learning parameterization can be obtained by use of
Lyapunov approach; (2) the possible controller singularity problem in some of the
existing adaptive control schemes met with feedback linearization techniques can
be removed. Finally, the proposed algorithm is verified through simulation.
1 Introduction
In recent years, there have been significant research efforts on the adaptive track-
ing control of nonlinear systems. The control scheme via feedback linearization
for nonlinear systems has been proposed in [1]. The fundamental idea of feedback
linearization is to transform a nonlinear system dynamic into a linear one. There-
fore, linear control or fuzzy logic control (FLC) techniques can be used to acquire
the desired performance [1]-[3].
The FLCs have well proven their broad potential in numerous practical indus-
trial applications and have attracted a great deal of public attention and a lot of
work has been published in the field. Although FLC systems have the advantages
such as no formal mathematical models are needed and the system uncertainties
can be coped with, an analytical understanding of this well demonstrated success
is lacking, and when designing controllers for an uncertain system, if the structure
of the uncertainties in the system is known and the upper bound of uncertainties
can be obtained, so it is understood that there exists a class of continuous-time ro-
bust controllers that insure the convergence of the state to an arbitrarily small
328328Y. Yang and C. Zhou
neighborhood of the origin in a finite period of time [4]. Unfortunately, in indus-

trial control environment, there were some controlled systems which were not
only nonlinear, but also characterized by the unstructured state-dependent (or non-
repeatable) unknown functions. Furthermore, as the design of FLCs involves a
great deal of heuristics, many theoretical questions regarding various analytical
properties of FLCs (e.g. stability, robustness) remain difficult to solve.
A solution to the above-mentioned problems was firstly presented by Wang [5].
The fuzzy systems were proposed to uniformly approximate the uncertain nonlin-
ear functions in the designed system by use of the universal approximation proper-
ties of certain classes of fuzzy systems presented in Lee [6] , Ying [7], Wang and
Mendel [8]. By using Lyapunov based learning law, an adaptive fuzzy control
strategy that ensured the stability of the overall system was developed. However,
the control method described in Wang [3] has two substantial drawbacks: (1) the
method is restricted to plants where the Lie-derivative L∆g Lnf0−1h( x) is constant,
which excludes many systems of practical importance; (2) many parameters need
to be turned in the learning laws when there were many state variables in the de-
signed system and many rule bases were used in the fuzzy system, which was used
to approximate the nonlinear and uncertain systems. In this paper, an adaptive ro-
bust fuzzy tracking control (ARFTC) scheme, that can evaluate the bound parame-
ters of the lumped non-repeatable state-dependent nonlinear function on line, is
presented. The main features of the control law proposed can be viewed from two
aspects: (1) the Lie-derivative L∆g Lnf0−1h( x) is not constant which can be consid-
ered in this method, so the prerequisite for understanding of the controlled system
is relaxed; (2) no matter how many rules in the fuzzy system are used, only one
parameter needs to be adapted on-line, so the computation load of the scheme has
been reduced.
This rest of this paper is organized as follows. In Section 2, we will give the
preliminaries for the configuration of MISO fuzzy systems and approximation
theorem. Section 3 presents the problem formulation for the description of a class
of nonlinear systems and tracking control problem. In Section 4, a systematic pro-
cedure for the synthesis of adaptive robust fuzzy tracking control (ARFTC) is de-
veloped. In Section 5, a simulation example is given to illustrate the effectiveness
of the proposed ARFTC for tracking control of an inverted pendulum system.
Concluding remarks are given in Section 6.
2 Preliminaries
The basic configuration of fuzzy logic systems consists of some fuzzy IF-THEN
rules and a fuzzy inference engine. The fuzzy inference engine uses the fuzzy IF-
THEN rules to perform a mapping from an input linguistic vector x to an output
linguistic variable y . In this paper, we consider a fuzzy system to uniformly ap-
proximate a continuous multi-dimensional function y = f ( x ) , where x is input
Adaptive Robust Fuzzy Tracking Control 329
329
vector with r independent x = ( x1 , x2 ,# , xr ) . The domain of xi is θ i = [ ai , bi ] . It

T
follows that the domain of x is

Θ = θ1 × θ 2 × # ×θ r = [ a1 , b1 ] × [ a2 , b2 ] × # × [ ar , br ]
In order to construct a fuzzy system, the interval [ ai , bi ] is divided into N sub-
intervals
ai = C0i < C1i < # < CN −1i < CNi = bi , 1≤ i ≤ r
On each interval θ i (1 ≤ i ≤ r ) , N + 1 continuous input fuzzy sets, denoted by
A ( 0 ≤ j ≤ N ) , are defined to fuzzify xi . The membership function of Aij is de-
i
j
noted by µ ij ( xi ) , which can be represented by triangular, trapezoid, generalized

bell or Gaussian type and so on.
Generally, the fuzzy system can be constructed by the K ( K > 1) fuzzy rules as
follows:
Rl IF x1 is Ahl1 AND x2 is Ahl2 AND …… AND xr is Ahl r
!
THEN y is Bhl1h2 #hr , l = 1, 2,# , K

where Bhl1h2 #hr is a singleton output fuzzy set for the output variable y . The mem-
bership function of Bhl1h2 #hr is 1 only at y = σ l (an arbitrary constant) and 0 at oth-
er position. The product fuzzy inference is employed to evaluate the ANDs in the
fuzzy rules. After being defuzzified by a typical center average defuzzifier, the
output of the fuzzy system is
K
# σ ( µ ( x ) ⋅ µ ( x )# µ ( x ) )
l
l
h1 1
l
h2 2
l
hr r
fˆ ( x, σ ) = l =1
K
= σ Tξ ( x) (1)
# ( µ ( x ) ⋅ µ ( x )# µ ( x ) )
l =1
l
h1 1
l
h2 2
l
hr r
µ hl ( x1 ) ⋅ µ hl ( x2 )# µ hl ( xr )
where ξ l ( x ) = and ξ l ( x ) is called a fuzzy base
1 2 r
K
#(µ
l =1
l
h1 ( x1 ) ⋅ µ ( x2 )# µhl ( xr ) )
l
h2 r
function. When the membership function µ ij ( xi ) in the ξ l ( x ) is denoted by some

type of membership function, ξ l ( x ) is a known continuous function.
When the fuzzy system is employed to approximate the continuous function,
two questions of interest may be asked: (1) whether there exists a fuzzy system to
approximate any nonlinear function to an arbitrary accuracy? (2) how to determine
the parameters in the fuzzy system if such a fuzzy system does exist. For the first
question, the following lemma is proven by Wang [3].
Lemma 2.1 Suppose that the input universal of discourse U is a compact set
in R r . Then, for any given real continuous function f ( x ) on U and arbi-
trary ∀ε > 0 , there exists a fuzzy system in the form of expression (1) such that
sup fˆ ( x, σ ) − f ( x ) ≤ ε (2)
x∈U
Remark 2.1 If using fˆ ( x, σ ) to estimate the function f ( x ) , there will be N n

unknown constants to be adjusted.
3 Problem Formulation
3.1 Descriptions of uncertain nonlinear systems
Consider an uncertain nonlinear dynamic system in the following form

-*ζ& ( t ) = f 0 (ζ ) + ∆f (ζ , w ) + ') g0 (ζ ) + ∆g (ζ , w ) 24 u ( t )
+ (3)
-, y (t ) = h(ζ )
where ζ ∈ R n is the state. u ∈ R is the input control. y ∈ R is the output of the
system. w is the model uncertainty belonging to a compact set, which includes the
parameter and the disturbance uncertainties of the system. f 0 (ζ ) and g 0 (ζ ) are
known functions and belong to smooth vector fields in a neighborhood of the ori-
gin ζ = 0 with f 0 ( 0 ) = 0 and g 0 ( 0 ) ≠ 0 . ∆f (ζ , w ) is the uncertain system func-
tion and ∆g (ζ , w ) is the uncertain control gain function, both of which are con-
tinuous functions depending on the state ζ .
For the purpose of designing the controller, the input-state linearization is
needed to be performed for the system first. The following assumptions are intro-
duced.
Assumption 3.1 The distributions
{ }
Di = span g0 , ad f0 g 0 ,# , ad if0 g 0 , 0 ≤ i ≤ n − 1
are involutive and of constant rank i + 1 in U1 .
Assumption 3.2 Suppose
{ }
Di = span g0 , ad f0 g 0 ,# , ad if0 g 0 , 0 ≤ i ≤ n − 1
uncertain function ∆f (ζ , w ) satisfies strict-triangle condition such that
') ∆f (ζ , w ) , Di 24 ∈ Di 0≤i ≤ n−2
T
Finding a diffeomorphism x = φ (ζ ) = ') h, L f0 h,# , Lnf0−1h 24 according to As-
sumptions 3.1 and 3.2,, then Eq. (3) can be transformed into
331
* x&i = xi +1 + ∆ i 1 ≤ i ≤ n −1
-
+ x&n = L f0 h (φ ( x ) ) + Lg0 L f0 h (φ ( x ) ) u + ∆ n
n −1 n −1 −1
(4)
-
, y = x1
where ∆ i = L∆f Lif−01h (φ −1 ( x ) ) , ∆ n = L∆f Lnf0−1h (φ −1 ( x ) ) + L∆g Lnf0−1h (φ −1 ( x ) ) .
In Eq. (4), ∆ i , i = 1, 2,# , n are the functions relating to the uncertainties of the
system. From Eq. (4), if ∆ i , i = 1, 2,# , n − 1 does not act within channels implicit
in the control input, i.e. unmatched uncertainty hold, it is difficult to design acon-
troller for the unmatched uncertain system. Therefore, the following assumption is
introduced in this paper.
Assumption 3.3 The uncertainties of the system satisfy the matched condition
such that ∆ i = 0, i = 1, 2,# , n − 1 .
According to Assumption 3.3, the uncertain nonlinear system (4) can be re-
written as follows
* x&i = xi +1 1 ≤ i ≤ n −1
-
+ x&n = f 0 ( x ) + ∆f ( x, w ) + ') g 0 ( x ) + ∆g ( x, w ) 24 u (5)
-
, y = x1
where x = [ x1 , x2 ,# , xn ] , f 0 ( x ) = Lnf0 h , ∆f ( x, w ) = L∆f Lnf0−1h , g 0 ( x ) = Lg0 Lnf0−1h ,
T
∆g ( x, w ) = L∆g Lnf0−1h .
Another assumption is introduced as follows.
Assumption 3.4 The uncertain control gain function ∆g ( x, w ) is confined
within a certain range such that
0 < bmin ≤ 1 + g0−1 ( x)∆g ( x, w ) ≤ bmax (6)
where bmin and bmax are the low and upper bound parameters respectively.
The Assumption 3.4 implies that smooth function g 0 ( x) + ∆g ( x, w) is strictly
either positive or negative. In the following, without loss of generality, we shall
assume 1 + g 0−1 ( x)∆g ( x, w ) ≥ bmin ∀x ∈ R n . Assumption 3.4 is reasonable be-
cause g 0 ( x) + ∆g ( x, w) being away from zero is the controllable conditions of sys-
tem (5). It should be emphasized that the low bound bmin is only required for ana-
lytical purposes, its true value is not necessarily known.
3.2 Adaptive fuzzy tracking control problem
In this paper, the control objective is to force y (t ) to track a given reference sig-
nal yd (t ) while keeping the states and control bounded, i.e. the output tracking er-
ror e1 (t ) = y (t ) − yd (t ) should be small. The given reference signal yd (t ) is as-
sumed to be bounded and has bounded derivatives up to the nth order for all t ≥ 0 ,
and yd( n ) (t ) is piecewise continuous.
T
Let ν = )' yd , yd(1) ,# , yd( n −1) 24 such that ν is bounded. Suppose e = x − ν . Eq.
(5) can then be described by
*-e&i = ei +1 , 1 ≤ n ≤ n − 1
+& (7)
-,en = f 0 ( x) + ∆f ( x, w) − yd (t ) + )' g 0 ( x) + ∆g ( x, w ) 42 u (t )
( n)
For Eq. (7), we have the following transformation for the control input
u (t ) = g 0−1 ( x) [ − f 0 ( x) + v (t )] (8)
where v(t ) is a new control input. From observation of Eq. (8), the control sche-
me proposed here consists of two terms: one is − g 0−1 ( x) f 0 ( x) which is used to
cancel the nonlinearity in the system and the other is g 0−1 ( x)v(t ) which is mainly
used to compensate the effect of uncertainty in the system. Substituting Eq. (8)
into Eq. (7), we have
*e&i = ei +1 , 1 ≤ n ≤ n − 1
+& (9)
,en = F ( x, w) − yd (t ) + G ( x, w ) v(t )
(n)
where F ( x, w) = ∆f ( x, w) − (1 + g 0−1 ( x)∆g ( x, w)) f 0 ( x) , G ( x, w) = 1 + E ( x, w) and

E ( x, w) = g0−1 ( x)∆g ( x, w) .
The problem of adaptive fuzzy tracking control design for v(t ) has recently re-
ceived attention with renewed interest (see, e.g. Wang [3], Fischle and Schroder
[9], Su and Stepanenko [10], Yang, Zhou and Jia [11]). If using fuzzy systems to
approximate F ( x, w) and G ( x, w) , then an "integral" control law is necessary,
i.e., a dynamic feedback
χ& = ϖ ( χ , ξ ( z ), e), ϖ (0, 0, 0) = 0 (10)
v(t ) = α r ( χ , ξ ( z ), e), α r (0, 0, 0) = 0 (11)
where ξ ( z ) is the known fuzzy base functions. And χ ∈ R χ , χ ( ' ) and α r ( ' )
n
n
are smooth functions on R χ × R K × R n . An important quality of the control law is
of course the property that the dimension nχ of χ is "as small as possible", and is
in particular not dependent on the dimension of the state. However, all the conven-
tional adaptive fuzzy tracking control laws available in the literature have the
property that the dimension nχ of χ is 2 N n when using the fuzzy system (cf.
Remark 1) to estimate the function F ( x, w) and G ( x, w) . We can state that the
dimension nχ of χ is strong dependent on the dimension n of the state. In such a
way, the learning time will tend to become unacceptable long for systems of
higher order. For the conventional adaptive fuzzy tracking control laws, there is
333
another main difficulty which comes from uncertainty G ( x, w) , which is usually

approximated by the fuzzy system Gˆ ( x, σ ) . Consequently, the esti-
mate Gˆ ( x, σ ) must be away from zero for avoiding a possible singularity problem.
In this paper, we will develop a novel adaptive robust fuzzy tracking controller
which does not require to estimate the unknown function Gˆ ( x, σ ) , and therefore
avoids the possible controller singularity problem usually met in the traditional
adaptive fuzzy tracking control laws. It will result in all the solutions of the
closed-loop system (9), (10) and (11), which are uniformly ultimately bounded.
Furthermore, the tracking error lim e ( t ) = 0 can be achieved. The outstanding fea-
t →∞
ture of the algorithm proposed in this paper is that the dimension nχ of χ is only
one, no matter how many states in the system are investigated and how many rules
in the fuzzy system are used.
4 Design of Adaptive Robust Fuzzy Tracking Controller
To simplify the design procedure of the adaptive robust fuzzy tracking controller
(ARFTC), we define a variable transformation for Eq. (9) as follows
n −1
S (e, t ) = en + k1e1 + # + kn −1en −1 = en + # ki ei (12)
i =1
where ki ’s are chosen in such a way that s n + kn −1 s n −1 + # + k1 = 0 is a stable

polynomial, hence, we get
n −1
S& ( e, t ) = e&n + # ki ei +1 (13)
i =1
Substituting Eq. (9) into Eq.(13), we have
*
-e& = e , 1≤ i ≤ n−2
-- i i +1
+e&n −1 = − k1e1 − # − kn −1en −1 + S (14)

- n −1
- S& = F ( x, w) + ')1 + E ( x, w ) 24 v(t ) − yd( n ) (t ) + # ki ei +1
-, i =1
Let ζ = [ e1 , e2 ,# , en −1 ] and we use kn −1 to multiply Eq. (12), and substituting

T
it into Eq. (14), we get
*-ζ& = Aζ + BS
+& (15)
,- S = F ( x, w) − yd (t ) + ')1 + E ( x, w ) 24 v(t ) + kn −1 S − Cζ
( n)
' 0 1 0 # 0 2 '02
( 0 3 (03 ' kn −1k1 2
0 1 # 0 ( k k −k 3
( 3 ( 3 ( n −1 2 1 )3
where A = ( $ $ $ $ $ 3 , B = ( $ 3 , C T = (( 3.
( 3 ( 3 $
( 0 0 0 # 1 3 (03 ( 2 3
() −k1 −k2 −k3 # −kn −1 34 ()1 34 ()( kn −1 − kn − 2 ) 34
Owing to A is stable, the positive definite solution P = PT of the Lyapunov

equation
AT P + PA + Q = 0 (16)
exists and Q > 0 is chosen by the designer.
Construct an ARFTC as follows
v = vequ + vN (17)
where vequ denotes a certainty equivalent controller and vN denotes a nonlinear
controller. Then we define vequ as follows
vequ = −(kn −1 + k 2) S (18)
where k > 0 is chosen by the designer.
Substituting Eq. (18) into Eq. (15), we can rewrite Eq. (15) as follows
*ζ& = Aζ + BS
-
+& 1 (19)
- S = − kS + F ' ( x, ζ , yd , w ) + (1 + E ( x, w) ) vN
(n)
, 2
where F ' ( x, ζ , yd( n ) , w ) = F ( x, w ) + BT Pζ − Cζ − yd( n ) (t ) .
When the controller is designed based on the feedback linearization technique,
the most commonly used control structure is vN = − F '( x) (1 + E ( x) ) if there are
no uncertain parameters in the system. When there are uncertainties in the system,
( )
the nonlinearities F ' x, ζ , yd( n ) , w and 1 + E ( x, w ) are unknown and many adaptive
control schemes have been developed, in which the unknown func-
tion 1 + E ( x, w ) is usually approximated by a function approximator 1 + Eˆ ( x, w ) .
Consequently, 1 + Eˆ ( x, w ) must be away from zero for avoiding a possible singu-
larity problem. In this paper, we will propose a method to design the adaptive ro-
bust controller without estimating 1 + Eˆ ( x, w ) so that we can not only reduce the
number of parameters needed to be adapted on-line for 1 + Eˆ ( x, w ) and but also
avoid the possible controller singularity problem usually met with feedback lin-
earization design when the adaptive fuzzy control is applied.
For Eq. (19), the uncertainties F ' and E are the bound function in the control
engineering. Hence there exists an unknown low bound for E . If ∆f and ∆g are
335
the non-repeatable functions that can be considered to be continuous, then the un-
certain function F ' is also a continuous function. In order to design control law, a
fuzzy system is employed to approximate the uncertain function F ' .
By using Lemma 3.1, the uncertain function F can be approximated by a fuzzy
system. Then we can obtain the bound function as follows
F '( z ) ≤ σ T ξ ( z ) + ε + D
T
where z = [ x, ζ ] , ξ ( z ) = ')ξ1 ( z ) , ξ 2 ( z ) ,# , ξ K ( z ) 24 is the fuzzy base function
T
which is known, σ is the weight parameters of the fuzzy system. ε is a parameter

for respecting approximating accuracy. yd( n ) (t ) ≤ D . Suppose σ , D and ε are
all unknown, above expression can be modified as follows
−1
bmin F ' ( z ) ≤ θψ ( z ) (20)
where ψ ( z ) = 1 + ξ1 ( z ) + ξ 2 ( z ) + # + ξ K ( z ) is a known function and

θ = max [ε + D, σ 1 ,# , σ K ] .
T
And E is a bound function which holds Assumption 2.4.

The nonlinear controller vN is expressed as follows
(
ˆ θψ ( z ) S
vN = −θψ ( z ) ( (21)
θψ ( z ) S + ς e − at
where ς > 0 and a > 0 will be specified by the designer.
For the parameter θ , let θˆ be the estimate of θ . Consider a constraint region
Ω for approximation parameter θˆ for all t ≥ 0 to be contained inside given con-
θ
straint region Ωθ , the parameter update laws with projection algorithms are nec-
essary. Here, the smooth projection idea proposed by Khalil [12] is used to tackle
this problem. Suppose the region Ωθ is a convex hypercube. That is, consider
( (
{ } { }
Ω 01 = θˆ 0 ≤ θ ≤ c and Ωθ 1 = θˆ 0 ≤ θ ≤ c + δ where all the values c and
δ > 0 are chosen by the designer. A smooth projection algorithm with respect to
θˆ can be obtained as
& -*lφ , if θˆ > c and φ > 0
θˆ = + (22)
-,lφ , otherwise
$ c − θˆ /
where l denotes the adaptive gain, φ = ψ ( z ) S and φ = %1 + 0φ .
% δ 01
&
Theorem 4.1 For the uncertain nonlinear system (3) subject to Assumptions
3.1 to 3.4, suppose that the uncertain function F ' ( z ) is approximated by the fuzzy
system, the bound function in form (20) is available, an adaptive robust fuzzy
tracking control scheme u (t ) is selected in Eqs. (8), (18) and (21) and the bound
parameter θˆ of uncertainty is estimated by Eq. (22), then the system tracking er-
ror converges to zero asymptotically, i.e. lim ( y (t ) − yd (t ) ) = 0 .
t →∞
Proof: Choosing the following Lyapunov function candidate
1 1 1
( 2
)
V ζ , S ,θ% = ζ T Pζ + S 2 + bmin l −1θ% 2
2 2
where θ% = θˆ − θ .
Differentiating V along the trajectory of Eq. (19) and substituting Eq. (18) and
(20), so we get
1 1 % %&
V& = − ζ T Qζ − kS 2 + S ( F '+ (1 + E )v ) + bmin l −1θθ
2 2
1 1 % &%
≤ − ζ T Qζ − kS 2 + bmin l −1θθ
2 2
(
ˆ (X )S ( θψ (X )S
+ S F − (1 + E )θψ
θψ ( X ) S + ς e − at
Substituting Eq. (20) into above equation, we get
1
2
1
2 ( &
V& ≤ − ζ T Qζ − kS 2 + bmin l −1θ% lψ ( x ) S − θˆ + bmin ς e− at ) (23)
&
(
Now, from the adaptive law in [12] we get l −1θ% lψ ( x ) S − θˆ ≤ 0 and θˆ ∈ Ωθ 1 )
for all t ≥ 0 if θˆ ( 0 ) ∈ Ω01 . Then Eq. (23) is
1 1
V& ≤ − ζ T Qζ − kS 2 + bminς e − at
2 2
≤ −Vn + bmin ς e − at
1 1
where Vn = ζ T Qζ + kS 2 . Then we get
2 2
t
V ( t ) − V ( 0 ) ≤ . Vn dt + bmin ς (1 − e − at ) ( 2a )
0
Therefore,
0 ≤ . Vn dt ≤ Vn (0) + bmin ς (1 − e − at ) (2a )
t
and
t
lim . Vn dt ≤ Vn (0) + bmin ς (1 − e− at ) (2a) < ∞
x →∞ 0
Since Vn is a uniformly continuous function, according to Barbalat lemma [4],

lim Vn = 0 . This implies that lim ζ = 0 and lim S = 0 based on the definition of
x →∞ t →∞ t →∞
Vn . As results we get lim ei (t ) = 0, i = 1, 2,# , n − 1 and lim S = 0 which implies

t →∞ t →∞
lim ( y (t ) − yd (t ) ) = 0
t →∞
So the desired result has been obtained.
337
5 Design of ARFTC for an Inverted Pendulum System
To demonstrate the effective-

ness of the proposed algorithm,
an inverted pendulum system is θ θ&
used for simulation. The Fig. 1
shows the plant composed of a
pole and a cart. The cart moves
on the rail tracks in horizontal
direction. The control objective
is to balance the pole starting u
from an arbitrary condition by
supplying a suitable force to the
cart. The same case studied has
been given in Hwang [13]. The
dynamic equations are described Fig. 1. An inverted pendulum system
by
*- x&1 = x2
+ (24)
-, x&2 = f ( x, w) + g ( x, w)u (t )
mlx22 cos x1 sin x1 cos x1
g sin x1 −
mc + m mc + m
where f ( x, w) = + w(t ) , g ( x, w) = .
$ 4 m cos 2 x1 / $ 4 m cos 2 x1 /
l% − 0 l% − 0
& 3 mc + m 1 & 3 mc + m 1
mc denotes the mass of the pendulum, m is the mass of the vehicle, 2l is the
length of the pendulum and u is the applied force. x1 = θ is the angular position
from the equilibrium position and x2 = θ& . The trajectory planning problem for a
weight-lifting operation is considered in this paper. The desired angle trajectory is
assumed to be yd (t ) = 0.1sin(2t ) . In this paper, we use following parameters for
simulation: mc = 1 kg , m = 0.1 kg , l = 0.5 m l=0.5. And w(t ) is assumed to be
w(t ) = 0.2sin(2t ) .
Define five fuzzy sets for each x1 , x2 , e1 with labels Ahi1 (NL), Ahi2 (NM), Ahi3 (ZE),
Ahi4 (PM), Ahi5 (PL) which are characterized by the following membership func-
tions
' $ x + π 6 /2 2 ' $ x + π 12 / 2 2
µ A1 = exp ( − % i
0 3 , µ Ahi2 = exp ( − %
i
0 3
hi
() & π 24 1 34 () & π 24 1 34
(25)
' $ x /2 2 ' $ x − π 12 / 2 2
µ A3 = exp ( − % i
0 3 , µ Ahi4 = exp ( − %
i
0 3
hi
() & π 24 1 34 () & π 24 1 34
' $ x − π 6 /2 2
µ A = exp ( − %
5
i
0 3 , i = 1, 2,3
hi
() & π 24 1 34
Let’s set
* f 0 ( x) = 0
- g ( x) = 1
- 0
+ (26)
- ∆f ( x, w) = f ( x1 , x2 , w)
-, ∆g ( x, w) = g ( x1 , x2 , w) − 1
If we get e1 (t ) = x1 (t ) − yd (t ) , e2 (t ) = x2 (t ) − y& d (t ) and k1 = 5 , Q = −2 , then
P = 1 and
vequ = − ( 5 + 0.5k ) S
where S = e2 + 5e1 .
S& = −0.5kS + F ( x, e1 , w) + (1 + E )uN
where F ( x, e1 , w) = f ( x, w) + 5e1 + g ( x, w) − y& d (t ) , E = g ( x, w) .
If g ( x, w ) has a lower bound and the system function F ( x, e1 , w) is a continu-
ous complicated formulation system function, then F is also a continuous func-
tion and a fuzzy system is applied to approximate function F .
Assuming that the fuzzy system is constructed from the following 75 fuzzy IF-
THEN rules
R1 : IF x1 is PL AND x2 is PL AND e1 is PL THEN y is α1
Ri : ....... ........ .........
R75 : IF x1 is NL AND x2 is NL AND e1 is NL THEN y is α 75
The fuzzy system can be obtained as follows
75
F ( x, e1 , w ) = # α iξ i ( x, e1 ) + ε (27)
i =1
So the bound function can be written as

F ( x, e1 ) ≤ θψ ( x, e1 )
Accordingly the adaptive robust fuzzy tracking control scheme can be obtained
for an inverted pendulum system as follows
( ˆ ( x, e ) S
θψ
u (t ) = −(5 + 0.5k ) S − θψ ( x, e1 ) 1
(28)
ˆ ( x, e ) S + 5e−0.01t
θψ 1
where k = 180 and θˆ can be calculated by Eq. (22). And l = 1 . c = 0.2 ,

75
δ = 0.01 . φ = lψ ( x, e1 ) S , ψ ( x, e1 ) = 1 + # ξ i ( x, e1 ) .
i =1
For the simulation, we choose the initial conditions as x1 (0) = 0.1 , x 2 (0 ) = 0 ,

θˆ(0 ) = 0 . The simulation results are shown in Fig. 2 and Fig. 3.
339
Fig. 2. Simulation results for Control algorithm in Eq. (28). (Position of the in-
verted pendulum system. Solid line: actual position, Dashed line: reference posi-
tion)
Fig. 3. Control force
Before presenting the outstanding advantages of ARFTC developed in this pa-

per, we will briefly review the control law proposed in Wang [3] as follows
1
u = uc + u s = − ' f ( x,θ1 ) − &&
yd (t ) + k T e 24
gˆ( x,θ 2 ) )
1 ' ˆ (29)
− I1 sgn(eT PB ) f ( x , θ1 + f U ( x )
g L ( x) )
+ gˆ ( x,θ 2 ) uc + g U ( x)uc 42
where k = [2,1]T , f ( x) ≤ f U ( x) , g L ( x) ≤ g ( x) ≤ g U ( x) , B = [0,1]T and
Q = diag[10,10] . Then, we solve Lyapunov equation and obtain
Fig. 4. Simulation results for the adaptive parameter ˆθ
Fig. 5. Simulation results for tracking error
'15 52
P=( 3
) 5 54
where I 1 = 1 if Ve > V (which is a constant specified by designer), I 1 = 0 if
1 T
Ve ≤ V . And Ve = e Pe .
2
Defining five fuzzy sets the same as those in Eq. (25) for each x1 , x2 , twenty-
five fuzzy rules for the systems fˆ ( x,θ1 ) and gˆ ( x,θ 2 ) respectively, and singleton
fuzzy model, the product inference and the center-average defuzzification are
used. Hence, the function f ( x) and g ( x) can be approximated by the fuzzy sys-
tems fˆ ( x,θ ) = θ T ξ ( x) and gˆ( x,θ ) = θ T ξ ( x) where
1 1 1 2 2 2
341
θ1 = ')θ11 ,θ12 ,# ,θ125 24 ∈ R 25

ξ1 ( x) = ')ξ11 ( x), ξ12 ( x),# , ξ125 24 ∈ R 25
θ 2 = ')θ 21 ,θ 22 ,# ,θ 225 24 ∈ R 25
ξ 2 ( x) = ')ξ 21 ( x), ξ 22 ( x),# , ξ 225 24 ∈ R 25
with components
ξ11 ( x) = µ h11 ( x1 ) µ h1 2 ( x2 ) D1
ξ12 ( x) = µ h11 ( x1 ) µ h22 ( x2 ) D1
ξ13 ( x) = µ h11 ( x1 ) µ h32 ( x2 ) D1 ,...
ξ125 ( x) = µ h51 ( x1 ) µ h52 ( x2 ) D1
and
5 5
D1 = ## µ hi 1 ( x1 ) µ hj2 ( x2 )
i =1 j =1
and the construction of ξ 2 ( x) is similar to ξ1 ( x) .

In Wang [3], the following adaptive law to adjust parameter vector θ1 was pro-
posed:
*−Γ1eT PBξ1 ( x), if ( θ1 < M f ) or ( θ1 = M f
--
θ&1 = + and eT PBθ1T ξ1 ( x) ≥ 0) (30)
-
-, P {−Γ1e PBξ1 ( x)} , if ( θ1 = M f and e PBθ1 ξ1 ( x) < 0)
T T T
where the projection operation P {'} is defined as

θ1θ1T ξ1 ( x)
P {−Γ1eT PBξ1 ( x)} = −Γ1eT PBξ1 ( x) + Γ1eT PB 2
θ1
And in Wang [3], the following adaptive law to adjust parameter vector θ 2 was
used:
-*−Γ 2 e PBξ 2 ( x)uc , if eT PBξ 2 ( x)uc < 0
T
θ&2 = + (31)
-,0, if eT PBξ 2 ( x)uc ≥ 0
Here, for the parameters M f , V , Γ1 and Γ 2 , please refer to Wang [3]. The simu-
lation results are shown in Fig. 6 to Fig. 8.
Fig. 5 shows the simulation results of tracking errors by use of the proposed
ARFTC and Fig. 8 shows the simulation results of tracking errors by use of the
controller given in Eq. (29). From the results, we can see that the control perform-
ance of ARFTC is better than the controller given in Eq. (29). Hence, we can state
that the ARFTC satisfies the following advantages that have been described in
Section 4: only one function f ( x) is needed to be approximated by fuzzy sys-
tems and no matter how many states in the system are investigated and how many
rules in the fuzzy system are used, only two parameters are needed to be adapted
on-line in ARFTC. However, for the traditional methodology (e.g. the control law
proposed in Wang [3], even based on five fuzzy sets for each state variable and
singleton fuzzy model aforementioned, there are 50 parameters needed to be
adapted on-line for the fuzzy system fˆ ( x,θ1 ) and gˆ ( x,θ 2 ) when the fuzzy logic
controller is implemented. And the traditional methodology can cause the increase
of the number of parameters needed to be adapted exponentially with that of num-
ber of state variables or fuzzy sets. The computational complexity can be lessened
dramatically and the learning time can be reduced vastly when using ARFTC de-
veloped in this paper. Then ARFTC has the potential to provide uncomplicated,
reliable and easy-to-understand solutions for a large variety of nonlinear control
tasks even for higher order systems.
Fig. 6. Simulation results for Control algorithm in Eq. (29). (Position of the in-
verted pendulum system: Solid line: actual position, Dashed line: reference posi-
tion)
Fig. 7. Control force

343
Fig. 8. Simulation results for tracking error
6 Conclusions
In this paper, the tracking control problem has been considered for a class of
nonlinear uncertain systems with the unknown system function and unknown gain
function, and fuzzy logic systems have been used to approximate the lumped non-
repeatable state-dependent nonlinear function and the fuzzy system is used as the
upper bound function of the uncertainty, then an adaptive robust fuzzy tracking
control (ARFTC) algorithm, that can guarantee that the system tracking error con-
verges to zero asymptotically, is proposed. The outstanding features of the algo-
rithm proposed in this paper are that it can avoid the possible controller singularity
problem in some of existing adaptive control schemes with feedback linearization
techniques and the adaptive mechanism with minimal learning parameterizations,
e.g. no matter how many states in the system are investigated and how many rules
in the fuzzy system are used, only one parameter is needed to be adapted on-line,
so that the computation load of the algorithm can be reduced and it is convenient
to realize this algorithm in engineering.
Finally, in order to research the efficiency of the algorithm proposed in this pa-
per, it has been applied to an inverted pendulum system. We compare the simula-
tion result by use of the algorithm proposed in this paper with the one by Wang
[3] . Simulation results have shown the effectiveness of the proposed ARFTC con-
trol scheme.
Acknowledgments
This work was supported in part by the Research Fund for the Doctorate Program
of Higher Education under Grant No. 20020151005, the Science Foundation under
Grant No. 95-06-02-22 and the Young Investigator Foundation under Grant No.
95-05-05-31 of the Ministry of Communications of P. R. China.
344 Y. Yang and C. Zhou
References
1. Sastry, S. and Isidori. A., Adaptive control of linearization systems. IEEE Trans. on
Automatic Control. Vol. 34, pp.1123-1131, 1989.
2. Isidori, A., Nonliear Control Systems. 2nd ed. Berlin: Springer-Verlay, 1995.
3. Wang, L. X., A Course in Fuzzy Systems and Control. New York: Prentice-Hall Inc.,
1997.
4. Slotine, J. E., and Li, W., Applied Nonlinear Control. New York: Prentice-Hall, Inc.,
1990.
5. Wang, L. X., Fuzzy systems are universal approximators. In proc. IEEE. International
conf. On Fuzzy systems, San Diego, CA, pp.1163-1170, 1992
6. Lee, C. C., Fuzzy logic in control systems: fuzzy logic controller—Parts I&II. IEEE.
Trans. Syst. Man. Cyber. Vol. 20, pp. 404-435, 1990,
7. Ying, H., Sufficient conditions on general fuzzy systems as function approximators.
Automatica. Vol. 30, No. 4, pp. 521-525, 1994.
8. Wang, L. X. and Mendel, J. M., Fuzzy basis functions universal approximation, and or-
thogonal least-squares learning. IEEE Trans. on Neural Networks. Vol. 3, pp. 807-814,
1992.
9. Fischle K. And Schroder D., An improved stable adaptive fuzzy control method. IEEE
Trans. on Fuzzy Systems. Vol. 7, No. 1, pp. 27-40, 1999.
10. Su, C. Y. And Stepanenko, Y., Adaptive control of a class of nonlinear systems with
fuzzy logic. IEEE Trans. Fuzzy Systems. Vol. 2, pp.285-294, 1994.
11. Yang, Y. S., Zhou, C. and Jia, X. L., Robust adaptive fuzzy control and its application
to ship roll stabilization. Information Sciences. Vol. 142, pp. 177-194, 2002.
12. Khalil, H. K., Adaptive output feedback control of nonlinear systems represented by
input-output models. IEEE Trans. on Automatic Control. Vol. 41, pp. 177-188, 1996.
13. Hwang, G. C. and Lin, S. C., A stability approach to fuzzy control design for nonlin-
ear systems. Fuzzy Sets and Systems. Vol. 40, pp. 279-287, 1990.
A Complete Algorithm for Attribute Reduction
B. Wang1, and S.B. Chen2
1. School of Materials Science & Eng., Shanghai Jiao Tong Univ., Shanghai
200030, P.R. China
2. School of Materials Science & Eng., Shanghai Jiao Tong Univ., Shanghai
200030, P.R. China
sbchen@sjtu.edu.cn
Abstract. Rough sets theory is an effective mathematical tool dealing with

vagueness and uncertainty. It has been applied in a variety of fields such as data
mining, pattern recognition or process control. It is significant to find high
performance algorithms for attribute reduction. Most of presented algorithms were
incomplete and the work on complete algorithms was comparatively little. This
paper presented a complete algorithm based on the principle of discernibility matrix
and discussed the time complexity of the algorithm. The proof of the completeness
of the algorithm was also given in the paper. Finally, the validity of the algorithm
was demonstrated by two classical databases in the UCI repository.
1 Introduction
Rough sets theory[3] introduced by Pawlak in 1982 is an effective mathematical tool

dealing with vagueness and uncertainty. It has been applied in a variety of fields
such as data mining, pattern recognition or process control[4]. Attribute reduction is
significant in most applications, so that it has been a major concern in the research
on rough set theory.
Let S be a decision table and we have S=<U, C, D, V, f>, where U is the uni-
verse, the disjoint attribute sets C and D are referred to as the condition and decision
attributes respectively. It is necessary to point out U, C and D should be non-empty
and finite sets. According to the definition given by Pawlak[3], if P⊆C is a reduction
of C, P satisfies follow two conditions:
(1)POSP(D)=POSC(D);
(2)POSP−{a}(D)≠POSC(D) for each a∈P.
In [3], Pawlak presented the original algorithm for attribute reduction whose
time complexity was O(2n), where n=|C|. So the algorithm is just a theoretical al-
gorithm and it is limited very much in actual applications. The output of the algo-
rithm, which is a subset of C, satisfies both two conditions above. Some attribute
346346B. Wang and S.B. Chen
reduction algorithms with polynomial-time complexity have been introduced, no-

tably those in [1, 2, 5]. These algorithms are more practical but their outputs only
satisfy the first condition above. If the output of an attribute reduction algorithm
satisfies both two conditions above, we say the algorithm is complete. If the output
of an attribute reduction algorithm only satisfies the first condition above, we say
the algorithm is incomplete. What we want are complete algorithms with polyno-
mial-time complexity. In [6], firstly, authors introduced a complete algorithm. It is
needed to define a total relation over the set of condition attributes beforehand and
the order can be considered as prior knowledge. In this way, one can fully use the
experience of experts. At the same time, the output of the algorithm is unique for a
given order of attributes. However, the algorithm still has limitation when no prior
knowledge is available. Secondly, authors presented a optimal reduction algorithm.
It is also complete and its time complexity is not lower than O(m3n2) in the worst
case. To the author's knowledge, there is no other published complete algorithm
with polynomial-time complexity. In this paper, an algorithm based on the principle
of discernibility matrix is introduced. We show that the algorithm is a complete
attribute reduction algorithm with polynomial-time complexity and its time com-
plexity is O(m2n2) in the worst case. The validity of the algorithm was demonstrated
by two classical databases in the UCI repository.
This paper is organized as follows. In Section 2, some concepts and definitions
are introduced. The algorithm and the analysis of the time complexity of it are given
in Section 3. In Section 4, we show that the algorithm is complete. In Section 5, we
give results of experiments and the conclusion is in Section 6.
2 Concepts and Definitions
Let S=<U, C, D, V, f> be a decision table. Suppose |U|=n. The discernibility matrix
of S denoted by MS is a n×n matrix defined thus:
(cij)={ a∈C f (xi a)≠ f (xj a) and w(xi,xj)=1}
!
" "
for i,j=1, 2, ..., n.
where
1 xi∈POSC(D) and xj∉POSC(D)
1 xi∉POSC(D) and xj∈POSC(D)
w(xi,xj) =
1 xi, xj∈POSC(D) and (xi, xj)∉IND(D)
0 otherwise
Table 1. An example of decision tables

U a b c d
x1 2 2 0 1
x2 1 2 0 0
x3 1 2 0 1
x4 0 0 0 0
x5 1 0 1 0
x6 2 0 1 1
347
Table 2. The discernibility matrix

1 2 3 4 5 6
1 ∅ {a} {a} {a ! {a b
! !
∅
b} c}
2 {a} ∅ ∅ {a ! {b c}! {a b
! !
b} c}
3 {a} ∅ ∅ {a ! {b ! c} {a b
! !
b} c}
4 {a b}! {a b}! {a b} !
∅ ∅ {a c}!
5 {a b
! ! {b c} ! {b c} !
∅ ∅ {a}
c}
6 ∅ {a b
! ! {a b
! ! {a ! {a} ∅
c} c} c}
For example, Table 1 is an example of decision talbes and Table 2 is its

discernibility matrix.
It is necessary to give follow definitions before discussing our algorithm in
detail.
Let MS be the discernibility matrix of decision table S. The set of non-empty
entries of MS will be called the discernibility set of S, denoted by $S. Obviously, the
discernibility set of a decision table is the collection of subsets of condition attrib-
utes. We associate a subset:
Core($S)={a: {a}∈ $S }
called the core of $S. Let B⊆C. A S− (B) be the B—elimination set of $S and
A S− (B)={A' A∈$S A'=A−B and A'≠∅ }. A S+ (B) is the B—selection set of $S
!
"
and A S+
(B)={A∈$S A∩B=∅}. It is clear that the B—elimination set and the
!
B—selection set are all subsets of condition attributes.

To illustrate these definitions, we take the discernibility matrix in Table 2 for
example. $S={{a}, {a, b}, {a, b, c}, {b, c}, {a, c}}. Core($S)={a}. Suppose B={a}.
A S− (B)={{b}, {b, c}, {c}}. A S+ (B)={{b, c}}.
3 The Complete Algorithm and Its Time Complexity
In this section, we will introduce a complete attribute reduction algorithm and

analyze its time complexity. The input of the algorithm is the discernibility matrix
$S of a given decision table S.
Algorithm 3.1
(1)R⇐∅, P⇐∅, T⇐∅;
(2)$=$S;
(3)If $=∅, stop, otherwise continue;
(4)P⇐ Core($);
348 B. Wang and S.B. Chen
(5)If P=∅, go to Step 6, otherwise go to Step 7;

(6)Select one attribute a in C–(R $ T) randomly, $⇐ A − ({a}) , T⇐T $ {a},
and go to Step3;
(7) R⇐R $ P, $⇐ A + (R) and go to Step 3.
When the algorithm stop, R is the output of the algorithm, namely the attribute
reduction. Fig. 1 is the flow chart of Algorithm 3.1.
START
R⇐∅ ! P⇐∅ ! T⇐∅ ! $
Y
$=∅ END
!
P⇐Core($) select a ! $⇐ $
−
({a}) !
Y
P=∅
!
R⇐R $ P ! $⇐ $
Fig. 1 The flow chart of Algorithm 3.1
Suppose |U|=n and |C|=m, where m and n are positive integer. In the worst case,
n2 − n
|$|= = N . The time complexity of computing Core($) is O(mN). Let B⊆C.
2
The time complexity of computing A − (B) and A + (B) are also O(mN). Algo-
rithm 3.1 will repeat m times at most so the time complexity of Algorithm 3.1 is
n2 − n
O(m(mN+mN))=O(2m2 )=O(m2n2) at the worst case. It is clear Algorithm 3.1
2
is faster than the optimal reduction algorithm[6] in the worst case.
349
4 The Completeness of the Algorithm
In this section, we show Algorithm 3.1 is complete, namely, the output of the al-
gorithm satisfies the two conditions presented in Section 1.
It has been proved those two conditions are equivalent to follow two condi-
tions[6]:
(1)For ∀ A∈$S, A # P≠∅;
(2)For ∀ a∈P, ∃ A∈$S, such that A # P−{a}=∅.
To prove the completeness of the algorithm, we can prove the output of the
algorithm, that is R, satisfies these two conditions. For this end, it is worth giving
some properties of the algorithm.
Step 3 to Step 7 of the algorithm form a loop. Since U and C are finite sets, it is
obvious the algorithm terminates absolutely. So we can safely suppose the algo-
rithm iterate k times, where constant k is a positive integer. During one iteration, if
Step 6 is executed in the loop, we say the iteration is an elimination iteration, oth-
erwise Step 7 is executed and we say the iteration is a selection iteration. In an
elimination iteration one attribute is added to T(the set of attributes eliminated by
the algorithm) and in an selection iteration some attributes are added to R. We use
the sequence $0, $1, ..., $i, ..., $k−1, $k to indicate the changing of $ during the algo-
rithm running, where i=0, ..., k. The first iteration changes $ from $0 into $1 and the
ith iteration changes $ from $i−1 into $i. Accordingly, the sequence R0, R1, ..., Ri, ...,
Rk−1, Rk indicates the changing of R and the sequence T0, T1, ..., Ti, ..., Tk−1, Tk in-
dicates the changing of T. We have $0=$S, $k=∅, R0=∅, Rk=R(the output of the al-
gorithm), T0=∅, Tk=T, and if i≤j, Ri⊆Rj and Ti⊆Tj. It is necessary to point out that
C−R=T is not always hold.
Let C={a, b, c, d, e, f} and the input $S={{a, b, d}, {a, c, d}, {b, c, d}, {a, b, c},
{e, f}}. The output of the algorithm is R={c, d, f}. Table 3 shows how the algorithm
runs.
Table 3. Running process of the algorithm

i $i Ri Ti
0 {{a, b, d}, { a, c, d}, {b, c, d}, { a, b, c}, {e, f}} ∅ ∅
1 {{b, d}, {c, d}, {b, c, d}, {b, c}, {e, f}} ∅ {a}
2 {{d}, {c, d}, {c}, {e, f}} ∅ {a, b}
3 {{e, f}} {c, d} {a, b}
4 {{f}} {c, d} {a, b, e}
5 ∅ {c, d, f} {a, b, e}
Proposition 1 If the ith iteration is an elimination iteration, ∃ A'∈ $i such that

A'⊆A for ∀ A∈$i−1, where i=1, ..., k.
Proposition 2 If the ith iteration is a selection iteration, A # Ri≠∅ if A∉ $ for

∀ A∈$i−1, where i=1, ..., k.
Proposition 3 A # Ri≠∅ if ¬∃ A'∈ $i such that A'⊆A for ∀ A∈$i−1 and
i=1, ..., k.
Proposition 4 A # Rj≠∅ if ¬∃ A'∈ $j such that A'⊆A for ∀ A∈$i , i < j and i,
j=1, ..., k.
Proposition 1 and Proposition 2 are easy to be proved. By Proposition 1 and
Proposition 2 we can show Proposition 3 is right and then we get Proposition 4.
Theorem 1 Let $S and R be the input and the output of Algorithm 3.1 respec-
tively. A # R≠∅ for ∀ A∈$S.
Proof
Let the algorithm iterate k times, where k∈Z and k≥0. If k=0, the theorem is
right obviously. If k>0, we have A0=AS and Ak=∅. For ∀ A∈AS, A∈A0 holds. Note
that Ak=∅, that is ¬∃ A'∈Ak such that A'⊆A. By Proposition 4, A # Rk≠∅, namely
A # R≠∅. !
Proposition 5 For ∀ A∈$i, ∃ A'∈ $0 such that A⊆A' and A'−A⊆Ti, where
i=0, ..., k.
Proposition 6 T # R=∅.
Both Proposition 5 and 6 are obvious.
Theorem 2 Let $S and R be the input and the output of Algorithm 3.1 respec-
tively. For ∀ a∈R, ∃ A∈$S such that A # R−{a}=∅.
Proof
Let the algorithm iterate k times, where k∈Z and k≥0. If k=0, the theorem is
right obviously. If k>0, we have A0=AS and Ak=∅. For ∀ a∈R, ¬∃ i(0<i≤k), such
that a∉Ri-1, a∈Ri, and {a}∈Ai-1. By Proposition 5, ∃ A∈A0, such that a∈A and
A−{a}⊆Ti−1⊆Tk=T. By Proposition 6, A−{a} # R=∅, namely A # R−{a}=∅. !
By Theorem 1 and Theorem 2, we can, therefore, conclude that Algorithm 3.1 is

complete.
5 Experiments
In this section, we will process two decision tables came from the UCI repository
using the approximate core algorithm.
The first decision table is formed from "1984 United States Congressional
Voting Records Database". It has 435 objects, 16 condition attributes and 1 decision
attributes. The missing values (the mark ? in the data file) in the original data will be
regarded as one new value since we do not concern how to deal with missing values
in this paper. The attribute reduction acquired by our algorithm includes the first,
second, third, forth, ninth, eleventh, thirteenth, fifteenth and sixteenth condition
attribute. The total is 9 and these attributes are listed in Table 4.
351
Table 4. The attribute reduction of the first decision table

No. Attribute information
1 handicapped-infants
2 water-project-cost-sharing
3 adoption-of-the-budget-resolution
4 physician-fee-freeze
9 mx-missile
11 synfuels-corporation-cutback
13 superfund-right-to-sue
15 duty-free-exports
16 export-administration-act-south-africa
The second decision table is formed from "Zoo database". It has 101 objects, 16
conditions attributes and 1decison attributes. The original data has no missing
value. The attribute reduction acquired by our algorithm includes the third, sixth,
ninth, thirteenth and sixteenth condition attribute. The total is 5 and these attributes
are listed in Table 5.
Table 5. The attribute reduction of the second decision table

No. Attribute information
3 eggs
6 aquatic
9 backbone
13 legs
16 catsize
Both two attribute reductions satisfy the complete condition, which validate the
completeness of the approximate core algorithm.
6 Conclusion
This paper introduced a complete algorithm for attribute reduction. The algorithm is
based on the principle of discernibility matrix and its time complexity is O(m2n2) in
the worst case. The proof of the completeness of the algorithm is also given in the
paper.
References
1. Hu X H, Cercone N. Learning in relational databases: a rough set approach. Computa-

tional Intelligence, 1995, 11(2): 333-338
2. Miao Duo-Qian, Hu Gui-Rong. A heuristic algorithm for reduction of knowledge.
Journal of Computer Research and Development. 1999, 36(6): 681 684 !
3. Pawlak Z. Rough sets. International Journal of Computer and Information Science,

1982, 11(5): 341-356
4. Slowinski R. Intelligent Decision Support—Handbook of Applications and Advances
of the Rough Sets Theory, Dordrecht: Kluwer Academic Publishers, 1992
5. Wang Jue et al. Data enriching based on rough set theory. Chinese Journal of Com-
puters. 1998, 21(5): 393 400
!
6. Wang Jue, Wang Ju. Reduction algorithms based on discernibility matrix: the ordered
attributes method. Journal of Computer Science and Technology. 2001, 16(6): 489-504
Preliminary Probe of Embedded Real-Time
Operating System and Its Application
in Welding Control
Wenjie Chen, Shanben Chen, and Tao Lin
Institute of Welding Technology, Shanghai Jiao Tong University, Shanghai,

200030, P.R. China
sbchen@sjtu.edu.cn
Abstract. This paper firstly introduces the concept of real-time and embedded
system, then analyses the character of welding and the advantages of adopting
RTOS in welding process control. In the end, simulation software was developed
to demonstrate the advantages of applying RTOS in welding process control.
1 Introduction
Generally speaking, there are two categories of computer application [1]: the gen-
eral-purpose computer application, such as PCs or workstations, and embedded
systems application, such as mobile phones or washing machines. Embedded sys-
tem means that the computer is built into a system and is not seen by the user as
being a computer. Real time software can run on embedded computers or on com-
puters that are doing control functions but are clearly freestanding and not built
into the target system. Real time software is a sub-domain of software generally
with distinct requirements and characteristics. Real time software differs from
conventional software in that its results must not only be numerically and logically
correct, they must also be delivered at the correct time. There are two types of
real-time systems [4]: hard and soft. In hard real-time systems, tasks have to be per-
formed not only correctly but also on time and missing a deadline has catastrophic
results for the system. In a soft real-time system, tasks are performed by the sys-
tem as fast as possible, but the tasks don't have to finish by specific times and
missing a deadline only causes a quality reduction.
Embedded systems have several common characteristics [2]:
1) Embedded system takes computer or microprocessor as its center and
cooperates with peripheral mechanical or electronic equipments to ac-
complish a particular purpose.
2) Reactive and real-time: Many embedded systems must continually re-
act to changes in the system’s environment, and must compute certain
results in real time without delay.
354354W. Chen, S. Chen, and T. Lin
3) Tightly constrained: Embedded systems often must be sized to fit a in-

commodious space, must perform fast enough to process data in real-
time, and must consume minimum power to extend battery life or pre-
vent the necessity of a cooling fan.
In a word, embedded real-time system is an important computer application
system. Relatively speaking, these systems have high even rigorous demand of
time character, reliability and security.
Real-time operating system (RTOS) is the platform of embedded application
software. Nowadays, there are many RTOS, both including commercial and indi-
vidual developed system. Such as pSOS, Vxworks and µC/OS [3]. In this paper,
µC/OS was used as platform for simulation software of welding process control.
2 Advantages of Implementation of RTOS

in Welding Control
The main advantage of adopting RTOS in embedded application is highly improv-
ing system’s reliability. Especially for control system, the non-breakdown is a ba-
sic requirement. For this problem, adopting some measures such as improving
anti-interference ability of hardware system to satisfy electromagnetic compatible
demand can do some better. On the other hand, taking some measures on software
can improve the reliability of system strikingly. For a long time, conventional de-
veloping style is: aiming at certain application and drawing procedure flow chart,
then coding. This type of program is so called structured program. While encoun-
tered strong interference, a hanging-up at any place of the program will cause
breakdown of system and must rely on watchdog of hardware to reset the system.
For RTOS managed system, however, this is not necessary. Because for RTOS
managed system, there are many independent tasks or processes, the interference
can only cause one task or process blocked, while the others are alive. Moreover,
the blocked task or process can be killed or reinstated by other tasks or processes.
The second advantage of adopting RTOS in embedded system is increasing de-
velopment efficiency and shortening development period. A complicated applica-
tion can divide into several independent tasks or modules, and each module’s de-
veloping and debugging does not affects the other modules. RTOS provides a nice
multitask debugging environment, and the application seems doing several task at
the same time.
In a sense, computer that has no operating system is no use. In embedded appli-
cation, only taking CPU embedded into the system meanwhile embedded in oper-
.
ating system is true embedded application [2]
3 Necessary of Implementation of RTOS

in Welding Control
As for welding, different welding technique has different requirement of output
characteristic of welding power source. For example, argon tungsten-arc welding
Preliminary Probe of Embedded Real-Time Operating System 355
355
need constant current external characteristic, manual arc electrode welding need
constant potential external characteristic and carbon arc air gouging and sub-
merged-arc welding often needs constant voltage output characteristic. These
welding techniques are widely used in boiler, pressure vessel and heavy-duty ma-
chinery manufacturing. Moreover, these welding techniques are often used on a
product even one joint at the same time. Developing such a power source that can
satisfy the various outputs characteristic can bring convenience for users and help
users to decrease investment of equipment. Now, embedded microcomputer con-
trolled power source provides this possibility.
The application environment of welding power source is very formidable. Most
of them are used in noisy workshop or field. In many cases, the high-frequency in-
ference is heavy, and it is very easy to cause microcomputer controlled power
source to breakdown. To avoid occurrence of this case, besides the measures such
as masking, optoelectronic isolating and electromagnetic compatibility designing,
the RTOS should considered to be used in microcomputer controlled welding
power source.
The character of welding process control is that the parameters needing adjust-
ment is much, for example, the welding current, the welding voltage, the wire feed
rate, the gas-flow rate and so on. If all of the adjustments adopt single process to
control, then not only the design is very complicated but also the level of reliabil-
ity is not high. It is easy to lead whole system breakdown just because a little pro-
cedure’s problem. While, the system that adopted RTOS is invulnerable. It can di-
vide these tasks into several independent processes; let each process to in charge
of each adjustment task. The scheduling, resource’s distribution and inter-process
coordination is up to RTOS. The problem of one process cannot lead to the whole
system’s breakdown.
Another character of welding process control is that needing rapid response to
external change. If adopt RTOS, the different process will be assigned a different
priority. The scheduling module of RTOS often designed as preemptive mode. As
for feedback information, the each process can reactive precisely and promptly. So
keep every adjustment in real-time. The real-time is often the critical requirement
of welding process control.
If adopt RTOS, the system’s transplant ability increase. Because the isolation of
RTOS, i.e. the hardware character has been encapsulated by RTOS, the applica-
tion can easy transplant from this type of CPU to another type of CPU without
modification of application codes. While, if not adopt the RTOS, the transplanta-
tion must involves massive modification of application codes even entirely recode.
4 Simulation of Welding Process Control Based on RTOS
µC/OS-II is a high performance, deterministic real-time kernel and can be embed-

ded in various products. Thousands of people around the world are using µC/OS
in all kinds of applications such as cameras, engine controls, network adapters,
ATM machines, industrial robots, and many more [5].
356 W. Chen, S. Chen, and T. Lin
To demonstrate the advantages of RTOS in welding process control, this paper

use the µC/OS-II as RTOS kernel and develop a simulation application of welding
dynamic process.
main
Taskstart
TaskController
TaskSpeed TaskCurrent
Fig.1 Task graph
In this paper, we developed a welding process control simulation application –

weldsimu. It’s consists of five independent tasks, µC/OS-II creates two ‘internal’
tasks: the idle task and a task that determines CPU usage rate. Weldsimu creates 3
other tasks. The TaskStart() task is created by main() and its function is to create
the other tasks, display the simulation statistics on the screen and save the simula-
tion data to files. The Taskcontroller() is created by TaskStart(). It is a double in-
put and double output fuzzy controller and its function is to control the dynamic
process of welding pool. The double inputs are error and error’s change respec-
tively; the double outputs are welding current and welding speed. As we known,
the welding current increase, the weld pool width increase, but there is a relatively
long lag. The welding speed increase, the heat input decrease, but the lag is rela-
tively short. Therefore, in some cases, for instance, there is a leap of heat trans-
mission condition of work piece, if only adjust welding current, the welding qual-
ity can not be ensured, welding current and welding speed be adjusted
simultaneously should be better. In this paper, we allocate for welding current and
welding speed adjustment each task. In conventional programming, these func-
tions are fulfilled in one and only one task. If there is some problem lead task to
hang up, the whole system is breakdown. But in the real-time multitask operating
system, one task out of function, the other is still work. The following is the de-
signed task graph.
From this task graph, we can know that the application’s update is easy. Be-
cause each task has independent stack space and CPU context status, the modifica-
tion of one task doesn’t affect the other tasks.
Fig.2 is the interface of simulation software. From this display, we can observe
the parameters of simulation process. There are 5 tasks running in computer and
10 task switches per second. The CUP usage rate is about 16%.
Preliminary Probe of Embedded Real-Time Operating System 357
357
Fig.2 The interface of simulation application
5 Conclusions
1. RTOS has many advantages; the main advantage of adopting RTOS in em-
bedded application is highly improving system’s reliability. The second advantage
of adopting RTOS in embedded system is increasing development efficiency and
shortening development period.
2. Different welding technique has different requirement of output characteris-
tic of welding power source. Embedded microcomputer controlled power source
can satisfy these requirements.
3. The characters of welding process control need the introduction of embedded
real-time operating system into welding process control.
4. The simulation of welding process results indicate RTOS is applicable to
welding control, and there are many advantages for adopting RTOS in software
programming and maintenance.
References
1. D.M. Auslander, J.R. Ridgely(2000), Design and Implementation of Real Time Soft-
ware for Control of Mechanical Systems.
2. Dr. Jürgen Sauermann, Melanie Thelen.(2000), Real-time Operating Systems.
3. Jack G. Ganssle(1999), The art of Designing Embedded systems.
4. Labrosse, Jean J.(2000), µC/OS, The Real-Time Kernel.
5. Ted Van Sickle(), Programming Microcontrollers in C, Second Edition.
358
Estimation of Oxygen Concentration

Using Dynamic Recurrent Neural Network
Based on PCA
Dazhong Sun, Fangjun Ye, and Xingsheng Gu
Research Institute of Automation, East China University of Science &

Technology, Shanghai 200237, P.R. China
sun_dazhong@163.net
Abstract. At present, much more research in the field of soft-sensing modeling

is concerned. In the method of building soft-sensing models, artificial neural net-
work is more frequently used. In this paper, a model based on dynamic recurrent
neural network is studied and the principal component analysis is also used to pre-
treat the data set before modeling. The generalization result shows that the model
is very accurate.
1 Introduction
In the process of continuous catalytic reforming (CCR),oxygen concentration is an

important variable. In this article, the objective is to build a soft-sensing model for
oxygen concentration so that it can be estimated using other easy-to-measure
process variables. High estimation accuracy for this model is demanded for eco-
nomic profit. Through mechanical analysis, we find that there are many secondary
variables relating to oxygen concentration. Some of the variables are correlated.
That is to say, there exists some redundant information and some information ex-
traction methods must be used to treat the sample data before they are used to
build the soft-sensing model. Principal Component Analysis (PCA)[1] among other
related techniques have been demonstrated as useful tools for analysis of data and
modeling of the systems.
2 Industrial Background
The catalytic reforming system is an important chemical process in oil refineries.

A typical catalytic reformer consists of a number of reactors, a regeneration tower
and a lot of other equipments as shown in Fig. 1. The continuous regeneration of
catalyst is an important symbol of the CCR system. Fig.1. shows the simplified
flow diagram of the catalyst. The catalyst to be regenerated firstly goes into the
Estimation of Oxygen Concentration Using Dynamic Recurrent Neural Network 359
359
catalyst collector from the reactor. In the lift line, the catalyst is lifted by N2 and
enters the regeneration tower. Coking, oxidize/chloridize reaction and dry take
place in the regeneration tower. At last, the catalyst reaches the reduction zone
lifted by highly pure H2.There, the platinum on the catalyst’s surface changes from
oxidation state to reduction state which is an important step for the catalyst to be
reactivating. The reactivating catalyst then enters the first reforming reactor for
use again. In the regeneration process of the catalyst, the oxygen concentration is a
very important variable. If the variable can not be well controlled, it is very harm-
ful to the regeneration tower and the catalyst itself. Because of this, many oil re-
fineries have spent much money to buy on-line analyzers. But on-line analyzers
have an analytical time delay which may degrade the control performance. Fur-
thermore, on-line analyzers require frequent and high maintain cost. Owing to all
of the above disadvantages, many oil refineries have considered using some effi-
cient methods to solve the problems which contribute to the progress of soft-
sensing techniques.
Fig. 1. Simplified flow diagram of the catalyst
3 Elman Network
In dynamic recurrent neural networks (DRNN)[2,3], Elman network[4,5] is com-

mon and used often. It is a simple recurrent neural network, as shown in Fig.2. It
can be seen that in a Elman network, in addition to the input, hidden and the out-
put units, there are also context units. The feedforward weights such as Wih, Wch
and Who are modifiable. The recurrent weights Whc is fixed. The output vector is
360360D. Sun, F. Ye, and X. Gu
u(k-1)
Wih
Who
Whc Y(k)
context Wch X(k)
Xc(k)
Fig. 2. Elman network
Y(k) Rm and the input vector is u(k-1) Rr. X(k) Rn is the hidden layer out-
! ! !
put vector. Xc(k) is a n×n matrix.

The relationship between the input and output can be represented as equation
(3.1) to equation (3.3). F and G are activation functions of hidden units and
output units respectively and they can be adopted as linear mappings.
X (k ) = F (Wch X c (k ) + Wih u (k − 1)) (3.1)
X c (k ) = X (k − 1) (3.2)
Y (k ) = G (Who X (k )) (3.3)
The dynamic back-propagation (DBP) learning rule is used to train the Elman
network. DBP can be presented as follows:
ji
∆who = ηδ io x j (k ) ( j = 1,2, # , n; i = 1,2, # , m) (3.4)
qj
∆wih = ηδ hj u q (k − 1) ( j = 1,2, # , n; q = 1,2, # , r ) (3.5)
m ∂x j (k )
∆w = η # (δ io who
lj
ch
ij
) lj
( j = 1,2, # , n; l = 1,2, # , n) (3.6)
i =1 ∂wch
∂x j (k ) ′ $ n
lj ∂xi ( k − 1) '
where lj
∂wch
= f j (⋅) x
% l ( k − 1) + # wch
∂wch lj ( (3.7)
& i =1 )
δ io = ( y di (k ) − y i (k )) g i ′ (⋅) (3.8)
m
′
δ hj = # (δ io whoji ) f j (⋅) (3.9)
i =1
361
At the beginning of the training process, the activation of the hidden units are
unknown. They are usually set to one-half of their maximum range. For a sigmoi-
dal activation function the initial values can be set to 0.5. For a hyperbolic tangent
activation function they can equal to 0.
4 Model Using Elman Network Based on PCA
In this paper, the application of Elman network combined with PCA is to build a
soft sensor for oxygen concentration. High estimation accuracy for this model is
demanded for economic profit. The data we used is from one oil refining factory.
Something must be done prior to build the sensor.
[6]
4.1 Data pretreatment
Data pretreatment is an important issue for developing soft sensors. Firstly, the de-
tection and deletion of singular samples is needed in order to improve generaliza-
tion results. The singular samples caused by human error or unusual system dis-
turbances are abnormal or wrong. Secondly, normalizing the input and output data
sets so that they will have means of zero and standard deviations of 1 in order to
avoid having important variables whose magnitudes are small from being over-
shadowed by less important but with large magnitude variables. The last is to ex-
tract meaningful information from the original noisy and redundant data sets. In
real process, the data is sampled two times every minute and in half an hour,60
samples are got. In order to eliminate the noise information from the original data,
the data is filtered every half an hour before used to build models.
4.2 Model and results
Through mechanism analysis, we select 21 secondary variables including tempera-

ture, pressure, flux and coke value which influence the oxygen concentration.
They will be used to build a model as input variables and the oxygen concentra-
tion is the objective variable.
Using the PCA algorithm, the data can be adequately described with far fewer
factors than the original variables with no significant loss of information. Table 1
lists the results of the PCA. From Table 1, it can be seen that the first five princi-
pal components (PC) include almost all of the variance (above 95%). It means that
the first five principal components use the most essential information of the origi-
nal data and the redundant information is eliminated.
The first five principal components are used as the input variables of Elman.
The selection of the number of hidden nodes of Elman is important and it is usu-
ally done by cross validation[7]. The optimal number of hidden nodes is deter-
mined according to the testing data root mean square error (RMSE). Usually, the
number of hidden nodes in which a local minimum in the testing RMSE is reached
is selected as being optimal. At last, we select 14 hidden nodes. 200 samples are
used to train the network and 100 samples are used to check the generalization re-
sult of the model. The training and generalization results for the PCA method are
shown in Fig.3. and Fig.4.. Fig.3. shows the results of training for the PCA model
and Fig.4. is the generalization result of the model The root mean square errors for
training and checking are 0.0160 and 0.0173 respectively which show that the
model’s generalization ability is very strong.
Table 1. The result of PCA
PC No. Eigenvalue Variance (%) Total variance

(%)
1 2547.493 56.227 56.227
2 880.820 19.441 75.668
3 420.269 9.276 84.944
4 380.932 8.408 93.352
5 149.907 3.309 96.661
6 51.721 1.141 97.802
7 35.866 0.791 98.593
8 27.770 0.613 99.206
9 17.180 0.379 99.585
10 7.477 0.165 99.750
11 4.769 0.105 99.855
12 2.475 0.055 99.910
13 1.530 0.034 99.944
14 1.200 0.026 99.970
15 0.619 0.014 99.984
-3
16 0.254 5.606×10 99.990
-3
17 0.172 3.796×10 99.994
-3
18 0.122 2.693×10 99.997
-3
19 0.075 1.655×10 99.998
20 0.041 9.049×10-4 99.999
-4
21 0.031 6.842×10 100.000
363
0.65 Analysis value

Elman output
Oxygen Concentration
0.6
0.55
0.5
0.45
0 20 40 60 80 100 120 140 160 180 200
Sample Number
Fig. 3. Training result of the Elman model
0.65 Analysis value

Elman output
Oxygen Concentration
0.6
0.55
0.5
0.45
200 210 220 230 240 250 260 270 280 290 300
Sample Number
Fig. 4. Generalization result of Elman model
5 Correction of Soft Sensor
In a real process, the operating conditions may often change, so if the process
drifts away from the conditions in which the soft sensor is built, the soft sensor
will not provide satisfying results. In this case, we should consider correcting the
model. There are two ways to correct the Elman model. We can change the
weights or to change the structure of the neural networks. We firstly collect many
new samples in the new operating condition and then use them to train the model.
The frequency that we correct the model depends on the stability of the process. If
the process is stable in general, the frequency is low. Otherwise, the frequency is
high.
6 Conclusions
To estimate hard-to-measure and important variables from the easy-to-measure

variables is a promising topic in the industrial fields. The model using Elman net-
work based on PCA has strong generalization ability. It also avoids the time delay
of on-line analyzer. The generalization result shows that the model is highly accu-
rate. This can be used for reference by some other similar problems.
References
1. Kaspar MH, Ray WH (1992) Chemometric methods for process monitoring and high-
performance controller design. AIChE Journal 38(10): 1593-1608.
2. Shaw AM, Doyle III FJ (1997) Multivariable nonlinear control applications for a high
purity distillation column using a recurrent dynamic neuron model. J.Proc.Cont 7(4):
255-268.
3. Karjala TW, Himmelblau DM (1996) Dynamic rectification of data via recurrent neural
nets and the extended Kalman filter. AIChE Journal 42(8): 2225-2239.
4. Wenhua Zeng (2001) On line estimation of aviation kerosene specific gravity based on
diagonal recurrent neural network.Computer automated measurement & control. 9(2):
50-51.
5. Pham DT, Liu X (1996) Training of Elman networks and dynamic system model-
ing.International Journal of Systems Science 27(2): 221-226.
6. Wei Zhong, Jinshou Yu (2000) MIMO soft sensors for estimating product quality with
on-line correction.Trans IchemE 78(A): 612-620.
7. Osten DW (1988) Selection of optimal regression models via cross validation. Journal
of Chemometrics 2: 39-48.
365
Robust Stability of Interval Lur'e Systems:

A Bilinear Matrix Inequality Approach
J. T. Sun
Dept. of Appl. Math., Tongji University, 200092, P.R.China

sunjt@sh163.net
Abstract. This paper provides a sufficient condition for an interval Lur'e system
to be globally exponentially stable with a damping factor. The Lur'e system
consists of an interval linear dynamical system and a sector-bounded memoryless
time-varying nonlinear term. The sufficient condition is described by a simple
bilinear matrix inequality. The relationship between the stability of symmetric
interval matrix and globally exponentially stability of interval Lur'e system is
established.
1 Introduction
Many researchers have investigated the stability of feedback systems whose

forward path is a linear time-invariant (LTI) system and whose feedback path is a
memoryless (possibly time-varying) nonlinearity (Vidyasagar 1993). These
feedback systems are called Lur'e systems. The simplest version of the circle
criterion guarantees that the Lur’e systems whose linear part is strictly positive
real and whose nonlinearity belongs to the sector [0,+∞) is globally exponentially
stable. Grujic and Petkovski (1987) discussed the robustness of the Lur'e system.
Tesi and Vicino (1991), Dahieh et al. (1993), Mori et al. (1995), Wada et al.
(1996,1998) and Sun et al.(2001) have investigated the parametric absolute
stability of Lur'e systems. Since the linear matrix inequality (LMI) can be solved
numerically efficiently (Boyd et al. 1994), Konishi et al. (1999) presented some
sufficient condition for the Lur'e system with the uncertainties satisfying a
matching condition to be globally exponentially stable by linear matrix inequality.
In the systems with structured parameter uncertainty, there are mainly two kinds
of representation of parameter uncertainty. The first representation is when the
uncertainty in the matrix A is given by ∆ =DFE with FFT ≤ I , D and E are known
matrix, i.e. the uncertainty satisfying a “matching condition”. The second
r e p r e s e n t a t i o n i s o n e i n wh i c h t h e u n c e r t a i n t y ∆ i s g i v e n b y
∆=#γi Ai , where the γi are uncertain parameters satisfying | γi | ≤ ri . Gu et al. (1990)
366
366 J.T. Sun
showed by an example that the second representation is more general than the first
one. In a practical situation one would have an uncertainty upon the components
of the matrix A, represented by an upper bound as |∆|# E, which means that
|)ij | $ eij , where )ij and eij denote, respectively, the ij-entries of ∆ and E. The
matrix E is assumed to be known with positive entries, i.e. the parameter
uncertainty is value bounded. It is worthy while noticing that this representation is
similar to the second one. Mehdi et al. (1996) presented a linear quadratic design
for uncertain systems in state space representation, the parameter uncertainty was
value bounded. But interval matrix N[P, Q] ={ A= (aij) n%m }| pij $aij$qij , i=1, & ,n;
j=1, &,m; P=( pij ) n%m, Q=( qij ) n%m } is more general representation than the value
bounded representation.
In this paper, we discuss the robust stability of the interval Lur'e systems
whose forward linear system has structured uncertainties and whose feedback path
is a sector-bounded memoryless time-varying nonlinearity. The BLMI approach
allows us to derive a simple sufficient condition for the interval Lur'e system to be
globally exponentially stable with a damping factor. The relationship between the
stability of symmetric interval matrix and globally exponentially stability of
interval Lur'e system is established.
2 Main Result
We consider the interval Lur'e system whose linear forward path is a system (1)
and whose feedback path is a memoryless time-varying nonlinearity ( as shown in
figure 1. The linear forward path is a linear interval dynamical system
'* x$ (t ) = N [ A, A]x(t ) + N [ B, B]w(t )

( (2.1)
*) z (t ) = Cx (t ) + Dw(t )
where x(t)' Rn is the state, ω(t) ∈Rm is the input signal and z(t) ∈Rm is the output
signal. N [ A, A] is n×n order interval matrix; N [ B, B] is n×m order interval
matrix.
Linear system
+ ω(t) (2.1) z(t)
Ψ(t, z(t))
Nonlinear feedback
Fig. 1 Lur’e system

367
Robust Stability of Interval Lur’e Systems 367
The feedback nonlinearity is described by
ω(t) = -Ψ(t, z(t)), Ψ: [0,+∞)×R m → R m (2.2)
We give definitions of the sector condition for the nonlinearity.
Definition 1 A memoryless nonlinearity Ψ: :[0,+∞)×R m → R m is said to belong to

a sector [0,+∞) if
ZTΨ(t, z) ≥ 0, ∀ t ≥ 0, ∀ z ∈ R m (2.3)
In this section we derive exponential stability conditions of the interval Lur'e

system (1) for the feedback nonlinearity Ψ belonging to a sector [0,+∞).
A+ A A− A B+ B B−B
Let A 0 = 2
,H = 2
= (hij ) n×n , B0 = 2
,H = 2
= (h ij ) n×m , E1 =
[ h11 e1 , # , h 1n e1 , # hn1 e n , # hnn e n ]; E 2 = [ h11 e1 , # , h1m e1 , # ,
h n1 e n , # , h nm e n ]; F1 = [ h11 e1 , # , h1n e n , # , hn1 e1 , # , hnn e n ]T ;
F2 = [ h 11 e1 , # , h1m e m , # , h n1 e1 , # , h nm e m ]T .
Where E1 , E 2 , F1 , e1 = (1,0, # ,0) T , # , ei = (0, # ,0,1,0, # ,0) T , # , e n =
(0, # ,0,1) T ∈ R n and F2 , ei ∈ R m .
#
*
Further more, we define ={ # ∈R nm×nm
# =diag[ε 11 , # , ε 1m , # , ε n1 ,
# , ε nm ], ε ij ≤ 1, i = 1,2, # , n, j = 1,2, # , m}, then N [ A, A] = A0 + E1 # F , N [ B,

1
B] = B0 + E 2 #F . 2
The proof is similar to those of Wu’s et al. paper(1999).
System (1) can be described by
'* x$ (t ) = ( A0 + E1
(
# F1 ) x(t ) + ( B0 + E 2 # F )ω (t )
2
(2.4)
*) z (t ) = cx(t ) + Dω (t )
Theorem 1 The interval Lur’e system consisting of (2.1) and the feedback (2.2)
belonging to a sector [0, +∞) is globally exponentially stable with a damping
factor ε if there exists a symmetric interval matrix
∆ $ T11 T12 ,
# =%
1 %T T
& 12 D + D .
- ≥ 0, where T11 = − PN [ A, A] − N T [ A, A]P − 2εP; T12 = C T −
-
PN [ B, B ]; T21 = C − N T [ B, B ]P
368
368 J.T. Sun
Proof Define a Lyapunov function V = xTPx ≥ 0, P > 0, then

$x ,
V$ ( x) = −[ x T , ω T ] #
% - + 2ω T z − 2εx T Px
1 %ω -
& .
$
where V (x) is evaluated along the trajectories of the system (1), if Σ1 > 0, ∀t ≥ 0
and Ψ(t, z) belongs to the sector [0, +∞) (i.e. ωTz ≤ 0), then we have
V$ ≤ −2εV (2.5)
Integrating (2.5)from 0 to t yields

t t
+ V$ ( x(t ))dt ≤ − 2ε V ( x(t ))dt then V ( x(t )) ≤ V ( x(0)) exp(−2εt ). From V ( x) ≥

+
0 0
( )
1
2 V ( x ( 0 ))
ρ min ( P) x , we obtain x(t ) ≤ ρ min ( P )
2
exp(−εt ).
This inequality indicates that the interval Lur’e system (2.1) is globally
exponentially stable with a damping factor ε. The proof of this theorem is
completed.
Remark 1 A relationship between the stability of symmetric interval matrix #1
and the globally exponentially stability of interval Lur'e system is established in
Theorem 1.
Remark 2 The symmetric interval matrix L[ A, A] = { A = (a ij ) n×n a ij ≤ a ij ≤ a ij ,
a ij = a ji , i, j = 1,2, # , n, A = (a ij ) n×n , A = (a ij ) n×n } is stable if and only if LV [ A, A]
= { A = (a ij ) n×n a ij = a ij or a ij , a ij = a ji , i, j = 1, # , n} is stable.
The proof is similar to those of Shi's paper (1986).
'a ij i= j
*
Remark 3 If M [ A, A] = (mij ) n×n , mij = (
*)max a ij , a ij , i ≠ j ( )
is stable, then
symmetric interval matrix L[ A, A] is stable.
Theorem 2 The interval Lur'e system consisting of (2.1) and the feedback (2.2)
factor ε if there exists a symmetric positive-definite matrix P > 0 such that
$ H 11
% C T − PB0 ,
-≥0 (2.6)
%C − BT P T
D +D+ F2T F2 -.
& 0
where H 11 = − PA0 − A0T P − 2εP + P ( E1 E1T + E 2 E 2T ) P + F1T F1 .

369
Proof Consider the equivalent system (2.4), Note
$ − PA0 − A0T P − 2εP

Y0 = %
C T − PB0 ,-
. Since ξ 1T ξ 2T (
$
)%%%
L11 PE 2 #F 2
,
-
--
%
& C − B0T T
D +D . -
&
F2T #E T
2 P 0
.
$ξ1 ,
%% -- = 2(ξ 1T PE1 )(
&ξ 2 .
# F ξ ) + 2(ξ
1 1
T
1 PE 2 )( #F ξ 2 2) ≤ ξ 1T PE1 E1T Pξ 1 + ξ 1T F1T F1ξ 1
ξ 1T PE 2 E 2T Pξ 1 + ξ 2T F2T F2ξ 2 = ξ 1T ξ 2T %% ( )$ H 11
F2T
0 ,$ ξ 1 ,
-% -, whereL11 = PE1
F2 -.%& ξ 2 -.
#F 1
&0
+ F1T #E T
1 P; H 11 = P( E1 E1T + E 2 E 2T ) P + F1T F1 ,
$ H 11 0 ,
hence Y0 + % ->0
% 0 FT F -
& 2 2.
by Lyapunov function V(x)=xTPx and similar procedure of proof of Theorem 1, we
can obtain the interval Lur's system (2.1) is globally exponentially stable with a
damping factor ε.The proof of this theorem is completed.
Lemma 1 If A ∈ N [ A, A], (1 − λ )( A − A) ≤ ∆A = A − [λ A + (1 − λ ) A] ≤ λ A − A) =
λT , λ ∈ [0,1], then for any positive number ε , P∆A + ∆A T P ≤ εP T P + nε −1 [max(λ ,
1 − λ1 )] 2 diag[( A − A) T ( A − A)]
Proof Since
x T {εP T P + nε −1 [max(λ , 1 − λ )] 2 diag (T T T )}x = εx T P T Px +
n[max( λ ,1− λ )]2
ε #x t
j ,k
j jk t jk x j ≥ εx T P T Px + 1
2ε
[ # x ∆a
j ,i , k
i ik ∆a ik x i +
# x ∆a
i, j ,k
j jk ∆a jk x j ] = εx T P T Px +
[max( λ ,1− λ )]2
2ε
[ #x t
j ,i , k
i ik t ik x i + #x t
i , j ,k
j jk t jk xj]
≥ εx T P T Px + ε1 # x ∆a
i, j ,k
i ik ∆a jk x j = εx T P T Px + ε1 x T ∆A T ∆Ax ≥ 2 x T ∆AT Px
T T
= x [ P ∆A + ∆A P ] x
then, Lemma is holds.
Theorem 3 The interval Lur'e system consisting of (2.1) and the feedback (2.2)
factor ε if there exists a symmetric positive-definite matrix P > 0 and scalar λi ∈ [0,
1] (i = 1,2), λ > 0, such that
370
370 J.T. Sun
$ T11
% C T − PB1 ,-
≥0 (2.7)
%C − BT T22 -.
& 1
where T11 = − PA1 − A1T P − 2εP + (λ + 1) PP + nλ −1 [max(λ1 , 1 − λ1 )] 2 diag[( A −

A) T ( A − A)], T22 = D T + D + m[max(λ 2 , 1 − λ 2 )] 2 diag[( B − B)]T ( B − B )].
Proof Note
$ − PA1 − A1T P − 2εP C T − PB1 ,
Y1 = % -, where A = λ A + (1 − λ ) A, B = λ B +
1 1 1 1 2
% C − B1 P D − D -.
T
&
(1 − λ 2 ) B.
$ K 11 P∆B ,$ x ,
(
Since x T ω T %% T) -% - = x T ( P∆A + ∆A T P) x + 2ω T ∆B T Px ≤
% -
0 -.& ω .
& ∆B P
x T {λP T P + nλ −1 [max(λ1 , 1 − λ1 )] 2 diag[( A − A) T ( A − A)]}x + ω T ∆B T ∆Bω +
x T PPx ≤ x T {(λ + 1) P T P + nλ −1 [max(λ1 , 1 − λ1 )] 2 diag[( A − A)]T ( A − A)]}x −
(
m[max(λ 2 , 1 − λ 2 )] 2 diag[( B − B ) T ( B − B)]ω T ω = x T )$ L0
ω T %% 11 0 ,$ x ,
-% -,
L 22 -.%& ω -.
&
where K 11 = P∆A + ∆A P; L11 = (λ + 1) P P + nλ [max(λ1 ,1 − λ1 )] 2 diag[( A −
T T −1
A) T ( A − A)], L 22 = m[max(λ 2 , 1 − λ 2 )] 2 diag[( B − B) T ( B − B )

hence
$ P∆A + ∆AT P P∆B ,-
Y1 + % ≥ 0, by similar procedure of proof of Theorem 1, we
% ∆B T P 0 -.
&
can obtain that the interval Lur'e system (2.1) is globally exponentially stable with
a damping factor ε. The proof of the theorem is completed.
Remark 4 (2.6) and (2.7) are bilinear matrix inequality (BLMI) of P for BLMI,
Goh. et al (1994) presented properties and computational methods.
3 Numerical example
We consider an uncertain system (2.1) with
$ − 0.06 0.97 , $ 0.06 1.03 , $ 0 .9 , $1.1,
A = %% --, A = %% --, B = %% --, B = %% --, C = (1 1), D = 0.98
& − 2 . 03 − 3 . 06 . & − 1. 97 − 2 . 04 . 0
& . . 9 &1.1.
the nonlinear feedback is set as
Ψ(t, z(t)) = z(t){1−sin[z(t)]} (3.8)

this nonlinear function belongs to the sector [0, +∞).
371
If we set ε =0.5 and λ =1, λ1=λ2 =0.5 there exists a positive definite matrix
!
$4 1 ,
P = %% -- which satisfies the condition in Theorem 2, hence the interval Lur'e
& 1 0 .5 .
system consisting of (2.1) and the feedback (3.8) belonging to the sector [0.+∞) is
globally exponentically stable with a damping factor 0.5.
References
[1]. Vidyasagar, M., (1993) Nonlinear Systems Analysis, Prentice-Hall, London

[2]. Grujic LJT, Petkovski DJ (1987) On robustness of Lur'e systems with multiple non-
linearities. Automatica 23: 327-334
[3]. Tesk A, Vicino A (1991) Robust absolute stability of Lur'e control systems in parameter
space. Automatica 27: 147-151
[4]. Dahleh M, Tesk A, Vicino A (1993) On the robust Popov criterion for interval Lur'e
system. IEEE Tansactions on Automatic Control 38: 1400-1405
[5]. Mori T, Nishimura T, Kuroe Y, Kokame H (1995) Comments on Òn the robust Popov
criterion for interval Lur'e system. IEEE Tansactions on Automatic Control 40:
136-137
[6]. Wada T, Ikeda M, Ohta Y, Siljak DD (1996) Parametric absolute stability of
multivariable Lur'e systems: an LMI condition and application to polytopic systems,
Proceedings of the 13th IFAC World Congress, San Francisco, California, USA,
pp19-24
[7]. Wada T, Ikeda M, Ohta Y, Siljak DD (1998) Parametric absolute stability of Lur'e
systems. IEEE Tansactions on Automatic Control 43: 1649-1653
[8]. Boyd S, Ghaoui LEL, Feron E, Balakrishnan V (1994) Linear Matrix Inequalities in
System and Control Theory, Philadelphia, Pennsylvania SIAM
[9]. Konishi K, Kokame H (1996) Robust stability of Lur'e systems with time-varying
uncertainties: a linear matrix inequality approach. Int.J. of systems Science 30: 3-9
[10]. Gu, K, Zohdy M.A, Loh NK (1990) Necessary and sufficient condition of quadratic
stability of uncertain linear system. IEEE Tansactions on Automatic Control 35:
601-604
[11]. Goh KC, Turan L, Safonov MG (1994) Bilinear matrix inequality properties and
computational methods. Proc. of Conf. on ACC, Baltimore, pp. 850-855
[12]. Goh KC, Safonov M G, Papavassilopoulos GP (1994) A global optimization approach
for the BMI problem .Proc. of Conf. Decision and Control, Tokoy, pp. 2009- 2014
[13]. Shi Z, Gao W (1986)A necessary and sufficient condition for the positive- definiteness
of interval symmetric matrices Int. J. of Control 43:325-328
[14]. Wu FX, Shi ZK, Dai GZ (1999) H∞ robust control for interval systems. Acta
Automatica Sinica 25: 705-708
[15]. Sun JT, Deng FQ ,Liu YQ (2001) Robust absolute stability of general interval Lur'e
type nonlinear control systems.J. of Syst. Engineering and Electronics 12:46-52
372
Thresholding Segmentation and Classification

in Automated Ultrasonic Testing Image
of Electrical Contact
H. D. Chen, Z. J. Cao, Y. W. Wang, and J. Xue
Welding Research Institute, School of Materials Sci. & Eng., Xi’an Jiaotong Univ.,
Xi’an 710049, P.R. China
hdchen76@hotmail.com
Abstract. This paper is concerned with the thresholding segmentation of

ultrasonic C-scan image for defect with artificial intelligence and support vector
machines techniques. Wavelet based method is used to remove the noise of image.
Segmentation techniques based on thresholding were performed in order to segment
the defect in ultrasonic testing image. Experimental results show that this
segmentation technique performs well in finding defects in C-scan image of
electrical contact.
1 Introduction
The increasing use of electrical contact requires that reliable non-destructive

evaluation (NDE) methods be available for the detection of defects and damage, or
the assessment of quality of these materials. Ultrasonic C-scan is a method
commonly used in NDE, which is based on propagation of ultrasonic waves through
the material. The waves penetrate the material, bounce at the bottom (or at the
defect if exist) and return to the transducer. The time at which the first echo arrives
determines the position of the bottom of joint or of the defect. The signal is then
digitized along the plate and is visualized in a manner of a gray level image. The
main difficulty is then to extract the defects from the background in C-scan images
[1]
. The images obtained are far from being ideal. Not only do these images contain
sensor noise, which is typically present in any kind of image acquisition, but they
are also corrupted by non-uniform illumination. That is due to the image acquisition
apparatus. For example, the specimen might not be perfectly horizontal when
ultrasonic transducer scans it.
Wavelet transform is of the ability called “digital microscope”. When ultrasonic
testing images are gotten, the original image attached with noise can be
decomposed by wavelet transform. There is a difference between the noise
frequency spectrum and the frequency spectrum of defect reflection signal, little
Thresholding Segmentation and Classification 373
373
useful information is with the low signal-to-noise ratio’s image, which can be
eliminated while image reconstruction. There is a large amount of useful
information in the sub-band image with high signal-to-noise ratio; the defect image
can be obtained after the appropriate defect inspection algorithm. The purpose of
image segmentation is to separate the defect image area from the whole image,
which is important for subsequent process. Therefore, research on ultrasonic image
segmentation technique is of significance for defect detection of automated
ultrasonic imaging system in weld.
With the rapid development of electronic technology and computer technology,
further requirement of joint’s ultrasonic testing is desired. It need not only get the
ratio of braze area (RBA), but also get the defect size defect location and defect
!
type in order to carry out quantitative and qualitative analysis. Thus research in
ultrasonic testing imaging segmentation method is essential to defect’s
identification in weld [2]. In spite of all modern automated inspection methods,
defect classification is still a difficult task. Much work, essentially based on pattern
recognition techniques and artificial intelligence methods using neural networks,
has been published on this topic.
The paper is organized as follows. In section two image techniques are talked
about and the methods that have been applied to this type of images, next a system
overview of the method that implemented is presented and the way that works is
introduced. Some theoretical explanations about the methods and techniques are
given. Then, segmentation results produced by the proposed method are shown. In
section six we conclude that this segmentation technique performs well in finding
defects in C-scan image of electrical contact.
2 Image Processing Technique
The image techniques utilized for C-scan image segmentation could be divided into
the following groups: threshold-based segmentation [3,4], statistical methods for
image segmentation and region growing methods.
In the category of threshold-based segmentation was proposed the use of:
iterative thresholding, histogram analysis and morphological operations. Some of
these techniques are followed by some refinement of the segmented zones.
Statistical methods represent another important category in the segmentation
process and most of the approaches proposed in this category are using some
statistical classifications combined with different image processing techniques in
order to segment the C-scan images. Also, the use of statistical methods in texture
analysis has been proposed. The region growing techniques applied to C-scan
images represents the final category.
All these techniques have its advantages and drawbacks, that’s why even more
research is demanded in this domain and maybe some combination of the already
existing methods or some new image processing techniques will lead to better
results. The segmentation problems that are widely encountered by the researchers
are the noise introduced with the acquisition of the image, the overlapping
374374H.D. Chen et al.
intensities (different defect structures have different characteristics which results in

various signal intensities and these intensities could overlap), and the partial
volume effect.
3 Image Segmentation System
Now, the system is introduced and all the theoretical implications with this system
will be explained in the next section. Fig. 1. shows the system used in order to
segment and detect the defect in C-scan images.
Fig. 1. System diagram
The C-scan image acquisition module is the first step in the image segmentation
system. In this step it is very likely that noise will be imported, thus the quality of
the image could be decreased. That’s why the next step is dealing with the
enhancement of the image in order to improve the quality or to perform a de-noising
operation. The de-noising method is a wavelet-based method that will be described
in the theoretical section.
After the de-noising has been performed the image is entering into the classifier
module that will distinguish, according to statistical parameters that describe the
image. In this step the classifier, which is a support vector machine should also be
able to distinguish between defect and background.
The last step is the segmentation module of the image to detect the defect. This
step uses some prior knowledge from the classification step in order to use an
adaptive histogram threshold for a better defect segmentation. After a C-scan image
has passed through the whole system, the result is the segmentation of the existing
defect.
4 Theoretical Backgrounds
[5]
4.1 Image De-noising
An important part of any image processing system is the pre-processing phase. This
phase could imply contrast enhancement techniques or methods for removing the
noise. The method that used to perform this task is a wavelet-based one. Lately,
wavelets have become a popular tool in many research domains, image-denoising
being only one of them. First, a wavelet transform is applied to the image, and then
the wavelet coefficients obtained are manipulated in order that the noisy points to be
375
eliminated and then inverse wavelet transform are performed in order to reconstruct
the image. The manipulation of the wavelet coefficients in a procedure of the
former type is based on a classification. This classification is often binary: the
coefficients are divided into two groups. The first group contains important, regular
coefficients, while the other group consists of coefficients that are catalogued as
“too noisy”. These two groups are then processed in an appropriate way. Zero often
replaces noisy coefficients. In order to perform this classification a decision is
needed in order to know what’s the threshold that distinguishes between these two
classes.
4.2 Classification
4.2.1 Statistical parameters

A region having a particular defect has a wide variety in its gray level. The question
is how we can capture this defect in order to distinguish between different defects.
In this case, the original C-scan image is split in small windows and each one of
these is characterized using statistical calculus. This area, also called window, can
also be varied in its size to capture a sample of different scales to be found here.
Note that the precision with which the boundary between regions is known is
still a function of the window used. This is always true, and so there is an advantage
in using methods that allow the use of small windows.
The mean gray level and the standard deviation are known as statistical
moments. The mean is related to the first moment, and the standard deviation
depends on the second moment. There are others, and these could also be used to
characterize defect areas. In general, we have:
(1)
where N is the number of data points, and n is the order of the moment. Now the
skewness can be defined as:
(2)
and the kurtosis as:
(3)
4.2.2 Support Vector Machines

Support Vector Machines (SVMs) are a new technique that introduced in 1995 by
Vapnik [6,7]. In terms of theory the SVM are well founded and proved to be very
efficient in classification tasks. The advantages of such classifiers are that they are
independent of the dimensionality of the feature space and that the results obtained
are very accurate, although the training time is very high.
Support Vector Machines are feedforward networks with a single layer of
nonlinear units. Their design likes the generalized radial basis function (GRBF)
networks with good generalization performance as an objective and follows for that
reason the principle of structural risk minimization that is rooted in VC dimension
theory.
The solution for a typical two-dimensional case to has the form shown in Fig. 2.
Those training points for which the equality in of the separating plan is satisfied
( i.e.( yt ( xt ⋅ w + b) − 1 ≥ 0 ∀i ) , those which wind up lying on one of the
hyper-planes H1 , H 2 ), and whose removal would change the solution found, are
called Support Vectors (SVs) and are indicated in Fig. 2 by the extra circles.
Fig. 2. Linear separating hyper-planes for the separable case, the support vectors are circled
4.3 Thresholding and Morphological Operations
According to the histogram of the C-scan image, the threshold to be used for
segmentation can be determined. For a pixel to remain in the image after
thresholding, its intensity value must be greater than the threshold value.
Thresholding is the most common segmentation technique and has received great
interest in the literature.
377
Another image processing technique used is opening which is a morphological

operation. The morphological operations are well known techniques for the
segmentation and enhancement of images and also for recognition purposes.
Opening is used to eliminate small details and smoothes the image boundaries.
4.4 Segmentation
The purpose of ultrasonic image segmentation is to separate the defect area from the
background. If the amplitude of reflection wave signal is large, then the intensity of
gray scale value of pixel in the image is large. Because the gray scale value of defect
area is different from that of the well-welded area, image segmentation can be
utilized to segment the defect image. The gray scale value range of ultrasonic image
is 0 255 suppose the gray scale value of a pixel in original image be
! " # "
T ( x, y ) after thresholding segmentation it becomes Tt ( x, y ) then

" "
#1 T ( x, y ) ≥ S
Tt ( x, y ) = $ (4)
%0 T ( x, y ) < S
where S , 0 < S < 255 , stands for thresholding.
! #
The image after thresholding process is a binary image. If the thresholding

selected is higher than the actual one, some defect area will be taken for welded area
and the calculated welded area would be large then the actual one, vice versa, it
would be less than the actual one. Therefore the key of image segmentation is the
thresholding selection.
The image segmentation techniques used in image processing can be divided
into the following groups: threshold-based segmentation, statistical methods for
image segmentation and region growing methods. Threshold-based segmentation is
widely used in image processing fields. In ultrasonic imaging testing,
threshold-based segmentation method is used to segment the defect area in
ultrasonic image. There are two ways to choose the threshold: iterative thresholding
that is independent of the image or image histogram-based threshold. In order to
extract defect object that interested in from the image we have to choose a threshold
in such a way that it separates those objects from the rest. One way to choose the
threshold is using the histogram of the image. In this paper we performed
experiments with histogram-based thresholds. According to the histogram of the
C-scan image, the threshold to be used for segmentation can be determined.
5 Experimental Results
The training of the Support Vector Machine network was done with 40 images, 20
being of artificial flaw workpiece and 20 of actual workpiece. The result of the
classification is around 70%, but it is very clear that they could be improved by
training the neural network with more images. The idea of using a neural network
was to build it and test it in order to use it in the future for a classification of the
images.
The first step was represented by the pre-processing phase of the images in
order to remove the noise that could be introduced to image acquisition. In the
figures below an example of image de-noising is shown. The next step after the
classification is represented by the segmentation of the C-scan image. The
segmentation is a threshold-based, but the threshold was applied after an opening
transform. The opening transform was used to enhance the defect and then the
threshold was applied. Fig.4 is the results after the segmentation.
(a) Image before de-noising (b) image after wavelet de-noising

Fig. 3. Experimental results of switch contact C-scan image
Fig. 4. Segmentation result of the C-scan image
6 Conclusions
An image segmentation approach based on wavelet de-noising and SVMs feature

extraction and classification has been developed to detect flaws automatically in
NDE image. The system implemented is used for detecting the defect in ultrasonic
C-scan images. The system involves two major modules, one that performs
classification and the other one that segment the defects from the images.
Classification module can be used either before the segmentation process or after,
depending on what the system is used for. In the segmentation phase the system
performed well and detected the defect in the images. But I had no extra information
that could be very useful in detecting some flaw patterns. Overall, the system
performed well, but there is still a lot to improve for the completeness of knowledge
base.
379
7 Future Works
The work will be continued about this system, so in the following lines I will
present some future research that to do. First, hopefully, I will have more artificial
defect images and actual ones. When the acquisition of these images will be done, I
want to use feature extracted from images in order to get an accurate classification
of the images. In my opinion, the classification would be better performed after the
segmentation method and the segmentation results to be used in classification.
References
1. D. E. Bray and R. K. Stanley, Nondestructive evaluation, a tool for design,

manufacturing and service. New York: McGraw-Hill, 1989.
2. D. T. Brown and G. Green, “Development of an ultrasonic system for submarine hull
acoustic cladding inspection,” Ndt&E Int., vol. 30, no.3, pp. 123-133, 1997.
3. P. Sahoo S. Soltani and A. Wong, “A survey of thresholding techniques,” Comput.
! !
Vision, Graph., vol. 41, no.3, pp. 233-260, 1988.

4. J. Kapur, P. Sahoo and A. Wong, A new method for gray-level picture thresholding
using the entropy of the histogram. Comput. Vision Graph. vol. 29, no.2, pp.
"
273-285, 1985.
5. L. D. Wu, Computer Vision. Shanghai: Fudan University Press, 1993.
6. V. N. Vapnik, The Nature of Statistical Learning Theory. New York: Springer Verlag,
1995.
7. V. N. Vapnik, “An Overview of statistical Learning Theory,” IEEE Trans. Neural
Networks, vol. 10,no.5, pp.998-1000, 1999.
Author Index
M. Barrett, 281 Y. J. Lou, 25

A. Boeing, 228 H. Luo, 110
T. Bräunl, 228 W. X. Lv, 123
H. Cai, 276 D. Maravall, 190

Z. J. Cao, 372
H. Q. Chen, 372 T. Qiu, 123
M. Chen, 144
Q. Chen, 152 A. B. Rad, 161, 304
S. B. Chen, 25, 56, 123, 345, 353
W. Chen, 353 P. Seyffarth, 14
X. Chen, 110 B. Sui, 152
D. Sun, 358
B. Da, 144 J. Sun, 365
X. Dai, 276 L. Sun, 276
D. Ding, 80 Z. Sun, 241
X. Ding, 144
D. Du, 152 Z. Tang, 241
J. S. Tian, 123
S. Emil, 1
M. F. Ercan, 212 B. Wang, 345
Y. W. Wang, 372
G. Fang, 263 Y. K. Wong, 161, 304
L. Wu, 25, 56, 123
M. Ge, 63
X. Gu, 358 P. Xiao, 281
Y. Xu, 63
Y. F. He, 152 Z. Xu, 263
F.H. Ho, 304 J. Xue, 372
J. Hoffmann, 14
D. Hong, 281 Y. Yang, 327
F. Ye, 358
Y. L. Ip, 161 J. Q. Yi, 56
U. Jasnau, 14 H. Zhang, 144, 152

Y. Zhang, 123
M. L. Lam, 80 D. B. Zhao, 25, 56
T. Lin, 123, 353 C. Zhou, 212, 241, 327
Y. H. Liu, 80 C. Zou, 144
W. L. Lo, 304

Robotic Welding Intelligence and Automation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Robotic Welding Intelligence and Automation

Uploaded by

Copyright:

Available Formats

Lecture Notes

in Control and Information Sciences 299

Editors: M. Thoma · M. Morari

Prof. Shan-Ben Chen

Cataloging-in-Publication Data applied for

Robotic welding systems have been used in different types of manufacturing.

Tzyh-Jong Tarn, Washington University at St. Louis, USA

Part 1: Intelligent and Automatic Techniques for Welding

Nd:YAG – Laser – GMA – Hybrid Welding in Shipbuilding

Computer Vision Sensing and Intelligent Control

Surface Shape Reconstruction of Weld Pool during Pulsed GTAW

A Novel Intelligent Monitor for Manufacturing Processes 63

3-D Grasp Analysis and Synthesis Using the Ray-Shooting Technique 80

Part 2: Intelligent Techniques for Robotic Welding

Intelligent Technologies for Robotic Welding 123

The Seam Tracking System for Submerged Arc Welding 144

Visual Measurement of Path Accuracy for Laser Welding Robots 152

Part 3: Intelligent Robotic Systems

Vision Based Motion Planning of Humanoid Robots 212

Evolution of Locomotion Controllers for Legged Robots 228

Gait Planning for Soccer-Playing Humanoid Robots 241

Fuzzy-Neural Impedance Control for Robots 263

On Singular Perturbation Based Inverse Dynamics Control

Part 4: Intelligent Techniques and Its Applications

Adaptive Neuro-Fuzzy Control of Systems with Unknown Time Delay 304

Adaptive Robust Fuzzy Tracking Control for a Class

A Complete Algorithm for Attribute Reduction 345

Preliminary Probe of Embedded Real-Time Operating System

Estimation of Oxygen Concentration Using Dynamic Recurrent Neural

Robust Stability of Interval Lur'e Systems: A Bilinear Matrix

Thresholding Segmentation and Classification in Automated Ultrasonic

Author Index 381

Alexander Binzel Schweisstechnik GmbH & Co. KG, Gießen/Germany,

Abstract. For modern transport systems Aluminum is getting of growing

Fig. 1. Employment of aluminium alloys in the automotive industry

2 Potentials and Problems of Aluminium Welding

segregated due to the considerably lower solubility in the solid aluminium,

3 Process Stability of Welding of Aluminium Materials

3.1 Welding power source- and welding torch technology

non-short-circuiting metal transfer

pulse time actual welding time

Fig. 2. Current-time-characteristic for impulsed welding of aluminium alloys

pulse period time (pulse frequency)

Fig. 3a. MIG-AC-Pulsating of voltage

Fig. 4b. MIG-DC-Pulsating of voltage

These process variants enable

Fig. 5. Cross-section of a fillet weld of AIMgSi

But also regarding conventional GMA welding of aluminium materials special

Fig. 6. Sectional drawing and photo of a double cooling-circuit-GMA-welding torch for

electrode at negative pole electrode at positive pole

Fig. 7. TIG-welding with negative and positive polarity at the electrode

TIG Welding Process

upper cooling circuit

gas diffusor shielding gas

lower cooling circuit

ceramic shielding gas nozzle

Fig. 8. Industrial plasma welding torch

The conventional inert-gas welding process stands in competition to the laser-

Positional tolerance ofder

Fig. 9. Comparison of GMA-, TIG, Plasma- and Laser welding processes

3.2 Wire Feeding

3.3 System Aspects

Process -depending Adjustment of wire

TIG small spool