FUNDAMENTALS OF
ROBOTICS
LINKING PERCEPTION TO ACTION
SERIES IN MACHINE PERCEPTION AND ARTIFICIAL INTELLIGENCE* Editors: H. Bunke (Univ. Bern, Switzerland) P. S. P. Wang (Northeastern Univ., USA)
Vol. 38: New Approaches to Fuzzy Modeling and Control — Design and Analysis (M. Margaliot and G. Langholz) Vol. 39: Artificial Intelligence Techniques in Breast Cancer Diagnosis and Prognosis (Eds. A. Jain, A. Jain, S. Jain and L. Jain) Vol. 40: Texture Analysis in Machine Vision (Ed. M. K. Pietikainen) Vol. 41: NeuroFuzzy Pattern Recognition (Eds. H. Bunke and A. Kandel) Vol. 42: Invariants for Pattern Recognition and Classification (Ed. M. A. Rodrigues) Vol. 43: Agent Engineering (Eds. Jiming Liu, NingZhong, Yuan Y. Tang and Patrick S. P. Wang) Vol. 44: Multispectral Image Processing and Pattern Recognition (Eds. J. Shen, P. S. P. Wang and T. Zhang) Vol. 45: Hidden Markov Models: Applications in Computer Vision (Eds. H. Bunke and T. Caelli) Vol. 46: Syntactic Pattern Recognition for Seismic Oil Exploration (K. Y. Huang) Vol. 47: Hybrid Methods in Pattern Recognition (Eds. H. Bunke and A. Kandel) Vol. 48: Multimodal Interface for HumanMachine Communications (Eds. P. C. Yuen, Y. Y. Tang and P. S. P. Wang) Vol. 49: Neural Networks and Systolic Array Design (Eds. D. Zhang and S. K. Pal) Vol. 50: Empirical Evaluation Methods in Computer Vision (Eds. H. I. Christensen and P. J. Phillips) Vol. 51: Automatic Diatom Identification
(Eds. H. duBufandM. M. Bayer)
Vol. 52: Advances in Image Processing and Understanding A Festschrift for Thomas S. Huwang (Eds. A. C. Bovik, C. W. Chen and D. Goldgof) Vol. 53: Soft Computing Approach to Pattern Recognition and Image Processing (Eds. A. Ghosh and S. K. Pal) Vol. 54: Fundamentals of Robotics — Linking Perception to Action (M. Xie)
"For the complete list of titles in this series, please write to the Publisher.
Series in Machine Perception and Artificial Intelligence  Vol. 5
FUNDAMENTALS OF
ROBOTICS
LINKING PERCEPTION TO ACTION
Ming Xie
SingaporeMIT Alliance & Nanyang Technological University, Singapore
U S * World Scientific
« • New Jersey • London • Singapore • Hong Kong
Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: Suite 202, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
Library of Congress CataloginginPublication Data Xie, M. (Min) Fundamentals of robotics : linking perception to action /Ming Xie. p. cm. — (Series in machine perception and artificial intelligence ; v. 54) Includes bibliographical references and index. ISBN 9812383131  ISBN 9812383352 (pbk.) 1. Robotics. I. Title. II. Series. TJ211.X84 2003
629.8'92dc21
2003043284
British Library CataloguinginPublication Data A catalogue record for this book is available from the British Library.
Copyright © 2003 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
Printed by Fulsland Offset Printing (S) Pte Ltd, Singapore
Dedicated in memory of my dear mother. To David and Sophie.
Preface
Purpose The word "Robot" comes from the Czech word "Robota" which means "labor doing compulsory manual works without receiving any remuneration" or "to make things manually". Oxford dictionary defines "Robot" as "a machine resembling a human being and able to replicate certain human movements and functions automatically." These days, more and more robot is living up to its name. Gone is the time when robots are merely mechanisms attached to controls. Today's robots are a combination of manipulative, perceptive, communicative, and cognitive abilities. They can be seen welding heavy machinery or repairing nuclear power plants; they can be seen crawling through debris underwater or crawling across the craters on Mars; they can be seen demining military zones or cleaning windows on high buildings. Today's robots are capable of so many tasks. Yet, there is so much more on the horizon. Tomorrow's robots, which includes the humanoid robot, will tutor children, work as tour guides, drive humans to and from work, do the family shopping. Tomorrow's robots will enhance lives in ways we never dreamed possible — Do not have time to attend the decisive meeting on Asian strategy? Let your robot go for you and make the decisions. Are not feeling well enough to go to the clinic? Let Dr. Robot come to you, make a diagnosis, and get you the necessary medicines for treatment. Do not have time to coach the soccer team this week? Let the robot do it for you. Tomorrow's robots will be the most exciting and revolutionary thing to happen to the world since the invention of the automobile. It will change the way we work, play, think, and live. Because of this, nowadays robotics is one of the most dynamic fields of scientific research. These days, robotics
vii
viii
The Fundamentals of Robotics: Linking Perception to Action
is offered in almost every university in the world. Most mechanical engineering departments offer a similar course at both the undergraduate and graduate levels. And increasingly, many computer and electrical engineering departments are also offering it. This book will guide you, the curious beginner, from yesterday to tomorrow. I will cover practical knowledge in understanding, developing, and using robots as versatile equipment to automate a variety of industrial processes or tasks. But, I will also discuss the possibilities we can look forward to when we are capable of creating a visionguided, learning machine.
Contents I have arranged the book, according to the following concerns: • Focus: I focus a great deal on analysis, and control of today's robots. I also delve into the more recent developments being made with the humanoid robot. • Educational: Whenever possible, I describe the philosophy and physics behind a certain topic, before discussing the concise mathematical descriptions. • Systems Approach: The robot is a tightlyintegrated entity of various engineering systems, ranging from mechanics, control, and information to perception and decisionmaking. Therefore, I use the systems approach to clearly organize and present the multiple facets of robotics. I follow the motioncentric theme of robotics, because motion is a visible form of action which is intrinsically linked to perception. On the other hand, I intend to stick to the original meaning of robotics, namely: the study of the robot which is shaped like a human. It is my aim that this book provides a balanced coverage of various topics related to the development of robots. Whenever possible, I relate our discussion to the humanoid robot. I have designed the flow of the chapters to be clear and logical, in order to ease the understanding of the motioncentric topics in robotics. I embrace both traditional and nontraditional fundamental concepts and principles of robotics, and the book is organized as follows:
Preface
ix
In Chapter 1, I introduce the robot, from a philosophical point of view, as well as the theme of robotics, from the perspective of manufacturing and automation. Additionally, I illustrate the motioncentric topics in robotics. I link the directing concept of taskactionmotion to perception in our discussion on artificial intelligence, and propose a new definition. In Chapter 2, I review the mathematical description of a rigid body's motion. Motion is the unifying concept in robotics which links various topics together, such as generation of motion, modelling and analysis of motion, control of motion, and visual perception of motion. Here, I unify the mathematical notations related to geometrical entities under consideration in both kinematic analysis and visual perception of motion. In Chapter 3, we study the pure mechanical aspect of a robot. We also cover, in detail, fundamental topics on mechanism and kinematics. I introduce a simple illustration on DH parameters, as well as a new term, called the simple open kinematicchain. This new term is useful for the kinematic modelling of the humanoid robot. Finally, I include a new scheme for solving inverse kinematics, called discrete kinematic mapping, which complements the classical solutions. In Chapter 4, we study the electromechanical aspect of a robot. After I discuss the origin of the rigid body's motion, I introduce actuation elements at a conceptual level. I also introduce a new concept of the dynamic pair and discuss the solutions of kinetodynamic couplings. I propose a new scheme on onetomany coupling (i.e. a single actuator for all the degrees of freedom) in comparison with the traditional scheme of onetoone coupling (i.e. one actuator per degree of freedom). In the latter part of the chapter, I focus on the fundamentals of establishing equations of motion, which relate force/torque to motion. I also mention the study of robot statics. In Chapter 5, we study the control system of a robot. First, I introduce the basics of the control system. Then, I focus on the control and sensing elements of the system, especially on how to alter the direction of motion, how to regulate the electrical energy applied to the electric motors, and how to sense motion and force/torque variables. I discuss at an appropriate level the aspects of designing robot control algorithms in joint space, task space and image space. In Chapter 6, we study the information system of a robot. Traditionally, this system aspect of the robot has been overlooked in robotics textbooks. Although an information system is not too important to an industrial robot, it is an indispensable subsystem of the humanoid robot. Here, we emphasize the basics of the information system, in terms of data processing, storage
x
The Fundamentals of Robotics: Linking Perception to Action
and communication. We study the fundamentals of computing platforms, microcontrollers and programming together with a conceptual description of multitasking. I also include discussions on various interfacing systems typically used by a microcontroller (i.e. I/O). In Chapter 7, we study the visual sensory system of a robot. It is a common fallacy among novices that it is simple to form a vision system by putting optical lenses, cameras and computers together. However, the flow of information from the light rays to the digital images goes through various signal conversions. Thus, a primary concern of the robot's visualsensory system is whether or not the loss of information undermines the image and vision computing by the robot's visualperception system. I start this chapter with a study of the basic properties of light and human vision. Then, I focus on the detailed working principles underlying optical imageformation, electronic imageformation, and the mathematical modelling of the robot's visual sensory system. I also cover the important topics of CMOSimaging sensors, CCDimaging sensors, TV/Video standards, and imageprocessing hardware. In Chapter 8, we study the visualperception system of a robot. I keep to the motioncentric theme of robotics while focusing on the fundamentals underlying the process of inferring threedimensional geometry (including posture and motion) from twodimensional digital images. This process includes image processing, feature extraction, feature description and geometry measurement with monocular & binocular vision. When discussing image processing, I introduce the principle of dynamics resonance. It serves as the basis for explaining various edgedetection algorithms. I discuss in detail a probabilistic RCE neural network which addresses the issue of uniformity detection in images. In our study of binocular vision, I present two new solutions which cope with the difficult problem of binocular correspondence. Finally, I provide a theoretical foundation for further study of a robot's imagespace control. In Chapter 9, we study the decisionmaking system of a robot. Decisionmaking is indispensable in the development of autonomous robots. After I introduce the basics of decisionmaking, I discuss the fundamentals of task and action planning, at a conceptual level. However, I emphasize motion planning. We first study motion planning in task space, and introduce a new strategy, known as backwardmotion planning. Then, we study imageguided motion planning in task space, discussing in detail, a unified approach, called qualitative binocular vision. I cover three important examples of imageguided autonomous behavior: a) handeye coordination,
Preface
xi
b) headeye coordination, and c) legeye coordination. Acknowledgement First of all, my sincere gratitude goes to: • Professor Dhanjoo Gista (Ph.D, Stanford), one of the founders of the scientific discipline of bioengineering, for his encouragement and initiative in persuading me to consolidate my teaching materials into this textbook. Without his moral support, this book would not have been possible. • Professor Horst Bunke, the Series Editor for World Scientific Publisher, for his kindness in not only reviewing this book, but also including it in Series in Machine Perception and Artificial Intelligence. • Mr. Ian Seldrup and Ms Tan Rok Ting, Editors at World Scientific Publisher & Imperial College Press, for providing me friendly editorial advice, and the final proofreading of the manuscript. • Dr. Hui Hui Lin (IBM) and his lovely wife, Jana, who have tirelessly helped in improving the readability of the manuscript and correcting errors. I would also like to express my gratitude to: • Nanyang Technological University and the SingaporeMIT Alliance, for providing me not only with a conducive environment to complete this book, but also generous research grants for research staff and various related projects. • My research staff and research students for their assistance in graphic works and experimental results. Finally, thanks to my wife and children for their love, support, and encouragement. M. Xie September, 2002
xii
The Fundamentals of Robotics: Linking Perception to Action
Note to Instructors This book can serve as a textbook for a simple, onesemester course on robotics, provided you do not want to delve into vision, or a twosemester course which highlights vision as well. I have designed this textbook specifically for undergraduate and graduatelevel students in engineering. It can also serve as a supplementary text for students having an interest in robotics, vision, and artificial intelligence.
Contents
Preface 1. Introduction to Robotics 1.1 Introduction 1.2 Manufacturing 1.3 Factory Automation 1.3.1 Automation and Robots 1.4 Impact of Industrial Robots 1.5 Impact of Humanoid Robots 1.5.1 Industry 1.5.2 Society 1.5.3 Space Exploration and Military 1.5.4 Education 1.5.5 Research 1.6 Issues in Robotics 1.6.1 Mechanism and Kinematics 1.6.2 Actuation Elements and Dynamics 1.6.3 Sensing Elements 1.6.4 Control 1.6.5 Information and DecisionMaking 1.6.6 Visual Perception 1.7 Exercises 1.8 Bibliography 2. Motion of Rigid Bodies 2.1 Introduction
xiii
vii 1 1 1 5 7 10 14 14 14 17 17 18 21 21 22 23 23 23 24 26 26 27 27
6.1.1 Links 3.9 Bibliography 3.2 Determination of Joint Velocities 27 29 30 31 34 34 35 38 40 41 43 46 49 52 53 57 57 58 59 59 59 59 63 64 67 69 71 71 74 76 78 80 80 81 82 86 87 92 .6.2.2 Cartesian Coordinate Systems 2. .6 Equivalent Axis and Angle of Rotation 2.3.8 Exercises 2. 3.3.1 Determination of Joint Displacements .1.1.3 Inverse Kinematics of Open KinematicChains .5 Rotational Motions 2.1 Introduction 3.2 Circular Velocity and Acceleration 2.2 Differential Homogenous Transformation 2.3.2 Linear Velocity and Acceleration 2.2.4.1 Kinematics of a Link 3.2 Robot Mechanism 3.1 Linear Displacement 2. .6.3.2 Joints 3.4 Multiplicity of an Open KinematicChain 3.2.6.1 Assignment of a Link's Frame 3.5. .3.3 Successive Elementary Translations 2. .2 Geometric Parameters of a Link 3.4 Linear/Circular Velocity of a Link 3.6 Composite Motions 2.3.2.2 Forward Kinematics of Open KinematicChains . .3.5 Euler Angles 2.3 Kinematic Chains 3.3.3.5 Angular Velocity of a Link 3.4 Translational Motions 2.6.1. . .5. 3.1 Circular Displacement 2.3 Robot Kinematics 3.3.xiv The Fundamentals of Robotics: Linking Perception to Action 2.3 Motion Transformation Matrix of a Link .2 Determination of the Endeffector's Velocity 3.3.3.1 Homogenous Transformation 2.1.3.1 Determination of the Endeffector's Posture 3. Mechanical System of Robots 3. 3.4. 3. .3 Projective Coordinate Systems 2.4 Successive Elementary Rotations 2.2.6.3. .2.7 Summary 2.
2 Discrete Inverse KinematicMapping .1.4.4 Summary 3.5 Exercises 3. .4 Dynamic Pairs and Chains 4.2.8 Origin of Motions 4.6 Bibliography 4.3. .2.2.2 Electric Stepper Motors 4.3 Torques 4.3 HarmonicDrive Devices 4.4.4 Effect of Singularity 3.2.1.1 Energy Conservation in a System 4.3 Actuation Elements 4.4 Formation of a Robot's Electromechanical System 4.3.1 Force and Torque Generators 4. .Contents xv 3.3.1 Discrete Forward KinematicMapping .2 PulleyandTiming Belt Assemblies 4.2.2 Forces 4.1 Working Principle of Electric Motors .2 OnetoMany Couplings 4.2.2 Origin of a Rigid Body's Motion 4. .3.2. 3.3.2.1 Gear Mechanisms 4.3.2.3 Numerical Solution for Joint Displacements 3. .3 Brushtype DC Motors 4.1 OnetoOne Couplings 4. .4.3.1.3.2.4.4.4.2 Parallel Splitter of Motion 4.3.3.4 Torque at Inertial Loads 100 102 104 106 107 108 109 110 112 113 115 115 115 116 117 119 120 122 123 124 124 125 126 131 134 136 137 139 141 142 143 144 145 147 148 150 151 153 .4. .2.2.1.3.2 Force and Torque Amplifiers 4.7 Kinetic Energy 4.1 Introduction 4. .3.1 Motion Distributors 4.4 Application of Discrete KinematicMapping 3.2.2.2.6 Potential Energy 4.4.4 Brushless DC Motors 4.5 Incremental Works 4. 3. .4.4. Electromechanical System of Robots 4.3.3 Serial Splitter of Motion 4.3.3 Bidirectional Discrete KinematicMapping . 3. 4.3.3.4 Discrete Kinematic Mapping 3.3.3.
2.2 154 155 156 157 158 160 161 161 162 164 165 166 167 168 172 173 174 174 176 176 177 177 178 179 179 182 182 183 188 194 195 197 199 199 200 200 201 .6.6.1 4.1 D'Alembert Principle 4.6.3 4.5.4 4. .6.4.6.5 Robot 4.xvi The Fundamentals of Robotics: Linking Perception to Action Open KinetoDynamic Chains Statics Dynamic System of Particles Generalized Coordinates and Forces Constraints of a Dynamic System Virtual Displacements Virtual Works Principle of Virtual Work for Statics Statics Against SelfInertial Loads Statics Against Inertial Loads at Endeffector .2.6.8 Exercises 4.6 Equation of Angular Motion of a Link .3.4.1 Introduction 5.6. .3 Equations of Motion 4.2 Lagrange Formula 4.3.3 NewtonEuler Formula 4.2 Potential Energy of a Rigid Body 4.3.3.4.3. .6. .2 Closedloop Control Scheme 4. .3 4. .1 Kinematic Parameters of a Link 4.5 4. .7 4.4.6.6.1 Motion Equations of a Rigid Body 4.6. 4. .7 Recursive Algorithm for Forces/Torques .5.2 AutomaticFeedback Control System 5.2. Dynamics Dynamic System of Rigid Bodies Dynamics of a Rigid Body 4.7 Summary 4. Control System of Robots 5.3 Sum of Forces of a Link 4.6.6 4.6.8 Recursive Algorithm for Velocities and Accelerations 4.6.5.6 Robot 4.1 System Concept 5.3.2 Dynamic Parameters of a Link 4.4 EulerLagrange Formula 4.9 Bibliography 5.5. 4.5.1 4.3.8 4.2 4.5. .2.5.3 Kinetic Energy of a Rigid Body 4.5.4 Sum of Torques of a Link 4. 4.6.3.6.5 Equation of Linear Motion of a Link .6.2.6.
2 Power Drives 230 5.3.2.1 Issues 266 5.1 Power Switches 223 5.4 5.1 Dynamics of a Robot's Mechanism 268 5.4.3 Detection of a Human's Presence 245 5. 228 5.3 Power Switches for Brushless DC Motors .2 Power Switches for Brushtype DC Motors .3 Absolute Optical Encoders 250 5.1 Power Switches for Stepper Motors 224 5.5.5.1 Indirect Measurement 259 5.2 Planning of Desired Outputs 267 5.4 Sensing Elements 239 5.4.2.3.2.4.4 Sensory Feedback 274 5.3.3.2.4.3.2 PWM Power Drives 235 5.1.2 Potentiometers 248 5.4.3 A Robot's System Dynamics 268 5.2.3 Control Elements 221 5.3.4.2 Safety Sensors 242 5.3 Digital Motion Controllers 237 5.1 Generic Sensing Principle 240 5.3.2 Proximity Sensing Systems 243 5.2.2 Direct Measurement 259 5.4.6.2.3.6 .4.3.Contents xvii System Dynamics 204 Transfer Functions 206 System Performance 209 Analysis of Absolute Stability 213 5.1 Placement of Motion Sensors 247 5.4.1 Emergency Stop Sensors 243 5.4.3 Motion Sensors 246 5.4.1.5.4.2.2.3 5.4 Incremental Optical Encoders 254 5.3 Dynamics of Electric Motors 271 5.1 RootLocus Method 213 5.7 Tuning of PID Control Algorithms 217 5.2 Dynamics of Speed Reducers 269 5.3.2.4.4.3.5 Control System Design 266 5.1 Linear Power Drives 232 5.5. 227 5.3.5 5.2 Lyapunov's Method 215 5.5 Tactile Sensors 264 5.2.3.6.4 Force/Torque Sensors 258 5.1.3.4.4.3.5.5.5.2.
4.7.4 Independent Joint Control Scheme 5. .2.4.7 TaskSpace Control 5.4 Hybrid Force and Trajectory Control .6. .3.6.1 Planning of Desired Outputs 5.1 PID Control Laws 5.4.7.2. .2.3 Computational Tasks 6.5.3 Imitating the Mind 6.5.2 Variable Structure Control 5.7.8 Bibliography 6. .5. 5.4.1 Planning of Desired Outputs 5.2 Imitating the Brain 6.2.1 Digital Computers 6.8.5 Impedance Control 5.6 CyclebyCycle Operations 6.1 Autonomous Actors or Agents 6.3 Arithmetic Logic Unit (ALU) 6.5.2 Autonomous Behaviors 6.2 Objectives 5.4 Bus 6.2 Number Representation 6.3 Methodologies 5. 5.8 ImageSpace Control 5.3 Centralized Joint Control Scheme 5.5.4.5.5.7.2 System Dynamics 5.5.5.2 Microprocessors 6.7.5.5.5 276 277 279 283 283 285 286 288 290 291 291 293 293 295 297 297 298 298 299 301 302 303 303 304 306 306 307 309 310 311 312 312 314 315 315 316 317 319 .6 JointSpace Control 5.2.5.5.5.5.1 Introduction 6.3.5.7 Generic Architecture of a Microprocessor .7 Exercises 5. Information System of Robots 6.6.4.8.2.3.4.6 Summary 5.5.4.4 DataProcessing Hardware 6.5.5.1 Scenarios 5.5.5 Generic Architecture of Computers 6.1 Memory 6.2.xviii The Fundamentals of Robotics: Linking Perception to Action Control Algorithms and Performances 5.6.4.3 Force Compliance Control 5.2 Posture Control in Task Space 5.8.
.1 Basic Concepts 6.1 Device Control with Parallel I/O Systems .5.4.7.7.1 Storage Devices 6.6. .7. .3 A/D Converters 6.7.4.5.3 Program/Data Files 6.5 Serial I/O 6.3 Strobed Parallel I/O 6.7.7. .2 Bitoriented Data Framing 6.4 Asynchronous Serial I/O 6.1.5.2.2 A File System 6.7.2 Cooperative MultiTasking 6.7.7.3 Data Storage Using Unformatted Files 6.1 Interfacing for Data Communication . .5 Data Storage Using Formatted Files 6.7.6 Programmable Timers 321 322 323 323 324 325 325 327 334 334 338 341 342 343 344 344 345 346 347 349 349 349 350 351 352 353 354 355 357 357 358 359 360 361 362 364 365 367 371 . 6.1.1.1.1. .4.5 DataProcessing Software 6.1.4 Parallel I/O 6.5.4 Data Storage Using a File System 6.1.Contents xix 6.6.5.2.5.4.5 The Basics of CProgramming Language .7.7.3 RealTime Operating Systems 6.4. .5. 6.7.1.5.1 Programming Language 6.7.6.5.2 Interfacing for Data Communication .3 Synchronous Serial I/O 6.2 I/O Ports 6.1 I/O Registers 6.7.3 Microcontrollers 6.7.2.6 Data Storage and Retrieval 6.6.5.3 Communication Networks 6.7. 6.5.5.7. 6.1 Programming 6.2 D/A Converters 6.4 Common Issues 6.4 Programming Procedure 6.6.2 MultiTasking 6.1. . 6.5 Procedure of Programming I/O Systems .1.2 Programming Environment 6.5.3 Preemptive MultiTasking 6.7.4 Full Handshake Parallel I/O 6.5.5.7 Data Interfacing and Communication 6.1 The Basics of MultiTasking 6.
5.4 Peripheral Vision 7.2.3 Light Splitting & Color Filtering 7.4.1 Formation of Optical Images 7.2.2 Digitization of Analogue Videos 7. Visual Sensory System of Robots 7.5 Modelling of Digital Images 7.5.1 Analogue Videos 7.4. .3 Selfgenerating Realtime Interrupt 6.5.4.7.5.3 CCD Imaging Sensors 7.4.3. .6.3 Formation of Digital Images 7.2.2 Assignment of Camera Coordinate Systems 7.10 Bibliography 7.1 Introduction 7.3. .5.2 CMOS Imaging Sensors 7.1 Representation in RGB Color Space 7.3.5.xx The Fundamentals of Robotics: Linking Perception to Action 6.2. 373 373 374 374 376 377 379 379 379 380 386 389 392 392 394 395 396 396 397 398 400 401 404 404 404 413 416 417 423 426 426 426 427 428 429 430 430 430 .3 Central Vision 7.2.1 Pinhole Model and Perspective Projection .1.1 Active Signal Output 6.2 Geometrical Properties 7.6. 7.2 Representation of Intensity Images 7.4.3.1 Chromatic Modelling 7.1.1 OpticalImage Focusing 7.4.1 Photoelectric Effect 7. 7.2.2 Photosensitive Cells 7.3.4.7.2.3 Refraction of Light Rays by a Thin Lens 7.1 Eyeballs 7.3.2 Active Signal Capture 6.3 Assignment of Image Coordinate Systems .4 Determination of Image Coordinates .1.2 Formation of Electronic Images 7.3 The Basics of the Human Eye 7.2 The Basics of Light 7.7.5.1.1 Physical Properties 7.2.4.6.2 Geometric Modelling 7.4.2.2.4 Digital Image Acquisition 7.5.4.8 Summary 6.9 Exercises 6.1.2 Control of Incoming Light 7.4.
1 Frequencydomain Transforms 8.1.2 Image Filtering 8.5.. .4 Image Feature Extraction 8.3 Integral of Convolutions 8.3 Criteria for Feature Detection .3.6 7.3..5 Determination of Index Coordinates 7.3 Image Processing 8.4. .2.3 Applications of Visual Perception 8.3 A General Scheme for Edge Detection .6.4. .2 Derivative of Convolutions 8.2 Attributes of Visual Perception 8.1 A Process of Visual Perception 8.4 Spatial Displacement of Convolutions .1 Introduction 8.4.2. 8.2 Timedomain Transforms 8.3.4 Information Processing in Visual Perception 8.4.1 Image Transformation 8.2.2. 8.2 DSP Processors 7.9 7.1.2.4.3.1 The Basics of Feature Detection 8.. 8.': 8. .3.3.2.2.2.2.6..1.2 Edge Detection '.2 Types of Edges 8.1.2.2 A Generic Procedure for Feature Detection .1.1 Feature Definition 8.6 Intrinsic Parameters of Cameras Digital ImageProcessing Hardware 7.7 7.Contents xxi 7. 8.3. .4.4 Sobel Edge Detector .1 Host Computers 7.2 The Basics of Visual Perception 8.2.4 Parallel Computing Platforms Summary Exercises Bibliography 433 434 435 436 436 438 439 441 442 444 445 445 446 446 447 448 450 453 453 453 454 457 457 458 461 463 464 465 465 466 467 468 471 472 472 475 476 8.2.6.8 7.3.3. .6.1 Definition of Edges 8.4.2. Visual Perception System of Robots 8.2.4.1 Convolutions 8.5.4.3 Pipelined Computing Modules 7.3 Spatial Transforms 8.1.
6.1 Forward ProjectiveMapping 479 483 486 489 490 492 496 496 497 497 500 508 509 510 513 513 514 515 516 518 518 520 522 523 524 525 526 528 528 529 530 530 532 534 538 540 546 547 .2. . .5.1 Definition of Image Templates 8.5.4.6.3.2.2 CircularArc Approximation 8.3 Inverse ProjectiveMapping 8.3 Corner Detection 8.5.5.5.1.3.6.1 Types of Neighborhoods 8.5.2.5.2 A General Scheme for Uniformity Detection 8.4.3.4.3 EllipticArc Approximation 8.1 The Basics of Feature Description 8.2 Zerogradient Corner Detector 8. .5.4 Spatial Uniformity Detection 8.1.4.1 Monocular Vision 8.6.2 Binocular Vision 8.4.5.5.4.1 Successive Linear Curves 8.4.4.4.6 Temporal Discontinuity Detection 8.5.4. 8.xxii The Fundamentals of Robotics: Linking Perception to Action 8.4.4.4.6.3 Boundary Tracing 8.6 Canny and Deriche Edge Detectors 8.4.3.5 Gaussian Edge Detector with Tunable Response and Localization 8.4 Edge Linking 8. 8.6.2 Successive Nonlinear Curves 8.2 Feature Grouping 8.2 Simulation of Monocular Vision 8.2.7 Summary of EdgeDetection Algorithm .2.5.1.1 Definition of Image Regions 8.5.2 Template Matching 8.6 Geometry Measurement 8.4.4.5.1 Forward ProjectiveMapping 8.5.1.1 LineSegment Approximation 8.2.3 Color Detection 8.6.5 GeometricFeature Description 8.1 Definition of Corners 8.4 Camera Calibration 8.2.5 Temporal Uniformity Detection 8.4.5 Determining the Parameters of Cameras .4.3 Other Corner Detectors 8.4 Curve Fitting 8.4.3.2 Pixel Clustering 8.4.6.4.1.3 Contour Splitting 8.2.
4 Neural Network 9.1 Uncertainty 9.3. Inverse ProjectiveMapping Unresolved Issue: Binocular Correspondence Continuous EpipolarLine Constraint .1.3 Decision Making for Autonomous Behaviors 9.4.4 Difficulties in DecisionMaking 9.2.3.6.6.3 Fuzzy Inference 9. .3 8.3.3 The TaskPlanning Process 9.2.3.3 The MotionPlanning Process 9.2.5.2.3 DecisionMaking Processes 9.1 An Action Description as Input 9.3.3 Motion Planning 9.3.6.3.4 A General Framework for Automated Planning .3. .4 8.2.3.2. .3. .2 An Action Sequence as Output 9.2 Action or Behavior Planning 9. .1 Task or Scenario Planning 9.2.9 Bibliography 9.2.2 The Basics of Decision Making 9.2 The Key to Automated Behaviors 9.3.2 8.7 Summary 8.2 Redundancy 9.2.3.2 Statistical Inference 9.2.5. .2 A Motion Sequence as Output 9. 549 553 554 559 562 566 568 570 573 573 574 574 575 576 577 579 580 580 581 582 582 583 585 587 588 589 589 591 591 592 593 594 594 596 596 597 597 600 DecisionMaking System of Robots 9.2.2.2.3.2. . Discrete EpipolarLine Constraint Differential EpipolarLine Constraint .5 Methodologies in DecisionMaking 9.1 The Key to Automated Actions 9.2.3.1.2.2.3.5. .1 Expert Knowledge 9.2 A Task Sequence as Output 9.1 Inputs of DecisionMaking 9.1 A Goal Description as Input 9.2.5.1 A Task Description as Input 9.3.2.6.4.2. .3.3.1 Introduction 9.2 Outputs of DecisionMaking 9.6 8.5 8.2.3 The ActionPlanning Process 9.6.Contents xxiii 8.8 Exercises 8.1.
1 A Discrete and Normalized Workspace .6.1.xxiv The Fundamentals of Robotics: Linking Perception to Action 9. .2.5.2 ImageGuided Target Following 664 9.6.2.3 A CollisionFree Path 614 9.4 Constraints of Robots 616 9.7 Summary 667 9.2.3 Formation Planning 608 9.6.2 Backward Planning 605 9.1.2.4 Iterative Approaches 656 9.5.1 Input 629 9.1 ImageGuided Road Following 661 9.3.4.1 Forward Planning 604 9.1 Input 645 9.1 Planning of CollisionFree Paths 611 9.3 A Closedform Solution 648 9.5 Motion Planning in Task Space 610 9.2.2 HeadEye Coordination 644 9.4.2 Motion Description 618 9.1 HandEye Coordination 629 9.6.2.5.4.5.5.6.4 Trajectories 622 9.8 Exercises 669 9.3 LegEye Coordination 660 9.2 MotionPlanning Strategy 604 9.6.2 Output 647 9.2.6 ImageGuided Motion Planning and Control 626 9. .3 Paths 621 9.1.3 A Closedform Solution 632 9.4.6.4 An Iterative Approach 632 9.2.2.6.1.5.5 Interpolation of Orientations 624 9.5.2 Circular Curves 619 9.2 Output 631 9.6.4 The Basics of Motion Planning 602 9.2 A Path Map 613 9.2.6.2.1.1 Linear Curves 619 9.6. Prospects Index 673 677 . 611 9.5.1.6.1.1 Path and Trajectory 603 9.5.1.9 Bibliography 670 10.5.4.5.6.2.3.
by making use of reprogrammable machine tools. ceramics. the literal meaning of manufacture is. the discovery of engineering materials. This creativity led to the invention of tools. Most importantly. In the later part of the chapter. property. I discuss the major concerns of robotics. I also describe a framework from which to understand human intelligence. A direct consequence of the activity of "making things by hand" was l . Thus. in making things by hand.1 Introduction This chapter gives an overview of robotics. which serves as a convincing basis to justify the usefulness and importance of robots in industry. "to make by hand" either directly. the engineering discipline which focuses on the study of robots. For those who are actively undertaking research in the area of artificial intelligence.2 Manufacturing The word manufacture comes from the combination of two Latin words. unifying theme. manus (hand) and factus (to make). 1. or indirectly. fuelled people's creativity in inventing various processes for making "things" of different complexity. which made the process of "making things" much easier and more efficient. and polymers. Since ancient times. such as metal.Chapter 1 Introduction to Robotics 1. by producing handicrafts. This. and scale. with or without the help of tools. motion. in turn. for the purpose of illustrating the simple. enlarged the scope of "things" which can be made by hand. I will introduce relevant topics from the angle of manufacturing. our ancestors exercised their creativity to the fullest. An understanding of this theme would make the rest of the chapters easier to read.
1. and parts or components. and appearance of materials for the production of goods of increased value. the exchange of goods or services. a formal definition of manufacturing can be stated. which goes into a dimension far beyond geographical. Part or Component ") Equipment  N \ 1 Manufacturing Processes / . With the advent and sophistication of finance and monetary systems. as follows: Definition 1. are still the same. If manufacturing is the art and science of "making things by hand.2 The Fundamentals of Robotics: Linking Perception to Action (T Initial Material. and cultural boundaries. Refer to Fig. Without exchange. commerce has steadily reached a scale. there would be no manufacturing. equipment.1.1 Manufacturing is the application of processes which alters the geometry. A material. which is measured by the rate of output products over a fixed period of time. This interaction results in energy consumption. property. This led to the creation of commerce. social. social. there would be no commerce. the basic fundamentals.1 Illustration of basic functional modules in manufacturing." directly or indirectly. Energy Labor n / l p ' Yl (^" Final Product of Increased Value J ) Fig. without commerce. which is a platform for stimulating the production of wealth for people and nations. Regardless of the scale and dimension of commerce. material. It is not exaggerating to say that today's commerce is the motor which drives all economic. The production of . Growing commercial activities have undoubtedly pushed up the demand for product quality and manufacturing capacity. and cultural activities. that craftsmen were able to produce a surplus of goods which far exceeded their needs. part or component is transformed from the initial state to the final state of product through the interaction among labor. 1. people began to exchange the surplus of one type of goods for the surplus of another type of goods. As a result.
components etc. Depending on the modes of interaction.. A process is called a serial process if the interaction occurs locally on a material. a process can be either a serial process or a parallel process. or electrical domains. chemical. construction equipment. automobiles. produced by Manufacturers. Capital products are used in manufacturing as facilities for the production of goods. devices. thermal. parts. TV. However. composite materials. and ceramicpolymer composites. part. parts. depending on the outcome of the interaction. such as metalceramic composites. The latter are products purchased by companies. processed materials. and input materials. rocks. The combination of these three basic engineering materials forms another category called. the interaction in a manufacturing process will consume energy in the mechanical. mainframe computers.Introduction to Robotics 3 goods in large volume has propelled manufacturing to evolve into a rigorous scientific discipline covering the following important aspects: Products Final products. A manufacturing process will change the geometry. beverages etc. machine tools. parts. Processes A manufacturing process consists of the interaction among labor. or components. or appearance of the input materials. Thus. such as personal computers. or component while a process is called a parallel process if the interaction occurs globally across a material. There are three basic categories of engineering materials. part or component. property. such as industrial robots. equipment. ceramics. Materials In addition to natural materials. can be classified into two categories: a) consumer products and b) capital products. bamboo. The former are products purchased directly by consumers. a process can fall into one of the four categories: . household appliances. and polymers. such as wood. namely: metals. video recorders. petroleum etc. metalpolymer composites. stones. or components. foods. the advent of engineering materials has undoubtedly enlarged the scope of things which can be manufactured.
(4) Deformation Process: This refers to the operation of altering the shape of a material. soldering etc. 1. property.4 The Fundamentals of Robotics: Linking Perception to Action (1) Removal Process: This refers to the operation of removing certain portions of input materials. die stamping etc. • Manufacturing Support System: This includes inbound logistics for ordering materials. heating etc. process equipment. Functionally. rolling. (2) Addition Process: This refers to the operation of adding other materials to input materials. or component. part. Equipment In order to satisfy the demand for high quality and high volume. Typical examples include die forging. parts. components and facilities. On the other hand. bending. coating. Examples include cutting. part or component through the application of either mechanical or thermal energy. metrology equipment. or joining parts or components together. (3) Solidification Process: This refers to the operation of creating objects through the transformation of thermal states of materials. assembly. Factories The manufacturing industry consists of factories and organizations which produce or supply goods and services. it is necessary to use machine tools or automated equipment to operate manufacturing processes. A factory is typically composed of the following entities: • Production and Automation System: This includes the factory layout. Refer to Fig. grinding. outbound logistics deals with distribution of the final products. 3D printing. the role of manufacturing equipment is to control the interaction in a manufacturing process which will alter the geometry. Examples include painting. materialhandling equipment and labor. or appearance of the initial material. . welding.2. A typical example of the solidification process is injection molding.
f ^\! V^Suppliers_^ V ^ ^ u r i l i e j s ^ ^ i l V ^ " ^ ™ ^ ^ ^Consumers^^ j .. • Design & Simulation j i j I proceSs Equipment I I Metrology Equipment Material Handling Equipment I Manufacturing Processes & Transportation I I Manufacturing Support System I ! Inbound Logistics iI 2k ^f Outbound Logistics " j j I . I j 1 I j : I I I ' Process Control ' ' Equipment Control ' ' Equipment Interface Material Control '  Manufacturing Execution System  !   I i I  I Manufacturing System I Design & Simulation ) •  Product/Process ^ • • . material flow control.—•. equipment control and deployment of labor. • Enterprise Information and Management System: This includes the management of resources.. It is possible to increase the production rate .^zzJg^=z^:. • Manufacturing Execution System: This includes process control.^j^L^. Facilities and Business Operation  [ j Manufacturing Design Enterprise/SupplyChain Design & Simulation  j j .L— —. facilities and business operations. \/^"^Materiii N.— . „ • .—^. yet most expensive way is to increase the number of production lines..3 Factory Automation There are two ways to achieve high yields in manufacturing. ^^ComponenT^X ! j f ~~N. The simplest. An alternative and more desirable way is to increase the rate of production in the existing production lines. • Manufacturing Design System: This includes product design & simulation. process design & simulation and supplychain design & simulation. . Fig.Introduction to Robotics 5 Management of Resources.y. 1.. 1.2 Outline of a factory in manufacturing.
. by reducing the cycle time needed to produce a single part or product. ^  \^TLJ / Sampling/Measurement ^J Monitoring Fig.3. There are also two ways to reduce cycle time. which operate at different rates.. the quality control can be achieved by first monitoring the output variation of a process. JL Inputs ~T* Proces/ Modeling [Outputs I *" I Feedback 1 Statistical Process Control f. Refer to Fig. one important aspect of factory automation is to automatically regulate or control the significant input variables of a process. 1.6 The Fundamentals of Robotics: Linking Perception to Action •°P"" » ^ — . and equipment control in manufacturing.Xn)} to its output Y = {(Yi. If we regulate or control these significant input variables. Ym)}. Therefore.. and then automatically acting on a process's inputs for the purpose of . . The second approach is to automate the manufacturing process by using reprogrammable and automatically controlled equipment. the dynamics of a process can be optimized. 1. Therefore.. Process control involves two cascaded control loops. which aims at maximizing product quality. which consists of establishing a mathematical description that relates a process's input X = {(X\.3. We can identify the subset of significant input variables by using the scientific method.3 Illustration of process optimization. The outer loop is the statistical process control. called the design of experiments. 1.. quality is inversely proportional to the output variation . *• CyclebyCycle Process Control . time responsiveness etc. process optimization always starts with process modelling.. . robustness.Y2.X2.. in terms of stability. process control. FT t/O *' I O?~~»' Equipment Control . The first approach is to improve the manufacturing process. By definition. As shown in Fig." ^ " ^ Disturbance / P r o c e s s / I ^/ °^% ^^~~~~^^^^ Significant inputs Process Optimization ^ 1 A lO1 I.
This interaction results in energy consumption in mechanical.3. In general. property.2 A mechanism is a set of (mechanical) elements arranged in certain configurations for the purpose of transmitting motions in a predetermined fashion. or components. A typical example of a machine is a motorcycle or car. Its control cycle will be longer than that of equipment control. Motion is the visible form of mechanical energy. As I mentioned before. the rate of equipment control depends on the dynamics of the process. If a system includes at least one element intended for energy conversion. 1. chemical or electrical domains. . measurement. A mechanism is not a machine. parts. the role of labor in the factory has shifted. this system is called a machine. is equipment control. A formal definition of mechanism is as follows: Definition 1. In this way. Therefore. in a process control. which aims at altering in a controllable manner. An element which converts electrical energy into motion is called an electric motor or actuator. The aim of automation is to eliminate the direct involvement of labor in the process interaction. thermal. The manifestation of energy in the mechanical domain takes the form of motions which can be altered by using a mechanism. the role of manufacturing equipment is to control the interaction in a process.3 A machine is a superset of mechanism(s). The most typical energydriven interactions are those which convert electrical energy to mechanical energy. The inner loop. equipment control must operate in realtime. or appearance of the initial materials. a manufacturing process consists of the interaction among labor.1 Automation and Robots As we discussed above. parts. or components.Introduction to Robotics 7 minimizing this variation. the geometry. This is only achievable by setting up automaticallycontrolled equipment. from direct involvement to indirect programming and/or monitoring of automated equipment. Equipment control output is the direct input to a process. equipment. However. A formal definition of machine can be stated as follows: Definition 1. and analysis of data. and materials. and contains elements which supply energy to drive this mechanism(s). statisticalprocess control depends on sampling. An example of a familiar mechanism is the bicycle.
The word robot became popular in 1921 because of a play named "Rossum's Universal Robots" by Czechoslovakian writer. and acceleration (a). which may or may not have a human shape. A large family of manufacturing equipment among the variety which exists.7 show some prototypes developed by Japanese companies. For example. an increasingly popular type is the industrial robot. Unimation Inc. From an engineering point of view. The American company. and took control of the world. drilling etc. multifunctional manipulator designed to move material. In the play. the robot is not only gaining more popularity in industry. assembly. Nowadays. communicative.8 The Fundamentals of Robotics: Linking Perception to Action In mathematics. perceptive. is the one which supplies the motion required by a manufacturing process. founded in 1962 by Joseph Engelberger and George Devol. killed their human masters. the formal definition of industrial robot is as follows: Definition 1. parts or specialized devices through variable programmed motions for the performance of a variety of tasks. which may or may not have a human shape. spray painting. a scientist called Rossum created humanlike machines that revolted. Accordingly. and cognitive system. Figs 1.v. controlling. polishing. a robot is the embodiment of manipulative. perceptive. a new definition of robot is necessary: Definition 1. meaning forced workers. and cognitive abilities in an artificial body. it is necessary to concisely define the term humanoid robot. locomotive. velocity (v). milling. industrial robots are merely a combination of an arm and a hand. A formal definition of humanoid robot is as follows: . but also slowly entering society. These motion parameters (p. perceptive. such as: arcwelding. The English word robot was derived from the Czech word robota. Of this class of equipment.4 A robot is a programmable. in view of the evolution of the robot into a sophisticated mechanical. deburring. Karel Kapek.5 A robot is the embodiment of manipulative. electronic. was the first company to actually produce industrial robots. According to the Robot Institute of America. It can advantageously be deployed as a tool. in the form of humanoid or animallike entertainment robots.4 — 1. However. communicative and cognitive abilities in an artificial body..a) are important input variables for many manufacturing processes. to make things in various environments. motion is fully described by the parameters: position (p). cutting. informative.
tasks or processes which can typically be accomplished by using robots include: • Welding: This is the process of joining two workpieces together by applying molten weld metal. communicative and cognitive abilities in an artificial body similar to that of a human.4 PINO: An open development platform for a desktop humanoid robot.Introduction to Robotics 9 Fig. Definition 1. In this book. Photo by Author. In the manufacturing industry. • Cutting: This is the process of applying thermal or mechanical energy to cut a . an additional important motion parameter is the speed of travel. 1. for arc welding. the important motion parameter is position. locomotive. which possesses skills in executing motions with a certain degree of autonomy. robot refers to both industrial and humanoid robots.6 A humanoid robot is the embodiment of manipulative. and can be advantageously deployed as agents to perform tasks in various environments. perceptive. For spot welding.
equipment program . In this process. In this process.5 TMSUK's humanoid robot prototype. Obviously. • Material Handling: This is the process of either packaging parts into a compartment (box) or loading/unloading parts to/from another station. can advantageously be deployed in the manufacturing industry to automate many processes. the important motion parameter is position. work piece into a specific shape. the important motion parameters are position and velocity. In this process. 1. or affixing components to a base unit (e.4 Impact of Industrial Robots The industrial robot. These indirect interactions include process monitoring. Photo by Author.g. automation using robots shifts the use of human labor from direct interaction with a process to various indirect interactions.10 The Fundamentals of Robotics: Linking Perception to Action Fig. position is an important motion parameter. 1. process diagnostics. which have motion parameters as input. to place components on a printed circuit board). • Assembly: This is the process of either adding components to form a single entity. equipment setting. a combination of arm and hand.
The initial goal of a company is to earn profits for the shareholders through the manufacturing and supply of goods and/or services. faced by both developed and developing countries. There is no simple solution to this issue. However. Despite all these new job opportunities. ming. We are now witnessing the third wave of the industrial revolution: Information Technology. it can be addressed at various levels: • Individual Level: It must be clear that we are living in a changing environment. one major concern in the manufacturing industry is that the proliferation of robots will cause displacement of human labor and eventually. there should be an adjustment in the company goals . It is clear that a second dimension should be added to this initial goal. 1. individuals must constantly learn new knowledge and skills in order to make use of the modern tools being deployed in the industry.Introduction to Robotics 11 £9B^HJ$5pN&wB^ Fig. This second dimension is the social responsibility of the company to constantly shape . This is a real problem. • Industrial Level: Perhaps. unemployment. the development of automation solutions etc. The tools for us to make things are constantly evolving with increased complexity and sophistication.6 FUJITSU'S humanoid robot prototype. Therefore. Photo by Author.
• Social Level: The deployment of robots in the manufacturing industry will undoubtedly increase the productivity and quality of goods.e. Photo by Author. service. generated by the deployment of robots. For an individual to be adaptable to a changing environment or job market. 1.12 The Fundamentals of Robotics: Linking Perception to Action Fig. will generate more wealth for companies and society. • Education Level: Perhaps. on top of understanding theories in a specific field of science and technology. physics. we should reemphasize the true nature of education. This. The development of brain power should be considered more important than the memorization of theoretical facts and data. It should be a process of developing brain power in mastering knowledge and acquiring skills. We should make sure that the redistribution of wealth. and computing. in turn. is wisely regulated.7 SHARP'S Animallike entertainment robot prototype. which will keep them relevant to the current market condition and the evolution of the industry (or emergence of new industry). it is necessary that he/she adequately develop his/her brain power in philosophy. entertain . mathematics. which is increasingly dominated by knowledge and skill. the knowledge and skills of its employees into a pattern. in order to create more jobs and activities for human beings in other employment sectors (i.
Introduction to Robotics 13 ment. but are immune to the emotional states which affect human's performance. is undoubtedly one of the best types of equipment for flexible automation. Therefore. education. robotics is an important subject in engineering.) Despite social concerns about the displacement of human labor. and is being widely taught in any university having an engineering department. For example. the ability to supply customized products or services in a timely manner. price. and difficult jobs. quality and service. • Scientific Discipline: The proliferation of robots in industry reveals the importance of robotics. In other words. because the biological system cannot deliver continuous physical effort without rest. healthcare. flexible automation or agile manufacturing is the key to success in today's manufacturing industry. we cannot perform motions with any reasonable degree of accuracy or replication. without accurate visual guidance. as well: • Productivity: It is impossible to achieve a high production yield using human labor. research etc. • Quality of Human Life: The use of robots can free humans from doing dirty. the study of robots as a scientific discipline for education and research. arts. dangerous. our vision does not make metric measurements when performing tasks. Indeed. The robot. being reprogrammable equipment capable of executing various motions within a reasonably large working space. There is no doubt that automation with robots will increase productivity. Robots can not only execute accurate motions repeatedly. And. • Quality of Products: Humans are not capable of making things with a consistent degree of accuracy. Automation using robots will add a fifth characteristic: choice. • Flexibility: The winning characteristics in manufacturing today are volume. A company which wishes to offer choices to its customers must have flexibility in configuring and operating its production line. . sports. the wide use of robots in the manufacturing industry will certainly have positive impact.
diagnostics. the greatest consumer product has undoubtedly been the automobile.1 Industry The enhanced and humanlike locomotive ability of the humanoid robot makes it possible to deploy humanoid robots to places where humans are still working like machines. we may discover the following trends: • The microprocessor is becoming smaller. it is foreseeable that the emergence of the humanoid robot will certainly automate more tasks. the question is: What will be the next great consumer product? If we look at the evolution of the computer as a consumer product.5. It provides us with unprecedented mobility. which could automatically be completed by specialized teams of humanoid robots under the supervision of human masters. security etc. 1. invehicle entertainment experiences among other things. . Additionally. that the humanoid robot will also benefit other industries. capable of selfdeveloping its mental and physical capabilities through realtime interaction with the environment and human masters. such as: maintenance. With the tight integration of perceptive ability. There is no doubt that the emergence of humanoid robots will have great impact on many aspects of our modern society. 1.5. precision surgery).2 Society Up until today.14 The Fundamentals of Robotics: Linking Perception to Action 1. In the healthcare industry. increases our sense of social status. humanoid robots could be of assistance in hospitals for certain tasks (e.5 Impact of Humanoid Robots The enhanced locomotive ability of humanoid robots will increase the range of the manipulative function. and offers us highquality.g. there would be no efficient transportation. It is clear as well. We can even imagine that an unmanned factory may become a reality one day. The inclusion of cognitive ability will make a humanoid robot a physical agent. a humanoid robot will gain a certain level of autonomy by interacting with human masters or executing tasks intelligently. humanoid robots could provide great service in the rehabilitation of patients. construction of buildings or houses can be considered as an assembly process. while computational power is increasing. For example. Now. Within the manufacturing industry itself. Without the automobile.
biped walking. yet it has increased manipulative ability. and provide the means for facial expressions and perceptive sensory input and output. These are basically preprogrammed interactions between users and artificial creatures in a virtual world. speech recognition. • The robot's hand is becoming more compact.g. or an animallike robot. such as: • Robotassisted Entertainment: Computerized games are very popular. visual perception etc. facial expressions). • The computer has more functions and better performance while the price is constantly decreasing. • The computer has the ability to express inner emotional states through visible motions (e. computation with words rather than digital logic).Introduction to Robotics 15 • The microprocessor is consuming less electrical power. which may influence the design of future computer's monitor. this will bring new benefits to consumers. certain animations or tours could also be automated . • The robot is becoming intelligent. The incorporation of demanding computational power and cognitive abilities enables the robot to perform realtime interaction with its environment and humans. • The robot has a head. computerized games will take on a new dimension: a preprogrammed interaction with artificial creatures in real space. despite the increased computational power. auditory communication. With the humanoid robot.e. • The robot is becoming the perfect embodiment of an artificial body with artificial intelligence. yet it has increased dexterity and sensing ability. will contribute to the emergence of the humanoid robot as the next great consumer product. • The robot has legs to perform humanlike. The evolution of the computer coincides with the evolution of the robot. with the humanoid robot. visual communication. each of which has its own group of manufacturers with sound and profitable track records. The parallel development of the computer and robot industries. Moreover. which is characterized by the following clear trends: • The robot's arm is becoming smaller. • The computer is constantly improving its cognitive abilities: soft computing (i. Undoubtedly.
more and more parents rely on private tutors to complement schooling. (e. through telepresence. With the humanoid robot. some prehospital diagnostics or treatments could be done at home or in a neighborhood community center. as well as at different skill and knowledge levels. having output and input functions. with preprogrammed and selectable knowledge and skills. it is possible to envision a virtual university for lifelong learning at home. but also relevant to the continuing education of adults. experience. One effective way to lower the cost is to increase the accessibility of healthcare services. at a much lower cost. • Robotassisted Education at Home: In our modern society. dentist). it is possible for us to envision in the future the concept of a virtual hospital. robotassisted education is not only appropriate for children. • Robotassisted Healthcare at Home: Healthcare is expensive. such as museums. or even a remote village. but also a collection of subsystems for input of sensory information. and is an ideal platform to extend the reach of a human's output functions. With the humanoid robot. theme parks.g. One obvious advantage to using the humanoid robot as a tutor is the ability for the same humanoid robot to be configured as a tutor in different disciplines. This would be possible. if humanoid robot technology advanced to the stage where we could deploy humanoid robots to homes. • Robotassisted Teleexistence: The human body is not only a collection of subsystems for output functions. for example. feeding back information to humans who can then feel. interpret. As a result. By simply activating the appropriate precoded programs. In the future. both locally and in remote areas. film studios etc. The humanoid robot is also a complex system. respond . In addition. Therefore. an alternative to private tuition may be the use of humanoid robottutors. parents are heavily occupied with their professional activities. hotels. There is limited time for parents to contribute to the educational process of their children. In order to compensate for this deficiency. shopping centers. It can act as a "complex sensor" placed at a remote location. a humanoid robot could instantaneously be configured as an "expert" in the necessary medical field. to a remote location. the humanoid robot would provide the means for a human medical expert to diagnose and treat a patient at a remote location.16 The Fundamentals of Robotics: Linking Perception to Action in places.
5. This research will have a direct impact on the manufacturing of smart artificial animals (e.3 Space Exploration and Military The humanoid robot is an ideal platform for telepresence or teleexistence. With the humanoid robot. dissuade. 1. SONY's AIBO. underlying the selfdevelopment of mental and physical abilities through realtime interaction between the environment and humans. The animals of the future will possess certain degrees of autonomy to selfdevelop emotional. It would also be possible to assign humanoid robots to maneuver military land. Because of this. This is what the commonly called teleexistence. air.Introduction to Robotics 17 etc. A formal definition of Mechatronics can be stated as follows: Definition 1. but interrelated engineering disciplines. affective and cognitive functions through realtime interaction with humans. it is easy for us to imagine its applications in space exploration and the military. humanoid robots could help to prevent.4 Education Today's engineering education emphasizes the integrative aspect of various engineering disciplines. a type of mechanized dog). yet controlled by humans at a ground station on Earth. It would be possible to dispatch a space shuttle commanded by humanoid robots. it may be possible one day for humans to travel or shop in a remote place without going anywhere. or Mechatronics. • Mechanized Buddy: More and more researchers in the community of robotics and artificial intelligence are studying developmental principles. . In the battle against terrorism. and rescue.5. 1. or underwater vehicles in a battle field without risk of human casualty. analysis and control of smart products and processes. Mechatronics originated in Japan in the 1970s and is gaining worldwide attention because of the importance of integrating different. It will not be long before we will see the market flourishing with smart toys or animals.g. This type of teleexistence will certainly enrich our lives.7 Mechatronics is the study of the synergetic integration of physical systems with information technology and complex decisionmaking in the design.
5 Research Besides education.18 The Fundamentals of Robotics: Linking Perception to Action • V Mental —j / World L k _ Reaj World —j^ J I LEARNING Modeling ____^ Optimization Representation I z1 ^ I PERCEPTION Sensory Systems Perceptual Filters I A . ordered arrangement). perceptive and cognitive systems.e. outcome). being a combination of mechanical. information.8 A proposed framework for human intelligence which encompasses four constituent elements: perception. a "behavior" is a sequence of actions with a specific syntax (i. Human life is an unceasing journey of achieving . Computational neuroscience is the study of how the human brain plans and controls behaviors to achieve desired results (i. is an ideal platform to illustrate the essence of Mechatronics. which propels research on the humanoid robot. By definition. The humanoid robot. electronic. thinking and action.t 1 [ S  STATE Mental States IZ~fl    Physiological States . control. 1.e. THINKING • II + ~ Belief System Inferring Process  \S ~T V ACTION  [ Physiology Mechanism  —  Menial Syntax  [sensory motor Control  Fig. two additional forces which simultaneously stimulate research in this fascinating field are: a) computational neuroscience and b) intelligent (physical) systems. 1. learning.5.
Our belief system also determines the configuration of our mental and physiological states. conceptual symbols etc). We all have similar bodies and neurological systems. our mental stateofmind. action is the source of results. but also skills which result in actions. and kinesthetic representations of the external real world. gustatory. the belief that one is resourceful places a person in a different stateofmind than the belief that one is . or disassociating them from effects. And. we can achieve similar results. However. knowledge describes the relationships between causes (e. mental and physical states affect the way the brain reacts to certain sensory input. thinking is a process of associating causes with. • Thinking: Human beings are capable of not only communicating with the real world. stimuli. conditions. Simply speaking. the ordered sequence of actions) with results. An important goal of computational neuroscience is to develop computational theories and models from an understanding of the brain's functions in these important areas. This form of learning encourages us to effectively imitate successful people. (See Fig.g. We all have our own mental syntax (i.g. auditory. facts. • Learning: The human brain has the ability to process filtered sensory data to derive structured representations which form our mental world.Introduction to Robotics 19 desired results through actions which are decided as a result of our internal representations of the real world. These sensory systems supply input to the brain to build the personalized internal representations of the external world. in terms of knowledge and skills. 1. results. mental world. Language is undoubtedly an important component in learning visual. situations. For example. Learning is a process for us to gain not only knowledge. and our physiological state. and olfactory. but also communicating with the internal.8 for illustration). The thinking process is dictated by our belief system. auditory. Hence. ordered sequence of mental actions) which conditions our thinking process. If we undertake the same actions under the same mental and physiological states. transformations. • Perception: Humans have five sensory systems. skill describes the association of behaviors (i.e.e. In addition. constraints etc) and effects (e. namely: visual. actions. because belief is preformed and preorganized predictions of achievable outcomes. kinesthetic. Generally speaking.
• Action: The human body is capable of performing actions. being vital. driven by our sensorymotor systems. and action. Undoubtedly. the humanoid robot is a perfect research platform for the study of the embodiment of artificial intelligence with an artificial body. humaninspired artificial intelligence must be tested by an artificial body. dynamic and excited certainly releases more physical energy than being depressed. precise definition of "intelligence" to prevent it from being misused. decisionmaking and integration in an artificial body. The ordered actions are behaviors acting on the real world for the achievement of desired results. scared or uninterested etc. thinking. Intelligence is a measurable attribute.20 The Fundamentals of Robotics: Linking Perception to Action miserable. An artificial body requires artificial intelligence in order to adapt to a changing environment for the purpose of better performing the assigned tasks. and is inversely proportional to the effort spent in achieving the intended goal. intelligent (physical) system is the study of computational principles for the development of perception. The study of computational neuroscience not only helps us have a better understanding of the brain's functions. How do we define intelligence? From an engineering point of view. One possible definition of intelligence is as follows: Definition 1. For instance. here comes the question of human intelligence. The performance of our sensorymotor systems obviously depends on the body's mechanisms. learning. .8 Intelligence is the ability to link perception to actions for the purpose of achieving an intended outcome. learning. The humanoid robot is an ideal testing ground for computational theories or models derived from the study of the human brain's mechanisms in perception. Now. However. as well as physical energy which is affected by our mental state. Intelligence and the ability to produce intended results are unique attributes associated with humans. it is constructive to form an objective. but also helps guide the engineering approaches in the development of artificial intelligence. On the other hand.
9 A set of related topics in robotics with a motioncentric theme. 1. a mechanism consists of joints and links. In robotics. the major issues in robotics will logically include: 1. As illustrated in Fig.9 Kinematics is the study of motion without consideration of force and torque. a link . being a visible form of action. this particular point is at the tip of an endeffector. the definitions of kinematics and dynamics are as follows: Definition 1. Literally.1 Mechanism and Kinematics From a mechanical point of view. the unifying concept in robotics is motion. 1. By default. A robot's degree of autonomy depends on its ability to perform the ordered sequence of perception.6 Issues in Robotics Robotics. a robot is a physical agent which is capable of executing motion for the achievement of tasks. Functionally.6. a robot's dynamics in motion execution is dictated by mechanical energy consumption in connection with kinematic constraints imposed by the robot's mechanisms. decisionmaking and action.9. while dynamics is the study of motion in relation to force and torque.^ ± ^ \ ^ ) ^T Sensing J I MOTION C Control JT ^^<«_<^^ Perception J (Decision making) Fig. As we know. a mechanism is a set of linkages without an actuator. is an engineering discipline. In general.Introduction to Robotics 21 ( Actuation ) Q Mechanism \ ^ ^ y*0 . Therefore. or the study of robots. The purpose of a mechanism is to impose kinematic constraints on the types of motion the mechanism can deliver at a particular point. 1.
the most widely available source of energy is electrical energy. Therefore. Today. which use the electromagnetic principle to induce mechanical movement from the interaction of two independent magnetic fields. This discipline aims at determining the equations of motion imposed by force and torque on a robot's mechanism. gears and bearings.2 Actuation Elements and Dynamics In the mechanical domain. any motion is produced by the conversion of mechanical energy. A magnetic field can be generated by either a permanent magnet or electromagnet. This leads to the study of electric motors. Hence. Being able to convert electrical energy into mechanical energy is important but not sufficient. • How do we determine the motion parameters of the linkages necessary to produce a desired set of motion parameters at a particular point on a mechanism? This is known as the inverse kinematics problem. 1. it is important to study the relationship between the motion parameters of the linkages and the motion parameters of a particular point on the mechanism.6. It is common at the output side to use a transmission mechanism which is a combination of reducers. which is converted into the corresponding mechanical energy. This study is the object of robot kinematics. . while a joint is the point of intersection between any pair of adjacent links. There are two problems with robot kinematics: • How do we determine the motion parameters of a particular point on the mechanism from the knowledge of the motion parameters of the linkages? This is commonly called the forward kinematics problem. Any changes in the relative geometry among the links will induce a specific type of motion. The study of the relationship between motion parameters and force/torque is the object of robot dynamics. Mechanical energy can be generated either through gravitational force or some other form of energy. We must also be able to modulate the mechanical energy being produced. There are two ways to modulate the final output of mechanical energy. It is common at the input side to use a power amplifier to regulate the amount of electrical energy.22 The Fundamentals of Robotics: Linking Perception to Action is a rigid body inside a mechanism. tendons. it is important to study the conversion from electrical energy to mechanical energy.
6. there are two other requirements for sensing: a) the measurement of the interaction force/torque between a robot's endeffector and its environment. and output accuracy. time responsiveness. Since a robot is designed to perform tasks through the execution of motions. and b) the motion parameters of a robot's workpieces and workspace. 1. Therefore. Sensing is the only way to obtain the actual output values. the dynamics of a robot must closely (if not exactly) follow the desired dynamics imposed by a task. any system in a real environment is invariably subject to certain types of noise. The beauty of the automaticcontrol theory is its versatility. it is desirable to have a mean to alter the robot's intrinsic dynamics externally. In robotics. Besides the sensing of motion parameters.6. Therefore.3 Sensing Elements From an engineering point of view. The hardware aspect of a machine's brain can be . in terms of stability. it would be inconceivable to attempt to build an automatic machine without a brain. You can alter a system's intrinsic dynamics through the insertion of a control element (controller) in a closedfeedback loop so that the system's actual dynamics meet a wide range of specifications. the discipline of automaticfeedback control is the study of methods and tools for the analysis and synthesis of system dynamics.6. it is impossible to design a robot which has intrinsic dynamics that meets the intended dynamics of any given task. In general. 1. no mathematical description of the input and output relationship is ever exact. (This latter will be the object of study when we discuss the robot's visualperception system).5 Information and DecisionMaking Today. sensing elements are used to convert physical quantities such as the motion parameters of actuators into corresponding electrical signals. Output from a sensing element provides feedback on the motion parameters regarding the linkages inside a robot's mechanism.Introduction to Robotics 23 1. uncertainty and disturbance.4 Control The robot's intrinsic dynamics are described by the relationship between the motion parameters of a robot's endeffector and the forces/torques applied to the robot's mechanism. In engineering. How to determine the desired dynamics from a given task will be studied under the topic of motion planning.
manifested in the form of visible motions. In fact. This is because associating a sequence of ordered actions/motions with the corresponding results is a skill which can be learned with the help of linguistic processing and programming.24 The Fundamentals of Robotics: Linking Perception to Action as simple as a microcontroller. research on developmental principles for acquiring perceptive. our abilities to act and learn would be tremendously weakened. The first function of a robot's brain is to perform computational tasks. endeffector) on the mechanism. In robotics. This ability to provide accurate visual feedback to the control loop of motion execution constitutes the first function of a robot's visualperception system. vision plays a vital role in any machine which intends to perform autonomous motions. action and motion. tasks. The second function of a robot's brain is to support not only the interaction between the robot and human master. Without vision. can be measured in two ways. The second approach is to use artificial vision to measure the motion parameters at a particular point on the robot's mechanism. robot dynamics. actions. The third and most important function of a robot's brain is to implant computational algorithms of artificial intelligence. or a microprocessor having a certain level of computational power.g. The first approach is to use sensors to measure the motion parameters of the linkages inside the robot's mechanism. One important aspect of a robot's intelligence is the ability to plan tasks. This is a crucial step . thinking and acting skills is aimed at developing a skillbased approach for the autonomous planning of task. and robot control. such as sensorydata processing (including visual perception) and the mathematical computations underlying robot kinematics. The actual dynamics of a robot. In a similar manner. the performance of a task implies that a robot's actual dynamics closely follows the dynamics required by the task itself.6. but also the communication between the robot and the outside world. The second important function of a robot's visual perception system is to describe an action in twodimensional (2D) image space instead of expressing it in threedimensional (3D) task space.6 Visual Perception Vision is the most important sensory channel for humans (as well as animals). actions and motions without human intervention. it is possible to use forward kinematics to derive the motion parameters at a particular point (e. or even behaviors. Subsequently. 1.
pure engineering principles. towards achieving the autonomous execution of a given action. 1. . it can be regarded as the fifth function of visual perception.Introduction to Robotics 25 LimbEye Coordination Skill Binocular Vision < System + target @ \ / \ Hand i—*—i Inverse ^yCw Kinematics ^O^ • Motion planner User i Motion Control 1 h rxi Vi r1 te LP Sensory j£ Feedback v. The fourth important function of visual perception is to provide visual input to support a robot's learning process. engineering approaches to limbeye coordination behavior. will be achievable. handeye coordination). This is because the automatic generation of an action/task plan requires knowledge of the geometric models of the workspace and workpieces. Of these behaviors. As we know. as illustrated in Fig. With welldeveloped perception capability. if one considers imagebased communication/interaction a special skill.10. The third important function of visual perception is to infer the 2D and/or 3D geometry of a robot's workpieces and workspace as a necessary step towards the robot's automated planning of an action/task. the most impressive ones will be: a) visionguided manipulation (i. / ' 1 Fig. Chapter 9 will not only cover wellknown. and c) visionguided locomotion (legeye coordination). but also report the latest findings on humanlike. 1. action is the source of results.e. b) visionguided positioning (headeye coordination). Finally. the limbeye coordination behavior.10 A Framework for LimbEye Coordination Behavior.
8 Bibliography (1) Bowyer. (1997). PrenticeHall. Introduction to robotics in CIM Systems. J. Addison. Lee (1987). A. E. 15.. (5) Groover. Introduction to Robotics: Mechanics and Control. K. R. (7) Rehg. McGrawHill. (4) What is a robot ? What is robotics? (5) How do you shape the motions exhibited by a machine? (6) What is kinematics? What is dynamics? (7) What is behavior? What is intelligence? (8) How do you address the issue of unemployment caused by increased automation? (9) What is the possible impact the humanoid robot will have on industry and society? (10) Explain the key topics in robotics.7 Exercises (1) Describe two key factors which pushed manufacturers to adopt automation. G. H. K. Gonzalez and C. Chang and J. Special Issue on Undergraduate Education and Computer Vision. Henderson (2001). S. T. Vision and Intelligence. Sutherland (1992). W. (3) DeVor. G. 5. (6) Knoll. (4) Fu.Wesley. 1. Statistical Quality Design and Control. 23. International Journal of Pattern Recognition and Artificial Intelligence. C. 37.. PrenticeHall. (8) (2001).. Stark (2001). Sensing. J. Tokyo. robotics and Autonomous Systems. (1996). and L. Special Issue on Humanoid Robots. A. . P. (2) What is a manufacturing process? (3) Identify some critical parameters of an arcwelding process. R. Japan. C. Proceedings of International Conference on Humanoid Robots.26 The Fundamentals of Robotics: Linking Perception to Action 1. (2) Craig. S. Bekey and T. M. PrenticeHall. Robotics: Control. (1986). Fundamentals of Modern Manufacturing.
A common approach for assigning the spatial reference to our physical 27 . Therefore.e. 2.) Thus. such as: motion generation (i. From a mechanical point of view.2 Cartesian Coordinate Systems In order to describe the motions of a rigid body. From a physics point of view. linear acceleration is generated when force is applied to a body which is free to move in the direction of the force (Newton's Law). we need to define two references: a) the time reference for velocity and acceleration. velocity and acceleration. a motion is the result of force or torque applied to a body.1 Introduction Motion is the unifying theme in robotics. A time reference is described by the onedimensional axis t. For example. motion planning & control (i.Chapter 2 Motion of Rigid Bodies 2. we will restrict our discussion to the motion of a rigid body. (See Chapter 3.e. and is applicable to all the real and physical systems on Earth. a robot is an assembly of links that are normally of a rigid body. If a body is fixed at a particular point. it is important to understand the mathematical description of motion in terms of displacement. and visual perception of motion. The values of time variable t depend on our common clock system which is unchangeable. Motion is the visible form of actions for the achievement of results in a real and physical world. This will also help as a sound basis for the study of some important topics in robotics. an applied force will cause the body to rotate about an axis which goes through the point. and b) the spatial reference for position and orientation. a robot's mechanism and actuation). decisionmaking and control). in this chapter.
1 Any spatial reference system with base vectors that are mutually perpendicular to one another is called a Cartesian coordinate system (or coordinate system for short). measured along the base vector i. measured along the base vector k. • z is the distance to the origin. These (x.z) are called the Cartesian coordinates (or coordinates for short).).k) where OQ — XQYQZQ having the assigned '•(s) J (i) *(i) the position of point P on a rigid body (see Fig. k]»\y\ =R*r (2.28 The Fundamentals of Robotics: Linking Perception to Action threedimensional (3D) world is to define an orthogonal reference system formed by X. These three axes are mutually perpendicular to one another and intersect at a common point called the origin. we must define a unit of measurement for each axis in the spatial reference system.1 can also be written in a matrix form. 2. Therefore. z)1.2) . centimeters etc.1) Interestingly. Any space with a Cartesian coordinate system as its spatial reference system is called a Cartesian Space. j .1) is represented by a vector r = (x. Accordingly.y. This unit vector is called a base vector. measured along the base vector j . as follows: r=[i. • y is the distance to the origin. Eq. and denoted by O. In a spatial reference system. This is normally done by associating a unit vector to each axis.j. In order to obtain a measurement with a physical meaning (in terms of meters. an axis serves as a reference to measure the position of an object along it. where • x is the distance to the origin. y. position vector r of point P can be written as: r = x * i + y • j + z»k. 2. a Cartesian coordinate system can be defined as follows: Definition 2. Y and Z axes. (2. In a (Cartesian) coordinate system base vectors (i.
1 A spatial reference system for a rigid body. which are useful for concisely describing general motion transformations by 4 x 4 matrices. which aims at inferring the threedimensional geometry of an object from its twodimensional projection onto an image plane. . such as translation. which is applied to coordinate system OQ ~ XQYQZQ. where I1 0 °\ # = [t. The projection of a geometric object along a set of straight lines has interesting properties and is useful in practice. rotation. One typical application of projection is the display of threedimensional objects onto a twodimensional screen. Another typical application is visual perception.*]= 0 1 0 . . Here. This concept of mapping can ~ be applied to any pair of coordinate systems in a Cartesian space.Motion of Rigid Bodies 29 Fig. It is an important concept. scaling. which is simple and helpful in understanding robot kinematics. 2. let me briefly introduce the concept of projective coordinates for the purpose of explaining the meaning of homogenous coordinates. and projection.?.3 Projective Coordinate Systems We can manipulate a geometric object in a Cartesian space in different ways. 2. \0 0 1/ The equation r = R • r means the mapping by matrix R.
2. The conversion from projective coordinates (X.fc)*to Cartesian coordinates (x. a threedimensional Cartesian coordinate system).k). Then we can form a new reference system represented by (X. we will make use of the equivalent projective coordinates to uniformly describe translational and rotational motion transformations in a matrix form. In robotics. *'Q Fig. Thus. Z. We draw a straight line which passes through the origin and point P. zY be a point in a spatial reference system (e. we call (x. the Cartesian coordinates can equivalently be represented .k»y. Let P = (x. by their projective coordinates.Y.30 The Fundamentals of Robotics: Linking Perception to Action __. The physical meaning of trans . £. k)1 are called the projective coordinates. In the following. Accordingly.k» zf where k is a scaling variable.Z. any complex motion of a rigid body can be treated as a combination of translation and rotation.2 Illustration of projective coordinate systems. the coordinates (X. element k is commonly called the homogenous coordinate. Y. 2. This new reference system is called the projective coordinate system.2. y. Refer to Fig.g. Y.zY is done in the following way: *= £ y= i *=f (23) When A = 1.4 Translational Motions In a Cartesian space.z)1. Z. 2. 1)* the equivalent projective coordinates with respect to the corresponding Cartesian coordinates (x.y.y. It turns out that the coordinates of any arbitrary point Q on this straight line can be written as Q = (k»x. When k = 1.y. Q = P.
£2]. it is sufficient to choose one point as representative for the study of the motion caused by a force or torque. the position vector of the origin O\ at time instant £2 will be O1(t2) = O1(t1) + T. to each rigid body. an orthogonal coordinate system located at a chosen point. Therefore.1 Linear Displacement A translational motion can be described by a linear displacement vector. In robotics. as one point in a threedimensional Cartesian space does not represent any orientational information. a point's position vector may have different values for its elements. it is common to assign. 2. it is necessary to assign two directional vectors to a point on a rigid body in order to represent the variations in orientation caused by a rotational motion.1 Fig.4.Motion of Rigid Bodies 31 lational motion is motion along a straight line. and a linear acceleration vector (if any).3 shows the assignment of frame O\ — X\Y\Z\ to a rigid body. These entities are equal at all points on a rigid body. oooooooooooooooooo With more than one frame in a Cartesian space. However. For the sake of clarity. Here. there is no unique way to do this. it is necessary to explicitly indicate the frame which a vector refers to. For the sake of consistency. In this way. It is worth noting that linear acceleration is the result of a force applied to a rigid body. Since motions at all points on a rigid body are equal. Example 2. If this body undergoes a translational motion described by displacement vector T = (txitytt2)t within time interval [£i. the origin of frame O\ — X\Y\Z\ can serve as a representative point on the rigid body for the study of its motion with respect to any other frame. Another approach is to use a superscript after the symbol. One approach is to use a superscript before each symbol to indicate the reference frame. depending on the choice of the reference frame. 2. the orthogonal coordinate systems are commonly called frames. the orientation of a rigid object will remain unchanged if the motion is translational. In robotics. a linear velocity vector (if any). we adopt . (2. when dealing with mathematic symbols.4) Thus. the study of a rigid body's motion becomes the study of the relative motion between its coordinate system and a reference coordinate system.
tzy the translational motion transformation from frame 1 to frame 0. Example 2.6) ••oooooooooooooooo This example clearly shows that vector °T\ describes the relative distance (or displacement) between the origin of frame 0 and the origin of frame 1.0. at time instant t. In general.2 Refer to Fig. the above equation becomes °O1 = 1O1+ °Ti. while the same position vector with respect to its own frame 1 is denoted by XO\.32 The Fundamentals of Robotics: Linking Perception to Action Y °A Y "A ^H N^ v I1. the first approach. frame 1 is fixed on the rigid body). In other words.ty. This vector is called the translation vector. 2.4.3.e. For example. 2. The position vector of origin O\ with respect to frame 0 is denoted by °Oi. we have °O1= 1O1+ 0OL (2. We denote °Ti(t) = {tx. if we imagine that frame 1 initially coincides with frame 0 at time instant to. 2."' (a) Before translation Fig. Example 2. If the coordinates of point P on a rigid body with respect to its . it describes the translational motion of frame 1 with respect to frame 0.0) and this position vector with respect to its own frame remains unchanged (i. Since lO\ = (0.5) If we define °T\ = °Oi. and an illustration of a translational motion.3 Refer to Fig.vffiF1 (b) After translation Frame assignment to a rigid body.3 I1 . the translational motion of frame j with respect to frame i can be denoted by %Tj. (2. °P means position vector P referenced to frame 0.
e.zo)t = lP+ °Ti(t). position vector ! P is a constant vector because frame 1 is fixed on the rigid body.4 Translational transformation from frame 1 to frame 0.7.Motion of Rigid Bodies 33 own frame (i. if we use the equivalent projective coordinates to substitute for the Cartesian coordinates in the above example. 000000000000000000 Interestingly enough. (2. Eq. ^ .yo. fP} . 2. the position vector of the same point P with respect to frame 0 will be °P(t) = (xo.7) In Eq. 2. frame 1) are lP = ( x i . ^ ) * .9) \0 0 0 1 / °P = {°Mi} . *p Fig.7 can be expressed in a matrix form as follows: ( or rr o \ / I 0 0 tx\ /iA yo 1/ =ioioty U V1 / (2. 2.
tz)1 the linear displacement vector due to translational motion. 2. with respect to frame 0.34 The Fundamentals of Robotics: Linking Perception to Action where ( 1 0 0 tx\ 0 10ty o o i tz • 0 0 0 1/ In fact.d{iTi^)} . This occurs when a force. Let us consider a rigid body to which frame j is assigned with respect to frame i. and Z axes. the corresponding linear velocity and acceleration vectors will be { 2.9 is called the homogenous motion transformation matrix. matrix °M\ in Eq. the first three column vectors of °M% are exactly the base vectors of frame 1 with respect to frame 0 while the last column vector is exactly the position vector of frame l's origin. If we ignore the elements of the fourth column. 2.10) in U\ _ dH'TjW} aj(l) — dt2 _ (d2tx ^ dHA* — \ dt2 ' dt2 ' dt2 J • Rotational Motions The physical meaning of a rotational motion is the rotation of a rigid body about any straight line called a rotation axis.4. ty.2 Linear Velocity and Acceleration Displacement only describes the effect of motions at discrete time instants among the frames under consideration. A typical example of rotational motion is the rotation about one of the X. is applied to a rigid body constrained by a rotation axis.Y. the instantaneous variations of a rigid body's motion are described by its velocity and acceleration. These physical meanings remain valid for any complex motion between any pair of frames. If we denote lTj(t) = (tx.5 »„ (t\ .(dtx dty auV vi\t>  d~t  \dt' dt ' I t ) (2. It describes the motion transformation from frame 1 to frame 0. or torque. However. .
and if any. but the orientation of its base vectors will be as well (except in the case when one base vector coincides with the rotation axis).1 Circular Displacement Rotational motion can be described by circular displacement. circular velocity. not only will the origin of a rigid body's Frame change (except in the case when d = 0).5.1)* Oo1(t1) [ZAi) = (d. Under the rotational motion..&. And.The base vectors and the origin's position vector of frame 1 at time instant t\ will be '°r1(t1) = (1.0. 2. or torque.0)* < °Ji(ti) = (0.5 Illustration of a rigid body's rotational motion to which frame 1 is assigned.0)* °*i(ti) = (0. Another important fact is that rotational motion will not cause the rotation axis to change. 0.5 shows a rigid body's frame undergoing rotational motion about the Z axis from its initial location at (d. ••''"s'. 2. Consider time interval [ti..Motion of Rigid Bodies 35 2.oy. these entities are equal at all points on a rigid body. Now. #2].0.o. let us examine rotational motion without regard to the force. Fig. If the rotational angle about the Z axis is denoted by 0 at time instant .0) (referenced by frame 0) to a new location.1. which causes the motion. circular acceleration (the sum of tangential and centrifugal accelerations). (a) Before rotation f'm\ (b) After rotation Fig.
0.0)'= xh l°Ai(*i) = (0. If we define matrix R as follows: R=l°ii(t2).15 into Eq. substitute t2 with t) can be expressed as follows: (°h(t)= °R1(t)m{1i1} I°j1(t)= ^ i W . f2Ux ' Now. (2.14.{°O 1 (t 1 )}. 2.°k1(t2)\= /cos{9) sin(0) sin(0) 0 \ cos((9) 0 . Of ' ok1(t2) = (o.36 The Fundamentals of Robotics: Linking Perception to Action t2.e. 008(6). (2.16) The physical meaning of matrix °i?i(i) is the rotational motion transformation at time instant t from frame 1 to frame 0. d»sin(0). This matrix is commonly called a rotation matrix. frame 0) at time instant to.12 can be rewritten in the following matrix form: ' o u ( i 2 ) = J R{°u(£i)} 0 J 1 (t 2 ) = i2«{°Ji(*i)} °fc1(t2) = ii. 0)*. because the three column vectors of °Ri (t) are the base vectors of frame 1 with respect to frame 0 . [ZAZ) (2. 2. A rotation matrix is a square and invertible matrix.1. then we have K i ( * i ) = (1.g.13) V 0 0 1/ Eq.0.ojl(t2). the base vectors of frame 1 at time instant t (i. these vectors will become 'oi1(t2) = (cos(0). 0)* °/i(*2) = (sin(0). 2.iy °01(t2) = (d»cos(6).{ 0 * 1 (t 1 )} _0O1(i2) = J R.15) If we denote °Ri(t) = R and substitute Eq. It is interesting and useful to note that this matrix describes the orientation of frame 1 with respect to frame 0.0)* = lh {°Ji(*i) = (0. if we imagine that frame 1 performs a rotational motion which always starts from its initial configuration. sin(fl).o. and this initial configuration is where the frame's base vectors are respectively parallel to the base vectors of a reference frame (e. f J i } (°h(t)= °R1(t)*{lk1}.1)*= %.
Y \ X Fig. We denote °Ri(t) the rotational motion transformation from frame 1 to frame 0 at time instant t. the inverse of °i?i(t) describes the orientation of frame 0 with respect to frame 1.6 Rotational transformation from frame 1 to frame 0.17 can be written as follows: °P= °Ml»{1P) with (2. i i? 0 (i) = {°i?i(t)r 1 . If the vector connecting origin O\ to point P on a rigid body is 1 P = (xi. Interestingly enough. 2.zi)1. 2. the same position vector P with respect to frame 0 will be °P(t) = {°R1(t)}. And. Example 2. OOOOOOOOOOOOO^O^ Now.17) = fiM*)}" 1 • {°P(t)} = {lRo(t)} • {°P(t)}.0. Eq. .6.18) (NOTE: 0 = (0.yi.4 Refer to Fig. it is logical to denote the inverse of °Ri{t) with 1R0(t).Motion of Rigid Bodies 37 at time instant t. 2. the inverse of the above equation yields the expression for 1P: XP (2.{1P}. if we substitute the equivalent projective coordinates for the Cartesian coordinates of point P. So. that is.0)).
There will be no change in the vector's orientation if there is no instantaneous variation in the rotation angle 8. we have examined the circular displacement of a rotational motion.38 The Fundamentals of Robotics: Linking Perception to Action From Eq.2. ( { % } x {%}) = ( 0 .6. ^ ) the instantaneous variation of the rotation angle.19) Then. 2.!&. substituting Eq.5. we can see that the translational and rotational motions can uniformly be described by the homogenous motion transformation matrix M.20 yields % =^ ^ = d. then we have ^ r = M0/i} (22i) Since °ji = °ki x° ix. { % } ) x (d*{°h}). it is in the tangential direction of the circular path and its magnitude of variation is equal to the instantaneous variation of the rotation angle. 2.21 into Eq. 2. (2. the circular velocity vector at this origin O\ with respect to reference frame 0 will be ^ = 4°°±. Since a vector's variation caused by rotational motion follows a circular path. the position vector of frame l's origin O\ is 0Oi=d»{°t1}.d.e. It is important to understand the mathematical description of circular velocity and circular acceleration because these motion parameters are directly related to the force and torque applied to a rigid body.18.5. (2. 2. Assume that we choose the Z axis as the rotation axis. (2.2 Circular Velocity and Acceleration So far. Refer to Fig. 2.20) at at The instantaneous variation of a vector's orientation is due to the instantaneous variation of the rotation angle 6.9 and Eq. In reference frame 0. If we denote 9 (i.22) If we define .
S = ^f^ . the expression of the circular acceleration vector becomes V . 2. it is important to understand the physical meaning of Eq. then we have <fci}=Q dt By applying this result into Eq.2.}.23. 2. (2. denoted by 9.24) By further differentiating Eq. we obtain V = (e »° ifei + 0 • ^~] x° Oi +° wi x ( V x° Oi).26.Motion of Rigid Bodies 39 and recall that °O\ = d» {%}.27) . which is also called the angular velocity or angular velocity variable of a rigid body. at at By applying the following equalities 0 (2.25) i *%* = V x {0Ol} into Eq. the expression of the circular acceleration vector will be d{V} 0 Q d{0Ox} ox — ——— x Oi + LOI x ———. This vector coincides with the rotation axis. Eq. Eq.26) Since a rigid body's rotational motion about an axis will not cause any change to the rotation axis.23 also means that a rigid body's angular velocity will cause a circular velocity at any arbitrary point P on the rigid body. 2. vector °u)\ is called the angular velocity vector of a rigid body to which frame 1 is assigned with respect to reference frame 0. This point's circular velocity vector is equal to the cross product between the rigid body's angular velocity vector and the point's position vector. The norm of °u)\ is equal to the instantaneous variation of the rotation angle. 2.^ x fF.(6 »° jfei) x° Ox +° wi x ( V x° Ox). (2.23.23) Now. (2. That is. (2.25. In fact.22 can be rewritten as follows: V =^ ^ = V x fd}.2.
This is called a rigid body's angular acceleration or angular acceleration variable. From Eq. 2.31) 2.29. d) . we can see that the circular acceleration vector is the sum of the tangential acceleration and the centrifugal acceleration. and an is in the opposite direction of °ii. In fact.6 Composite Motions As I mentioned before.{°j1} { I an = V x ( V x° Ox) = (92 . at is called the tangential acceleration vector and an is called the centrifugal acceleration vector. any complex motion can be treated as the combination of translational and rotational motions.d). it is possible to prove the following equalities: (<H = °a1x°O1 = (9.40 The Fundamentals of Robotics: Linking Perception to Action If we define the following terms: ' °ai = 0 .° kx < at = °ax x° Oi an= V x ( V x°Oi) (2.29) where °ai is the rigid body's angular acceleration vector to which frame 1 is assigned with respect to frame 0. .27 can be rewritten as follows: V = % x° Oi +° Wl x ( V x° Ox) =at + an (2. it is easy to study the composite motions among the frames. (230) where at is in the direction of °ji. {0?!}. denoted by 6. From Eq. 2. 2.28.28) then Eq. Vector °a\ is parallel to the rotation axis and its norm is equal to the secondorder derivative of the rotation angle. With the help of frames. In general. (2. a rigid body's angular acceleration lotj will generate circular acceleration at any point P in the following way: laj =l OCJ xi Pj +i ujj x Ccoj xi Pj).
let us assume that the rotational motion occurs within time interval [icb^i] while the translational motion occurs within time interval . In this example. and the reference frame is denoted by frame 0. z Wm ' \ (a) Initial configuration z y%\ (b) Atter rotation X° ' \ i *p\ y ' v^ *pN (c) After translation (d) Final configuration Fig. encapsulates the relative position and orientation of these two frames.Motion of Rigid Bodies 41 2. 2. 2.7 illustrates the fact that it is possible to configure any pair of frames by applying a rotational motion followed by a translational motion to the initial configuration where the two frames coincide. Here. frame 1 is assigned to a rigid body.6. The configuration of a frame. Fig.1 Homogenous Transformation For simplicity's sake. with respect to another frame.7 Illustration of composite motions. we commonly use the term configuration or posture to refer to the relative geometry of a frame with respect to another frame.
and substitute t2 for time variable t. (2. (2. z2y = °P(h) + °TL The combination of Eq.37) and ({°Ri}1 1M0{t) {"iii}1. we can derive the expression for lP as follows: lP = {°i?i}.z1)t = °R1»{1P}.^]."^^ . its position vector remains unchanged.34 and Eq. {XP} < (236) I lP = (x1.32) where 1P = (x0.32 and Eq. these two equations can be written in a compact matrix form as follows: r °P(t) = (x0.34) Since rotation matrix R is formed by a set of orthogonal base vectors. Now. zo. 2.y1. (2.f i ? ! } " 1 .38) = V 0 1 / . with respect to the frame assigned to the rigid body. z0Y is point P's position vector with respect to frame 1.y1.t2\.42 The Fundamentals of Robotics: Linking Perception to Action \t\. let °Ti be the translational vector of frame 1 with respect to frame 0 within time interval [ii. 2. If we denote °R\ as frame l's rotation matrix with respect to frame 0 in time interval [£o.2/2. j/o. I) 4 = 0M!(£) . Since P is a point on the rigid body.z1. { 0 7\}.33 yields 0P(t2)= (2.^i].35) If we substitute the equivalent projective coordinates of P for the Cartesian coordinates of P in Eq. (2.1 • {°P(t2)} . point P's position vector at timeinstant ti will be °P(t1) = (x1. 2. 2.35.Then. y0. 2.34. From Eq. its inverse exists. If = lMQ(t) • {°P(t)} with (2. point P's position vector at timeinstant t2 will be °P(h) = (x2.33) °R1*{1P}+ °Ti.
Motion of Rigid Bodies 43 Matrix °Mi(t) describes mapping from the equivalent projective coordinates of a point expressed in frame 1 to the equivalent projective coordinates of the same point expressed in frame 0. we have °R1(t) =({°r i }. 2. the origin) and orientation of frame j with respect to frame i. In general. {%}). matrix lMj{t) describes mapping from frame j to frame i.e. Assume that the rigid body.39. If we denote °R\ (t) the continuous rotation transformation. we obtain the expression for point P's velocity vector with respect to frame 0.7. to which frame 1 is assigned. and °Ti(t) the continuous translation transformation of frame 1 with respect to frame 0.40) dt dt dt Since a rotation matrix's column vectors are a frame's base vectors. matrix 1Mo(t) describes inverse mapping by °M\{t). Vt) = <™ = d?R«» . An alternative interpretation of matrix lMj(t) is that it represents both position (i. (2. Mapping by this type of matrix is commonly called the homogenous transformation because it describes the homogenous transformation applied to the equivalent projective coordinates between two frames.6. (2. 2.39) •*<«) = cRf °Tf) By differentiating Eq. undergoes a continuous composite motion. point P's instantaneous coordinates with respect to frame 0 will be °P(t)= °Ri(t)*1P+ °Ti(t) or in a matrix form: °P(t)=°M1(t)»{1P} with (2. 2. {%}. that is.2 Differential Homogenous Transformation Refer to Fig.41) . i p + < T ^ » . And.
44) Accordingly. 2. Eq.f. = (uix.43 can also be written. 2.45 and Eq. .fi).40 will become %{t) = 5(V) • ({°hh {°U {°h}) • i'P} + ^ P = 5(V){° J Ri(i)}«{ 1 P} + f ^ = 5(V).v»(^mft).(«}. we have (f. the above equation can be concisely written as follows: M (246) . < ) By applying Eq. Eq.^i)^).42.u}y..(«)_<£*<!».{° J P(i)} + ^ p . 2. If S(°u>i) denotes the skew symmetric matrix of vector °wi.m^)). as follows: (^i. <.47.<. If we use the equivalent projective coordinates.„ with (2. 0 / (2. 2.^. then we have / 5(V) =\UJZ \UJy 0 LOZ UJX u!y \ 0 ux .u!z)t.44 The Fundamentals of Robotics: Linking Perception to Action Differentiating the above rotation matrix with respect to time gives d(°iZi(t)) _ /d{°ii> d{°l} d{%}\ dt \ dt ' dt ' dt ) ' [ ' is °LOI If the angular velocity vector of the rigid body. with respect to frame 0.
In other words.«§I.49) where A is a 3 x 3 matrix which has an inverse.}.{^h}) (2. A . Proof of the above expression is as follows: First. A .51) = 5(^»°w 1 )«( J 4«°/2 1 ). By applying the following equalities: 'A.e. .48) where °uii is frame l's angular velocity vector. 2. From Eq.{*].50.A. A»S{°u>1)»A1 =S{A»°u1) (2.° Wl) x (A • {%}. The above expression is useful in the study of motion kinematics (i. = A. i0^}) (2. we can derive one interesting property of the skewsymmetric matrix S(°u>i).° Wl) x (A «° ki) into Eq. even though point P has no motion with respect to frame 1.50) (A. 2. at time instant t. matrix dtl describes the differential homogenous transformation from frame 1 to frame 0. and °Ri(t) describes frame l's orientation.48. it determines how the velocity of point P in frame 1 is perceived from frame 0. to derive a compact expression of the Jacobian matrix).^A. we compute the derivative of matrix A »° R\: ^rhl = i{A'{0h}.*£p=A*{°w1xoi1) = (A*oUl)x(A*oi1) < A. That is. & M = S(°u1).Motion of Rigid Bodies 45 Therefore. A. with respect to frame 0. we have ^ ^ = (A . That is. {«£}.<£I).{0R1(t)} (2. Another useful result from the above development is the expression for the differential of the rotation matrix. *&£ = A* ( V x°h) = (A»°ux) x (A*0h) A• ^ i = A • ( V x° ki) = (A .
it provides a sound basis for describing the relationship among a set of frames assigned to a set of rigid bodies which are connected (i. However. it provides a better understanding of the minimum number of variables necessary to describe a pure translation or pure rotation. Fig. . (2. 2. 2.8. the links inside an arm manipulator).46 The Fundamentals of Robotics: Linking Perception to Action By applying the following equality into Eq. and multiplying A~l to both sides. we can conclude that the minimum number of variables to fully describe a translational motion in a Cartesian space is 3.g. Y and Z axes. If we denote °Ti(t) = (tx.tzy. Eq.ty. we obtain A • 5 ( V ) »° Ri = S(A »° u>i) • {A m° i?i). 2. 2.51. where each rigid body has only one degree of freedom (e. and the translation vector of frame 1 with respect to frame 0. Secondly. it is clear that the translation vector °Ti(t) can be decomposed in the following way: G)"C)+0+0 (2. we can also treat these as a series of spatially arranged frames which are consecutively supporting each other. 2.53) We can see that any arbitrary translation will be the sum of three elementary translations along the X. Elementary motion denotes the motion of a frame having only one independent (motion) parameter or degree of freedom (DOF). constrained) in a series. A graphic illustration of this fact is shown in Fig.52 finally becomes Am 5 ( V ) • A'1 = S(A»° coi).6. Consider the translational motion between reference frame 0 and frame 1 (assigned to a rigid body in a Cartesian space).8 shows the temporal series of frame 1 in a Cartesian space after three successive translations.3 Successive Elementary Translations The study of successive elementary translations and/or rotations is of a particular interest in robotics. First of all.e.52) By eliminating °Ri from both sides. Therefore.
we can rename the temporal series of frame 1 in the following manner: • Frame 1 after the translation along the X axis is renamed as frame i. 2. A typical scenario is the assignment of frames to links which are serially connected to form an arm manipulator.O. o.8. tzy = t z . iYi Y . .O)t = tx*{°ii} < iTi+i = (O.oy=ty i+1Ti+2 •{%+!} {i+1ki+2}.ty. • Frame 1 after the translation along the Z axis is renamed as frame i+2. i. If we examine the consecutive translational motions among frames 0. (c) Translation along Y axis (d) Translation along Z axis Fig. • Frame 1 after the translation along the Y axis is renamed as frame i+1. i + 1 and i + 2. In fact. the following result is straightforward: '°Ti = {tx.54) = (o.Motion of Rigid Bodies 47 Yo »Yi Y AFi X\ 0 /O 0 Y xQir X\ Y z»/z & 1 (a) Initial configuration z> / < z X 1 (b) Translation along X axis Y0. as shown in Fig.8 Illustration of three successive translations. 2. (2.
Thus. 2.54 into Eq.e.} . These expressions are helpful to better understand motion kinematics which will be studied in Chapter 3.57) ( ( 1 0 0 J. 2.e.48 The Fundamentals of Robotics: Linking Perception to Action where we explicitly emphasize the direction of translation °Xi (i. just as translational motion does not cause any changes in a base vector. In robotics. 2. Eq. it will also not cause any changes to a directional vector. the above 0 0 0 1 / \0 0 0 1/ X. Y and Z directions respectively.e. differentiating Eq.54 with respect to time yields ' ^ = ^{^+1} (255) By applying Eq.\ /I 0 0 tx\ 0 1 0 ty 0 0 0 1 / 1 0 0 0\ oM _ 0 1 0 0 \0 0 0 1/ /I 0 0 0\ i+1. {<Mi+1} • {i+1Mi+2} with (2.53. I 0 1 0 0 0 1 0 L kinematics of a Cartesian or XYZ robot having three links moving in the oequation is the+ exact+ expression fort the •forward o i o M ' 2 = o o i . 2.56 is valid for any set of spatially arranged frames. Eq. the base vector k of frame i + 2). the direction of translation J Tj + i (i.. (2. the translational motion transformation from frame i + 2 to frame 0 can be expressed as follows: °Ti+2 =° T% + %+1 + i+1Ti+2.56) In general. the base vector i of frame i). 2.56 can be rewritten as follows: °M l + 2 = {°M. For example. and the direction of translation l+1Ti+2 (i. If we use the equivalent projective coordinates. . the base vector j of frame i + 1).
and it is that of frame 1 (at the current configuration) about its own Y axis. At the initial configuration. a temporal series of frame 1 can also be treated as a series of spatiallyarranged frames which are consecutively supporting each other.58) . one elementary rotation does not represent a general rotation. 2. This elementary rotation is fully determined by rotation angle 0x. two consecutive elementary rotations are not sufficient to produce a general rotation. This elementary rotation is fully determined by rotation angle 9y. is renamed frame i.4 Successive Elementary Rotations Fig. k) remain inside frame 0's YZ plane. is renamed frame i + 1. {*Ri+1} . the minimum number of variables necessary to describe a general rotational motion between two frames in a Cartesian space is 3. Since each elementary rotation only depends on its corresponding rotation angle about one of the X. After the second elementary rotation. frame l's base vectors (j. and it is that of frame 1 (at the current configuration) about its own Z axis. • The second elementary rotation follows the first one. frame l's base vector j still remains inside frame 0's YZ plane.Motion of Rigid Bodies 49 2. The three successive elementary rotations are as follows: • The first elementary rotation is that of frame 1 (at its initial configuration) about its own X axis. • Frame 1. Clearly. {l+lRi+2} (2. Therefore. This elementary rotation is fully determined by rotation angle 6Z. after the third elementary rotation. frame 1 coincides with frame 0. Similarly. as follows: °Rl+2 = {°Ri} . After the first elementary rotation.9 shows three successive rotations of frame 1 in a Cartesian space referenced to frame 0. after the first elementary rotation.6. • Frame 1. • The third elementary rotation follows the second one. after the second elementary rotation. Y and Z axes. is renamed frame i + 2. Only after three successive elementary rotations will frame 1 reach an arbitrary orientation with respect to frame 0. the orientation of frame 1 with respect to frame 0 after three successive elementary rotations can be expressed. This can be done by renaming the frames in the following way: • Frame 1. Hence.
s i n ( ^ ) 0 cos(9v)J ( cos{9z) sin(0 z ) 0 \ sin(0z) cos(6»z) 0 .V t1 7 / v Y Z ^/ z (c) Rotation about Y axis Fig.9 (d) Rotation about Z axis Illustration of three successive rotations.50 The Fundamentals of Robotics: Linking Perception to Action Y Y f t (a) Initial configuration (b) Rotation about X Axis Z' oX / . with /I 0 0 cos(6':c) \ 0 sin(^) / cos(6>y) 0 \ sin^x) cosCflx) / 0 sin(0 s )\ °Ri= \ . 2. 0 0 1/ .
58 yields < ! p i = S(V +2 ){ o i? i+2 } = fe). 2. + l } . 9y. = 5(°i? i + 1 .* wi+1) .61 into Eq. 2. we finally obtain the following interesting and useful expression: S(°cui+2) = S ( V ) + S(°Ri . { 0 ^ + 2 } . and 6Z can undergo instantaneous variations with respect to time. 2.59) In robotics.59 describes the forward kinematics of a spherical joint having three degrees of freedom.60. Imagine now that 0x. {°i? i+2 } d{i+1df+2) {0Ri} . {^R1+2} = 5(V) • {°Ri+2} (2. ' ul+1) + S(°Ri+1 .Motion of Rigid Bodies 51 If we consider the equivalent projective coordinates. J + ! u. Eq. By applying Eq. Differentiating Eq.62) .i+1 UJ1+2) (2.{l+iRi+2} By applying the following property of skewsymmetric matrix: A»S(°u1)»A1 we can derive the following equalities: =S(A*°u>1) ' a£M • { ^ <{o^}.t+2) . {^+1^2} = 5 ( 0 ^ .{*iW#{*+iiW (2.*CR^l.58 can equivalently be written as follows: °Ml+2 = {"Mi} . 2. {^ i + 1 } . f M 1 + i } • { l + 1 M l + 2 } with (2. Eq.60) +{0Ri}. 2.61) fei) .
It simply means that frame i + 2's angular velocity vector with respect to frame 0 is the sum of the angular velocity vectors of the intermediate frames between frame 0 and frame i + 2. They are: {9x9y6z) (9y9z8x) (9z9x9y) {0x9y0x) (9v9z9y) (9Z9X9Z) (0x8z9y) (9y9x9y) (9z8y9x) {9X9Z9X) (9y6x9z) (9z9y9z). 2. • The second elementary rotation is that of frame 1 about its Y axis at the current configuration (i. Assume that the actual orientation of frame 1 is reached through three successive elementary rotations. Therefore. (2. Similarly. again. As a result.9. the initial configuration). The rotation angle is (3. Example 2.5 Refer to Fig. The first elementary rotation can choose X.64) These sets are commonly called Euler Angles. it has only two axes to choose from.52 The Fundamentals of Robotics: Linking Perception to Action or 0wi+2 = V + °Ri • fuf+i} + °i? i+1 • {i+1uJi+2}. as follows: • The first elementary rotation is that of frame 1 about its Z axis at the current configuration (i. .e. 2. As for the second elementary rotation.5 Euler Angles Refer to Fig. the third elementary rotation also has two axes to choose from. 2.6.63) The physical meaning of the above expression will become clearer in Chapter 3. we have a total of 3 x 2 x 2 (or 12) possible combinations to form a sequence of three successive (elementary) rotations. after the first elementary rotation). The rotation angle is a. because its rotation axis should not be the same as the previous one. Each combination has its own set of three rotation angles as the minimum number of variables to describe a general rotation. (2. we have 12 possible sets of rotation angles. because its rotation axis should not be the same as the previous one.e.9. all being expressed with respect to frame 0. Y or Z axes as its rotation axis.
g.6. /?. Since a set of Euler Angles contains three angles. called angular variables. Thus.e. with respect to another frame. •ooooooooooooooooo 2. the rotation matrix describing the orientation of frame 1 at its actual orientation with respect to frame 0 will be i?i = with Ri+2 = { Ri} • {lRi+i} • {l Ri+2} ( i+lRi+2 cos (a) — sin (a) 0\ sin(a) cos(a) 0 I 0 0 1/ / cos(/3) 0 sin(/3)\ zRt+i = I 0 1 0 Vsin(/?) 0 cos(/3)/ /cos(0) sin(^) 0 \ sin(0) cos(0) 0 . it is necessary to interpolate the orientation of a frame from its initial orientation to an actual orientation so that a physical rigid body (e. The rotation angle is <p. Each set has three angles which make three successive elementary rotations. the orientation of a frame. the endeffector) can smoothly execute the rotational motion in real space. the three angles (a. depends on three independent motion parameters even though the rotation matrix is a 3 x 3 matrix with 9 elements inside. = V 0 0 1/ In this example. and frame i + 2 after the third rotation. In robotics. the interpolation applied to three variables may take a different .6 Equivalent Axis and Angle of Rotation Euler angles are the set of minimum angles which fully determine a frame's rotation matrix with respect to another frame.Motion of Rigid Bodies 53 • The third elementary rotation is that of frame 1 about its Z axis at the current configuration (i. after the second elementary rotation). <j>) are called ZYZ Euler Angles. frame i + 1 after the second rotation. If frame 1 is renamed frame i after the first rotation.
is determined by the YXZ Euler angles.) And assume that the orientation of frame i with respect to frame 0.ry. and is denoted Based on the fact that a rotational motion by unit vector r = {rx. intermediate orientations of a frame between its initial and actual orientations. will not cause any changes to the rotation axis itself. can be treated as a rotation about an equivalent axis from its initial configuration to its actual configuration. the following equality must be valid. Y V Axis of rotation Y Axis °f rotation (a) Initial configuration (b) Actual configuration Fig.ry. with respect to another frame. with respect to frame 0. there is an equivalent axis of rotation which brings a frame from its initial orientation to its actual orientation with a corresponding angle of rotation. imagine that there is an intermediate frame i which has the Z axis that coincides with the equivalent rotation axis r.65) V = °r= (rx. if r is the equivalent axis of rotation for the rotational motion between frame 1 and frame 0: °R1m{1r} = °r with (2. Interestingly enough. Now.54 The Fundamentals of Robotics: Linking Perception to Action path or trajectory in a space defined by these three variables.11. Therefore.rz)t. The truth of this property can be verified in the following way: Assume that such an equivalent axis of rotation exists. (See Fig. it is not easy to manipulate Euler angles for the purpose of determining smooth. 2. If °Ri denotes the orientation of frame i.10 Equivalent axis of rotation: any orientation of a frame. we .rzy. 2.
Motion of Rigid Bodies 55 Axis of rotation Fig. V 0 0 1/ In the above expression.11 An intermediate frame i which has the Z axis that coincides with the equivalent rotation axis. angles (a. 2.66) ( Rz = cos(a) 0 sin(a) \ 0 1 0 — sin(a) 0 cos(a) / /I 0 0 \ Rx = 0 cos(/3) . (3) must take the following values in order for the Z axis of frame i to coincide with the equivalent rotation axis after two successive elementary rotations about its Y and then. have °RZ = Ry*Rx*Rz with (2.sin(/3) \ 0 sin(/3) cos(/3) / /cos(6>) sin(6>) 0 \ sin(0) cos(6») 0 . X axes: a = arccos (T^I ) P = — arccos (\/r%~+~r%) • .
56
The Fundamentals of Robotics: Linking Perception to Action
When the equivalent rotation axis coincides with the Z axis of frame i, then
lr=
°Rl*{ikl} = Ry*Rx»Rz*{iki}
(2.67)
and °r = °Ri.z k = Ry* Rx» Rz* {%} (2.68)
where % = (0, 0,1)*. Since the angle 9 inside matrix Rz is unconstrained, if 9 = 0 (i.e. Rz = hx3), Eq. 2.67 becomes
1r
= Ry»Rx*{%}
(2.69)
Now, by applying Eq. 2.69 and Eq. 2.68 to Eq. 2.65, the following equality holds: °R1»Ry»Rx = Ry»Rx*Rz, or more explicitly we have °R1= Ry»Rx*Rz*R~1 •R1. (2.71) (2.70)
The above equation proves the validity of the assumption that an equivalent rotation axis exists for any given orientation of a frame with respect to another frame. And, the rotation matrix can be formed by five successive elementary rotations which are fully determined by the rotation axis's unit vector and the rotation angle 9 (i.e. the Rz matrix). If °i?i is given as the input:
(
and
1
rii
712
r13\
T2\ r22 r 2 3 I ,
7*31 732 7"33 /
we can derive the solutions for r and 9. From Eq. 2.71, we can obtain
o (T\\ +r 2 2 + r 33  I N I 9 = arccos I
(2.72)
/r32r23\
r=2M9)'
h'"^ '
\r 2 i r12j
(2J3)
Motion of Rigid Bodies
57
2.7
Summary
This chapter provides a review of the fundamentals of a rigid body's motions. The descriptions cater to the need in robotics, to precisely establish kinematic relationships among a set of kinematicallyconstrained rigid bodies. The concept of frame is introduced to summarize a rigid body's mathematical abstraction in space. This abstraction indicates the need to adopt framecentric notations when dealing with the symbols which denote motion transformations. The framecentric notations also serve as a sound basis to unify motionrelated notations in robotics and computer vision. Motion is caused by a force or torque applied to a rigid body which is free to move in certain way (i.e. straight path/trajectory, circular path/trajectory, or a combination of both). A motion can be described in terms of displacement, velocity and acceleration. All these motion parameters could be studied by working with the frames assigned to the rigid bodies in space. Interestingly enough, the relationship between any pair of frames is consistently described by a homogenousmotion transformation, which uniformly describes translational motion, rotational motion, or the combination of both. In this chapter, we also discussed the concept of successive elementary motions. This helps to understand the minimum number of motion parameters necessary to describe any combination of translational and rotational motions. On the other hand, it also serves as a sound basis to justify the most advantageous way of assigning frames to kinematically constrained bodies (i.e. links). This will be discussed in more detail in Chapter 3.
2.8
Exercises
(1) In a Cartesian space referenced by a frame 0 a rigid body, to which frame 1 is assigned, is free to perform any composite motion. At time instant t, the rigid body reaches a configuration which is described by this homogenous motion transformation matrix:
(
7n r12 ri3 tx\
T2i r22 r23 ty r3i r32 r33 tz I
0 0 0 1/
58
The Fundamentals of Robotics: Linking Perception to Action
What is the expression of the base vectors of frame 1 with respect to frame 0 at time instant tl (2) Explain the meaning of homogenous coordinate. (3) °Ri(t) describes the actual orientation of frame 1 with respect to frame 0 at time instant t. At this moment, the rigid body associated with frame 1 undergoes a rotational motion about the Z axis of frame 0. Assume that the angular rotation angle is 8. Prove the following expression:
&*)l
=
e.s(%).{°R1(t)}.
(4) We normally denote S(v) the skewsymmetric matrix of vector v. Assume that vector v is a 3 x 1 vector which has an inverse. Prove the following equality: R»S(v)»R1 = S(R*v).
(5) Prove the solutions in Eq. 2.63 and Eq. 2.64. 2.9 Bibliography
(1) Baruh, H. (1999). Analytical Dynamics, McGrawHill. (2) Nakamura, Y. (1991). Advanced Robotics: Redundancy and Optimization, AddisonWesley. (3) Rogers, D. F. and J. A. Adams (1990). Mathematical Elements for Computer Graphics, McGrawHill.
Chapter 3
Mechanical System of Robots
3.1
Introduction
Action is the source of results. Thus, mechanical system is a part of any physical system, such as a robot. In general, a mechanical system consists of: a) a structure and b) a mechanism. A structure is a set of elements with static couplings. There is no motion between any two adjacent elements in a structure. On the other hand, a mechanism is a set of elements with flexible couplings. The purpose of mechanism is to constrain types of motion produced by a mechanical system. Thus, it is necessary to study a robot's mechanism in order to better understand the robot's functions during motion execution. Because it takes a lot of experience and knowledge to create a mechanism, we will study the robot mechanism at the basic level. However, in this chapter, we will focus on the principles and solutions underlying the robot mechanism's kinematic analysis. 3.2 Robot Mechanism
Fig. 3.1 shows a conceptual design of a humanoid robot. Fig. 3.2 shows an actual prototype of a singlemotordriven (SMD) arm manipulator with 20 degrees of freedom. From these two examples, we can see that a robot's mechanism is an assembly of elementary rigid bodies, called links, which are connected through entities called joints. 3.2.1 Links
A formal definition of a link can be stated as follows:
59
60
The Fundamentals of Robotics: Linking Perception to Action
Fig. 3.1 A conceptual design of a humanoid robot.
Definition 3.1 A link is a rigid body with at least one particular point, called a node, which supports the attachment of other link(s). According to the number of nodes each link possesses, we can divide the links into these common categories: • Unary Link: This refers to a link with a single node. In Fig. 3.2, the big conic entity is a unary link (the virtual model offers a better view). The tip unit (i.e. tip link) of a finger assembly is also a typical unary link (see Fig. 3.3). • Binary Link: This refers to a link with two nodes. The robot's arm manipulators, legs, and fingers are composed of seriallyconnected binary links (Figs. 3.1 and 3.2 show two examples with many binary links). • Ternary Link: This refers to a link with three nodes. A typical example is the hip of the humanoid robot, shown in Fig. 3.2. One of the nodes in the hip link supports the connection to the upper body of the humanoid robot, while the other two nodes connect to the assemblies of the two legs in the humanoid robot.
Mechanical System of Robots
61
(a) Virtual Model
(b) Real Prototype
Fig. 3.2 The first prototype of the singlemotordriven robot arm manipulator: a) the virtual model and b) the actual prototype with 20 degrees of freedom, which are simultaneously and independently driven by a single motor located at the base (on top).
• Quaternary Link: This refers to a link with four nodes. A typical example is the upper body of the humanoid robot, shown in Fig. 3.2. The node on the top is connected to the head/neck assembly of the humanoid robot. The node at the bottom is connected to the hip assembly. The remaining two nodes, at the left and right, support the attachment of the assemblies of the left and right arms. In practice, there are cases when a link needs more than four nodes. A typical example is the palm link inside a robot's hand. If a robot's hand has five fingers, then the palm must have 6 nodes: 5 nodes for the connection of the five fingers and one node for the attachment to the endpoint of a robot's arm. Example 3.1 Fig. 3.3 shows three examples of links: a) a binary link, b) a unary link, and c) a ternary link.
•ooooooooooooooooo
62
The Fundamentals of Robotics: Linking Perception to Action
(a) Link with two nodes
( < & •
/
(b) Link with single node
L
J^
(c) Link with three nodes
Fig. 3.3 Examples of unary, binary, and ternary links.
Example 3.2 Fig. 3.4 shows a conceptual design of a multiplefingered hand. Notice one link with 6 nodes (the palm link), five unary links, and ten binary links (i.e. two links per finger with the first binary link of the thumb in the shape of a crosscylinder).
(a) Front view
(b) Back view
Fig. 3.4 A closeup view of the conceptual design of a multiplefingered hand.
•ooooooooooooooooo
Mechanical System of Robots
63
3.2.2
Joints
A formal definition of joint is as follows: Definition 3.2 A joint is the connection between two or more links at their nodes. It constrains the motions of the connected links. If a joint connects only two links, the entity is also called a kinematic pair. Depending on the degrees of freedom (DOF) allowed for the kinematic pair, a joint can be classified as: oneDOF joint, twoDOF joint, threeDOF joint etc. In robotics, for the simplicity of kinematic and dynamic analysis, we use oneDOF joints because any joint with a higher order of DOFs can easily be treated as the combination of multiple oneDOF joints. In a kinematic pair, one degree of freedom (DOF) needs one independent parameter, which uniquely determines the relative geometry (position and/or orientation) of the two links in the kinematic pair. Generally speaking, a mechanical system's degrees of freedom refer to the number of independent parameters, which uniquely determine its geometry (or configuration) in space and time. In threedimensional space, any complex motion with respect to a reference coordinate system can be treated as the combination of two basic motions: translation and rotation. Therefore, if a oneDOF joint imposes a translational motion, it is called a prismatic joint. If the constrained motion is rotational, it is called a revolute joint Example 3.3 Fig. 3.5 shows a mechanism with one base, three links, and a robotic hand. The base has a slotted track so that Link 1 can undergo linear motion along it. The joint constraining the base and Link 1 is a prismatic joint. In a similar way, Link 1 also has a slotted track so that Link 2 can undergo linear motion along it. Thus, the joint constraining Link 1 and Link 2 is also a prismatic joint. Since Link 3 is a rod which is fixed inside the inner cylindrical tube of Link 2, Link 3 moves up and down along the axis of Link 2. Therefore, the joint constraining Link 2 and Link 3 is another prismatic joint. We can see that the three moving directions of Links 1, 2, and 3 are mutually perpendicular to each other. A robot with such a mechanism is called a Cartesian robot or XYZ robot.
oooooooooooooooooo
Example 3.4 Fig. 3.6 shows a mechanism with one base and three links. Link 1 is fixed inside the inner cylindrical tube of the base, and can only
64
The Fundamentals of Robotics: Linking Perception to Action
7.
"..v^****?.:  •' : \£S£*
Fig. 3.5 A mechanism with three prismatic joints.
undergo a rotary motion about the axis of that tube. Hence, the joint constraining the base and Link 1 is a revolute joint. Now, look at the connection between Link 1 and Link 2. The forkshaped node (attachment point) of Link 1 is joined using a pin to the corresponding node (attachment point) of Link 2. The joint constraining Link 1 and Link 2 is a revolute joint. In a similar way, the joint constraining Link 2 and Link 3 is also a revolute joint. A robot with such a mechanism is called an articulated robot or RRRtype robot (R stands for revolute).
oooooooooooooooooo
The above examples illustrate that robots may structurally be different. One way to differentiate between the various robots is to identify the first three joints in a robot's mechanism. Because a joint may be either prismatic or revolute, there are eight possible combinations. For example, an RRRtype robot means that the first three joints in the robot's mechanism are all revolute joints.
3.2.3
Kinematic Chains
As a robot is a physical agent which performs given tasks through the execution of motions, it should have a sufficient number of degrees of freedom in order to be flexible in a threedimensional space where any freemoving rigid body exhibits six degrees of freedom: three for rotary, motions and
Mechanical System of Robots
65
^T
'
Link .*
^ ^ ^ ^ H H R ^ ^ ^ I Base
Fig. 3.6 A mechanism with three revolute joints.
three for translation^ motions. In general, a robot should have at least six degrees of freedom in order to be able to position a tool at a certain location. However, robots dedicated to specialized tasks may require more or less than six degrees of freedom. And, it is a challenge to figure this out at the design stage. The common approach to arranging degrees of freedom is to form an ordered sequence of links connected in a series through the joints. This type of mechanism is called a kinematic chain. A formal definition of kinematic chain can be stated as follows: Definition 3.3 A kinematic chain is an assembly of links connected in a series through joints, the output motion of which at any chosen point, only depends on the motion parameters of the joints. A kinematic chain is said to be open if there is no joint connecting the first and last links inside the chain. Otherwise, it is called a dosed kinematic chain. Example 3.5 Fig. 3.7 shows an example of a mechanism with one base link and six other links. These elements are all connected in a series to form an open kinematicchain. In robotics, this mechanism is called an articulated arm manipulator in robotics. Refer to Fig. 3.4. It is easy to see that each finger in a robotic hand is also an open kinematicchain.
•ooooooooooooooooo
66
The Fundamentals of Robotics: Linking Perception to Action
Link 3
A
Fig. 3.7
An open kinematicchain with one base and six links, connected in a series.
Example 3.6 Fig. 3.8 shows a conceptual design of a parallel robot mechanism. In this example, the circular disk at the top is a ternary link which is supported by three open kinematicchains. The base links of these open kinematicchains are fixed on a common ground plane, and are treated as a single ternary link. It is clear that this mechanism contains three closed kinematicchains.
ft
^Kf^w
n
Fig. 3.8
An example of a parallel robot with three closed kinematicchains.
oooooooooooooooooo
Mechanical System of Robots
67
3.2.4
Multiplicity of an Open
KinematicChain
When we deal with an arm manipulator, we can treat the underlying mechanism as a simple open kinematicchain. This is because the base link (i.e. the first link) and the endeffector link (i.e. the last link) are clearly indicated, and will remain unchanged. However, it is more complicated when dealing with a humanoid robot. Here, we use the term simple open kinematicchain to differentiate it from the more general term open kinematicchain. A definition of a simple open kinematicchain can be stated as follows: Definition 3.4 Any open kinematicchain with designated base and endeffector links is called a simple open kinematicchain. Clearly, an open kinematicchain can form different simple open kinematicchains. As a result, it is necessary to emphasize the multiplicity of an open kinematicchain in a humanoid robot's mechanism for at least two reasons: (1) An open kinematicchain inside a humanoid robot should not be treated as a simple open kinematicchain. This is because the designations of the base and endeffector links depend on the types of activities the humanoid robot is performing. (2) If we divide a humanoid robot's activity into a sequence of phases, an open kinematicchain in a humanoid robot can advantageously be treated as different simple open kinematicchains corresponding to different phases. Example 3.7 Fig. 3.9 illustrates a simulated humanoid robot in a seated position. The right arm can be treated as a manipulator with its base link attached to the upper body. Similarly, the right leg can also be treated as a manipulator with its base link attached to the hip. In this example, the right hand and the right foot are both the endeffector links.
oooooooooooooooooo
Example 3.8 Fig. 3.10 illustrates a simulated humanoid robot in a standing position. For the activities performed by a humanoid robot in such a standing position, it is easy to study the motions if the right leg, the body, and the right arm/hand form a simple open kinematicchain, like an arm manipulator. In this case, the right foot serves as the base link, and the right hand is the endeffector link.
68
The Fundamentals of Robotics: Linking Perception to Action
2D Humanoid Robot in Posture of Seating
_, Mpha(Arm)
' " n i m M j U j L U J J I I I I I ITTm I I I I I I I I I I II I I I I I I I I I I
J lA^ra (Legl
_
j Beja jam)
">::::::
flWBBBBB
'•'•'•'•'•'•''''':::::::::::
Bete (Leg)
•
_j 179.9089
"° 2 0
°
20
40
6°
80
> H36216
d
1w»eI
I E* I
Li
Fig. 3.9 A simulated humanoid robot in a seated position, where the right arm and the right leg can be treated as two independent manipulators.
oooooooooooooooooo
Example 3.9 A robot's humanlike biped walking can be treated as a combination of two sequences of successive cycles: a) the sequence of the left leg's walking cycles and b) the sequence of the right leg's walking cycles. Each cycle has three successive phases. Fig. 3.11 shows the first two phases of the left leg's walking cycle (the third phase overlaps with the first phase of the right leg's walking cycle). A cycle begins with the first phase. The first phase starts when the left heel elevates off the ground, and ends when the left toe elevates off the ground. The second phase starts at the end timeinstant of the first phase, and ends when the left heel touches the ground again. Within the left leg's walking cycle, it is clear that we can consider the right foot to be the base link of the simple open kinematicchain formed by the two legs.
OOOOOOOOOOOOOOOOOO
Mechanical System of Robots
69
^j
20 Humanoid Robot in Posture of Standing
NphaG*rm)
;
1OO 1111111111
jTTiwnTjTn
111111111 TTTnnTffl
is^i80*)
Beta(Arm)
70
: : : : : : : : : : : : :  J M H B B ' I 1 "•'"•''''•
"
jAlpho(Ug)
(3amraa(Afml
«::::::iB^MI:::::::::::::::::::::::::::::::::
jesfefLeg)
_J 147
tjg^
20
0
20
40
60
80
J
 Home 
[
Exit

Fig. 3.10 A simulated humanoid robot in a standing position, where the right leg, the body, and the right arm/hand can advantageously be treated as a simple open kinematicchain.
Phase 1
Phase 2
Phase 3
Fig. 3.11 Illustration of a simple open kinematicchain for the study of the left leg's walking cycle when the right foot is treated as the base link.
3.3
Robot Kinematics
From a mechanical point of view, a robot is an assembly of a set of links where any pair of two adjacent links is connected by a oneDOF (Degree Of Freedom) joint. Every joint inside a robot's mechanism can be either a prismatic or revolute joint which imposes one degree of freedom and can be
Mapping from joint space to task space is called forward kinematics.12). 3.70 The Fundamentals of Robotics: Linking Perception to Action described by one input motion parameter. while the reverse mapping (from task space to joint space) is called inverse kinematics. All the joint variables in a mechanism form a vector of joint coordinates which define a space commonly called a joint space. a joint imposes one kinematic constraint on the corresponding kinematicpair. . (See Fig. As for the robot mechanism's input motions of a robot's mechanism. that the robot mechanism's output motion is specified and described with respect to a Cartesian space which is commonly called task space in robotics. it is important to understand the relationship between the robot mechanism's input motion parameters and its output motion parameters. Otherwise. Since the purpose of a robot's mechanism is to shape the output motion at a particular point on the mechanism itself. where a general motion involves six parameters: a) three for translation and b) three for rotation.12 An illustration of kinematic mapping. the robot mechanism's kinematic analysis is simply the study of the mapping between task space and joint space. 3. The motion parameter of joint i is commonly called a joint variable and is denoted by cfo. It is worth noting at this point. Therefore. The joint variable q^ refers to an angle if joint i is a revolute joint. And a threedimensional Cartesian space can be represented by the orthogonal coordinate system O — XYZ. it refers to a linear or circular displacement. I Task Space \ \^^~ F° r w a r d Kinematics ^f / Joint Space \ / ^"""^ Inverse Kinematics ^ \ / Fig. The motion parameter of a oneDOF joint is normally represented by a variable which describes the relative position between the two links in the kinematic pair.
motion transformation between any pair of frames involves six motion parameters: three for the elementary translations and three for the elementary rotations. a simple solution is to constrain one of the X. the joint at the proximal node is called the proximal joint and the one at the distal node is called the distal joint.Mechanical System of Robots 71 3.3. the node of a binary link which is closer to the base link (when measured along the open kinematicchain) is called the proximal node. the open kinematicchain has one base link and three binary links. As for kinematic analysis. it is important to develop a theoretical framework for the modelling and analysis of an open kinematicchain. This method is called DevanitHartenberg (or DH) representation. an open kinematicchain can be treated as a simple open kinematicchain with a clear designation of its base and endeffector links.3. the question is: Where do we place frame i assigned to it? If we want to reduce the number of motion parameters.1 Assignment of a Link's Frame As we discussed in Chapter 2. The basic idea behind DH representation can be explained as follows: At a time instant. Y. With reference to a link. 3. it is desirable to have a systematic method. In general. An open kinematicchain consists of a set of serially connected links. the starting point for kinematic analysis is to study the kinematics of a link. it is imperative to constrain a frame's orientation and origin as much as possible. In robotics. In robotics. while the other node is called the distal node. a common way is to align the Z axis of a link's frame with . For clarity's sake.1. the two nodes of which are merged into one. Thus. the study of motions among rigid bodies is equivalent to the study of motions among frames assigned to the rigid bodies under consideration. We treat a unary link as a degenerated binary link. or Z axes of frame i by aligning it with the motion axis of the joint. there is a simple method which only requires four motion parameters to represent the relative posture between two consecutive links. Since a robot's mechanism can be treated as the combination of a set of open kinematicchains.1 Kinematics of a Link It is a common practice in science and engineering to develop a systematic approach to describe any complex system. Since two consecutive links form a kinematic pair (the corresponding joint has one degree of freedom).3. For link i inside a simple open kinematicchain. For the example shown in Fig.7.
and °fcj_i (i.72 The Fundamentals of Robotics: Linking Perception to Action the motion axis of one of its two joints.13 Designation of link i and joint i. It is important to note that the endeffector link in a simple open kinematicchain is physically driven by its proximal joint when an actuator applies force/torque to it.13. 3.13. this common normal vector defines the X axis of frame i. Example 3. so that a Cartesian coordinate system is formed for frame i. 3.e. the question is: Should we align the Z axis of frame i with the motion axis of the proximal joint or the distal joint? The obvious answer is to align it with the motion axis of the distal joint.14 shows a link which has nonparallel motion axes . This can be done by placing the origin at the intersection point between the Zi axis (its direction is represented by °ki with respect to frame 0) and the common normal vector from the Zj_i axis to the Zi axis. 3. as illustrated in Fig. the direction vector of the Zj_i axis with respect to frame 0) are parallel to each other (as shown in Fig. 3. including frame assignment to link i. For easy notation.10 Fig. Finally. Once we have imposed a constraint on the Z axis of a link's frame. because the robot mechanism's output motion is usually at the distal joint of the endeffector link. If °fc. And. Now. the Y axis of frame i is determined by the righthand Rule. Proximal Joint Distal Joint Fig. 3. This fact is also true for the rest of the links inside a simple open kinematicchain. it is also easy to impose a constraint on the link frame's origin along the Z axis.13). the proximal joint of link i is called joint i. the origin of frame i is chosen to be at the center of the distal joint. The designation of link i and joint i is also illustrated in Fig.
11 Fig. and the X axis is determined by the crossproduct of the Zi and Zii axes' base vectors: ti = Ki X Ki — \ • 2.15 A link which has motion axes at the proximal and distal joints that are coplanar and intersect at a point. 3. Fig.15 shows a link which has motion axes at the proximal and distal joints that are coplanar and intersect at a point. oooooooooooooooooo Example 3. In this case.Mechanical System of Robots 73 at the proximal and distal joints. 3. In this case. Fig. A. it is easy to determine the origin of the frame assigned to the link.14 A link which has nonparallel motion axes at the proximal and distal joints are not parallel. z. It is at the center of the distal joint. the origin of the frame assigned to the link is at this intersection point.. 3. .
there is no reason to design two consecutive links in such an arrangement. This normal vector indicates the X axis of the frame assigned to the link.}. 3. we make use of two parallel planes to better illustrate the frame assignment to link i. 3.74 The Fundamentals of Robotics: Linking Perception to Action ••OOOOOOOOOOOOOOOO Example 3.. the Xi axis). plane pi contains the Zi axis while plane pii contains the . And.e. If we know the common normal vector °ij (i. 3. In practice. » _ i Fig. Let us assume that the Zii and Zt axes are not collinear (i. It also shows in a clearer way that the minimum set of four parameters is necessary to fully describe the relative posture between the two consecutive links.1.e.16 A link which has motion axes at the proximal and distal joints that are in a general configuration. both using this vector as their normal vector. the common normal vector from the Z^x axis to the Zt axis is % = {%} x {%_•.3.12 Fig. we can define two parallel planes pi and pii. oooooooooooooooooo 3. not on the same line). In this case.16 shows a link which has motion axes at the proximal and distal joints that are in a general configuration. Z.2 Geometric Parameters of a Link In Fig.17.
By applying two successive elementary motions: the elementary rotation about the Xi axis followed by the elementary translation along the Xi axis. This angle ft indicates an elementary rotation of frame i about the Xi axis if we want to bring frame i back to frame i — 1.Mechanical System of Robots 75 z. . Now. Zii axis.17. 3. If joint i of the simple open kinematicchain is a revolute joint. 3. it becomes clear that the third DH parameter is rotation angle 0j about the Zk axis which is aligned with the Zj_i axis.17 Illustration of DH parameters. Qi = 8i).e. Obviously. Frame z — 1 is placed in a general orientation which depends on link i — 1.. In fact. axis if we want to bring frame i back to frame i — 1. This rotation aligns frame k parallel to frame i — 1. frame i will reach an intermediate configuration which is denoted by frame k in Fig. Refer to Fig. this distance 6* indicates the linear displacement of an elementary translation by frame i along the X. using two parallel planes.. 3.17 again. It is clear that the first DH parameter is the rotation angle ft about the Xi axis which makes the Zi axis parallel to the Zii axis. ^ Zlj7T" I f™> / / Pii /S Fig. 3. then 6i is the joint variable corresponding to link i (i. Refer to Fig.17. the second DH parameter is the distance bi between these two parallel planes.
13 In Fig. (2) &. 3. The linear displacement di of this elementary translation is the fourth DH parameter. Based on the frame assignments. shown in Fig. 3. 3. As a result. In summary.3. rotation angle about the Zi—\ axis. 3. DH representation only requires a set of four parameters to fully determine the relative posture between two consecutive frames. we can see that the DH parameters are not the same if we alter the designation of the base link and endeffector link for the same open kinematicchain.1. the right arm and right leg are simple open kinematicchains with four planar links (including the common base link). (3) 9i'. which is the joint variable if joint i is a prismatic joint. linear displacement along the Z^\ axis. it is necessary to make a final elementary translation along the Zi\ axis in order to move the origin of frame k to coincide with the origin of frame i — 1. The frame assignments and the geometrical parameters of the right arm and right leg.9. the DH parameters of the simple open kinematicchain shown in Fig.: linear displacement along the Xi axis. These four parameters are: (1) fa: rotation angle about the Xi axis. Let us denote l~1M$ .3 Motion Transformation Matrix of a Link If we know the DH parameters of link i.18 are: Link 1 2 fii bi 0~~^ 0 l2 di di 0[ 0~ 02 0 3 I 0 \ h  6>3 1 0 oooooooooooooooooo From the above discussions. Example 3.18. 3. an open kinematicchain requires two sets of DH parameters to fully describe its kinematic property. especially in the case of the humanoid robot. (4) di'. & = di). are generically illustrated by Fig.9. which is the joint variable if joint i is a revolute joint. we can quickly and systematically establish the motion transformation matrix which describes the position and orientation of frame i with respect to frame i — 1.76 The Fundamentals of Robotics: Linking Perception to Action Finally. It is clear that this fourth parameter di will be the joint variable if joint i is a prismatic joint (i.e.
the position and 0 the following 1 0 *•«. [6A) 0 sin(A) cos(ft) 0 • frame i — 1. 3. which describes the position and orientation of frame i with respect to frame k as follows: ( transformation matrix: 1 0 0 bi\ (3. 3. It is easy to write the homogeneous motiontransformation matrix kMi. An angle takes a positive value if the rotation is clockwise about the axis of rotation.18 Simple open kinematicchain with four planar links (including the base link). 3.) T \ 0 sin(6i) co f J :: 0 0 0 0\ M 1/ . we have to pay attention to the signs of the values of the DH parameters. Frame w is a world coordinate system. is fully determinedorientation 0of frame/ k. =/cos(<9.1) 0 cos(A) sin(A) 0 I . where 9i is the joint variable. (/3i. and a linear displacement takes a positive value if the translation is towards the negative axis of translation. Refer to Fig. the motion transformation matrix from frame i to frame i — 1. For the example shown in Fig.17.17. Based on this notation.Mechanical System of Robots 77 ik w •*„ Fig.di) take negative values and bi has a positive value. with respect to by homogeneous motionSimilarly. Refer to Fig.17 again. 3.
This is called motion kinematics or differential kinematics.C0i o spi Spt. The position of frame i.78 The Fundamentals of Robotics: Linking Perception to Action The combination of Eq. However.2 yields iiA/r _ l~ where Spi = sin(/3i).3 describes the configuration (or posture) of frame i with respect to frame i — 1. with respect to frame i — 1.Set bi»C0i\ Spi.SOl sei Cf3z. 9i is the joint variable). matrix in robotics. 3.1 and Eq. 50i = sin(fii). 5/3< CA / (3.5) \ 0 From Eq. 3. di). regardless of whether it is a revolute joint (meaning 0j is the joint variable) or prismatic joint (meaning di is the joint variable).e. 3. V di (3. is i1Ri=\sei cfc^cOi spt.ceA. 3. /COi Cpi.4 Linear/Circular Velocity of a Link Eq.C6ih.3 is called the /mA. is (k.3. CPi = cos(Pi).4.COA i~1O1 =\bi*Sei\.SoA cpt di {66) Vo o o i / Eq. C0i = sin(0i). with respect to frame i — 1. and summarizes both the position and orientation of frame i with respect to frame i — 1. 3. we can see that the position of frame i depends on (#*. 3. this is not the case for the orientation of frame i which will only undergo changes if the joint is revolute (i.1. If joint i is a prismatic joint.4) ) And the orientation of frame i. we have qz = di. This means that the position of frame i always undergoes changes. Another important property of robot kinematics is the differential relationship between the joint variable and the configuration of frame i. .
= (0.4 with respect to time. is { I1 fcj_i • <ji • qt if prismatic joint (3. That is. as follows: i~1Oi= i~1h1*qi.7) Obviously. On the other hand.H) In summary. = C " 1 ^ ) x ^lOi) (3. we can obtain the linear velocity vector at the origin of frame i with respect to frame i — 1.12) {(i~1fcj) x i^Oi)} if revolute joint. the circular velocity vector of any point P in frame i with respect to frame i — 1 will be iip where i~l^i = dV^P} = at ( i _ l w i ) x( i _ l p ) (3 g) (3.e. (3.9) If we choose point P to be the origin of frame i. qi. with respect to frame i — 1. 3.* ^ . a prismatic joint will not cause any changes in the orientation of frame i.0. the velocity vector at the origin of frame i. " A .10) or ^di = {r%i) x riOi)}. ) • ( • ) • * • < ' " i~1kii This linear velocity vector is in the direction of the ^ _ i axis (i. (3. If we denote l~1u>i the angular velocity vector of frame i with respect to frame i — 1. (312) . Let l~1u>i be the angular velocity vector of frame i.Mechanical System of Robots 79 By differentiating Eq.( . (3. we have i"1O. The above equation can be rewritten. we have l~xu>i = 0 if joint i is a prismatic joint.8) = *%! • ft. the joint variable for a revolute joint i is ql = 9i.1)*). Due to the instanta/ neous change of joint variable <$ about the Zi\ axis.
gi. . Link 0 is the base link and link n is the endeffector link.3.14) In summary.2 0 if prismatic joint (3.15) l~1k <ji Forward Kinematicsii of• Openif revolute joint. the angular velocity vector of frame i with respect to frame z — 1 is { 3. If we denote qt the joint variable corresponding to link i.92.1. That is..3.9) i~1wi= '"1fcl_1.19 A simple open kinematicchain with n + 1 links.13) If joint i is revolute.. it is clear that there is no angular velocity if joint i is prismatic. (3. the angular velocity of frame i with respect to frame i — 1 will be (see Eq. 3. KinematicChains Let us consider a simple open kinematicchain having n + 1 links as shown in Fig 3. 7 (3.16) Joint n Joint 4 \ Joint 2 /^t^— Js^l «^~~W^^^ / Endeffector Link n / S a Joint 1 v M Base Link Fig.«„)*. (3. .. then joint variable vector q of the simple open kinematicchain will be < = (?i.80 The Fundamentals of Robotics: Linking Perception to Action 3.19. 3. i~1uji = 0.5 Angular Velocity of a Link From the above discussions.
20cm and 8cm respectively. °Mn(q) describes forward kinematics mapping from joint space to task space. which is also a function of the joint variable vector q. frame n) if the joint variable vector and its velocity are known. (9) ) = f{rlMM. If the input of the joint angles are as follows: C qi = 37.0.. position and orientation) and velocity of the endeffector's frame (i. 3. n — 2.Mechanical System of Robots 81 Here.3.18.. The orientation of frame n. can be directly computed from Eq.9.5° { q3 = 36.17) °Mn(q) = CRfQ) ° T . The lengths of the upper link (Link 1). with respect to frame 0.0)). The origin of the base link's frame (frame 0) is at (0. which is a function of the joint variable vector q.14 Refer to Fig. which pass through frames n — 1. 3. the objective of the study of forward kinematics is to determine the configuration (i.e.1 Determination of the Endeffector's Posture If we denote l~1Mi(qi) the homogeneous motiontransformation of frame i with respect to frame i — 1. is represented by rotation matrix °Rn(q). . the position of frame n's origin. The physical meaning of the above equation is that frame n will coincide with frame 0 after a series of successive motions. Example 3.. and hand link (Link 3) of the right arm are 20c?n.. ) } or (3. and 1. the position and orientation of frame n. with respect to frame 0. then the homogeneous motiontransformation of frame n (the endeffector link's frame) with respect to frame 0 (the base link's frame) will be as follows: °Mn(q) = f M ^ i ) } • {lM2(q2)} .e.0°. 3. . is represented by translation vector °Tn(q). We have joint variable vector q.5° < q2 = 98. Therefore. And. 80cm) in the world frame. with respect to frame 0..2. lower link (Link 2). . { " " ^ ( f t . .18) (NOTE: 0 = (0. The right arm is treated as a simple open kinematicchain with the upper body as its base link.)} (3.
with respect to frame 0. 3. If the input of the joint angles are as follows: r qi = 72.9 again. The lengths of the upper link (Link 1). the linear velocity vector (the combined effect of linear and circular velocities) at the origin of frame n will be the sum of results obtained from all the joints. 2. •••OOOOOOOOOOOOOOO Example 3. and joint 2 will independently move the subassembly consisting of links 2. Refer to Fig.. joint 1 will independently move the assembly consisting of links 1.19.15 Refer to Fig.7946cm \y = 13. In other words. and foot link (Link 3) of the right leg are 20cm. with respect to the world frame. with respect to the world frame. until n. when the velocities q of the joint variables q are given? For simplicity's sake. 3.. will be at f x = 33.9089cm.2 Determination of the Endeffector's Velocity In a simple open kinematicchain. As the joints independently act on their corresponding subassemblies of links.0° ) q2 = 90. The right leg is also treated as a simple open kinematicchain with the upper body as its base link. OOOOOOOOOOOOOOOOOO 3. we use the term linear to mean the combined effect of linear and circular velocities which act at the origin of a frame. 3. 20cm and 15cm respectively. until n etc.2.6216cm.0° [ q3 = 103.40cm) in the world frame.9469cm \y = 79. Now. .5°. is at ( x = 27. then the origin of the foot link's frame. the question is: What will the linear and angular velocities of frame n be.3.. lower link (Link 2). the constraints imposed by the joints are independent of each other. The origin of the base link's frame (frame 0) is at (0.82 The Fundamentals of Robotics: Linking Perception to Action then the origin of the hand link's frame. .
By applying Eq.i n[qi> °^i~1 * ^ [{(lilx^.x _ [ t~1kii • Qi \ {('%!) x C'^n)} • qt if prismatic joint if revolute joint . the linear velocity vector caused by joint i at the origin of frame n.1 On = °On with /0\ °On = ° M n . That is.12. (3. 3.) = Joi»9i. . with respect to frame 0. 3. 3. let us compute the sum of the results obtained from all the joints.21) W If we substitute Eq.21 into Eq. let us examine the contribution of joint i to the linear and angular velocities of frame n: We consider that the subassembly from links i until n forms a single link supporting frame n. l e oue ' As i~1On is the position vector from the origin of frame n to the origin of frame i — 1.19. °h C \ . By multiplying rotation matrix °iJj_i to both sides of Eq._!)} we have °6 n (q.20. l n[qt) " ' where l~10n is the position vector from the origin of frame n to the origin of frame i — 1.^ if P r i s m a t i c J ° i n t (o 9 n \ if r v l t joint. will be HA . with respect .°O. we have °Ri1 . ' .! = "Mi! * /0\ o • (322) 0 Oi_i (3. 3. The final result of the linear velocity at the origin of frame n. °Q OO.23) l J (3. \i/ if prismatic joint if revolute joint.^J}. we obtain the linear velocity vector at the origin of frame n. with respect to frame i — 1.24) Now. and define f ofc2_i \ {(°fcii) x (°On .Mechanical System of Robots 83 Now.
^>n(Qi) (0 = S i_i^ . • • . will be °On(q) = £{°O n fe)} = itVoi • %} »=i j=i (3.28 can be rewritten as follows: °Lon{qi) = J^i .26) {Joli I • • Jo2j •••> J o r a J • \t /„ o 7 \ l<7= (9l. 3. and If we multiply rotation matrix °Rii define _ / 0 I Kj_i if prismatic joint it revoiute joint. the final result of the angular velocity vector of frame n. qt. (O. ^ w l ) ^o)2> •••> Jujn) 1 • / • • • \t /o o o \ (o.29) then Eq. to both sides of Eq.Zl ) Eq. with respect to frame i — 1.15. 3.26 describes the mapping of the joint variables' velocities to the linear velocity at the origin of frame n. Similarly. if prismatic joint . the angular velocity vector of frame n caused by joint i. will be n n <W<7) = £{V>fe)} = £ i J « i • Qi} i=l i=l (3.25) or in matrix form °On(q) = Jo*q with I Jo 1 • = (3. (3. if we compute the sum of results obtained from all the joints.28. with respect to frame 0.. will be ii (•. 3.31) or in matrix form °un{q) = Ju*q with I "^OJ = = ( . .84 The Fundamentals of Robotics: Linking Perception to Action to frame 0.r . by applying Eq.ZOJ I Kj_i • % if revolute joint. (3.oo) (3. 3.30) Now.32) .?n) • lo.92.
This mapping is fully conditioned by matrix J(q).36) Eq.° O0) {°k) x (°O3 . with respect to frame 0 (the upper body). It is clear that the inverse mapping is manytoone if n > m.9. and further define Aq)=(t)' the combination of Eq.36 describes the mapping from the joint variables' velocities to the velocity of frame n. 3. •ooooooooooooooooo .32 will yield P = J(q) • q. the lower link and the hand link of the right arm as Links 1. We name the upper link. 3.26 and Eq. If the dimension of P is m (m < 6). /0\ °\ °O 2 = °M 2 . ('5 33) (3. J(q) is aTOx n matrix. Example 3. 3.34) the velocity vector of frame n. 3. \e2 = J(q) . The velocity of frame 3. It is worth noting that the Jacobian matrix is not a constant matrix and it depends on the actual values of the joint variables.Mechanical System of Robots 85 Eq.32 describes the mapping of the joint variables' velocities to the angular velocity vector of frame n. If we denote P=(°o£) (3. 3.° O2) \ % 0\ % °k ) ( /0\. q W with J(q) = and { / (%) x (°O3 . 2 and 3 respectively.° Oi) (%) x (°O3 . ° °O1= "Mf I . will be P = J(q) . commonly called the Jacobian matrix. with respect to frame 0.16 Refer to Fig.
From the illustration shown in Fig.3 Inverse Kinematics of Open KinematicChains The robot mechanism's output motion is usually at the endeffector link of a simple open kinematicchain. which is the robot mechanism's input motion. Then.86 The Fundamentals of Robotics: Linking Perception to Action 3.3. The motion specification in task space needs to be mapped into joint space. it is simple to compute the robot mechanism's output motion because of the simple formulation of forward kinematics. The purpose of robot control is to apply energy into the robot's mechanism which will alter the dynamic behavior of the robot. as the direct input to robot control is the desired motion in joint space. The direct outcome of robot control is the executed motion in joint space. \^^ I  1 V^ in Joint s \ ^ ir J "^/ ^^y ^~— Forward Kinematics A Robot Mechanism ^ "^ ~ ^ ~ ( Executed Motion \^ V • T • c J~ \ m Joint S p a c e / Robot Control ^r  Robot Dynamics Fig. the robot mechanism's . If we know the joint variables and their velocities. 3.20 The motionrelated data flow inside a robot.> T Inverse Kinematics y I ~~~~W^^ ^^X 1 Y Desired Motion A V in Task Space . 3.20. Therefore. the executed motion in joint space is physically manifested by the robot mechanism's output motion in task space. it is clear that the desired motion is specified in task space. the question is: Why is there a need to study inverse kinematics? ( Tasks J Motion Planning & Programming /< / Motion ^ ^ V . Let us examine the motionrelated data flow inside a robot. Through the robot's mechanism.
the question is: What should the joint angles . ( ax sx nx tx\ (3.19. Frame 3 is the hand's frame for the right arm (or the foot's frame.. Now. 3. it is unlikely for us to derive a closed form solution from Eq. the sliding vector and the normal vector of the endeffector's frame.9 and Fig. . 3.3. Example 3.1/3. for a given task at a timeinstant. what we want to determine is the set of joint angles q. The input for the determination of joint displacements is the posture or configuration (position and orientation) of the endeffector's frame with respect to the base frame. it determines the ability of a robot to fulfill a task through the execution of motion in joint space. as shown in Fig.0)' is the origin of frame 3 with respect to frame 0. 3. s = (sx^y^z)* and n = {nx.38 describes the equality of two matrices.17 Refer to Fig. It implies that their corresponding elements must be equal to each other.1 M n (g n )}. (3.38.Mechanical System of Robots 87 inverse kinematics is crucial.19. s and ft coincide respectively with the X.38) Eq. SP = sin(/3) and (23. .18. 3. Y and Z axes of frame n. the configuration of frame 3 needs to be (CP ~S/3 0 x3\ oM 3 = (% ° J 3 °fc 3 ° O 3 ^ I SP CP 0 2/3 ^ o o o i y I o \ 0 o 0 i 0 o 1/ where CP = cos(/3). For consistency of notation with the planar open kinematicchain.az)1.nzY are called the approaching vector. 3. Now. the objective of inverse kinematics is to compute the desired motions of the joint variables for a given motion specification of the endeffector's frame in task space. In general.37) ay sy ny ty az sz nz tz I 0 0 0 1/ where a = {aX)ay.. Accordingly. which satisfies the following equation: °Mn = f M i f o ) } • {lM2(q2)} . Assume that.3. 3. That is.1 Determination of Joint Displacements Refer to Fig. in the case of the right leg).ny. {". we consider that a. 3.
88 The Fundamentals of Robotics: Linking Perception to Action (01.l3 • {%}• If we denote 0 O 2 = (a. frame 2's origin.02. That is.03) be? Refer to Example 3. 3.14 and Eq. with respect to frame 0. The three link matrices of the right arm are as follows: ( C6i SOi 0 J x C0i\ 50! ce1 0 0 502 o i1sel 0 ' 0 1 / 0 Z 2 C0 2 \ 1 ( 2jl.0)*. °O 2 = °O 3 . we can easily establish the following equation: f i2c(e1 + 02) + i1cel = x2 \i2s{el + e2) + hS6l = y2 (66y) . 3. 0 0 C02 50 2 0 0 C02 0 0 C03 0 12S02 1 0 ' 0 1 / /C0 3 50 3 Ms~ 503 o o 0 0 i i3ce3\ o 1 /  0 13S03 o V 0 (NOTE: CO = cos(0) and 50 = sin{6)). can be expressed in terms of frame 3's origin with an offset of l3 along the X axis.y2.{ 1 M 2 } yields (C{e1 + e2) 5(0i + 02) o i2c(e1 + e2) + i1ce1\ On. 5(0x+ 02) C(0i+02) 0 Z25(0! + 0 2 )+Zi50i M 2 = o o i o \ 0 0 0 1 / According to the frame assignment shown in Fig.2. Substituting °M\ and xMi into the equation °M2 = {°M 1 }.3.18.
\CViJ Analytically.39 in a matrix form in terms of the unknown 8\ as follows: fh+hCOi hS02\ [C0i\ { i2se2 il + i2ce2)*{sej or = fx2\ \y2) \sej~ with k \ i2se2 h + i2ce2 )\y2) k = {h + i2ce2f + {i2se2)2 = i\ + i22 + 2hhce2. If we know CQ\ and S#i.Mechanical System of Robots 89 with / x2 = x3 . from the equality of the corresponding elements at location (1. the mapping from joint space to task space is usually a manytoone mapping). the solution for the unknown 62 is ^ = ±arCCOS( 2^2 J Two possible solutions for 62 mean that the mapping from task space to joint space is usually a onetomany mapping (or conversely. 3.39. the solution for the unknown Q\ will be &i = arctan ( —^. From Eq. we can easily obtain the solution for the unknown 63 as follows: t/3 — p — U\ — u2.kS/3. 3. 1) of the matrices in the following equation: 0M3 = {°M1}»{lM2}»{2M3}. oooooooooooooooooo . the computation of x\ + y\ yields 2lxl2C82=x\+yll\l\. Knowing 82.) . Hence. we can now rewrite Eq.l3Cf3 12/2 = 2/3 .
2922°.6 c m \y3 — xw = 29cm.21. expressed in frame 0. 3.5844° [ 03 = 57.O. 1 and 2. 3. is described by ( (33. coordinates of this point. / 2 \ . with respect to the base frame (frame 0) of the right arm.18 Refer to Fig.3972° I 62 = 114. and choosing j 13 = aictan ( g ) the computed joint angles of the right arm (as shown in Fig.l)t The f £3 = 80 . . Example 3. By applying the inversekinematic solution discussed in the previous example.yw = .1)*= or 0 . we want to move frame 3's origin to J xw =29cm \ yw = 86cm where the coordinates (xw.0. The posture of the world frame.21) are r <9i = 44. will be °Mw»(xw. 2_ X 2 2 ^ The above expression is precisely the proof of the cosine theorem applied to the triangle formed by the origins of frames 0. And angle 7 is the inner angle between the two edges corresponding to link 1 and link 2.yw.03.1 0 80\ 1 0 0 0 o o i o 0 0 0 1/ Now.yw) are expressed in the world frame.90 The Fundamentals of Robotics: Linking Perception to Action If we examine the solution for #2 and define 7 — IT ± #2 we have COS(7) = COs(7T ± 0 2 ) = / l + 72 .
is described by /0 oM _ 1 Mw~ o \0 .. If we choose J f3 = arctan 0&) + 60 .EM!  [»J Examples of an inversekinematic solution for t h e right a r m and right leg.1 0 o 0 0 0 i 0 40\ 0 o • 1/ For the posture of the right leg shown in Fig. the posture of the world frame. Similarly. \C£ j Fig.21. 3. 3. 40 ffHH^MHijtt :::::::::::::::::::::::::::  iGatwna(Leg) "* oc " °20 0 20 40 60 Bo 80 LJ . with respect to the base frame (frame 0) of the right leg.yw = 18cm \ 2/3 = xw = 30cm. the origin of frame 3 of the right leg is {x w = 30cm yw — 22cm expressed in the world frame.21 [ Home  . with respect to the base frame (frame 0) are f x3 = 40 . Its coordinates.Mechanical System of Robots 91 •I 2D Humanoid Robot in Posture of Seating tf 100 pi _j iAipNa(Arm} 111111  ^ ^ S J T T T I T n T  111111] 111111 IITTI11111 F ^ WphaM^ — iBeta (Arm) J 70 ::::: flSi^fflfBR:::::::::::::::::::::::::::: 8ete_CfS) Gamma(Arm).
all motions intrinsically involve velocity. Let us denote q(t) the joint velocity vector. In general. N . 000000000000000000 For a simple open kinematicchain with n+1 links. Thus.41 can also be written in the following recursive form: q{tk+l) = q{tk) + q{tk) • At. in joint space) to a desired final configuration within time interval [£».42) In the next section. But. will be { 0i = 74.92 The Fundamentals of Robotics: Linking Perception to Action the computed joint angles of the right leg.2 Determination of Joint Velocities The dynamics of task execution is more appropriately illustrated by the behavior of a trajectory following.0677° 03 = 125. Mk £ [0. 3.1].3. 3. 3.41) with N = tf^i • If we denote tk = U + k» At. Eq.e. as shown in Fig.2736° 62 = 81. it is clearly difficult in practice to obtain a closedform solution for determining the joint variables. (3.21. then the joint variables at time instant tf will be q{tf) = q{U) + f ' q(t)dt. we will discuss in detail the numerical solution for the computation of inverse kinematics. (3.40) Ju If we perform the integration in a discrete time domain with the sampling step of At. For a given task.8303°. If q(t) follows a trajectory from an initial configuration (i. as shown in Fig. 3. it is crucial to know how to determine the joint variables' velocities q if the endeffector .3. 3. Eq.22. it is possible to derive a general numerical solution from knowledge of the velocities of the joint variables. the desired motion for the fulfillment of the task is normally described in the form of a trajectory which is a spatial curve with a time constraint.40 can numerically be computed as follows: JV1 q(tf) = q{U) + J2 4(U + * • At) • At fe=0 (3.£/].
e ^ Fig.36.22 Trajectory following by the endeffector's frame in task space. the endeffector's frame is considered to coincide with the tool's frame.44) . as shown in Fig. Thus.22. Recall Eq.Mechanical System of Robots 93 frame's velocity vector P is given as input (NOTE: P is a vector which denotes the posture or configuration of a frame). Let us assume that frame n has already been taken into account the offset between the frame of the last link in a simple open kinematicchain and the frame assigned to a tool. 3. We will study how to plan the corresponding trajectory or spatial path for a given task in Chapter 9. robotic task execution is accomplished by the endeffector's frame which follows the planned trajectory represented by the time evolution of frame e. A given task's planned spatial curve describes both the position and orientation of the tool's frame denoted by frame e. 3. The relationship between joint velocity vector q and velocity vector P of the endeffector's frame is as follows: P = J(q) • q with (3. (3.43) P=C°:). In this way. 3.
It is clear that the pseudoinverse does not exist if matrix J*J is not of full rank. In other words. it is easy to see that the unknown q is overconstrained if 77i > n. when determining the joint velocity vector. the simple open kinematicchain has more degrees of freedom than necessary to configure the endeffector's frame to a desired posture. 3. But there is one approximate solution exists based on an optimal estimation. In this situation. From a mathematical point of view. it is useful to find a solution among the infinite number of possible solutions that requires the (3. 3. 3. This is a very advantageous situation in robotics because the redundancy allows us to impose constraints. m < n)./*</) to both sides of Eq. In this way. it is necessary to pay attention to whether or not the simple open kinematicchain is in a singular posture.e. From Eq. For a given posture of the endeffector's frame. there may not be any feasible solution. In this case.e. The first three elements of P are the endeffector frame's linear velocity vector. we obtain q = ji»P with jt = (jtjy1 • j \ Matrix j t is commonly called the pseudoinverse of the Jacobian matrix J. there is no unique solution for q if there is a kinematic redundancy (i.43 again.45) • Step 2: By multiplying the inverse of (. we can derive the solution for joint velocity vector q as follows: • Step 1: Multiplication of the transpose of J to both sides of Eq. The last three elements of P are the endeffector frame's angular velocity vector. 3.45. In this case.43. (3. the simple open kinematicchain is said to be kinematically redundant.46) . In general.94 The Fundamentals of Robotics: Linking Perception to Action If P is aTOx 1 vector in a threedimensional Cartesian space. the maximum dimension of P will be six (i. the number of feasible solutions is infinite. the simple open kinematicchain is said to be in a singular posture. Therefore. Refer to Eq. ifTO> n. the dimension of the Jacobian matrix J will beTOx n.44 yields Jt*P = (JtJ)»q. optical solutions can be obtained by optimization. If a simple open kinematicchain has n+1 links (inclusive of the base link).TO= 6). Joint velocity vector q is underconstrained ifTO< n. In fact.
50.49) • Step 3: Substituting Eq. q) (3.Mechanical System of Robots 95 least amount of energy.48 into Eq.J .. 3.51 provides the solution for q as follows: q = J^P (3.52) . a new cost function using Lagrangian Multipliers is constructed as follows: /(<?. Substituting it into the second equation in Eq. • Step 2: The solution for q that we are interested in must satisfy the following necessary conditions: dq u (3. The minimumenergy solution is the one which guarantees the shortest path in joint space. it is the solution which minimizes the following cost function: f(q) = \tf»4) (3.51) A= {J*Jt)1*P.\ A : 0 ° • Step 4: From the first equation in Eq. substituting A into Eq.43. 3.48) where A is an m x 1 vector whose elements are called Lagrangian Multipliers. A) = ~ (q* . we have q = Jt»X.49 yields {r/. 3. 3. In other words.50 yields P=(JmJt)mX or <»•») (3. • Step 5: Finally. 3. q) + A* • (P . 3.47) The minimumenergy solution for q can be derived by using Lagrangian multipliers as follows: • Step 1: Considering the primary constraint expressed by Eq.
it is obvious that the sum of qn + qa is generally not a solution for q satisfying Eq. The physical meaning of internal motion can mathematically be described as the difference between any two feasible solutions of q for a given input of P. This requirement can easily be translated into a cost function as follows: f{q) = \{{iqa)t*(qqa)}(353) By applying Lagrangian multipliers. Since there are an infinite number of feasible solutions to q for a redundant open kinematicchain. 3.96 The Fundamentals of Robotics: Linking Perception to Action with J t = J * . 3. it has to be clear that not any given velocity vector of q can be treated as an internal motion vector. 3.43 and Eq. we have full freedom to impose an internal motion. Another useful constraint in robotics is called the internalmotion of a kinematicallyredundant open kinematicchain. the corresponding internal motion vector will be ? = ??• If qn is the minimumenergy solution of q. there is an analytical method for us to map any given velocity vector of q to its corresponding internal motion vector.53 can be derived as follows: • Step 1: Considering the primary constraint expressed by Eq.qa)} + \* • (P . a new cost function using Lagrangian Multipliers is constructed as follows: M A) = \{{q . the solution satisfying Eq. 3. J*)l where J t is also called the pseudoinverse of J. What we can hope to obtain is a feasible solution which is as close to qa as possible. If we denote q1 as feasible solution 1 and q2 as feasible solution 2. ( J . the sum of «n+<r is also a feasible solution of q which satisfies Eq.43.43. Fortunately. This powerful method can be described as follows: For any given velocity vector qa in joint (velocity) space.43.QaY • (q . 3. However.J • q) (3.54) .
Now. Substituting it into the second equation in Eq. substituting A into Eq.A or A = ( J . and / is an identity matrix. (3.58) Eq.55 yields H*aJt'A = 0 \PJ*q = 0. 3. 3.57) • Step 5: Finally. assume that the endeffector frame stands still. 3. ( J .56 yields P = J*qa + {J* J*). 3. Jt)1»{PJ*qa). • Step 4: Prom the first equation in Eq.n (3.54 into Eq. as follows: q=ji»P+[Iji»J]»qa with Jt = J * .57 gives the solution for q. 3. 3. we have q = qa + Jl • A. (3.Mechanical System of Robots 97 where A is an m x 1 vector. Since P = 0. J]mqa. J ' ) " 1 where j t is again the pseudoinverse of J. 3.56.58 becomes q=[IJ t .56) V ' (3.59) (3. the elements of which are called Lagrangian Multipliers.59 physically means that the mapping of any given velocity vector in joint (velocity) space into its corresponding internal motion vector will . Eq. • Step 2: The solution for q that we're interested in must satisfy the following conditions: (910 \ dq ~ U I 9/ .55) • Step 3: Substituting Eq.
q°_ (3. It is denoted by N(J). 3. Jacobian Matrix J maps the null space into the zerovelocity vector in task (velocity) space. In terms of the null space of J. such as: collision with obstacles. The general guideline for choosing an internal motion constraint is simple.43 are called the range of J.23. and denoted by R(J). or joint limits. As illustrated in Fig. all the internal motion vectors in joint (velocity) space form a subspace which is called the Null space of J. J] . If we construct an objective function g(q) which measures the extent to which a simple open kinematicchain is reaching a particular configuration. If we denote ql the internal motion vector corresponding to qa.e. 3. Eq. One common method is to align the internal motion vector in the direction that makes a simple open kinematicchain either approach or move away from a particular configuration. How to advantageously explore kinematic redundancy depends on the applications. J • q = 0). the internal motion vector can . On the other hand. The range of J is the reachable subspace of task (velocity) space from the mapping using Jacobian Matrix J. And. we have qi = [I .Jt . 3.60 means the mapping from any given velocity vector qa into the null space.98 The Fundamentals of Robotics: Linking Perception to Action not cause any change to the endeffector's frame (i.23 Mapping from joint space to task space using the Jacobian J. 3. singularities.60) For a posture of a simple open kinematic chain: / Joint Space \ J ^ ^ f Task Space \ Fig. all feasible velocity vectors P obtained from q by Eq.
If we choose the "+" sign in the above equation. Example 3. and d) effect due to the internal motion of four joint variables. •••••••ooooooooooo . 3. two.24 Illustration of the effects of internal motion on a simple open kinematicchain with 20 degrees of freedom: a) the initial posture.19 Fig. this means that the simple open kinematicchain should approach a desired configuration. it means it should move away from a particular configuration. c) effect due to the internal motion of two joint variables.Mechanical System of Robots 99 be chosen as dq where c is a coefficient which determines the magnitude of the internal motion vector. (a) (b) (c) (d) Fig. Otherwise. Notice the endeffector frame stands still after successive applications of internal motions on one. and four joint variables.24 shows the results of internal motions applied to a simple open kinematicchain with 20 degrees of freedom. b) effect due to the internal motion of one joint variable. 3.
These displacements correspond to a series of incremental motion transformations eMe(tk. 3. the joint velocity vector can be computed analytically from Eq.{l[leMe(tk.3. q(tf)). 3. In practice.tk+1)}} fc=0 (3. 3. Refer to Fig.46 or Eq. which is denoted by °Mn(tf). Assume that the initial posture or configuration of the endeffector's frame is at time instant £$. These solutions are crucial for velocity control in joint space.61) = {°Me(tk)}1 • {°Me(tk+1)} (3. we can choose (U.e.tk+i\. 3. 3.tk+i) of the endeffector's frame within a series of equal time intervals [tk.52 for any given velocity vector of the endeffector's frame.tk+i) indicates the posture or configuration of the endeffector's frame at t^+i with respect to itself at tk. the input will be the final posture or configuration of the endeffector's frame at time instant tf. eMe(tk.tf) . What we want to compute are the joint angles at time instant tf (i. In order to determine joint displacements.62) and At ' (NOTE: At is the sampling step in time. If we know a desired series of incremental motion transformations along a chosen spatial curve. The symbol "e" highlights the fact that the incremental motion transformations are obtained from any chosen spatial curve regardless of the geometrical property of the simple open kinematicchain under consideration.tk+1) 0Me(ti). These solutions also serve as the computational basis for determining the joint displacements if the endeffector frame's posture or configuration is given as input. the desired final posture of the endeffector's frame will be JVl °Me(tf)= with eMe(tk.22 again.3 Numerical Solution for Joint Displacements Refer to Fig.22.100 The Fundamentals of Robotics: Linking Perception to Action In summary.3. The basic idea underlying the numerical solution for determining q(tf) is to compute the series of incremental joint displacements q(tk) (see Eq.42). denoted by °Mn(ti). The origin of the endeffector's frame will travel along a spatial curve which connects the initial posture to the final posture. 3.
63) In fact. Vfce[0. Eq.JV1]. 3. That is. a good strategy is to replace °Me(tk) in Eq.tk\i). Now. tfc+i) to be a constant matrix if the spatial curve is linearly interpolated with a series of piecewise segments.tk+1).61 can equivalently be rewritten as follows: °Me(tk+1) = {°Me(tk)}*{eMe(tk}. (3.1). is expressed as follows: Vc e [O.42 yields q(tk+l)=q(tk) + {ji.tk+i].58 into Eq.67) .65) and we denote (A9. 3. 3. let Pe{tk) be the velocity vector of the endeffector's frame which executes the incremental motion transformation eMe(tk. The accuracy of the computation depends on the scale of the time interval At. Eq.Mechanical System of Robots 101 to be (0.63 is a discrete representation of a given spatial curve because one can choose eMe(tk. efe) the equivalent rotation angle and equivalent rotation axis of eRe(tk.tk+x) within time interval [tk. Eq. 3. In a recursive form.64) f If eMe(tk. 3.tk+1)} \ V = If .tk+i) (3. Substituting Eq. °M n (i f c )= °Mn(q(tk)) (3. {°Rn(tk)} . the velocity vector Pe{tk) of the endeffector's frame at tk will be Pe(tk) = (°o0n) with (366) (°dn = ±* {°Rn(tk)} • {eTe(tk.Pe(tk) + [IJ1»J]*qa}^t. In order to avoid an accumulation of numerical errors. 3. (3. {«fe} where °Rn(tk) is the rotation matrix which represents the orientation of the endeffector's frame at tk with respect to frame 0 (base frame). Then At is automatically computed once we decide the number N of iterations).62 with the computed °Mn(tk) from the solution of forward kinematics.JVl].64 is the recursive algorithm for the computation of the joint variables if the endeffector frame's final posture is given as input.
it also means that (J • J*) loses one rank. 3.25 Example of the use of a numerical solution to solve inverse kinematics: a) robot at its initial posture and b) the final posture reached upon application of the numerical solution. At time U. when one eigenvalue is zero.e.4 Effect of Singularity The determination of joint velocities and the numerical solution for estimating joint displacements all depend on the computation of the inverse of (J • J1). When one or more eigenvalues of (J • J*) approach zero.65 for the computation of Pe(tk).3. <X>0<X>0<X>000<X>00<X>0 3.64.tk+1) = fM^fc)}" 1 •{°Me(tk+1)} (3. ( a) Initial posture ( b ) Final posture Fig. the physical meaning is that the endeffector's frame of a simple open kinematicchain loses one degree of freedom or one mobility. This leads to the use of eMe(tk. In fact. the robot is at its initial posture and the endeffector frame's final posture at time tf is given as input.25 is an example of the use of a numerical solution to solve inverse kinematics. there is no inverse in the strict mathematical sense.20 Fig. the endeffector's frame reaches its desired final posture after 50 iterations (i. Example 3. 3.3. When there is .102 The Fundamentals of Robotics: Linking Perception to Action where q(tk) is the actually computed joint angle vector from Eq. N = 50). When applying the numerical solution to iteratively solve the inverse kinematics. 3. When (J • J*) loses one rank. 3.68) to replace Eq. The arm manipulator has 12 degrees of freedom.
We can see that the endeffector's frame deviates a lot from the desired trajectory at iterations 2 . this example is equivalent to the case when the endeffector's frame follows a predefined trajectory. despite the length of the path and the singularity of the initial posture of the simple open kinematicchain. As a result. • It is even more difficult for the robot to quit the singular posture when approaching final posture 3 because a larger amount of motion in the vertical direction is required. The rank of (J • J*) is 3 if the simple open kinematicchain is not in a singular posture.27 shows the benefit of using error compensation in the numerical solution for joint displacements. it is clear that the endeffector's frame does not have any degree of freedom or mobility in the Y direction (i. a frame in a 2D plane has three degrees of freedom: two for translation and one for rotation.68. • It is a bit difficult for the robot to quit the singular posture when approaching final posture 2 because more effort is needed to move in the vertical direction. it is important to know the effect of singularity on inverse kinematics. 3. The endeffector's frame can closely follow the chosen spatial path which connects the initial posture to final posture 1. Therefore. Fig. the simple open kinematicchain is said to be in a singular posture.e. In this position. As for the initial posture at tit as shown in Fig. vertical axis). Now.21 Fig.26 shows a simulation of a humanoid robot in a standing position. We can see that the endeffector's frame reaches precisely the desired final posture. numerical error and singularity.26. oooooooooooooooooo Example 3. Example 3. If we imagine that the desired path is a real trajectory. As a result. the singularity causes error to the endeffector frame's posture. the rank of (J»Jl) is 2. it is necessary to compensate for the accumulated error in the numerical solution with the help of Eq.Mechanical System of Robots 103 a loss of any mobility or degree of freedom. the endeffector frame's posture is subject to a large amount of error due to the same singularity. Normally. we choose three different final postures for the endeffector's frame to reach: • It is easy for the robot to quit the singular posture when approaching final posture 1 because it does not require too much effort of movement in the vertical direction. 3.22 Due to disturbance. 3. The robot has been shifted away from its initial posture in order to have a better view of the first three iterations. 3.
• Error compensation should be turned off when a simple open kinematicchain is approaching singularity. 3. iiiiiiiiziiiiii==== ^ J 0 — j 70 : : : : : : .._ :::::: J F j I J i i S ! { f ! i : : : : : : : : : : : : : : : : : : : : : : : : : : : : = n3 tlrJ3^II ^~^ 4jHHr Initial posture _LiUHUI LH 21 111 i^ ^ i J P iiiiiiiiiiiiiiiiiiiiiiiiziiiiii 11 i ^ Bt + Q 11111i^^^Bfiiiiiiiiini IIIIIII::::IIIII _ ! ^. Based on the above two examples.26 Effect of singularity on the numerical solution of inverse kinematics without error compensation: "+" indicates the desired path of the endeffector's frame. and 3.. we can highlight the following points regarding the conventional solutions of inverse kinematics: ..( 20 0 20 40 60 80 ll Home  Exit  Fig. we can make the following two statements: • Error due to singularity can effectively be compensated by Eq.. 3. This implies that the combined effect of error compensation and singularity will be disastrous for the scenario of the trajectory following.4 Discrete Kinematic Mapping From the above discussions.— ..3. 3. and "o" indicates the actually computed path of the endeffector's frame. .68 in the numerical solution for computing joint displacements.flBHHBp— .104 The Fundamentals of Robotics: Linking Perception to Action d 2D Humanoid Robot in Posture of Standing Id an :::::" X^^B^fr 111 z:::: z ' l i i i i : z'' z yu H^^^^ffl^ so ..— 1 1 1 fll^^B§tt I I I I I I I I I I I I I 111   M ^ : : : : : : : : =======: d beta(Arm) . =  Ujpho(Leq) *l i3amma (Arm) j —1 ^ 40 Bl^BSlE — ~ll "** z~~ii0'~ ~~: — z * _j Beta(Leg).
(2) In general. it is advantageous to explore this kind of kinematic redundancy.. 3.Mechanical System of Robots 105 d _ .::: d " J JAIpha(Boefy) '«n Zl __ i:::: : : ! : : : : :::::::::::::: :::.p a i TTTT B H l B B P n " T T ::::: [Beta (Leg) g1 .. Alpha (Arm) 2D Humanoid Robot in Posture of Standing 1 0 0  I I I I 11 I I I I I I I I I I I I I I I I I I I I I 11 I I I I I I I I I 11 I I I I I I I I I I 90. (3) The range of joint variable qi is specified by the joint limits.max).:. it is difficult to obtain a closed form solution for inverse kinematics.::::::::. (4) If we apply the solution for joint velocities to obtain a general numerical solution for recursively computing joint variables.:::::::::.i:::::i.:. (5) The numerical solution for computing joint displacements will not work properly if the initial configuration of the simple open kinematicchain . In practice. it is necessary to be careful about singularity due to the inverse of (J • J1).. In practice.. The robot has been shifted away from its initial posture in order to give a better view of the first three iterations.. and "o" indicates the computed path of the endeffector's frame.4 ?216 20 i 0 Home [ 20 40 Exit 60  80 Fig.. denoted by {qi..The solutions of inverse kinematics usually do not explicitly take into account the limits of the joint variables.^ . <li. (1) In general.:.min.. one must pay attention to the joint limits when programming a real or virtual robot. :::::: i o Ll d 80 :::MHHMMMWM^ijpiHHEH! ::: * Gamma (Arm) 40 _.27 The effect of singularity on error compensation used in the numerical solution of inverse kinematics: "+" indicates the desired path of the endeffector's frame. mapping from task space to joint space is a onetomany mapping.
69) We denote K = (ki. kn) the index vector. 3. the corresponding posture of the endeffector's frame can be precalculated using the solution of forward kinematics (see Eq. this principle appears to be an interesting approach to the study of a timevarying robot mechanism..1] * (3.min.17). there is room to improve the efficiency of the inversekinematic solutions. ty .. with ~j Qi. The study of a developmental principle in robotics is emerging as a new research topic which directs the way towards the realization of an intelligent and autonomous robot.1 Discrete Forward KinematicMapping For a simple open kinematicchain with n + 1 links (inclusive of the base link). .max Qi. and stored if needed. Clearly.4. the corresponding joint space can be divided into a set of regular partitions which form a discrete joint space. If the range of joint variable qi is {qi. (6) The numerical solution is computationally expensive as N is usually not a small number.. learning and realtime interaction with its environment. 3. the idea of discrete kinematicmapping naturally serves as a sound basis for the study of a developmental principle for a robot to acquire motion skills and kinematic modelling through experience. All the prestored postures of the endeffector's frame can be arranged in the form of a reference table which is indexed by index vector K. i = 1.max). the kinematic modelling of which is not available in advance. the discrete representation of this variable in joint space will be qt{kl) = qi.2. Most importantly. One interesting idea is to employ discrete kinematicmapping.n}. &2. This table describes discrete mapping from joint space to task space.3.min + ki*Aqi.min Aft A e [0.e..106 The Fundamentals of Robotics: Linking Perception to Action is in singularity (i. ... in terms of {Aqi. of the subdivision of joint space can incrementally be adjusted and finetuned during the developmental process of a robot.. such as a humanoid robot. qi. the rank of (J'J1) at U is less than the admissible number of degrees of freedom at the endeffector's frame). When we have an index vector K. Since the granularity.
can be represented by • X{ix) < Y(iy) . The discrete locations in task space. = Zmin + iz • AZ.Nxl] iy e [0.1] iz e [0.AY. ixe[0. in the form of voxels. having the corresponding set of joint angles. this table describes discrete mapping from task space to joint space.4. Clearly. Let / = {ix.iy. we have set: Ax = 10cm and Ay = 10cm.3. Ny . Z{iz) with • 1\J — JVa: "" Iyy = Xmm + ix»AX.AZ).Mechanical System of Robots 107 3. are marked by the diamond symbol. Given an index vector / . we can divide task space into a set of regular partitions which are commonly called voxels in computer graphics. Nx . ••oooooooooooooooo . the corresponding joint variable vector q can be precalculated by a conventional inversekinematics solution. 3. one can use the numerical solution. For example. In this example. All the precalculated joint variables can be arranged in a reference table indexed by index vector / .28 shows the process of constructing discrete inverse kinematicmapping by using the numerical solution. of the subdivision of task space can also be incrementally adjusted and finetuned during the developmental process of robot. Example 3. the granularity.iz) be the index vector. in terms of (AX. = Ymin + iy • Ay.1] (3.23 Fig.2 Discrete Inverse KinematicMapping Similarly.70) XmaxXmin AX Kf — YmaxYmin ~ AY I M _ 1Sz  AZ ZmaxZmm Again. If the endeffector's effective workspace is within the volume of {{Xmim Xrnax)) {'mini *max)i \^mim ^rnax)j task space.
= p S  j ^ S i : : : : : : : .. The results are not exact. This mapping can be prestored into a database..J bg ""20 0 20 40 60 80 J Home ' I Exit  Fig.28 Construction of the discrete inverse kinematicmapping by using the numerical solution. The diamond symbol indicates the discrete locations in task space. inverse kinematics solutions are known in advance and a basic computer can effortlessly retrieve a solution from the database._ : : : ^ _ : : : ^ : : : : : : : : : : : : : : iBetadea) .. 3. 3. the discrete inverse kinematicmapping can be established through a learning process based on realtime interaction between the robot and its environment. If there is any change in the level of granularity of discrete task space or joint space..4.3.17. but at a certain level of accuracy. This is . depending on the levels of granularity.108 The Fundamentals of Robotics: Linking Perception to Action J 2D Humanoid Robot in Posture of Standing id a{ WphafArm) 100 11 [ j j M M I I I I I 11 I I I I I I I I I I I 11 I I I I ITJj I 11 I I I I I I I 111 t*fata®«*d _ Genrrifl(Arm) 40 .3 Bidirectional Discrete KinematicMapping If there is no concern about the size of memory. it is theoretically possible to precalculate the mapping between index vectors I and K from the forward kinematic solution in Eq... In the spirit of the developmental principle.. With the mapping between index vectors / and if at a certain level of granularity. it is sufficient to recalculate the mapping and update the database. if the precise kinematic modelling of a robot is not available. 3.
In addition. The difference depends on the granularity of discrete task space. From this intermediate posture. Example 3. After 26 iterations. the endeffector's frame reaches its final posture.4 Application of Discrete KinematicMapping The numerical solution for determining joint displacements is very powerful but computationally demanding. In addition. the endeffector's frame is instantly moved over to an intermediate posture at (50cm. 54cm). oooooooooooooooooo Example 3. as shown in Fig. Its final posture is at (46cm. the computational load with performance enhancement is three times less than the computational load required by the numerical method alone. Instead of starting from the singular initial posture.30. we apply discrete inverse kinematicmapping first to instantly obtain the joint angles which bring the endeffector's frame to an intermediate posture °Mn(tj). In fact.25 For the same example as shown in Fig.4. 3. Instead of using a numerical solution to interpolate a chosen spatial curve which connects the endeffector frame's initial posture °Mn(ti) to its final posture °Mn(tf). which is close to the final posture °Mn(tf).29 shows an example of the use of the numerical solution to determine joint displacements without any performance enhancement. we now explore the knowledge of discrete inverse kinematicmapping. it is necessary to pay attention to the effect of singularity. 3. the endeffector's frame reaches its final posture within 5 iterations. 3. Subsequently. we apply the numerical solution to compute the joint displacements which bring the endeffector's frame from the intermediate posture °Mn(tj) to the final posture °Mn(tf). 3. Therefore.Mechanical System of Robots 109 similar to the effortless way that human beings solve inverse kinematics.24 Fig. discrete kinematicmapping automatically eliminates concern about joint limit and singularity because all these have been implicitly taken into account by the computation. oooooooooooooooooo . In other words. the performance of the numerical solution can greatly be enhanced with the inclusion of discrete inverse kinematicmapping.3. The initial posture of the endeffector's frame is singular. it is useful to attempt to further improve the performance of the numerical solution. The idea is very simple.29. 60cm).
110 The Fundamentals of Robotics: Linking Perception to Action 2D Humanoid Robot in Posture of Standing «J . known as the DH representation. In this chapter. joint. we started with the study of a robot's mechanism. In order to facilitate the kinematic analysis of a humanoid robot. we covered in greater detail. Then.1 Alpha(Armj ._ 70:::::::: BHBj^^^ffl::::::::::::::::::::: Alpha (Leg) Gamma (Arm) 40 :::::::::::^^^^K:::::::::::::::::::::::::::::: iBetaiteq) _ l . We learned the concepts of link. the kinematic formulation of a simple open kinematicchain.29 ment.4 Summary A robot is able to perform motions because of its underlying mechanism.54 ~ 20 0 20 40 60 80 J Home . 1 °° I I I I 1111 I I MUM I 11111 ITTTTTTTTTTTTI I I I I I I I I 1 i m 1 ^halBocM SetaJAim) . we studied a systematic way of representing a mechanism with a set of kinematic parameters. we discussed a new illustration of the DH representation. and open kinematicchain. 3. . These parameters are sufficient to describe the relationship between the input and output motions governing the behavior of a robot's mechanism. we discussed the concept of simple open kinematicschain. Exit  Fig. Numerical solution for determining joint displacements without any enhance 3. It makes use of two parallel planes to clearly explain the four kinematic parameters which uniquely describe the motion transformation between two consecutive links' frames. Subsequently. For the sake of clarity. after the study of the kinematics of a single link.
: : ^ : : : <_:::::::::::::: Beta(Leg)_ —I t i 20 0 20 40 60 80 J . This method effectively complements the conventional numerical inversekinematics solution.. We know that the objective of forward kinematics is to determine the output motion (in task space) of a robot's mechanism from the knowledge of its input motion (in joint space).Mechanical System of Robots 111 2D Humanoid Robot in Posture of Standing j •J NphnArm) I2Z2. Subsequently. .TO81 100 n T m T T T  I I L L U J l I I I I I I I I I I 11 I I I 111 I I I I I I I I 11II I I I 90 ::::::::±:^^S:::::::. 4° : : : : : : : : i J h M i J : : : " . We learned that there are two related problems under the topic of robot kinematics: a) forward kinematics and b) inverse kinematics.. Home . the most critical issue is how to determine the robot mechanism's input motions in joint space if the desired motion in task space is given. we studied different methods for solving inverse kinematics including discrete kinematic mapping. Exit j Fig.. 3. As input to a robot system is a task. This is known as the inversekinematics problem. AIpho(Body) IH56B6 Na(Arm) . the next questions are: How do we physically produce the input motions to a robot's mechanism? And..::::::::::::::::::::::.30 Numerical solution for determining joint displacement with performance enhancement by discrete inverse kinematicmapping.:::^:::::::::::::: Alphs (Ug}_. 70 :::::::::^^B^^ffij. how do we relate the input motions to the forces and torques applied to the joints? We will study the answers to these questions in the next chapter. _ ^amma(ArmL. Now that we know the relationship between the input and output motions of a robot's mechanism.
(4) What is the acceleration vector of a single link? (5) Under what situation will Jacobian Matrix J(q) be independent of joint variables ql (6) Find the Jacobian matrix for the simple open kinematicchain shown in Fig.5 Exercises (1) What is the purpose of studying a robot's mechanism? (2) What is a simple open kinematicchain? (3) Illustrate the possible number of simple open kinematicchains inside a humanoid robot's mechanism. . • Step 3: Multiply 2Mi(<?2) to both sides of the above equation to obtain: 2 Mi(g 2 ) .21.10. cannot be treated as an internalmotion vector to a kinematicallyredundant open kinematicchain.. 3.. (10) How do you determine internalmotion vector qa for a kinematicallyredundant open kinematicchain which is required to avoid the joint limits? (11) How do you determine internalmotion vector qa for a kinematicallyredundant open kinematicchain which is required to avoid the singularity caused by the inverse of (J • J*)? (12) Look at Fig. {nlMn{qn)} and solve for q^• Step 4: Repeat until qn is solved.112 The Fundamentals of Robotics: Linking Perception to Action 3.9: • Step 1: Multiply 1M0{qi) to both sides of Eq. as shown in Fig. Among them. • {"^Mnte. 3. 3. ^Motei)} • {°Mn} = {2M3(q3)} . (8) Verify the results of the inversekinematic solution applied to the right arm and right leg.. 3. the endeffector of which stands still in task space.)} • Step 2: The above equation describes 12 equalities between the corresponding elements of the two matrices of both sides.38 to obtain: lM0{qi) . 3. . choose the simple one(s) to solve for q\. (9) Explain why an arbitrary velocity vector.. (7) Comment on the applicability of the following algorithm to compute the joint angles of the planar open kinematicchain shown in Fig..27. in joint space. Discuss the reason for the large deviation of the endeffector's frame at iteration 2 from its desired trajectory. {°Mn} = fMM} • . .
15. (8) Whitney. 12. A Mathematical Introduction to Robotic Manipulator. J. Li and S. S. CRC Press. AddisonWesley. John Wiley and Sons. R. Resolved Motion Rate Control of Manipulators and Human Prostheses. (5) Murray. 10. J. (1969). June 12 . Sastry (1994).. Fundamentals of Robotics: Analysis and Control.6 Bibliography (1) Asada. P. MIT. H. M. (1991). and R. Automatic supervisory control of the configuration and behavior of multibody mechanisms. IEEE Transaction on Systems. 2. (4) McKerrow. (1999). Man and Cybernetics. (2) Denavit. (1977). Design of Machinery. Proceedings of the IEEE Second International Conference on Development and Learning. R. Robot Analysis and Control. E. S. 7. 215. Hartenberg (1955). A. L. Introduction to Robotics. D. and J. Z. E. (7) Schilling. (1990). Slotline (1986). A Kinematic Notation for LowerPair Mechanisms Based on Matrices. IEEE Transaction on ManMachine Systems. (3) Liegeois. (9) (2002).Mechanical System of Robots 113 3. R. (6) Norton. Journal of Applied Mechanics. . X. J. J. PrenticeHall. McGrawHill. 22.
.
Prom a systems point of view. In this chapter. Then. it is indispensable to study the relationship between motion and the force/torque applied to a robot's mechanism.1 Introduction The pure mechanical aspect of a robot is its underlying mechanism and structure. It can only be transformed from one form into another. a robot is a machine which executes motion. as the result of energy consumption. Accordingly. For example. we consider the case in which a robot's mechanical energy is obtained from electrical energy. we will present. A discussion about robot statics will come first before we move on to studying robot dynamics. the mathematical principles for the establishment of equations of motion. The study of the relationship between the input and output motions of a robot's mechanism is fully covered by kinematic analysis. Naturally.e. while a portion of the chemical energy is wasted in the form 115 . 4. in detail. because of the use of electric actuators.Chapter 4 Electromechanical System of Robots 4. On the other hand. electric actuators).2 Origin of a Rigid Body's Motion We all know that energy cannot be created nor destroyed. we will first study the concept of energy and the fundamentals underlying energy conversion from the electrical domain to the mechanical domain (i. in order to drive a car. it is also logical to study the actuation elements together with a robot's dynamic and static analysis. it is logical to study the robot mechanism together with its kinematic analysis. the internal combustion engine converts chemical energy into mechanical energy. Thus.
a system can be treated as an energy storage device. Thus. Fig. Fig. Obviously. In order to better understand this issue in robotics. 4. Then.2. This principle of energy conservation is also applicable to a system. it is important to study the origin of a rigid body's motion. which corresponds to the amount of energy contained in the system. we call the energy stored inside a system the internal energy. 4. the principle of energy conservation can be stated as: The change of internal energy (denoted by A£jrej) inside . the answer is negative. Another example is the rechargeable battery. This is concisely described by the Einstein's famous equation E = me2. which converts energy from the electrical domain to the electrochemical domain and viceversa. It can only be converted from one domain into another. 4. We can see that there are many components and devices in addition to the robot mechanism. a robot's mechanism must be complemented with the extra actuation elements. the question is whether a robot's mechanism can move on its own without any extra element added to it.1 shows a prototype of a humanlike robot (called HARO1). any substance is an embodiment of energy.1 Energy Conservation in a System Energy cannot be created nor destroyed. In fact. For convenience. Clearly.116 The Fundamentals of Robotics: Linking Perception to Action of thermal energy.1 A prototype of a humanlike robot (HARO). a system has its own energy state. As any substance is an embodiment of energy. Now. at a time instant.
The nature of force is well described by Newton's second law.2 Forces In robotics. If we denote fi(t) the individual force i acting on a particle and assume that there are n external forces (see Fig. p(t)=m»v(t).Electromechanical System of Robots 117 a system is equal to the sum of the energy added to the system (denoted by Ein). The way in which this change is related to the acting forces is precisely described by Newton's second law. the mass measures the capacity of a body to store kinetic energy. the following equality holds: ^Eint = Ein — Eout — Edis(41) 4. the linear momentum p(t) will change if there is an external force acting on it. In other words. then the linear momentum of the particle is defined as follows: Definition 4. Newton's second law states that the rate of change of a particle's linear momentum is equal to the sum of all the (external) forces acting on it.2). Alternatively.1 A particle is a body without a physical dimension. the energy removed from the system (denoted by Eout) and the energy dissipated from the system (denoted by Edis). (4. Now.2) In fact. Newton's first law simply states that a particle retains its linear momentum if no (external) forces act on the particle. In other words. the linear momentum p(t) of a particle characterizes the ability of the particle to maintain its body at a constant linear velocity. And. then . 4. an important concern is mechanical energy. The entire mass of the body is concentrated at a single point. If we denote m the mass of a particle and v(t) its linear velocity vector with respect to a reference frame. Before we briefly introduce this law. it is useful to examine the particle and its linear momentum. a particle can be denned as follows: Definition 4.2. and how to relate mechanical energy to motions. Mechanical energy is normally manifested in the form of force or torque.2 The linear momentum of a particle is the product of its mass and its linear velocity. the mass of a particle describes the resistance to the change in its linear velocity. Formally. that is.
4. a body without any internal deformation) can be mathematically treated as a particle.2 External forces acting on a particle and the total sum of forces. In the mechanical domain. m is a constant).3 yields (4.4 indicates that the motion of a rigid body originates from the forces acting on it. This law states that when two particles mutually exert acting and reacting forces upon each other. As a rigid body (i. The formal definition of force is as follows: A force is the effect of one body acting upon another In the universe. If a particle conserves its mass (i.2 into Eq. 4. 4. (4.118 The Fundamentals of Robotics: Linking Perception to Action we have F{t) = ±Mt)=*&. substituting Eq. there are two types of forces: a) contact force and b) field force.e.3) /. Eq. a typical example of contact force is the elastic force generated by a spring. 4.. When body A exerts a force upon body B. these two forces are equal in magnitude and opposite in direction.e. F I Definition 4. And a typical example of field force is gravitational force.4) where a(t) = ^^ is the acceleration vector of a particle. the reaction force exerted by particle . Jn Fig. This phenomenon is precisely described by Newton's third law. body A will receive a reaction force from body B. If we denote fij the force exerted by particle i upon particle j .3 body.
H = m»uj — m •[f x v(t)} = f x [m • v(t)] (46) where r'is the position vector of the particle. 4.2. v(t) its linear velocity vector.Electromechanical System of Robots 119 j upon particle i will be denoted by fjfi. and w its angular velocity vector.3. 4.4 The angular momentum of a particle moving along a circular path is equal to its mass times its angular velocity.3 Torques (45) If a force acts on a particle which is constrained to follow a circular path.i4. let us examine the variation of angular momentum. Differentiating Eq. Now. we have fij = ~fi. Its definition is as follows: Definition 4.6 with respect to time gives l[ = ftX[m' V{t)] + "X(jt[m' Vit)]) • ^ . that is. 4.3 A particle moving along a circular path. the energy state of the particle is conveniently described by angular momentum. as shown in Fig. According to Newton's third law. Y ^ \ > X F Tm« Fig.
or torque for short.4. is called the acting body. 4. the body delivering the force. called a dynamic pair and a dynamic chain. In other words. For simplicity's sake. 4. (4. vector r is called the moment of force. . Substituting Eq. Eq. 4.8) (4.9 becomes ^L=T (XV = rxF(t). any acting torque exerted by body A upon body B will also have a corresponding reacting torque exerted by body B on body A. we use Tij to denote the torque exerted by body i on body j .8 into Eq.4 and Eq. 4. Eq. Inside a dynamic pair. 4. the coupling between an acting force (or torque) and a reacting force (or torque) leads us to define two new terms. or torque. In robotics.e.10 indicates that the angular momentum of a particle will remain unchanged if there is no external torque acting on particle (i. 4. r = 0). In fact. the dynamic pair's acting body will deliver either a contact force or a field force on the reacting body. the crossproduct of a particle's position vector and the force vector acting on it.2.10) By definition. while the one receiving the force or torque is logically called the reacting body. For convenience of description. we have df — x[m*v(t)] = 0. Since a force can be a contact force or a field force.7 yields ~=rxF(t) (4. the rate of change of a particle's angular momentum is equal to the torque acting on the particle. quantity r depends on force F because the torque is equal to. Physically. If we define T = rx F(t). we only consider the case where there is just one force or torque delivered by the acting body.4 Dynamic Pairs and Chains We use the term dynamic pair to define any pair of rigid bodies which exerts force or torque upon each other. We know that any acting force will have a corresponding reacting force.9) where F(t) is a force vector expressed with respect to a reference frame. as illustrated in Fig. by definition.120 The Fundamentals of Robotics: Linking Perception to Action Since the derivative of position vector ^ is in the direction of linear velocity vector v(t). the acting body of a dynamic pair can exert multiple forces upon the reacting body. In practice. 4. Similarly.
frame 0 of the base link in an open kinematicchain). on each other within a consecutive pair. Due to gravitational force. In this example.i = —fifi I /i. these three links mutually exert acting and reacting forces and torques on each other.e. The definition of a dynamic chain is similar to that of a kinematic chain.1 Fig. and mutually exert forces . a formal definition of dynamic chain can be stated as follows: Definition 4. that implicitly indicates the reference frame. When we are not using the superscript. which form an open dynamic chain as well.5 shows an example of an open kinematicchain with three links at rest. ' fj. links). Otherwise. 4.2 = —/2.i Bodyj (a) With contact force (a) With field force Fig. or torques. all forces in a dynamic chain are expressed by default with respect to the same reference frame (e.Electromechanical System of Robots 121 Bodyi ou i '^Tjj Bodyj Body i o o . Thus. When dealing with robot dynamics. it is called a closed dynamic chain. The interactions among the rigid bodies are governed by the acting and reacting forces/torques. A dynamic chain is said to be open if there is no direct coupling of force or torque between the first and last bodies in the chain. A dynamic chain is a set of rigid bodies which are arranged in a series. we can break down the open kinematicchain into a set of independent rigid bodies (i.5 A dynamic chain is a set of rigid bodies arranged in a series which mutually exert forces.4 Illustration of a dynamic pair: a) the acting body delivering a contact force. 4.or torques on each other within a consecutive pair.i . we have f /o.g. and b) the acting body delivering a field force. To study the effect of the forces and torques. Example 4.
oooooooooooooooooo 4. linear or angular velocity) is proportional to the applied force or torque if the masses of the bodies inside a dynamic pair remain constant.i.i = —71.e. a n d body j undergoes a displacement along a linear or circular path. 4. and b) the corresponding open dynamic chain formed by the three rigid bodies with mutually acting and reacting forces/torques. 4.2 . or torque.122 and The Fundamentals of Robotics: Linking Perception to Action f r o.4.2. related to the physical quantity: energy? Refer to Fig. Assume that force fij acts on body j within time interval [ii.5 Incremental Works Motion originates from the force or torque of a body acting on another body. Now.^].2 = —T2. Link 2 CT^^~~^ Link 1 / / I Bodyl i a—— ' J. If we denote dfj the differential displacement of body j caused by force faj (or its corresponding torque if the path is circular). The rate of change of motion (i. fj / / / / '' I \ g I Base Link (a) I  " I Body 0 (b) Fig..11) . Body 2 A. by definition. the question is: How is force.5 Example of an open dynamic chain: a) an open kinematicchain. the differential work done by body i exerting a force (or torque) on body j will be dW = fitj • drj (4.0 I ^1.
15 becomes AQ = Q(t2) + Q(h) with Q(t) = fiAri(t))*rj(t).12.13) By definition.16) Eq. AW = (2 dW= (2 fa • dfj.15 expresses incremental potential energy done by force fij on body j during time interval [ii. energy is removed from body j if the displacement is in the direction of the force.^ = Vj(t) (i./ Jfj(. The integration of Eq. 4. £2] gives the expression of the incremental work done within the time interval. work done by a force which only depends on a position vector is called potential energy. 4. Eq. if force fij only depends on the position vector of body j and is independent of the time variable. And. Assume that body i is the Earth. 4.2. For example.vjW.17) (4.12 can also be written as follows: AW= [^[f^. the linear velocity vector of body j). 4. Eq.17 expresses potential energy done by gravitational force fij.e. Jti Jti (4. Eq.15) In physics. Then fij is the gravitational force acting on body j and is a constant force. 4.ti) (4.12 can be reformulated as follows: AW = AQ with AQ = . 4. (4. that is.11 over time interval [ti. In this case.6 Potential Energy In Eq. when . (4. the expression fij • Vj(t) is called power. 4.14) fu{rj)*drj.^]A typical example of potential energy is the work done by gravitational force.Electromechanical System of Robots 123 where dW denotes the differential work done by force fij on body j . which characterizes the ability of body i to do work on body j . 4.dt. Jtt (4. The negative sign inside the expression of potential energy indicates that the work done by the force is negative.12) Since dtj — ^~ • dt and . Eq.
A simple example to prove this point is to look at the case when a person applies force to a building.8 Origin of Motions (4.19) or AW = ^mj[v](t2) If we define K(t) = \rnj[vtj(t)*vj(t)]. 4. 4. 4. Then. In the discussions above. 4. j . its kinetic energy increases. . Eq.20 becomes AW = K(t2) .2. there will be no motion even if there is an applied force. under normal circumstances. we are able to precisely answer the question of what the origin of motion is. Obviously. At this point. its kinetic energy increases.21 is an expression of kinetic energy. Force fij acting on body j will cause variation in the linear momentum. it is clear that this answer is not completely correct.7 Kinetic Energy Refer to Eq.124 The Fundamentals of Robotics: Linking Perception to Action an apple falls from a tree to the ground due to gravitational force. It describes the capacity to produce motion.20) Now.12. For example.12 yields /"*2 AW = Jti rrij* Vj(t) • dVj (4. Eq. Assume that the mass of body j is rrij and its linear velocity vector is Vj(t).^mj[v](ti) • Vj(<!)]. a person pushing a building will not cause any displacement. it loses potential energy.18 into Eq.21) • VJ(t2)} .2. 4. (4. we have dvAt) > ' d t  f ^ = m (418) Substituting Eq. By definition.K(h). we mentioned that a rigid body's motion originates from force or its corresponding torque applied to the body. 4. This lost potential energy is converted into kinetic energy. When the displacement vector or motion of body j is in the direction of force / . when we accelerate a car. In other words. 4.
This definition is helpful for those lacking a mechanical engineering background. Since a robot is a mechanical system. the motion of any mechanical system is created by the addition or removal of mechanical energy to or from the system. the kinematic pair is formed by the body of the vehicle and the wheels (treated as part of the ground). A simple way to create controllable motion for a kinematic pair is to couple a dynamic pair to it. or torques.Electromechanical System of Robots 125 From the expression describing work done by an applied force or its corresponding torque. the mechanical system of a robot (or any machine) can be formally defined as a combination of kinematic pairs coupled with their corresponding dynamic pairs. Conceptually.3 Actuation Elements The purpose of a robot's mechanism is to shape the output motion. It . The corresponding dynamic pair is formed by the pistons and the engine block of the internal combustion engine.2 For a land vehicle powered by an internal combustion engine. In other words. It is also helpful in better understanding the design principle of the device called actuator or simply force/torque generator. It is necessary to add extra elements to a robot's mechanism so that the motion can be generated in a controllable manner. We know that a robot's mechanism can be treated as a set of kinematic pairs. As the motion originates from the addition or removal of energy. a robot's mechanism may include many kinematic chains. Example 4. it becomes clear that the origin of motions is energy or work added to or removed from a body. 4. This philosophy is concisely illustrated in Fig. one may ask these questions: • How do we add energy to or remove it from a robot's mechanism in order to produce the desired motions? • How do we describe the relationship between the motions of a robot's mechanism and the forces/torques applied to it? 4. as the two bodies in a dynamic pair can exert force or torque upon each other.6. an actuator or force/torque generator is the realization of a dynamic pair which consists of two bodies and their acting/reacting forces. Each kinematic pair determines the type of motion (prismatic or revolute) between its two links. a robot's mechanism cannot produce any motion on its own. which is a function of the input motions of the robot's mechanism. Conceptually. In general.
if pressure p\ in the left chamber is not equal to pressure p2 in the right chamber of body . oooooooooooooooooo 4.3.1 Force and Torque Generators As mentioned earlier.e. goes without saying that a vehicle will stand still if there is no coupling between the kinematic pair and the dynamic pair. A common method for the realization of a dynamic pair governed by a contact force or torque is illustrated in Fig. (2) The generated force or torque must be repeatable (i.e. one can produce a periodic and/or continuous force or torque as output). As we can see.126 The Fundamentals of Robotics: Linking Perception to Action Force Field i Bod> i .7. In order to power a robot or any machine. 4. 4. Body j I I • •\r «J ^ _' i» f I Dynamic Pair v Coupling A Coupling A (oV'" 'ink' Kinematic Pair Fig. there are two types of force: a) contact force and b) field force. its magnitude and direction are functions of certain controllable variables). a force (or torque) generator must satisfy the following two conditions: (1) The generated force or torque must be controllable (i. 'n r* .6 The coupling of a dynamic pair with a kinematic pair in a robot's mechanical system.
g. 4.7 A principle for the realization of a dynamic pair governed by a contact force.Electromechanical System of Robots 127 Body i ••••••••MP8*—1 Bodyj Pi Fig. i. follows this layout of the dynamic pair. air).g.8 Possible principles for the realization of a dynamic pair governed by a field force. 4. or liquid (e. Body i Body i I m ) • U" '\~V \ * I + 1 + .' Bodyj [2J (a) Gravitational force Bodyi (magnet) Bodyj (*O (b) Electrical force B°dy' (magnet) ^•) ( • • ' • ' Bodyj A (Material with iion)JB (c) Electromagnetic force Bodyj $0P*^ cT^B (d) Electromagnetic force Fig. The generated force is both controllable and repeatable because it depends on quantity (pi — P2) • The design of the pneumatic actuators. as well as hydraulic actuators. body i will exert a force on body j through the medium of compressed gas (e. . oil).
23) where g = 9.22 yields fij =mJ*g (4. 4. rrii will be about 5. it is clear that the density of the gravitational force on the surface of the Earth is almost constant. This force is proportional to their electric charges (measured by C and called Coulomb) and inversely proportional to the distance between them. 4. If we treat the pairing of the Earth and the moon as a dynamic pair. From Eq.23. In other words. If we consider the gravitational force field on the surface of the Earth. kg for kilogram. the conversion between force and potential energy is reversible. will exert electric force upon each other. two bodies in space (i and j) will attract each other. if their masses are rrii and rrij respectively. or relative motion. two electrically charged bodies. we cannot make use of this type of field force to realize a dynamic pair.378 x 106 m. Another way to explain this conclusion is the fact that a gravitational force is a conservative force. 4. according to Coulomb's law.803 m/s2. and not controllable.8): Gravitational Field Forces According to Newton's Law of Gravity. and s for second). assume that body i is the Earth. Electric Field Forces Similarly. their interaction. r will be about 6. When we have two bodies i and j separated by distance r. i and j . The magnitude of the attraction is proportional to the product of the two bodies' masses and inversely proportional to the squared distance between them. an atom . we have the following choices to consider (see Fig. g can be interpreted as the gravitational force per unit mass on the surface of the Earth. At an atomic level. It characterizes the density of the gravitational force of the Earth at its surface.128 The Fundamentals of Robotics: Linking Perception to Action As for the realization of a dynamic pair governed by a field force. is governed by a gravitational field force. the gravitational force fij acting on body j by body i will be hj~G—y2— yA22> where G = 6. As a result.673 x 10~ n m3 / (kg • s2) (m stands for meter. Now.976 x 1024 kg. The substitution of values of rrii and r into Eq.
m for meter and C for Coulomb). E{r) describes the density of an electric field at distance r. the atom is said to be positively charged. the electric force fcj acting upon body j by body i will be /. However. fcM.988 x 109 N • m2/C2 (N stands for Newton. it is said to be negatively charged. Electromagnetic Field Forces We are all familiar with the phenomenon that two magnetic bars in proximity will attract or repel each other. When we have two electrically charged bodies i and j . Any incidental discharge may be dangerous and undesirable. 4. If some electrons are removed from an atom. A magnetic field will always exert an attractive force upon an object containing ferrous materials. The electric force is controllable as it is possible to manipulate the electric charge qj and the density E(r) of the electric field. the number of electrons is equal to the number of protons. And the direction of the force will point towards the source of the magnetic field. The protons and neutrons of an atom form the nucleus. (4.. proton(s) and neutron(s). created by body i having electric charge <?j. For an electrically neutral atom. = fei^i (4. if we can design the two bodies in a cylindrical shape and choose an electromagnet to be the acting body. In theory. But. it is possible to alter the direction of the force acting on a body containing . Conversely.Electromechanical System of Robots 129 is composed of electron(s).24) where k = 8.24 can be rewritten as fiJ = \qj\*E(r). it is possible to design a device which forms a dynamic pair based on the principle of electric field force. the interaction between a magnet and a body containing ferrous materials is not controllable because one cannot alter the direction of the interacting force between them. if more electrons are added to an atom. this is not a practical solution because the high density of an electric field poses a safety challenge.25) In fact. if their charges are qt and qj respectively. If we define E{r) = Eq. Strictly speaking.
This is precisely the working principle underlying electric motors. the electric charges contained inside an infinitesimal length dl will be dq = I • dl.27. . coils wound around a ferrous core). Most importantly. Moreover. If body j contains an electric charge q travelling inside it with velocity vector v. Nowadays.e. Assume that body j is placed inside body i's magnetic field.26. (4. A magnet will always produce a magnetic field B around it.(ifxB).130 The Fundamentals of Robotics: Linking Perception to Action ferrous materials. that is.27) From Eq. If an electric charge travels inside a current conductor. The integration of the above equation over the entire length of body j yields the total force acting upon body j by body i due to the magnetic field.I. we can see that the electromagnetic force between two bodies is controllable and repeatable. force dfoj acting on the infinitesimal length by body i will be dfii:j = I»(vx B)»dl where v is a unit vector indicating the direction of current inside body j .26) The net flow per unit time of electric charges at a conductor's crosssection area is called (electric) current. Because of these desirable features. this charge will receive a force proportional to the density of the magnetic field and proportional to the travelling speed of the charge. b) cylindrical coils and c) electromagnets (i. 4. By applying Eq. If body j is a straight current conductor with a constant current / which flows inside it. There are three types of magnets: a) permanent magnets found in nature. / y = J. the working principle of all the electric motors is based on the use of electromagnetic force. (4. 4. it can be easily manipulated by varying quantity / and/or B. which is measured in Amperes (or A for short). electric motors are widely used in robotics for the realization of dynamic pairs in a robot's mechanical system. force fij acting on body j by body i will be fitj=q*(€xB). the direction of the force acting upon an electric charge is perpendicular to both the velocity vector of the charge and the direction of the magnetic field.
Let us denote . Refer to Fig.—L—!__• ^—•x : : : Fig. which is a controllable variable. a coil normally has multiple loops. From Eq. body j is a rectangular coil having a single loop. Here. Assume that current / flows inside the coil. ~T~f—•? . motors are classified in different ways.3. However. it is called a linear motor if the relative motion is along a straight line (i. Body i in the dynamic pair is an electromagnet. In this example..e. The density B of the generated magnetic field is a function of current I o . 4.9. A AB k /* fWiiii)PlMg* Body i \ Body j fb /) < \ '"^Z4^^ . As a result. 1 /*. And. 4.9 Illustration of the basic working principle underlying electric rotary motors. devices which physically implement this principle of interaction are very popular in industry and are commonly called electric motors. 4.Electromechanical System of Robots 131 4. we know that the four edges of the coil will receive an electromagnetic force. It is called a rotary motor if the relative motion between the two bodies in a dynamic pair is rotational about a fixed axis."Axis of rotation / / id '^~/l/. Depending on the types of motion. Body j in the dynamic pair is a rectangular closedloop ABCD which is constrained by an axis of rotation. we will discuss the basic working principle underlying electric rotary motors. linear trajectory).1 Working Principle of Electric Motors It is easy to control an interacting force caused by an electromagnetic field.1.27.
4. T = A • I • \B\ • sin(9) with A = la*lb (i. body j will not undergo any linear displacement.fb.vc. we have fa — —fc and fb = —fd (429) This leads to the conclusion that fx = 0. (4. BC. Thus.27 to edges AB. 4. CD and DA. the total forces received by body j in the X and Y directions will be <fx = fb + fa L Jy — Ja T JcSince va = —vc and vb — —Vd. B). BC. we can make the following observations: • If the electromagnetic field is constant.fd). • (va.e.31. Now.30) Zi By substituting the value of /„ in Eq. the rectangular area of the coil). Then.30. • sin{6) + / c  • ^ • sin(6) = \fa\ .Vd): the unit vectors indicating the directions of current I along edges AB. BC. That is.28) fc = I • la « («c X B) Jd = l*lb*{vd.31) . and fy = 0. we obtain the final expression for the torque exerted upon body j by body i.fc.28 into Eq.the electromagnetic forces induced on edges AB.9. lb): the lengths of edges AB and BC of the coil respectively. 4. from Fig. lb • sin{6). it is clear that the net torque exerted on body j by bodyi is T = I/a • i. 4.y. From Eq. we will obtain ' fa = I • la • (va X B) fb = I»lbm(vbX B) (4.Vb. This means that the net force exerted on body j by body i is zero. CD and DA of the coil respectively. 4.132 The Fundamentals of Robotics: Linking Perception to Action • (fa. the net torque exerted upon body j is a function of current I and angle 6. • (la. ZJ (4. CD and DA of the coil respectively. If the angle between the plane containing the coil and the XY plane is 9. By applying Eq. assume that the plane containing the singleloop coil is not parallel to the XY plane.
body i is called the stator and body j is called the rotor. • If body j is an object containing ferrous materials. The controllable element inside an electric motor is the electromagnetic field which is produced by a set of coils wound on either the stator or rotor. Fig. 4. which we will discuss in more detail in the next chapter. the torque exerted upon body j is solely a function of the magnetic density B. and is expressed as r = kt • / (4. 4. DC brushtype motors. and DC brushless motors. Externally. 4. the total net torque received by body j in a constant magnetic field is almost a function of the single variable /. which are uniformly and symmetrically arranged about the common axis of rotation.32 mathematically describe the basic working principle underlying all electric rotary motors. a motion sensor is an indispensable element required by motion control.. which is controllable by current IQ. In the following sections. we will discuss conceptual design solutions for electric stepper motors. Fig 4.10 A sectional view of the conceptual design for an electric rotary motor.:! Shaft !• " (axis of rotation) If ^ T * I \ Coils or Magnets > Rotor (Body j) Motion Sensor Fig. Eq. 4.zzzzzz.Electromechanical System of Robots 133 • If body j is composed of a large number of rectangular loops.31 and Eq.11 shows one image of an electric motor. all electric motors look similar.32) where kt is called the electric motor's torque constant.10 shows a sectional view of the conceptual design of an electric rotary motor. If we use terminology from electrical engineering. Stator (Body i) S m izzzzz. . All design solutions for electric motors invariably follow this principle. In addition.
the rotor (body j) is a toothed cylinder which can be either a permanent magnet or an object containing ferrous materials.e. 4. stator and rotor).12 A crosssectional view of a stepper motor.134 The Fundamentals of Robotics: Linking Perception to Action Fig. 4. Fig. And. an electric stepper motor is a torque generator which can produce stepwise motions between the two bodies in a dynamic pair (i.11 An external view of an electric motor. 4.2 Electric Stepper Motors Conceptually. In this example. The coils . the stator (body i) is a toothed cylindrical tube.12 shows a crosssectional view of a conceptual design solution for a stepper motor.3.1. Fig. 4.
the rotor will rotate in a clockwise direction and a stepwise manner as well. If we sequentially energize these phases one at a time.13 Illustration of the working principle of a stepper motor. teeth 1 and 4 of the rotor will align with the magnetic flux generated by phase AA"s coils. the rotating magnetic field will attract the rotor to rotate at the same speed as the magnetic field but in an opposite direction. BB'. CC and DD'. If the pitch of the rotor is different from the pitch of the stator. At the next time instant. This is clearly illustrated by Fig. When phase AA' is energized at a time instant. . we will create a rotating magnetic field around the shaft of the motor. 4. we switch off the electrical power of phase A A' and turn on the electrical power of phase BB'. this will make teeth 6 and 3 of the rotor align with the magnetic flux of phase BB'.Electromechanical System of Robots 135 wound around the teeth of the stator are grouped into four phases: AA'. The angle between two consecutive teeth is called the pitch.13. When the magnetic field of the stator rotates in a counterclockwise direction and a stepwise manner. 4. (a) Phase AA1 is energized (b) Phase BB1 is energized (c) Phase CC is energized (d) Phase DD1 is energized Fig.
136 The Fundamentals of Robotics: Linking Perception to Action If the numbers of teeth on the stator and rotor are Ns and Nr respectively. as this issue is closely related to a robot's control system. In this example.3.3 Brushtype DC Motors An electric stepper motor rotates in a stepwise manner. Fig 4. long wire is wound around the surface of the rotor according to a special winding diagram so that multiple rectangular coils are formed. that is. 4. A single. cc'.23) How to control the direction of a stepper motor's rotation will be discussed in the next chapter. there are four rectangular coils: aa'.JJHHHHB^^ (a) Section view of DC motor Commutator plate (b) Front view of commutator Fig. 4. it cannot run at a high speed. ^^^^^ Stator ^djHSB^^^A—" Carbon brushes MBr P " '•HI ^ ^ ^ Rotor /\ . (4. Obviously. Each coil has two terminals which are connected to an electrical power supply. bb'. the product of the torque and angular velocity describes the mechanical power. which an electric motor can deliver. slow velocity means low mechanical power output. In fact. This design results in an electric motor known as a brushtype DC motor. the step angle AO for each move will be the difference of their pitches. One way to overcome this drawback is to have the stator hold a pair of permanent magnets and the rotor carry a large number of rectangular coils. and dd!.14 shows the conceptual design solution for a brushtype DC motor.14 Illustration of the working principle behind the brushtype DC motors. Thus.1. a special mechanical device called commutator groups all the . As we know.
And. c'. out of the paper. the direction of force induced on edges a. c' and d! is towards the lefthand side of the rotor. and edges a'.3. edges a. c and d is towards the righthand side of the rotor. b. which places the coils on the stator and uses a cylindrical . as marked by the dots).14b. If we make the voltage at terminal B higher than the voltage at terminal A.Electromechanical System of Robots 137 terminals of the coils into a disk known as a commutator plate.14) are electrically connected to the electric power supply's terminal A and the terminals on the upper half of the commutator plate (above the dotted line in Fig 4. As a result. b'. the terminals on the lower half of the commutator plate (below the dotted line in Fig 4. and d will receive the current which goes inward (i. there is friction between the carbon brushes and the commutator plate when the rotor is rotating.4 Brushless DC Motors There are several drawbacks to the design solution of a brushtype DC motor. 4.27. 4. If terminal A has a higher voltage than terminal B (as shown in this example). In this example.e. b'.14) are electrically connected to the electric power supply's terminal B. A second drawback is the high inertia of the rotor if a large amount of coils have to be wound on the rotor in order to deliver high torque. as shown in Fig 4. A common solution to these drawbacks is to adopt a design. according to Eq. b. as marked by the crosses). at any timeinstant. but the two brushes remain stationary. the net torque induced on the rotor will make it rotate in a clockwise direction. Since the direction of the magnetic field generated by permanent magnets is towards the upper side of the stator. The most notable one is the risk of generating electric sparks due to the friction between the commutator plate and the carbon brushes when the rotor is rotating at high speeds. the rotor will rotate in a counterclockwise direction.1.e. and d! will receive the current which goes outward (i. and the direction of force induced on edges a'. Two carbon brushes are placed symmetrically against the commutator plate for the purpose of supplying electrical power to the coils. into the paper. Certain environments or workplaces may not tolerate this type of risk. Thus. c. it is obvious that the rotor will continuously rotate as long as the voltage at terminal A is not equal to the voltage at terminal B. The commutator plate rotates together with the rotor. How to control the direction and velocity of a brushtype DC motor's rotation will be discussed in the next chapter as these issues are closely related to a robot's control system.
4. The magnetic polarity is shown in Fig. which go inside these coils. The rotor inside a brushless DC motor is a permanent magnet in a cylindrical shape. 4. We call these coils UU'. W and WW. can be independently controlled in order to form a certain type of sequence. The directions of the currents. only two coils are energized. At this time instant. the magnetic force induced on edge U points towards the righthand side. There are three independent coils. The result of this design is the brushless DC motor. This reacting force will make the rotor rotate in a counterclockwise direction as well. In order to keep the rotor continuously rotating.15 Illustration of the working principle of brushless DC motors. Fig. Since the stator is stationary. are as shown in Fig. each having multiple loops. there will be a reacting force at the south pole of the rotor due to the induced magnetic force at edge U'.15 shows a crosssectional view of a conceptual design solution for the brushless DC motor. Assume that the directions of the currents going inside the two energized coils.15 (the axis of the north and south poles is perpendicular to the axis of the rotation of the rotor). at time instant t. Similarly. 4.15a. Stator U / U (a) At time t (b) At time t + 1 Fig. This force will have a corresponding reacting force on the north pole of the rotor. the rotor will rotate in a counterclockwise direction as a result of the torque produced by the reacting force. 4. which are symmetrically wound around the cylindrical inner surface of the stator. At a time instant. it is necessary to turn "on" the current which goes inside coil VV at an appropriate next time .138 The Fundamentals of Robotics: Linking Perception to Action permanent magnet as the rotor.
g. as these topics are closely related to a robot's control system. torque r times angular . In order to gain maximum efficiency.16 illustrates how an electric motor converts power from the electrical domain to the mechanical domain. An electric motor is an electromechanical device which takes electrical power as input and produces mechanical power as output.3. Force acting on a body times the linear velocity of the body describes power in the mechanical domain. the electromagnetic field will keep the rotor continuously rotating with an induced torque at its maximum value. it is clear that the sequence of energizing the coils must be synchronized with the angular velocity of the rotor. Refer to Eq. 4. In this way. an electric motor is a force or torque generator which forms a dynamic pair for the purpose of creating motion. If there is no dissipation of power.2 Force and Torque Amplifiers Conceptually.Electromechanical System of Robots 139 instant. For maximum efficiency.13. Because of this. we need to examine their relationship.4.16 Power conversion in an electric motor. the output power must be equal to the input power. the larger the induced force will be. the current in coil VV' should be turned "on" at timeinstant t + 1 when the north pole reaches a position underneath edge V. 4. 4. Since force and velocity are closely related to each other. In the electrical domain. a resistor). We will discuss methods of controlling the magnitude of electrical power and the directions of the currents supplied to a brushless DC motor in the next chapter. Voltage I • Current ^ 1 Electromagnet (coils) — '\ _Power_^> K Stator . it is easy to understand that the electronic drive circuit for the control of a brushless DC motor will be quite complicated. The closer a current conductor is to a magnetic pole. Fig . power is the voltage times the current supplied to an electrical load (e. If the input power in the electrical domain applied to an electric motor is a constant p. ^ Torque • Velocity Fig.
one needs to apply strong force or torque to overcome the static frictional force.17 shows the relationship between the input and output in a speed reducer. or torque amplifier.34. In order to trigger the relative motion between two bodies at rest. If an electric motor does not have any inertial load. 4. To reduce speed or amplify torque related to a rotary motion. or torque amplifier. 4. Prom Eq. _ Input shall : . that is. a speed reducer can also be called a force or torque amplifier. That is an indispensable element inside an actuation system. will also be a rotational motion which is transmitted through the output shaft. we connect this rotational motion to the input shaft of a speed reducer. we can see that the only way to increase torque is to reduce velocity if the power is to remain a constant. If the velocity of the input motion is LUQ and the velocity of the output motion is w.. or torque amplifier. its tendency will be to output a very high angular velocity (thousands of rotations per minute). the torque which an electric motor can deliver is very low. tmmmmmmmmmmmmmmm * • ^ " Body 2 ..17 Illustration of the inputoutput relationship in a speed reducer or torque amplifier for rotary motion. T»UJ =p. it is necessary to alter the characteristic of an electric motor so that it can deliver the torque at a reasonable range of magnitude. This is an undesirable feature because the static frictional force between two bodies at rest is usually very large. JI Fig.34) It is easy to see that the higher the velocity is. r ' • Output shaft Body 1 L . the important parameter . As a result. (4. Fig.140 The Fundamentals of Robotics: Linking Perception to Action velocity u is a constant as well. A device which allows us to reduce velocity without loss of power is commonly known as a speed reducer. The output motion from a speed reducer. 4. I i / Speed Reducer or Torque Amplifier . With reference to force or torque. Therefore. the lower the effective torque will be.
4. 4. By coupling device.35) If the torque of the input motion is T 0 . kr = ^ .17. a speed reducer or torque amplifier is a coupling device because it is composed of two independent bodies.Electromechanical System of Robots 141 of the speed reducer is the reduction ratio kr.1 Gear Mechanisms A simple solution for speed reduction. In the following sections.18 A gear mechanism for speed reduction or torque amplification. or torque amplification. — Input shaft ______^_^ ' ••. then the amplified torque of the output motion will be T = kr»r0. In this way.• ' • • ' ^•EELJI Output shaft (a) Gear mechanism (b) Real example Fig. we will discuss conceptually some examples of speed reducers or torque amplifiers. The input motion is supplied to the shaft of the gear having a smaller diameter.3. a pair of links) through a coupling device such as a speed reducer or torque amplifier. that is. (4. is to use a gear mechanism as shown in Fig. IV (4.36) In fact. If the numbers of teeth of the larger and smaller . as shown in Fig. we mean a device which can be physically connected to another pair of bodies without losing any degree of freedom. and the output motion is transmitted from the shaft of the gear having a larger diameter. 4. For example. 4.g.18. the degree of freedom of the kinematic pair is preserved. we can interface a dynamic pair (e.g. a motor) with a kinematic pair (e.2.
As a result. the ratio of speed reduction will be kr is also the ratio of amplification for the output torque. More gears must be added if one wants to alter this distance. the ratio of reduction is not very high (normally less than 100). However. the timing belt is a flex spline with teeth. In a pulleyandtiming belt assembly.2 PulleyandTiming Belt Assemblies In order to overcome the first two drawbacks of a gear mechanism. . • Thirdly. • Secondly. the teeth of the two gears are in direct contact.g.3. there will be the problem of wearandtear. 4. 4. Timing belt Y Input shaft ^^^^^m^^mi ^^ i** •* r >HBH Output shaft (a) Pulley and timingbelt Fig. After longterm use. 4. rubber). there are three notable drawbacks with regard to the gear mechanism: • First of all.19. the distance between the input shaft and the output shaft depends on the diameters of the two gears. 'If (b) A real example Pulleyandtiming belt assembly for speed reduction or torque amplification. the effect of wear and backslash is not as serious as that of a gear mechanism. In addition to imprecision of machining and assembly.19 MBSHE. The whole timing belt is normally made of soft material (e. a common solution is to use the pulleyandtiming belt assembly as shown in Fig.2. the undesirable problem of backslash (i.e. hysteresis) becomes inevitable.142 The Fundamentals of Robotics: Linking Perception to Action gears are No and Ni respectively. The direct contact of the two pulleys is therefore replaced by the soft contact between the timing belt and the two pulleys.
is the harmonic drive. the speed reduction is equal to the ratio between the numbers of teeth of the two pulleys.2. The axis of rotational motion is the output shaft. and c) a rigid circular spline with teeth. Its axis of rotation is the input shaft. if the flex spline is fixed onto the supporting body.20 Harmonic drive for speed reduction or torque amplification. 4.section view •• . the ratio of speed reduction (or the ratio of torque amplification) will be k . ^ ^ £ j ^ B B S ^ ^ ^ ^ ^ ^ ^ ^ ^ B I B f r ' .20..Electromechanical System of Robots 143 Since the length of the timing belt is selectable. A harmonic drive is composed of: a) an elliptical wave generator. the rigid circular spline with teeth is the output motion carrier.3 HarmonicDrive Devices The best device for speed reduction. It is easy to see that there will be zero displacement between the flex spline . Assume that the output motion carrier is the flex spline. as shown in Fig. Input motion earner Wave generator * ^. Rigi(J circular' sp. 4. and d) the supporting body.^ Again. or torque amplification. 4. Alternatively. And. the flex spline is the output motion carrier. Refer to Fig. if the circular spline is fixed onto the supporting body. If the number of teeth on the larger pulley is No and the number of teeth on the smaller pulley is Nt.20.3. The elliptical wave generator is the input motion carrier. Just as with a gear mechanism. b) a flex spline with teeth. The role of the wave generator is to keep the flex spline in contact with the circular spline at the two ends along the major axis of the elliptical wave generator. it is easy to alter the distance between the input shaft and the output shaft.ine (b) Real example Fig.plI" H r V ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ H t 'f* \ ^~"~"^. ' a f a : \ ™ "j^^HSaP^^KM^^^^^r^flBi^k ^i 1 Output motion carrier (a) Cross . this ratio is not very high (typically less than 100). 4.
electrical) into the mechanical domain. kr is linearly proportional to Nf. If this is not desirable. If Nf = Nc + 1.e. If Nf = Nc + 2. Similarly.144 The Fundamentals of Robotics: Linking Perception to Action and the circular spline if they have the same numbers of teeth. The only problem with a harmonic drive is that the input and output shafts are coaxial. the design of a robot not only embraces mechanism design but also deals with machine design (or mechatronic design of a machine). the circular spline will shift two teeth after one full rotation of the wave generator. . the kinematic pairs of a robot's mechanism must be coupled with a set of dynamic pairs in order to produce controllable motions. the wave generator has to make Nf/2 rounds of full rotation.4 Formation of a Robot's Electromechanical System We now know that a robot's mechanism cannot move on its own without the addition of actuation elements. 4. As it is necessary to couple kinematic pairs of a robot's mechanism with a set of dynamic pairs. a good solution is to combine the use of the harmonic drive (for achieving better ratio and no backslash) with the pulleyandtiming belt assembly (for achieving the offset between input and output shafts). The role of a dynamic pair (e. Interestingly. if Nf = Nc + 2. it is easy to see that the circular spline will shift one tooth after one full rotation of the wave generator. Since the contact forces between the flex spline and the circular spline are along the major axis of the wave generator (perpendicular to the tangential direction of each pair of teeth in contact). the ratio of speed reduction or torque amplification will be kr = Nf/2. In order to make the flex spline shift Nf teeth. there is no backslash effect.g. such as motors and torque amplifiers. Let Nf be the number of teeth on the flex spline and Nc the number of teeth on the circular spline. In other words. motor) is to convert energy from one domain (i. This explains why the ratio of speed reduction of a harmonic drive can be very high (easily over several hundred). as long as there is no slippage between a pair of teeth in contact. This is because all motions must originate from a source of energy.
If we denote TJ the torque received by joint i..Electromechanical System of Robots 145 4. In this way.4. _ _ _ i : i . Thus. On the other hand.i.37) .i * J L f ••• 1 I § « I ""me amplifier J .iii. ^S^BQI : i I Joint i Kinematic Pair ! Fig. the motion of link i comes from the torque applied to joint i.e. 4. In this illustration.. And the output motion of the motor (i. ' 1 i 1 Pulley and timing belt Torque BB» u s » P*"™* I genei. we have Ti = kri»Tmi.21 illustrates the coupling of a dynamic pair with a kinematic pair.. Fig. both the stator of the motor and the supporting body of the torque amplifier are fixed onto link i — 1.21 Kinetodynamic pair inside a robot. Dynamic Pair > . ! . an entity called kinetodynamic pair is formed. dl. The output shaft of the torque amplifier coincides with the axis of rotation of joint i. a robot can also be treated as the combination of a set of kinetodynamic pairs. (4.1 OnetoOne Couplings A robot's mechanism can be treated as the combination of a set of kinematic pairs. A common way to form the coupling between the dynamic and kinematic pairs is to independently connect a dynamic pair to a kinematic pair. Assume that the kinematic pair contains link i — 1 and link i.. 4. torque generator) is coupled with the input motion of the torque amplifier through a pulley and timingbelt assembly. Tmi the output torque of the motor and kri the total ratio of torque amplification (including the one due to the pulleyandtiming belt assembly). i J i " d.
000000000000000000 One notable advantage of the onetoone coupling scheme is the fast motion response at each joint. each kinematic pair is coupled to its corresponding dynamic pair. 4. An educational robot is used to serve as the arm manipulator. One serious drawback is the weight of the overall robot . 4. . the robot has five degrees of freedom and five kinetodynamic pairs. it is easy to implement velocity feedback control which is necessary for applications involving the behavior of a trajectory following.3 Fig.^** ~* T ^. there are several drawbacks associated with this onetoone coupling scheme. Example 4. \ i l TUB I i"s'SBhi j H ~ . we will have u>i = — »Lomi. As a result. if we denote u>i the angular velocity vector of link i and u>mi the angular velocity vector of the motor. All the commercially available industrial or educational arm manipulators make use of the onetoone coupling scheme to form the kinetodynamic pairs. (4. each of which is formed by applying the onetoone coupling scheme. In this example.• » "" * Fig.22 shows an experimental setup for research on robotic handeye coordination. 4.146 The Fundamentals of Robotics: Linking Perception to Action Similarly.38) As shown in Fig. However. The kinetodynamic pairs of this robot are designed by applying the onetoone coupling scheme. The robot has five kinetodynamic pairs.22 An educational arm manipulator in an experimental setup for research on robotic handeye coordination.21. We call this type of kinetodynamic coupling the onetoone coupling.
is applied to joint i of the kinematic pair consisting of link i — 1 and link i. In general. a major portion of the hardware cost of a robot is spent on electric motors.e. the total cost and weight of a robot is greatly reduced as well.23 Illustration of onetomany coupling scheme. and power amplifiers. A common scheme to achieve this objective is illustrated in Fig. In this way.2 OnetoMany Couplings In order to overcome the drawbacks of the onetoone coupling scheme. one motor) and to couple it with all. As a consequence. Moreover. torque amplifiers. the effective payload of a robot is largely compromised due to the heavy weights of the electric motors.4. an electric motor. If we attempt to use a single motor to independently drive multiple inertial loads. it is easy to consider how the motion from the motor is split and distributed to multiple inertial loads. the number of motors inside a robot is largely reduced. Single Motor Single $ Motion =) Splitter Multiple Motion z ) Distributors Multiple Motion Reducers =) Multiple Inertia] Loads Fig. 4. an alternative solution is the onetomany coupling scheme. or a subset of independent kinematic pairs inside a robot's mechanism. link i is called an inertial load of the motor. it is possible to miniaturize a robot to suit medical needs such as a robotic arm for minimallyinvasive surgery. 4. after the coupling between the motor and torque amplifier. is an expensive device.Electromechanical System of Robots 147 system because an electric motor is a heavy device compared with other materials and components. The philosophy of a onetomany coupling scheme is to make use of a single dynamic pair (i. A second drawback is the cost. The additional new elements are: a) the motion splitters and b) the motion distributors. In the mechanical domain. 4. When there is no motor inside an open kinematic chain. which must move together with the kinematic pairs. .23.21. The output torque. 4. Refer to Fig. together with its torque amplifier and power amplifier. In fact.
the requirement on the motion directions at the multiple shafts is not fixed and is timevarying. meaning the direction of the output motion is exactly the same as the direction of the input motion.24 Working principle of a bidirectional clutch. we must address the following issues: • How do we split the motion of the motor into multiple motions. 4. . Contact Disk Solenoid 2 Shaft of Input Motion Solenoid 1 ^^^—^^—^—/ .148 The Fundamentals of Robotics: Linking Perception to Action 4.e. This poses a problem. " ^^H ^ ^ H ^ H ^ H ' i Shaft of Output Motion mechanism A' Pulley / Timingbelt assembly Fig.1 Motion Distributors In order to implement the onetomany coupling scheme. For a bidirectional clutch. A simple solution is to build a new device called a bidirectional clutch. either clockwise or counterclockwise).4. the direction of the output motion is independent of the direction of the input motion and can be switched to any one of the two possible directions: clockwise or counterclockwise. Since we are using a single motor. the direction of the motion delivered by this motor is unique at one time instant (i. All the available clutches on the market are omnidirectional. However. visible at the multiple shafts supporting the multiple inertial loads? • How do we handle the conflicting requirement on the directions of motions at these multiple shafts? Let us examine the second issue first. — "~^^^^^^^f ^"/'"^^•^H^H^K' jf.2.
In this way. the contact disk. . and the multiple speed reducers are composed of worm gears and cables. The contact disk rotates with the input shaft. nanism ^^•SHPUWMK^V J Pulley and Timing Hell Assemhh WHB^BE^m^^^SA Fig.25 Singlemotordriven multiplefingered hand which makes use of bidirectional clutches for motion distribution.il Cliikh diMi \k<. is in contact with the pulleyandtiming belt assembly on the output shaft.e. Fig.24. Alternatively. Due to the use of bidirectional clutches. the total number of independent degrees of freedom for the three fingers is seven. the whole hand (including the motor coaxially located at the middle of the wrist) is very compact. 4. a pair of gears will invert the direction of the input motion).Electromechanical System of Robots 149 The working principle of a bidirectional clutch is shown in Fig. In this example. 4. made of ferrous materials. the input motion is transmitted to the output shaft. 4. The inversion of the input motion is due to the effect of the gear mechanism (i. ^ ^ ^ ^ ^ H i l t ' One liiiliiivlion.25 shows a prototype of singlemotordriven multiplefingered hand. When solenoid 1 is switched on. when solenoid 2 is switched on. while the direction of the output motion is the same as the direction of the input motion. the contact disk is in contact with the gear mechanism on the output shaft and the input motion is inverted and transmitted to the output shaft.
Fig. found in some automobiles. Each output motion of the splitter is coupled to a motion distributor. which requires the ability to smoothly vary the transmitted motion (i. the motion is further coupled to the inertial load through a speed reducer and a cable (or a pulleyandtiming belt mechanism).27 shows the application of the parallel splitter of motion to .e.2. Fig. 4. It does not produce smooth motion. to regulate the amount of energy released to the corresponding inertial load). as was discussed above. 4. How to miniaturize a CVT device is still a research issue.e. There are two ways to duplicate the input rotary motion. either "on" or "off"). \ • v y : Output Shafts Output Shai /" A Fig. One way to overcome this drawback is to introduce a device called a continuous variable transmission (or CVT).2 Parallel Splitter of Motion The purpose of a motion splitter is to duplicate the input motion into multiple output motions. their axes of rotation are parallel). it is not suitable for velocity control.26 shows an example of a planargear mechanism which splits the input rotarymotion into a set of four output rotarymotions at four output shafts.26 Parallel splitter of input motion using a planar gear mechanism.4. The first method consists of using a planargear mechanism to split the input motion into multiple output motions at multiple output shafts which are positioned in parallel (i.e. 4. The input rotarymotion to a motion splitter comes from the output motion of a single motor. Subsequently. /'" / /" Input Shaft \ * >  ' ' \ . A device implementing this method is called a parallel splitter of motion. 4.150 The Fundamentals of Robotics: Linking Perception to Action The drawback of the clutchbased motion distribution is that the mode of motion transmission is binary (i. Therefore.
The first one is the shaft for twist rotation.Electromechanical System of Robots 151 implement the singlemotordriven multiplefingered hand.2. Fig. However. However.28 conceptually illustrates a serial splitter of motion having n consecutive units. Each unit has two output shafts aligned in a series. However. 4.27 Parallel splitter of motion inside a singlemotordriven multiplefingered hand. In practice. the simplest solution is the one which makes use of the bevelgear mechanism. 4. and the second one is the shaft for pivotal rotation.3 Serial Splitter of Motion A humanoid robot normally has four limbs (two legs and two arms) which are open kinematicchains. The advantage of a parallel motion splitter is its simplicity. it is not advantageous for the implementation of an open kinematicchain. a notable drawback is that a long cable or timing belt has to be used if the inertial load is located far away from the motion splitter.4. If one adopts the scheme of onetomany coupling for the kinetodynamic pairs in a limb. the parallel splitter of motion is suitable for the implementation of a singlemotordriven multiplefingered hand. A serial splitter of motion is a mechanism which duplicates the input rotary motion into multiple output motions at multiple output shafts aligned in a series. it is better to employ a device called a serial splitter of motion. The twist and pivotal axes of a unit are perpendicular to . like an arm manipulator. In this example. Fig 4. the motion of the single motor is duplicated into seven output motions at the output shafts of the planargear mechanism. there are many different methods of implementing a serial splitter of motion. Therefore.
• :.152 The Fundamentals of Robotics: Linking Perception to Action Input  \  Twist jAxisl \  Twist !Axis2 K  Twist lAxisn ':\J Pivotal Axis 1 \. Example 4.n .29. each other.7 j j i i a i . because the number of motors ...'•< i •(. •ooooooooooooooooo It is advantageous to employ the scheme of onetomany coupling to design the mechanisms of a humanoid robot.1 ' .4 An example is shown in Fig.28 Serial splitter of input motion using a series of bevelgear mechanisms. 4. " " • : ' •" ' ••''L'?*saiJ^^Bi T^T^^ ^^MIB^I! 'i 1 ^—MjHr^lr^^iBi^BB (a) Real example (b) Pivotal rotation (c) Twist rotation Fig.29 Real example of an open kinematicchain having a serial splitter of motion. In this example. an open kinematicchain can be easily formed by putting together a set of these units in a series. iSlfr* . . : • • . 4.••• •• . l f l  g g E M M i i:^1 :?"'.. Therefore. v* . .:'>. • ./.. we display two units. M Pivotal Axis 2 \\ 1/ 1 Pivotal Axis n \' Fig. Each unit has two degrees of freedom: a) a twist rotation and b) a pivotal rotation. 4. j S .
Electromechanical System of Robots
153
will largely be reduced. For example, a single motor may be sufficient to supply the necessary amount of energy to all the degrees of freedom inside one limb, including the multiplefingered hand or foot. Fig. 4.30 shows a prototype of a humanoid robot. In this prototype, each of the two arm systems has six independent degrees of freedom. Each arm is driven by a single motor located inside the shoulder.
^^^^^^^^H'
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ H . ;
A*M Laboratory
IWQlMttontei Laboratory
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 
'P'SS 1 "•=?' Oisrr "•'"»•
HIS*
A > ^t\ M I
Fig. 4.30 A prototype of humanoid robot where each of the two arm systems is driven by a single motor.
4.4.2.4
Torque at Inertial Loads
Under the onetomany coupling scheme, the output torque TO and the angular velocity LOQ of the motor are the same for all the inertial loads. Refer to Fig. 4.22. Let us denote ks the ratio of speed reduction from the motion splitter, kdi the ratio of speed reduction from the motion distributor connected to inertial load i (link i in an open kinematic chain), and kr% the ratio of speed reduction from the speed reducer coupled to inertial load i.
154
The Fundamentals of Robotics: Linking Perception to Action
Then, the torque T; received by inertial load i will be _ ( ks • kdi • kri • r 0 \ 0 if the clutch is "on"; otherwise.
(4.39) '
It is clear that the torque applied to an inertial load is not linear. The only way to obtain a linearlyvariable output torque to an inertial load is to introduce a CVT device which has a ratio of speed reduction that is a real number and is controllable. 4.4.3 Open KinetoDynamic Chains
A robot's mechanism is used to shape the output motion as a result of the input motions. In Chapter 3, we learned that the relationship between the input and output motions of a robot's mechanism is fully described by kinematics. We also know that a robot's mechanism cannot move on its own because all the mechanical motions must originate from the work done by the forces and/or torques acting at the joints of a robot's mechanism. In order to apply forces or torques to a robot's mechanism, there must be a coupling between the kinematic pairs of a robot's mechanism and a set of dynamic pairs (or one dynamic pair if adopting the onetomany coupling scheme). Accordingly, the kinetodynamic pairs of a robot form the electromechanical system. For kinematic analysis of a robot's mechanism, we employ the concept of an open kinematicchain to make the description clear. In a similar way, it is helpful to define the concept of open kinetodynamic chain to ease the study of robot statics and dynamics. A formal definition of open kinetodynamic chain can be stated as follows: Definition 4.6 An open kinetodynamic chain is the entity formed by the coupling of a simple open kinematicchain with a single or a set of dynamic pair(s). Based on the concept of kinetodynamic chain, the definition of robot statics can be easily defined as follows: Definition 4.7 Robot statics is the study of forces or torques acting on a kinetodynamic chain which keep the kinetodynamic chain at rest. Similarly, the definition of robot dynamics can be defined as follows: Definition 4.8 Robot dynamics is the study of forces or torques acting on a kinetodynamic chain which drive the kinetodynamic pairs to produce the
Electromechanical System of Robots Table 4.1 Parameters of links.
155
Link 1 Weight I Length I Center of Gravity (CG) I Offset of CG 1 mi h rcl lci 2 m2 h r_c2  lC2
desired input motions to a robot's mechanism, so as to obtain the desired output motion of the robot's mechanism. In other words, robot dynamics is the study of the relationship between the forces (or torques) applied to a set of kinetodynamic pairs, and the motions of the corresponding simple open kinematicchain. For the study of robot statics and dynamics, all physical quantities must be expressed in a common reference frame. By default, we consider that all the parameters and variables are expressed in the base frame of the simple open kinematicchain. We do not explicitly indicate the common reference frame with any superscript, whenever it is not necessary to do so. 4.5 Robot Statics
Fig. 4.31 illustrates the problem of robot statics. The humanoid robot is standing still, and its task is to hold the wheel which has a rotation axis that is fixed at point A. If the robot releases its hold, the wheel will rotate in a clockwise direction due to the inertial load hanging onto the wheel. By holding the wheel, the robot receives a force and torque at the endeffector of the arm. Let us consider the case in which the robot's arm consists of two rigid links in the XY plane of the base frame. We assume that the parameters of the two links are as shown in Table 4.1, where the offset of the center of gravity (CG) of a link is measured from the link's proximal node. In this example, the robot's body and link 1 of the arm forms the first kinetodynamic pair, and links 1 and 2 of the arm form the second kinetodynamic pair. Assume that the two joints are revolute joints. And, we denote re the acting point of the endeffector of the robot arm upon the wheel. Now, the question is: What should torques T\ and T^ applied to the two revolute joints be in order to keep the wheel at rest, if the endeffector receives force fe from the wheel ? and Te = (Tx,Ty,TzY). In the following In general, fe = {fxjyjzf), sections, we will discover the answer.
156
The Fundamentals of Robotics: Linking Perception to Action
I
Torque
r
mii
V ^ ^^
Fig. 4.31
Illustration of the problem of robot statics.
4.5.1
Dynamic System of Particles
The inertial load of the wheel can be treated as a force or torque generator. When the endeffector holds the wheel, it forms a dynamic pair and also, a special kinematic pair. The kinematic pair is special because the endeffector has more than one degree of freedom. In general, it has six degrees of freedom. For simplicity's sake, we can conceptually treat this coupling as another kinetodynamic pair. Therefore, for the example shown in Fig. 4.31, there are three kinetodynamic pairs which form an open kinetodynamic chain. Since all the bodies under consideration are rigid bodies, we can mathematically treat these kinetodynamic pairs as a dynamic system of particles (three particles, in this example) with their masses concentrated at their centers of gravity. For the third kinetodynamic pair, the weight of the wheel is supported by its rotation axis. It has no effect on the static relationship between the robot's arm and the wheel. As a result, the mass of the third kinetodynamic pair is treated as zero, and its center of gravity is considered at reFig. 4.32 illustrates the mathematical abstraction of the open kineto
Electromechanical System of Robots
157
^ ^ ^ ^
System of Particles
^ ^ ^ \ ^
/A /*
Z
*"Y
rcl
Tor q ue ^ 
\ \
\
©
I
\ .
©
Force ^~~~^^_^^ Force
Force^S1^
y/ ^
Fig. 4.32 Mathematical abstraction of an open kinetodynamic chain by a dynamic system of particles.
dynamic chain by a dynamic system of particles. Since in this example, there are three kinetodynamic pairs inside the open kinetodynamic chain, the corresponding dynamic system has three particles. In the dynamic system of particles, each particle has a mass and receives an externally exerted force (i.e. gravitational force). However, for the particle corresponding to the endeffector, it does not have any mass but may receive both externallyexerted force and torque. Here, we only consider externallyexerted forces (or torques) to a system because all the internal forces (or torques) will not do any work to the system itself. Hence, we can ignore the effect of internal forces if all the bodies are rigid bodies (i.e. having no deformation).
4.5.2
Generalized Coordinates and Forces
In a threedimensional space, the position of a particle is normally described by the X, Y and Z coordinates which are called the physical coordinates with respect to a Cartesian frame. For the study of certain problems (e.g. statics and dynamics), it is not helpful to directly use these coordinates. One idea, which is helpful, is to see whether there is an alternative set of coordinates which are easy to manipulate for mathematical formulation, and also permit the unique recovery of the physical coordinates (X, Y, Z). If a set of n new variables qt (i — 1, 2, ...n) is related to the physical coordinates
158
The Fundamentals of Robotics: Linking Perception to Action
(X, Y, Z) in the following manner:
{
X =
fx(qi,q2,,qn)
Y = fv{qi,q3,...,qn)
(4.40)
z = fz{qi,q2,,qn), and these new variables completely determine the positions of the particles in a dynamic system, these new coordinates are called generalized coordinates. The space defined by the generalized coordinates is called the Configuration Space. In general, there are many sets of variables which satisfy the above requirement. This is an advantage because we have the freedom to choose whichever variable is easiest to manipulate. For example, in robotics, we choose the joint variables to be the generalized coordinates of a robot's dynamic system. However, one must be cautious and makes sure that the generalized coordinates can recover the physical coordinates without any ambiguity. Example 4.5 Refer to Fig. 4.31 and Fig. 4.32. We choose the two joint variables 9\ and #2 to be the generalized coordinates. For the position of particle 2 in Fig. 4.32, its physical coordinates will be
Tc2
= fXc2 \ = (ll * cos (0i) + l& • cos(6i + 02) \ ~~ Uc2 ) \k • sin(0!) + lc2 • sin(0i + 82) J '
oooooooooooooooooo
For a generalized coordinate, we can imagine that there is a force associated to it. This force is called the generalized force. In robotics, if we choose joint variables to be the generalized coordinates, the generalized forces will be the forces or torques applied to the joints by the motors. The purpose of introducing the concept of generalized forces is to be able to compute the power which is equal to the generalized force times the displacement of the corresponding generalized coordinate. 4.5.3 Constraints of a Dynamic System
The mathematical abstraction of an open kinetodynamic chain by a dynamic system of particles is valid if and only if the particles satisfy the constraints imposed upon the corresponding open kinetodynamic chain. A constraint refers to the restriction imposed upon a pair of bodies which interacts with each other. For every constraint, there is a constraint equation
Electromechanical System of Robots
159
and a constraint force or torque. Thus, a kinetodynamic pair is a perfect constraint imposed upon a pair of bodies. And the constraint equation simply indicates the kinematic relationship among the generalized coordinates. Example 4.6 Refer to Fig. 4.31 and Fig. 4.32. We choose the two joint variables B\ and 92 to be the generalized coordinates. Assume that the endeffector's position is re. Then, the generalized coordinates of the dynamic system of particles must satisfy the following constraint: _ (xe\ re~\Ve)~ _ fh •cos(6»1)M2«cos(6>i + 92)\ {h»sin(91) + l2»sm(01+02) ) •
OOOOOOOOOOOOOOOOOO
For a system with n generalized coordinates qt (i = 1,2, ...,n), the constraints can be described in the following form: /(gi,g2,...,g n ,t) = o. (4.41)
One can omit the time variable if the constraint is independent of time. If Eq. 4.41 is differentiable, we have
f = 7T * + ^
+  + 7T4» + % = °
< 4  42 )
dqi dq2 dqn dt A constraint expressed in the form of Eq. 4.41 is called a configuration constraint. A constraint expressed in the form of Eq. 4.42 is called a velocity constraint. If a constraint can be expressed in both the forms of Eq. 4.41 and Eq. 4.42, it is called a holonomic constraint. Otherwise, it is called a nonholonomic constraint. In robotics, all open kinetodynamic chains impose holonomic constraints. Example 4.7 Refer to Fig. 4.31 and Fig. 4.32. We choose the two joint variables 6\ and 02 to be the generalized coordinates. Let us consider the position vectors of all the particles in the system. Then, the generalized coordinates of this dynamic system must satisfy the following constraints: (1) For position vector rc\ of particle 1, we have = (Xcl\ or rci=rcl(0i,02).
=
/^ci«cos((9i)\
160
The Fundamentals of Robotics: Linking Perception to Action
(2) For position vector rc2 of particle 2, we have _ fxc2 \ _ fh» cos(0i) + lc2 • cos(0i + 62) \ ~ \Vc2 ) ~ \h • 8111(0!) + lc2 • Sin(0! + 02) )
^ or
rC2 = rc2(61,62). (3) For position vector re of particle 3, we have
Te
= (XA = (h • cos(0i) + l2 • COB^! + 02)\ \Ve) \h»sm(61)+l2»sm(01+d2) )
or
re = re(91,62).
oooooooooooooooooo
4.5.4 Virtual Displacements
In the above example, we can see that a position vector of a particle in a dynamic system can be expressed as a function of the generalized coordinates. Assume that a dynamic system of particles is fully described with a set of n generalized coordinates % (i = 1, 2,..., n). Then, position vector r of a particle can be written as r = r(q1,q2,...,qn). Differentiating Eq. 4.43 with respect to time gives Or , dr v—> f dr , 1 dr , dr = ~^dq1 + ^—dq2 + ... + ^—dqn = } < ^dqi } . (4.44) (4.43)
dq2 dqn ~ 1% J dqi dr is the infinitesimal displacement within time interval dt. This displacement satisfies the velocity constraint in the form of Eq. 4.42. For the study of robot statics, the system is at rest and there is no real displacement. If we examine the tendency of particles' displacements, time interval dt can be ignored. In this way, the infinitesimal displacement dr is called the virtual displacement. In other words, it is not a real displacement. In dynamics, the virtual displacement corresponding to the differential dr is
Electromechanical System of Robots
161
denoted by 5r. If we denote 5qi the virtual displacement of the generalized coordinate &, Eq. 4.44 becomes
*r = §{!;•**}•
(4'45)
Based on the above discussions, it is clear that a virtual displacement is an infinitesimal displacement. These virtual displacements are obtained by differentiation. As a result, all virtual displacements satisfy the velocity constraint of a dynamic system in the form of Eq. 4.42. 4.5.5 Virtual Works
Refer to Fig. 4.32. Assume that all the particles receive an externallyexerted force, and the particle corresponding to the endeffector may also receive an externallyexerted torque. Here, we assume that the endeffector receives an externallyexerted force only. We denote (fci,fc2) the external (e.g. gravitational) forces exerted on particles 1 and 2 respectively, and fe the external force exerted on particle 3. By definition, the work done by these forces to the system will be SW = fcl • Srcl + fc2 • 5rc2 + fem 5re. (4.46)
Since the work done is not real because the displacements are not real, we call it virtual work. 4.5.6 Principle of Virtual Work for Statics
It is easy to understand at this point that a static system (i.e. a system at rest) will remain at rest if the virtual work done by the externallyexerted forces is zero. This is because the motion can only originate from work done to a system. Refer to Fig. 4.32. Forces (/ c i,/c2,/e) are physically real. If there are no other external forces, the virtual work expressed in Eq. 4.46 will not be zero. In this case, real work will be done to the system and it will make the robot's arm and the wheel undertake motion. If we want the robot to hold the wheel at rest, there must be some other external forces applied to the robot's arm in order to make the total virtual work be zero. (NOTE: A work can be negative if energy is removed from a system). In this example, the generalized coordinates of the system are (0i)#2) If we denote (TI,T2) as the generalized forces associated with these
162
The Fundamentals of Robotics: Linking Perception to Action
two generalized coordinates, the virtual work done by the generalized forces will be 6WT = n • 661 + T2 • 662. (4.47)
Now, we can state the principle of virtual work for statics as follows: Principle of Virtual Work: For a static system to remain at rest, the total virtual work done by all external forces to the system must be equal to zero, i.e. SW — 6WT = 0.
4.5.7
Statics Against SelfInertial Loads
If the goal is to keep the system at rest (i.e. statics), the purpose of the generalized forces will be to cancel out virtual work done by external forces acting upon the system. Refer to the example shown in Fig. 4.31 and Fig. 4.32. We first examine the generalized forces required to cancel out virtual work done by gravitational forces acting upon links 1 and 2. In Example 4.7, the centers of gravity of links 1 and 2 can be expressed as functions of the generalized coordinates. That is,
(rcl=rcl(91,02) \rc2=rc2(Oi,02).
Differentiating Eq. 4.48 yields (5rcl =^.50
1 +
(4.48)
[
'
^.502
{ {Src2 = ^.6e1
(449)
+
^.602.
These are expressions for the virtual displacements of the two centers of gravity. By applying the principle of virtual work for statics, we have n • 661 + T2 • S02 = fci • Srcl + fc2 • Src2. (4.50)
Substituting Eq. 4.49 into Eq. 4.50 yields the following two equalities:
{ T2 = / c l . f ^ + / c 2 .  
(451)
Electromechanical System of Robots
163
In general, in order to keep an open kinetodynamic chain with n links at rest, the generalized force at joint i will be
*£{'«•£}•
<«2>
where <?; = 8i if joint i is a revolute joint, and fcj is the external (gravitational) force acting on link j at its center of gravity rCj. Some textbooks treat Eq. 4.52 as the definition of generalized forces. Here, we treat it as the solution for determining generalized forces. The work done by a gravitational force is called potential energy. An alternative way of deriving the expression in Eq. 4.52 is to make use of the concept of potential energy. For the example shown in Fig. 4.31, the potential energy of the robot's arm is V=~fclTclfc2*rc2. (4.53)
Since the centers of gravity of the arm's two links are the functions of the generalized coordinates, the potential energy can be implicitly expressed as follows:
V = V(61,62).
This means that the potential energy only depends on the positions which are the functions of the generalized coordinates. The differentiation of the above equation yields virtual work done by gravitational forces fcl and /C2 That is, dV 5WV = 5V = —*591Ou\ Ou2
dV — .562.
(4.54)
(NOTE: The negative sign at expression 5W = —SV means that potential energy is removed from a body if the displacement of the body is in the direction of the gravitational force). By applying the principle of virtual work (i.e. 8WT = SWV), we obtain
(4.55)
* >
=
 & •
Eq. 4.55 is equivalent to Eq. 4.51. In general, in order to maintain an open kinetodynamic chain with n links at rest, the generalized force at
164
The Fundamentals of Robotics: Linking Perception to Action
joint i will be n = —. dqi In fact, Eq. 4.56 is equivalent to Eq. 4.52. 4.5.8 Statics Against Inertial Loads at Endeffector (4.56)
Refer to the example shown in Fig. 4.31 and Fig. 4.32 again. We will now examine what should be the generalized forces required to cancel out virtual work done by external force and torque acting upon the endeffector's frame in order to keep the robot's arm at rest. From the study of robot kinematics, we know that velocity vector Pe of the endeffector's frame in a simple open kinematicchain is composed of two vectors: a) the linear velocity vector Oe of the endeffector's frame and b) its angular velocity vector we. That is,
(NOTE: Symbol Pe stands for the posture of the endeffector's frame). If the Jacobian matrix of the robot's arm is J, we have Pe = J » q (4.57)
where q = (0i,02)*. From Eq. 4.57 and by definition, the virtual displacement of the endeffector's frame will be 5Pe = J» 5q. (4.58)
If we denote £e = (/e,Te)* the vector of the external force and torque exerted upon the endeffector's frame, the virtual work done to the system (i.e. robot) at rest is as follows: 5W = £*5Pe. (4.59)
(NOTE: In general, & = (fx, fyJz,Tx,Ty,TzY). In order to cancel out virtual work done to the system at rest, we must apply generalized forces (T\, T2) to the system at rest. We know that virtual work done by generalized forces is 5WT = n • 0i + r2 • 02 = r* • 5q (4.60)
Electromechanical System of Robots
165
where r = (71^2)*.
By applying the principle of virtual work (i.e. 5WT = SW), we obtain r* = ee • J(4.61)
If we compute the transpose at both sides of Eq. 4.61, this equation will become
T
= Jt»£e.
(4.62)
Eq. 4.62 describes the static relationship between external force/torque exerted upon the endeffector's frame, and generalized forces required on the open kinetodynamic chain in order to keep the whole system at rest. This static relationship is very useful for the study of compliance or interactive control because £e encapsulates the interacting force and/or torque between the robot's hand and its environment. In general, Eq. 4.62 is also valid for any open kinetodynamic chain with n rigid links. Here, we consider the case of revolute joints. If joint i is a prismatic joint, the corresponding generalized coordinate will be linear displacement di and the associated generalized force will be force ft itself. 4.6 Robot Dynamics
The purpose of robot dynamics is to determine the generalized forces required to not only overcome the selfinertial loads (due to the weights of the links inside the robot) but also to produce the desired input motions to the robot's mechanism. Fig 4.33 illustrates the problem of robot dynamics. In this example, we assume that the robot's body stands still. The robot's arm is holding a tool, which can be a laser welding gun or any other industrial tool. For the robot to fulfill a task (e.g. arc welding), the robot needs to move the tool to follow a predefined trajectory. It is clear that the robot's arm has to execute motions in order to carry the tool along that trajectory. Therefore, a certain amount of work (i.e. energy) must be done to the robot system. The work done to the robot will not only overcome the selfinertial load of the arm but also generate the desired motions to the joints of the arm so that the tool will follow the trajectory. Now, the question is: What should the generalized forces applied to the joints of the robot arm be in order to produce the desired velocity and
166
The Fundamentals of Robotics: Linking Perception to Action
n
m\g
•v
mz8
s^
(^^^^
y
J
U
Trajectory
Z_H
Fig. 4.33 Illustration of the problem of robot dynamics.
acceleration at the generalized coordinates (joint variables), which will subsequently produce the desired velocity and acceleration at the endeffector's frame (forward kinematics of the robot's mechanism)? 4.6.1 Dynamic System of Rigid Bodies
Refer to Fig 4.33. The robot's arm is an open kinetodynamic chain consisting of two rigid links. For the study of robot dynamics, an open kinetodynamic chain can be conveniently treated as a dynamic system of rigid bodies. These bodies satisfy the constraints imposed by the kinematics of the robot's mechanism, and also the constraints imposed by Newton's third law (the relationship between a pair of acting and reacting forces). Fig. 4.34 shows the mathematical abstraction of an open kinetodynamic chain by a dynamic system of rigid bodies. Link 0 is the supporting body of link 1, while link 1 is the supporting body of link 2. In this example, links 1 and 2 are represented respectively by a mass, a center of gravity, a sum of all externallyexerted forces upon the body and a sum of all externally
35 Dynamics of a single rigid body. let us examine the solution of dynamics for a single rigid body. Table 4.2 Parameters of links. 4. . Angular motion M — .1 " " ~'T ' ' Linear motion Fig. 4.'•illU V  O •* I \ • cv t —™r' / Fig.' . f r Z A.34.Electromechanical System of Robots 167 ^^"^ / LinkO System of Rigid Bodies ^^\^ ^\ \ \ r l .2 summarizes the parameters of links 1 and 2 in Fig. Table 4.2 Dynamics of a Rigid Body Before we develop the solution for robot dynamics.34 Mathematical abstraction of an open kinetodynamic chain by a dynamic system of rigid bodies.6. 4. 4. Link 1 2 Mass mi m2 Center of gravity rcl rc2 Summed force Fi F2 Summed torque Mi M2 exerted torques upon the body.
frame 0). By default. Now. If we know the mass of the rigid body.65) This equation illustrates that the sum of displacement vector rd weighted by differential mass element dm is equal to zero. 4. the following result is proven: / J body rd • dm = 0. . The sum of the external forces is denoted by F and the sum of the external torques is denoted by M. potential and kinetic energy)? 4. how is the motion of a rigid body related to its energy state (i. If the mass density of rigid body is p(r). r is the position vector of differential mass element dm on the body.e. r and rd.35 shows a single rigid body in a reference frame. rd is the displacement vector from the center of gravity to differential mass element dm. Eq.64) Jbody Since r = rc + rd. rc. then the position vector indicating the center of gravity of the body is 1 f rc = — / m r • dm.2.63) where dm is a differential mass element. m Jbody rn Jbody m Jbody Therefore. the questions are: What will the relationship be between the motion parameters of a rigid body and the sum of (external) forces/torques exerted upon the body? And. (4. 4. are expressed in the common reference frame (i. l f rc = — / rc • dm H / rd • dm = rc\ / rd • dm.e.6. (4.168 The Fundamentals of Robotics: Linking Perception to Action Fig.1 Motion Equations of a Rigid Body By definition. the mass of the body is m= / J body dm (4. all these vectors. Assume that the body is subject to some external forces and torques. rc is the position vector indicating the center of gravity of the body.64 becomes 1 f 1 f . then dm is expressed as dm = p(r) • dx dy dz.
the differentiation of Eq. the linear momentum of the rigid body becomes p= J body vc» dm = vc • m. the linear velocity vector at position r of differential mass element dm will be dr dt Since ^ = vc and ^ drc dt drd dt = u x r<j. From the relationship r = rc + rd. angular acceleration vector. that is.Electromechanical System of Robots 169 Assume that the motion parameters of the body at the center of gravity are as follows: vc to ac a : : : : linear velocity vector. dv dt or a = ac + LO x {UJ x rd). the variation rate of the linear momentum of a body is equal to the sum of external forces exerted upon the body. the .e. = 0 (i. Then.65).66) Similarly. S i n c e Ibody (u x rd) * dm = u) x (Jbody rd • dm) and Jbody rd*dm Eq 4. angular velocity vector. the linear momentum at differential mass element dm. (4. is v • dm. acceleration vector.66 yields the expression for the acceleration vector at position r. we have v — vc + to x rd. 4. As a result. (467) dvc dt d dt By definition. the linear momentum of the rigid body will be p= Jbody v • dm = I Jbody (vc + to x rc) • dm or P= / Jbody Vc* dm + I Jbody (to x r<i) • dm. By definition.
4. As a result. we have / Jbody (rd x rc) • dm = Sl(rc) • Jbody rd* dm — 0 (i. by definition. Substituting Eq. and rd x (to x rd) = S(rd) • S*(rd) • w. will be H = / Jbody (rd x v) • dm.66 into the above equation yields H = / J body (rd x rc) • dm + J body (r^ x (u x r^)) • dm. 4. the angular momentum of the rigid body. and angular motion about an axis at its center of gravity. By using the property of the skewsymmetric matrix.65). (4. the crossproduct of any two vectors a and b can be expressed as a x b = S(a) • b or a x b= (b) x o — 5*(6) • a. is rj.e.68) Now. with respect to the center of gravity of the body. Eq.170 The Fundamentals of Robotics: Linking Perception to Action equation of motion governing linear motion will be dp dvc F = — = m» —dt dt or F = m»ac. x (v • dm). with respect to its center of gravity. we consider the general case in which the motion of the rigid body is composed of linear motion at its center of gravity. . Eq.69 can be simplified to: H= I J body S(rd)»St(rd)»uj*dm.69) If S() denotes the skewsymmetric matrix of a vector. If the linear momentum at differential mass element dm is vdm. 4. the angular momentum of differential mass element dm. Then. (4.
the variation rate of the angular momentum of a body is equal to the sum of external torques exerted upon the body. matrix / is the inertial tensor or mass moments of inertia (or simply inertial matrix).St() = St(). From Eq. 4. will be . the equation of motion.70. 4. When the rigid body changes its position or orientation. the angular moment H can also be written as H= f Jbody St(rd)»S(rd)*LO*dm. Finally. it is convenient to choose a frame assigned to the center of gravity of a rigid body. inertial matrix I is computed with respect to frame 0.70. By definition. or dH T dw dl M = I»a+^mu.. Computationally. dt (4.70) the final expression for the angular momentum of the rigid body becomes H = I»u. (4. 4.35. In Eq.Electromechanical System of Robots 171 By applying the following property of a skewsymmetric matrix: S().71) In fact.S(). Refer to Fig. In practice. it is clear that the inertial matrix depends on the choice of reference frame because vector rd is expressed with respect to the reference frame chosen. it is not desirable to keep calculating an inertial matrix based on Eq. 4. If we define 1= f J body St(rd)»S(rd)»dm. this matrix has to be recalculated. Let crd denote the displacement vector of .72. (4. and then to calculate inertial matrix I with respect to frame 0. Frame c is attached to the rigid body at the center of gravity. governing the angular motion.72) (NOTE: / is not a constant matrix). An alternative way is to compute the inertial matrix of a rigid body with respect to a fixed frame assigned to the rigid body itself.
Eq. 4.70 yields 1=1 •Jbody (4. —9. Assume that the density vector of gravitational force on the surface of the Earth is g. inertial matrix CI of the rigid body. 4. with respect to frame c.76) Eq. (4.803)*.76 also allows us to evaluate the derivative ^ in Eq. its differential potential energy dVm will be dVm = g* • r •dm= gl • (r c + rd) • dm.75) Since S{RC «c rd) = Rc • S(crd) • i?* (see Chapter 2). and is expressed in the common reference frame.76 is a computationally efficient solution to calculate the inertial matrix of a rigid body with respect to a reference frame because inertial matrix CI of the same body.dm. let us examine the relationship between the position of a rigid body and its potential energy.74) Sl {Rc »c rd) • S(Rc •crd)»dm. Substituting Eq. 4.74 into Eq. 0.172 The Fundamentals of Robotics: Linking Perception to Action rd. (4. 4.73) If rotation matrix Rc (or more precisely °RC) describes the orientation of frame c with respect to frame 0. .dm\»Rtc ) or / = Rc • {CI} . g = (0. By definition.75 can be simplified as I = Rc*{f I Jbody St(crd)»S(crd). Eq.72. g is a vector pointing towards the center of the Earth. 4.2 Potential Energy of a Rigid Body Now. then vector rd expressed in frame 0 is related to the same vector crd expressed in frame c as follows: rd = Rc » c rd. Rl (4. If we consider differential mass element dm. expressed in frame c. 4. with respect to frame c assigned to the body. is CI= f Jbody St{crd)»S(crd).2. is a constant (and symmetric) matrix. If the Z axis of the reference frame is pointing upward and perpendicular to the ground. 4.6.
(4. we have dKm = (vc + to x rdf • (vc + to x rd) • dm. that is. the kinetic energy of this differential mass element will be dKm = v1 »v» dm.Electromechanical System of Robots 173 As a result.6. the kinetic energy of a rigid body is very important in the understanding of robot dynamics. it stores kinetic energy. if we want to move a rigid body which is initially at rest. of a rigid body will be V = TO • gl • r c .. In other words.2. the total potential energy of the body is V= f Jbody dVm = gt» I Jbody (rc + rd) • dm. Consider differential mass element dm at position r on the rigid body. the potential energy Since Jbod rd • dm = 0 and Jbod rc • dm = rc»m. one can easily prove /&«*. K = I Jbody dKm = ~ / ^ Jbody {VC + UJ x rdy • (vc + u> x rd) • dm or K = o / 1 ivcvc + v *( w x rd) + {OJX rafvc + (w x rd)*(w x rd)} • dm. If v is the velocity vector of this differential mass element. In robotics.78) Jbody By applying the property Jbod rddm = 0. The integration over the entire body of the above equation yields the total kinetic energy of the body. Since v = vc + UJ x rd (i.e.K(w x rd) + (w x rdyVc} •dm = Jbody{vlS(uj)rd =0 + ^ ^ ( w j w j • dm where S(UJ) is the skewsymmetric matrix of u>. 4.3 Kinetic Energy of a Rigid Body (4.77) When a rigid body is in motion. Eq. we must apply energy to it. .66). 4.
dm. and the second term refers to kinetic energy from the angular motion. 4.34. 4.80) K = \m • v\ • vc + Jo/ • [Rc . The first term in the equation refers to kinetic energy from the linear motion. c / • Rlc\ • u.3 NewtonEuler Formula If we know the solutions to a single rigid body's dynamics. For the purpose of better illustrating kinematic and dynamic constraints. the kinetic energy of the rigid body. 4. Zi Zi (4.81) Eq. 4.1 Kinematic Parameters of a Link An open kinetodynamic chain is the superset of the corresponding simple open kinematicchain. we redraw Fig.79 yields K= m»vtc»vc+ or LUt»I»w (4.63.174 The Fundamentals of Robotics: Linking Perception to Action If we use the property of the skewsymmetric matrix. 4. Now. 4. Obviously. we must know the kinematic parameters .3.} . The basic idea behind the NewtonEuler formula is to treat an open kinetodynamic chain as a dynamic system of rigid bodies which satisfy: a) the kinematic constraint imposed upon the robot's mechanism. Therefore.6. Eq. 4. 4.36. the robot's arm has three links (n = 2).76 into Eq. (4. it is easy to derive the equations of motion for an open kinetodynamic chain.36. let us consider an open kinetodynamic chain with n + 1 links. x rd) = [S*(rd)w]t[Srt(rd)w] = ^ [ 5 ( r d ) ^ ( r d ) ] W .78 can be simplified as K =\ f 1 Hvc + w*[5(rd)5*(rd)]u. 4. or removed from. The result is illustrated in Fig. 4. we have (w x rd)*(o. 4. Eq. For the example shown in Fig.6. A simple way to produce the desired motion on a rigid body is to control the amount of energy added to. The first method is the NewtonEuler formula. From this equation. it is clear that the motion of a rigid body is directly related to its kinetic energy.70 and Eq.81 describes the kinetic energy stored inside a rigid body in motion. and b) the dynamic constraint imposed upon the kinetodynamic couplings.79) Jbody Substituting Eq.
jP* • *v!ml • 1 ^ Joint 2 . 4. By default. . Acting torque Cb Acting force ' \cting force ' ' /2 1 / •'^^jlti" / y I . In general. the origin of link i's frame. of an open kinetodynamic chain even if the purpose here is only to study the dynamics. #. As shown in Fig. we must know where the center of a link's gravity is. each link is assigned a frame according to the DH representation (see Chapter 3).36 Illustration for NewtonEuler formula. a*: the linear acceleration vector of link i's frame. The frame assigned to a link is located at its distal node. Rf. x~J .36.e. Usually. For the study of dynamics. frame 0) is chosen as the common reference frame. link i inside an open kinetodynamic chain should have the following kinematic parameters: • • • • • • Of. 4. . the base link's frame (i. the rotation matrix describing the orientation of link i's frame.Electromechanical System of Robots 175 ^ Acting torque . all the kinematic parameters of a link are expressed with respect to a common reference frame. Therefore. LOi\ the angular velocity vector of link i's frame which has Z\—\ as its axis of rotation. the linear velocity vector of link i's frame. Link 0 Fig. • on: the angular acceleration of link i. rci: the center of gravity of link i. the center of gravity of a rigid body plays a useful role. if i2 7. Vim. 1 !U U />.
Therefore.3. By default. the sum of forces F.176 The Fundamentals of Robotics: Linking Perception to Action 4. When joint i applies an acting force or torque to link i. the inertial matrix of link i.6. we must also know what the dynamic parameters related to each rigid body inside an open kinetodynamic chain are. If we know the dynamic parameters of link i. • g: the density vector of gravitational force on the surface of the Earth. will be Fi = fi + {fl+1)+mi. • Tii the acting torque exerted by joint i on link i. the acting and reacting forces/torques are very important dynamic parameters as well. As a result. we need to know the sum of forces/torques exerted upon a link. Now. • cIi. • fc the acting force exerted by joint i on link i. 4.g. all dynamic parameters are expressed with respect to the common reference frame (frame 0).3. it is logical to call the force or torque applied to a joint the acting force or torque.82) g is a vector expressed in the common reference frame (frame 0). • Ii. calculated with respect to the common reference frame (frame 0).6.2 Dynamic Parameters of a Link Similarly.3 Sum of Forces of a Link Before we derive equations of motion for each link. the mass of link i. (4. expressed in the common reference frame (frame 0). The externallyexerted force or torque (except gravitational force) is applied to a link through the supporting joint. joint i will also exert a reacting force or torque to link i — 1 according to Newton's third law. When . In general. We denote Fi the sum of all exerted forces on link i. • — Tj+i: the reacting torque exerted by joint i + l o n link i. calculated with respect to the center of the link's gravity. the inertial matrix of link i. the dynamic parameters of link i inside an open kinetodynamic chain will include • mf. the dynamic parameters of a rigid body include mass and inertial matrix. Due to the dynamic constraint imposed by kinetodynamic couplings. • —fi+i\ the reacting force exerted by joint i + 1 on link i. consider an open kinetodynamic chain with n + 1 links where link i is supported by link z — 1 through joint i.
Electromechanical System of Robots 177 studying a humanoid robot. Here. .r c M x fi+1). we consider the case of revolute joints and assume that all links can undergo angular motion. We further denote Mi the sum of all torques exerted on a link i. which can make rotation.3. If we denote • 7"ci. we obtain drCi dOi d ~JT — ~HT . Further differentiation of the above equation yields the expression for the computation of the linear acceleration vector at the center of gravity. 4. By differentiating the equation rci = Oi — rCiti. this force will also induce torque on the body with respect to a reference point.1 (r ci)i _i = Oti — rci). Here. the induced torques corresponding to acting force /» and reacting fi+i will be rciti^i x /* and rciyi x ( .6.i — Oi .rci). 4.ii: the vector connecting the center of gravity rCi of link i to origin Oii of frame i .3.83) By direct application of the result in Eq. When force is applied to a body.6. and • rciti\ the vector connecting the center of gravity rci of link i to origin Oi of frame i (rCi./ i + 1 ) respectively. If we know the kinematic and dynamic parameters of link i. 4.5 Equation of Linear Motion of a Link (4. it will not apply any torque to link i.4 Sum of Torques of a Link If joint i is a prismatic joint.68.^rci. we consider the center of gravity of a body as a reference point.i dt dt dt (4.84) Vd = = ViUJiX rciti. this vector may not be a constant vector because the reference frame will have relative motion with respect to the Earth when a humanoid robot is in motion. the equation of linear motion of link i is h + (fi+i) + ^i • 9 = rrn • aci where aCi is the acceleration vector at the center of gravity of link i. the sum of torques Mi will be Mi=Ti + (T i + 1 ) + r d i i _ i x fi + ( .
4. Eq. 4.6. 4.72 yields the equation of angular motion for link i.87) By applying the property ^ = S(u>) • R where R is a rotation matrix describing the orientation of a frame.• LOi = LUi X (Ii • UJi) +Ii» (S*(Wi) • Ui).6 = mi»{aiai x r cM . Finally.e.178 The Fundamentals of Robotics: Linking Perception to Action That is. and u> is the angular velocity acting on the frame (see Chapter 2 for detail). (4. that is. dvci dt or aci = a% . the direct application of the result in Eq.3.w .87 can be expressed as —^•ujt = S{uJi) • (Ii • u>i) + /j • (S^oJi) • iOi) or —. the term ^ • u>i in Eq.Qi x rci. Ti + (TJ+I) + rcitii x fi + (r ci)i x fi+1) = Ii»ai Since U = Rt • { C /J • i?* (i.86) (4. we have + j±»ui. (4. x (u>i x r c i > i ) . dt .i . x ( ^ x rci<i)}.85) dvi dt du>i dt f d \dt \ ) Equation of Angular Motion of a Link Similarly. the derivative of inertial matrix Ii becomes ^ or = S(ui) • Ri • {ch} • ^ + i2i • f / J • i?* • 5*(o.w . the expression of the equation of linear motion becomes fi + (/j+i) +mi»g 4.0 As a result.76).
8 Recursive Algorithm for Velocities and Accelerations Strictly speaking. 4. x r cM . x (w.3.88) A robot is used to perform actions and tasks through the execution of motions.87 yields Tj + (T i + 1 )+r c M _i x / j + ( .uii.l . (Vi = n .7 Recursive Algorithm for Forces/Torques (4.^ gQ . 4.90 does not explicitly relate the generalized forces (the forces or torques produced at the joints) to the generalized coordinates (the joint variables and their derivatives). .6.m* • g \n= ri+1 .89) Eq.3. = u>i x W = 0. .a . This is because the motion parameters (vi. we will make use of a computer to do the complex computations.w . .r c i ] i x / 1 + 1 ) = / I » a 1 + w I x (/i»Wj). Eq. the derivative of inertial matrix Ii can j be simplified as ^ • u ^ W i X (£•<*). 4. It is obvious that Eq.88 into Eq. It is a tedious process if we have to do the computation by hand.6.89 in the following recursive form [fi = fi+i + mi • {at . As a result. given the kinematic parameters. l ) . n . The desired motions of the links inside a robot's mechanism are normally planned based on the input task or action that a robot must undertake.86 and Eq.90 describes a backward recursive algorithm for the computation of the required forces and torques at the joints in order to produce the desired motions to the links inside the robot's mechanism. and the planned motions of a robot.r ciii _i x ft + rci>i x fi+1 +Ii» on +WjX (I. the required forces and torques can be computed by using the equations of linear and angular motion in a recursive manner. let us write Eq.i.89 is the final expression of the equation of angular motion.ai) are related to the frame assigned to link i and they are not the derivatives of the generalized coordinate qi of joint i. .a. x r ciii )) . 4. In order to make this statement clear. Substituting Eq. 4. 4. .Electromechanical System of Robots 179 Since St(cJi) • w. • Ui). Therefore. 4. 4. we need to know how these motion parameters are related to the generalized coordinates and their derivatives. (4. 4. In practice.
2. Eq. the axis (or direction) of motion at joint i is the Zii axis (i.92) l~1kii} (NOTE: £(&_!) =w i _ix£_ 1 ). Let us first derive the recursive solution for the computation of the angular velocity vectors {u.. qi = 0 if joint i is a prismatic joint... we have _ (0i \di if revolute joint.91) And.ra} of link frames.. Differentiating Eq. Refer to Fig.93) (4. i = .92 and Eq.n} and acceleration vectors {ati..e. if prismatic joint. . If joint i is a revolute joint..n} and acceleration vectors {m.180 The Fundamentals of Robotics: Linking Perception to Action Assume that 5. i = 1. we derive the recursive solutions for the computation of linear velocity vectors {vi.. the relationship between the angular velocity vector of frame i and the angular velocity of frame i — 1 will be u>i = cuii + Rii • {qi • or LJi=cuii+qi»kii. If Rii is the rotation matrix describing the orientation of frame i — 1 with respect to the common reference frame (frame 0).. Obviously... A joint can be either a revolute joint or a prismatic joint. 4. is the joint variable of joint i.. .93 are the forward recursive solutions for the computation of angular velocity vectors and angular acceleration vectors of link frames (i.92 with respect to time gives (*i = ttii +q\» fcji + qi • (uJii X kii). i = 1. Now..2. The relative angular velocity vector of frame i with respect to frame i — 1 will be Qi» i~1kii. its two joint variables will be: 9i and dt = 0 (0 means that this variable is not in use). According to DH representation. A?ii). 4. 4. we imagine that each joint has two joint variables: 6 and d.e i = 1. 4. If joint i is a prismatic joint..2. Here. (4.. For joint i. its two joint variables are: ^ = 0 and di.36 or Chapter 3. we set qt = Qi. 2. i = 1. (4. n if there are n+1 links in the open kinetodynamic chain).. .
Electromechanical System of Robots 181 1. 4. n} of link frames. we obtain at = ai. From a simple geometry.1+aixOilti+uiiX~{Oiiti}+qi»ki^i+qi»(uJiixkii). Let Oiiti denote the displacement vector from origin Oj_i of frame i — 1 to origin O* of frame i. if joint i is a revolute joint. + qt • kix. we know Oz = Oi1+Olhl. Then.94 becomes Vi = Vi_i + S{u)i) • Ri • 'Oji.97) (4. In this case. 4. By default.. we set qi = di for joint i. (4. 4.95 also proves the following equality ^ { O i .96) (NOTE: ^ki_1)=cji1 xfc^j).i} (494) Since link i is a rigid body.. Eq. at By differentiating Eq.i + Ri»qi» or Vi = Vix +UjX Oj_i. Eq.i .95) %i Because Uj = i>i_i + Jj{O»i. qi = 0._i. 2. we have (NOTE: it will be zero if joint i is a revolute joint).. origin Oi can be rewritten as Ox = Oi_i + Ri • 'OiLi. i } = ^ x O^!. Differentiating the above equation with respect to time yields Vi = v^ +^ • 'OiLi + Ri • JtVOti.i}. (4. displacement vector Oji^ can be expressed as Oiij = Ri • lOiitiTherefore. . If rotation matrix Ri describes the orientation of frame i with respect to the common reference frame (frame 0).j + 9i • fc.95.. all vectors are expressed with respect to the common reference frame..
According to Newton's Third Law. Assume that particle i is represented by the following physical quantities: • • • • • • m. n if there are n +1 links in the open kinetodynamic chain). linear velocity vector.. For a particle having mass m and moving at velocity v.97 are the forward recursive solutions for the computation of linear velocity vectors and acceleration vectors of link frames (i. we can apply the EulerLagrange Formula to derive a compact description of motion equations in a matrix form.i\ mass.6.6. its linear momentum is p = m»v.. then F + (—p) = 0. The sum of the acting and reacting forces is equal . every acting force will have a corresponding reacting force.1 D'Alembert Principle The EulerLagrange Formula is derived from the D'Alembert Principle. The basic idea behind the D'Alembert Principle is simple. is equal to zero.95 and Eq. this means that the sum of external forces acting upon a particle. Newton's second law). Now.e. it may be desirable to have a closed form solution of robot dynamics.e i = 1. 4. 4. rf.4 EulerLagrange Formula The equations of motion derived from the NewtonEuler formula is recursive in nature. F[: sum of internal forces acting upon the particle. we have F = p (i. Ff...182 The Fundamentals of Robotics: Linking Perception to Action Eq. In fact. If F denotes the sum of external forces acting on the particle.2. position vector. 4. sum of external forces acting upon the particle. plus its inertial force. Conceptually. Vim. let us consider a system of n particles. 4. It is computationally efficient and is fundamentally based on Newton's Third Law governing the relationship among acting and reacting forces in a dynamic system.4. pi: rate of change in its linear momentum. In some cases. The D'Alembert Principle governs the balance of forces acting on a system of particles in motion. If we define — p the inertial force of a particle.
we can derive the wellknown Lagrange Formula from the D'Alembert Principle. 4. 4. In other words. 4. (4. i=l (499) If Srt denotes the virtual displacement of particle i at a timeinstant. (NOTE: A work can be negative if energy is removed from a system). becomes 5WF = SWk. .7=i(Fi*Sn) and 5Wk = ^=1{Pi»Sri). (4. we have i=l Since Fi + F[ + (—pi) = 0 for particle i. It states that the sum of all forces inside a dynamic system of particles is equal to zero.101 illustrates the balance of virtual work done to a system in dynamic equilibrium. a dynamic system is always dynamically in balance.100) Eq. 5Wk — 0 for a system at rest). 4.4.99 yields n n £(*<•<&•<) = £ & • i=l i=l to). 4. + F[ + (pi)) = 0.2 Lagrange Formula If the particles in a system undergo linear motion. In other words. Eq. The principle of virtual work for statics is a special exception to the D'Alembert Principle (i. Therefore.Electromechanical System of Robots 183 to zero. the multiplication of to to both sides of Eq. 4.101) If we denote 6WF = T. the D'Alembert Principle also states that virtual work done by external forces to a system is equal to virtual work done by inertial forces.98 describes the D'Alembert Principle for the force balance governing a system of particles. then we have n X>.e.98 can be rewritten as n n $>< t=l = $>. Eq. As the sum of internal forces is equal to zero.6.100 (4.98) Eq.
184 The Fundamentals of Robotics: Linking Perception to Action Virtual Displacements Assume that a system of n particles is presented by n generalized coordinates {qi.. Eq.106) From Eq. Then. 4.. the virtual displacement Sri can also be expressed as * • < = E ( l  • **) • Kinetic Energy Given particle i. its kinetic energy is 1 mi»Vi (4105) t »Vi.103.i = 1. 4. the total kinetic energy stored by the system of n particles will be (4. 4. . ( 4 ..q2. the virtual displacement Sri will be ^ = E(J:*%) (4102) As position vector r^ is a function of the generalized coordinates.104) From Eq. that is: * = * = E(^*)P = P.qn) And.103. it is easy to see that the computation of the partial derivative ^ yields % dqi As a result.n}..103 ) (4. Then. position vector r^ of particle i is a function of the generalized coordinates as follows: n =ri(qi. we know that velocity vector Vi is a function of the generalized coordinates and their velocities. Hence.2. its corresponding velocity vector Vi will be a function of the generalized coordinates and their velocities...106 allows us to .
virtual work done to the system by gravitational force will be 6WV = 5V = J2(d^' 51*) • ( 4 .Electromechanical System of Robots 185 compute the following partial derivatives fg'EM^^g.107) [Ht = E T = i " * ^ . It is obvious that the following equation holds: dV — =0. . motions) of the generalized coordinates.e.. i=l (4.q2. (JQi VtG[l. And. Potential Energy For particle z.109 ) The potential energy is independent to the velocities (i.. its potential energy is Vje[l. the potential energy will also be a function of the generalized coordinates as follows: V = V(qi.n].108) (NOTE: g is a vector expressed in the common reference frame).111) . z=l (4. ] (4. Since position vector r^ is a function of the generalized coordinates.n].110) Virtual Work done by External Forces We denote TJ the generalized force associated to the generalized coordinate Qj.. (4.^ . Virtual work done by generalized forces to the system of n particles will be n 6WT = Yd{ji*5q^.5*«ri). I Ve ln j [. the total potential energy stored by the system of n particles will be n V = 53(mi.qn) And.
114 can be expressed as n I A \ n f)' l\ .113 can be rewritten as SWk =AB where (4. the external forces exerted upon the system of n particles include: a) gravitational forces and b) generalized forces. virtual work done to the system by external forces is 5WF = 5WV + 6WT or 5WF = y2(~+Tl). As a result.{vi* at 5ri\ ~vi* Sfi. Therefore. (4. 4. 4.i • vi). 4. the first term in Eq.5qi. • Vi. i=1 (4. at we have Vi • Sri = T. By applying Eq.112) V aQi J Virtual Work done by Inertial Forces The linear momentum of particle i is m. virtual work done by inertial forces will be n 5Wk = J2(mi*i)i»Sri).113) (NOTE: There is no negative sign.186 The Fundamentals of Robotics: Linking Perception to Action In general. Eq. From ~{vi • Sri} = Vi» Sri + vt • 5rt. This means that the system absorbs energy).105. Then. and its corresponding inertial force is (—m.114) B — YH=l(mi*vi*Sri).
114. 4.107. 4. or ^ d (dK\ (4. the second term in Eq.107. So.Electromechanical System of Robots 187 The sequence of summations is permutable. we examine the second term in Eq. Therefore. we obtain or (4. Now. the above equation can be rewritten as By applying the second equality in Eq. virtual displacements of the generalized coordinates will cause virtual displacement of Vi. As velocity vector Vj is a function of the generalized coordinates. we have .116) .115) (NOTE: f  = f g because v% = h). 4. Svi can be expressed as follows: As a result. 4.114 becomes n I n /a \ I or By applying the first equality in Eq.
Newton's Third Law on the relationship between acting and reacting forces). Thus. substitution of Eq. it is easy to obtain the expression for the wellknown Lagrange Formula.3 Equations of Motion Refer to Fig. This means that potential energy does not depend on any velocity variable.w .188 The Fundamentals of Robotics: Linking Perception to Action Finally.£ . This extension is called the EulerLagrange Formula.e. This expression is also valid for a system of n rigid bodies.£ — l ( f ) .120) Eq.36. In this case. we can directly apply the .119 can be compactly expressed as dt \dqij dqt (4. we have d_ /dV\ dt \dq~) ~ ' If we define L = K — V.120 is the famous Lagrange Formula.112 and Eq. An open kinetodynamic chain can be treated as a dynamic system of rigid bodies which satisfy: a) the kinematic constraint imposed by the robot's mechanism and b) the dynamic constraint imposed by the kinetodynamic couplings (i. It is a very useful formula for the derivation of a closedform solution to the motion equations in robotics. that is.<» «> Gravitational force is a conservative force.4. 4. virtual work done by inertial forces can be compactly expressed as a function of kinetic energy. 4. 4.101 yields £{(£HH^(f)a4 ^ It can also be equivalently written as . If a robot's mechanism is in the form of a simple open kinematicchain. 4.6. (4.117 into Eq. Eq. 4. In fact. the kinematic constraint will be a holonomic constraint. 4. which satisfy a holonomic constraint such as the kinematic constraint imposed upon a robot's mechanism.117) Expression of Lagrange Formula Now. 4.
q. and b) a 3 x n . •••. Now. given link i.e. One simple way to do this is to duplicate the link's frame i and translate it to location rci. we first assign a new frame ci at its center of gravity. Jacobian matrix J is a 6 x n matrix and is composed of two submatrices: a) a 3 x n matrix J o governing linear velocity.81 to determine the kinetic energy of a link. If we treat frame ci as an endeffector frame.36. we denote q = (qi. Given joint i. the Jacobian matrix Jci describes the motion kinematics of frame ci with respect to frame 0 (the common reference frame). if Pci denotes the velocity vector of frame ci with respect to frame 0. vci: linear velocity vector at rci. 4. Let us consider an open kinetodynamic chain with n + 1 links which are connected in a series by revolute joints. n = 2). the Jacobian matrix of frame ci with respect to the common reference frame. In addition. u)i. For the example shown in Fig. 4. and define the following parameters for it: m. and can be computed from the kinematic solutions discussed in Chapter 3. rCi: its center of gravity. aIci'. the robot's arm has three links (i.Electromechanical System of Robots 189 EulerLagrange Formula to derive the closedform solution to the motion equations. These motion equations relate torques (or forces) to the input motions of the robot's mechanism. ••. in frame ci. • RCi: rotation matrix describing the orientation of frame ci with respect to the common reference frame (frame 0). And.T2. • Jci~.inertial matrix calculated with respect to the center of gravity.qn)t the joint variable vector. and r = {T\. we need to assign frame ci at the center of gravity rCj of link i. angular velocity vector. we have • • • • • Pcl=(VA=Jci.q2.121) Normally. Frame at the Center of Gravity of a Link In order to apply Eq. and b) the torque Tj (generalized force). we represent it with two variables: a) the joint variable qi (generalized coordinate).Tn)t the joint torque vector. (4.: mass.
Substituting Eq. If we denote Ja=[ ). the kinetic energy stored in an open kinetodynamic chain will K = ±Kt i=i or K = ^* • I E[m» • (Jri)* • j s + ( ^ • ^ • r ^ } • ^ • ^ ]  • «• If we define J3(«) = E ^ • ( J )' # J S + ra* • *» • {ci/c0 • Ki • JSJ.(4123) Then. Eq. Ki = mi • vfci • vci + w\ • RCi • {clId} • Ki • Ui. i=l (4124) the expression for the total kinetic energy /i" can be compactly written as K=±qt*B(q)*q (4. that is.190 The Fundamentals of Robotics: Linking Perception to Action matrix Jw governing angular velocity.125) . 4.122) The direct application of Eq.122 into the above equation gives Ki = Jm.81 yields the expression for kinetic energy stored in link i. 4.121 can be rewritten as f vci = J°ci • q I Kinetic Energy of an Open KinetoDynamic Chain (4. 4. • (JS • QY « {J°a « 9) + \(% • <?)* • Ra • {"/«} • i& • ( ^ • <i) or ^ be = ^ ' • {mi • (J° )* • J° + (J^)* • i? ci • {clIn} • i?*e • Jci) • 9.
Vje[l.Electromechanical System of Robots 191 where B(q) is a n x n matrix. we have dK ^7 = yhu ( \ • Z. From Eq. It is expressed with respect to the common reference frame (frame 0). the potential energy stored in an open kinetodynamic chain will be n n (4.lin. which depends on the generalized coordinates q.119 (or Eq.n]. 4.128. 4. Vie[l. B(q) is commonly called inertial matrix of a kinetodynamic chain. • g* • rci. the potential energy stored in link i is Vi = . It is not a constant vector if the reference frame changes during the action or motion undertaken by a robot.v. 4. Accordingly. As a result. In robotics.120) to derive the motion equations.. n n ^=2EEM?)«ft»?)i=l j = l (4126) Potential Energy of an Open KinetoDynamic Chain Let g be the density vector (or acceleration vector) of the gravitational force field on the surface of the Earth. we can represent B(q) as follows: {biiiq). we are ready to apply Eq. 4.126. In fact.n]}.M«)* . we need to evaluate the expressions of the three terms on the left side of Eq.128) dt \dqij dqi dqi Now. 4.m . Eq.e.125 can also be expressed as . let us rewrite Eq. For easy reference.127) i=l i=l Motion Equations of an Open KinetoDynamic Chain If we know the expressions of potential and kinetic energies stored in an open kinetodynamic chain. By definition. 4. (4.119 as follows: !(£)_«C + £_n.
For convenience. .T ^ • 5fc • gj.126 and express it as: ^ = ^ E E M 9 ) •«*•*•• j=lk=l (4131) Since the partial derivative of bjkiq) • <f • qj with respect to variable qi 7c is . 4. we also change the index of summation in Eq.129 can be computed as follows: (4129) or Accordingly.129 becomes 5(f)i:M«). .128.132 ) This is the expression of the second term in Eq. the partial derivative of Eq.192 The Fundamentals of Robotics: Linking Perception to Action The computation of the firstorder derivative of the above equation yields Jt ( f f ) = T)f>iM) • < + M<?) • %•]• & The derivative bij(q) in Eq. Finally.fc. 4. Eq. 4. 4.LF'?*»4 ( 4 .. we have to determine the expression of the third term in Eq. 3—1 3 — 1 rC—1 (.3) 4 0 1 Now.. For convenience.„.128. 4. ^=22.131 with respect to variable qt yields OK l^^dbjk .ft + x:t^.127 and express it as n V= ^2mj»gtTcj.128. 4. we change the indices of summation in Eq. . let us examine the second term in Eq. . 4.*. 4.
Electromechanical System of Robots 193 The partial derivative of the above equation. 4. J c ° » ) 3 x n where J°j (i) denotes the ith column of matrix J°j.133 becomes ^ = X>WW (4134) Now. 4.(2) .135 can be rewritten as n n T..135) Vie [l.104. 4.e. substituting Eq.132 and Eq. Eq. with respect to variable qu yields f^gWtf.128 yields 3=1 3=1 U = l L Hk Hl J J j=l (4. 4.136) 9i(Q) = Eq.137) .n]. From Eq.. we have (33) Accordingly. 4. we have drcj _ drcj _ dvcj dqt dqt den ' We know vcj = J°cj • q (i. Eq. 4.130. 4. Eq.122)."=imj»gt»J^(i). If we define (4.q)mq + G(q)=T (4. 4.134 into Eq. 3=1 3=1 or in a compact matrix form as: B(q)»q + C(q. If we define J°cj = (J°CJ(1) J°.
vector G{q) accounts for the effect caused by gravitational force. Eq.7 Summary In this chapter. It is a function of the generalized coordinates and their velocities.n]. 4. it is difficult to obtain an exact expression for the dynamic model of a robot's electromechanical system because the computation of inertial matrices poses a challenge..q) = {cvfaq). VjG[l. in detail.T2.n]}. Matrix C(q..Tny < B(q) = {btJ(q)yi e [l. The first scheme consisted of onetoone coupling for each kinematic pair and its corresponding dynamic pair.G( g ) = {ffi(g). after a presentation of the important concepts on force.. q) accounts for centrifugal and Coriolis effects. such as force or torque amplifier. 4. In practice. Nevertheless. we discussed a new concept on dynamic pairs and dynamic chains. Then.194 The Fundamentals of Robotics: Linking Perception to Action where T = {n. work and energy.n]} C(q. Vi 6 [l.138) . And. ViG[l. each degree of freedom in a robot's mechanism has one actuator and a set of associated actuation elements. knowledge of robot dynamics is important and useful for dynamic simulation and control system design. The advantages of this scheme are the fast response of force or torque output and the efficiency in performing velocity and force/torque . as well as the associated actuation elements. This motivated the study of various schemes of generating the force field underlying the design of force and torque generator.137 describes the dynamic model of an open kinetodynamic chain with n + 1 links. Matrix B(q) is the inertial matrix.. two ways of forming a robot's electromechanical system through the coupling of kinematic and dynamic pairs.n]. We learned the working principles behind electric motors.n]} (4. We studied. we started with the study of the origin of a rigid body's motion.Vj G [l. Under this scheme.
52. (4) What is a dynamic pair ? Apply the concept of a dynamic pair to propose a design solution for an electric linear motor. (8) Search the website of US Patent Databases (it is a free service) to survey the various design solutions for CVT devices (CVT stands for Continuous Variable Transmission). From a kinematic point of view. 4. The first method is the NewtonEuler Formula. (5) From Eq. 4. a robot is also an electromechanical system. 4. But. advantages and drawbacks between the onetoone coupling and the onetomany coupling schemes of kinetodynamic pairs.8 Exercises (1) Explain the origin of motion. from a dynamic point of view. a robot is a mechanism or pure mechanical system. and allows us to develop a recursive solution for the computation of required forces and torques which will produce required motions from a robot's mechanism. (7) Prove the expression in Eq. Propose a new solution for onetomany coupling between kinematic pairs and one dynamic pair which incorporates CVT devices for motion distribution.13. prove that the mechanical power of a rotary motor is equal to the product of its torque and angular velocity. The notable advantages of this new scheme are the robot's compact size. Mathematically.Electromechanical System of Robots 195 control. (6) Comment on the difference. We have discussed the issue and solution of robot statics. (3) Explain the physical meaning of potential energy and kinetic energy. we have learned two methods for formulating equations of motion. light weight. This method is based on Newton's Second and Third Laws. A second method is the EulerLagrange Formula which allows us to derive a compact expression for the equations of motion. (2) Give a conceptual definition of what the mechanical system underlying a robot (or any machine) is. The notable drawbacks include the weight of the robot system and the cost of the robot's hardware. Subsequently. An alternative scheme is the onetomany coupling. and low cost. (9) If S(b) is a skewsymmetric matrix of threedimensional vector b. and b) how to apply forces and torques to produce the desired motions from a robot's mechanism. it is important to know: a) how to apply forces and torques in order to keep the robot at rest. prove .
In other words. Explain what will be the spatial distribution of the external force/torque exerted upon the endeffector's frame in order to keep the robot at rest.S(b)= (y2 + z2 xy xz \ xy x2 + z2 yz . Also. (14) Refer to Eq. Vi £ [l. (13) Write the equations of motion of the robot arm shown in Fig. i. prove that matrix N(q. explain the meaning of this distribution. (11) In the study of robot statics of an open kinetodynamic chain. the joint velocity vector is on a spherical surface of unit radius.196 The Fundamentals of Robotics: Linking Perception to Action the following equality: S(b)*St(b) = St(b). riij = —riji. assume that a robot is at rest and starts to exert a unit torque vector. 4.e. we have the relationship where r is the joint torque vector required to balance the externallyexerted force/torque £e acting upon the endeffector frame in order to keep a robot at rest.«]).q) is represented by {n^. and. also. Explain what the spatial distribution of the endeffector frame's velocity ve will be.e. If we define a new matrix N(q.137.e.q) = B(q)C(q. q) is a skewsymmetrical matrix (i. . r* • r = 1. assume that the robot moves with a unit joint velocity vector (i. the unit torque vector is located on a spherical surface of unit radius.n]. according to the EulerLagrange formula. Geometrically. we have the relationship ve = Pe = J • q where ve is the endeffector frame's velocity vector and q is a robot's joint velocity vector. q) as follows: N(q.Vj 6 [1.q). \ xz yz x2 +y2 J (10) In the study of differential kinematics of an open kinematicchain. if N(q. its meaning. At configuration q(t). 4. 4.36. qt • q = 1). Now. (12) Write the equations of motion of the robot arm shown in Fig.36 according to the NewtonEuler formula.
(2) Ryff. Dynamics of Active Articulated Mechanisms and Synthesis of Artificial Motion. D. Proceedings of 5th World Congress on Theory of Machines and Mechanisms. (4) Stadler.. AddisonWesley. (3) Spong. Freedman (1996). McGrawHill. Recursive Solution to the Equations of Motion of an nLink Manipulator. 34. (1967). P. Mechanism and Machine Theory. Karnas (1987).Electromechanical System of Robots 197 (15) If N is a n x n skewsymmetrical matrix and q is a n x 1 vector. Analytical Robotics and Mechatronics. J. Vidyasagar (1989). A. Canada. Montreal. prove the following result: ql • N • q = 0. ASME Journal of Applied Mechanics. F. 4. W. A. (5) Uicker. M. (6) Vukobratovic. M. (1978). University Physics. 13. D and R. H. PrenticeHall. W. Platnick and J. . (1979). W. and M. (1995). (7) Young. M. Electrical Machines and Transformers. Robot Dynamics and Control. J. John Willey & Sons.9 Bibliography (1) Armstrong. Dynamic Force Analysis of Spatial Linkages.
.
resistance to noise. land and onwater vehicles are indispensable for the proper functioning of our modern industry and society. A robot is a dynamic system. a desired output can be easily achieved even if the intrinsic dynamic model of a system under control is not accurately known. we will discuss how to apply the automaticfeedback control scheme to control a robot's motion execution even if the dynamic model of the robot is not accurately known. national security etc. economic growth. On the scale of enterprise. The magic of control arises from the scheme of an automaticfeedback control loop which responds to a cost function of errors (and statefeedback if working in a state space). In this chapter. control is exerted at various levels from process control. At the macroscopic level. the duty of a government is to control the social stability. 199 . and parameter variation). we know that it is quite difficult to obtain an exact dynamic model of a robot because it is almost impossible to theoretically compute the inertial matrices for all the links inside the robot. the primary job of which is to accomplish tasks or actions through the execution of motion. equipment control. production control. financial order.e. and all the way up to supplychain control and humanresource management. In the study of electromechanical aspect of the robot system. On a large scale. disturbance. In this way. traffic control of air. it is useless to attempt to run a robot for motion execution without invoking the automaticfeedback control scheme.Chapter 5 Control System of Robots 5.1 Introduction Control is a buzzword employed almost everywhere. Consequently. The ultimate goal of control is to minimize the cost function in an efficient and robust manner (i.
The output motion from a robot's mechanical system is at the endeffector. The relationship between input and output is known as the dynamic model of a robot. the system's output varies with time when being stimulated by the system's input or a disturbance). a system has input and produces output according to the characteristics of the system's dynamics. Therefore.2. or more precisely. and c) the sensing elements (a set of elements which measure actual output as feedback to enable the computation of error signals). (i. Normally. In the real world.1 A system is a combination of elements which act together and perform a certain task. 5. As for a robot's electromechanical system. and its input motions are at the joints of the underlying mechanism. The emphasis of this chapter is on the application of the automaticfeedback control scheme for efficient motion execution by a robot. In general. we know that input is the forces or torques applied to the joints of the robot's mechanism and output is the joints' motions. the dynamic model of a robot's electromechanical system. a system can be defined as follows: Definition 5.1 System Concept Using the robot as an example. 5.2 AutomaticFeedback Control System So far. Indepth coverage on the theoretical aspects of an automatic control system is beyond the scope of a robotics textbook as there are many textbooks dealing with this vast body of knowledge in control engineering. however. it is easy to formally introduce the concept of system. The internal parameters of a system may also undergo change as a function of time. the internal state of the system can be described by a set of variables. a system is often subject to noise or disturbance. we have treated the robot as the combination of a mechanical system and an electromechanical system. it is helpful to study all these related topics together. b) the control elements (a set of elements which transform error signals into corresponding power signal to be applied to the system under control). These .e.200 The Fundamentals of Robotics: Linking Perception to Action It is wellknown that the core of a feedback control system consists of three basic building blocks: a) the system under control (the plant). If a system exhibits certain dynamic behaviors.
1. When a system at rest is stimulated by input or noise. 5. the control . 5. Fig. the system is considered to be unstable (i.1 illustrates the concept of a system.2 The steadystate response of a system is the inputoutput relationship when the system is at rest. as shown in Fig. Under the openloop control scheme. whether a system is stable or not). the study of the design. the steadystate response directly indicates a system's absolute stability (i.1 Illustration of a system concept. how smooth a system is evolving to a new state). analysis and control of a system aims at achieving the desired steadystate and transient responses.e. Formally. And the transient response describes a system's relative stability (i.e.2 Closedloop Control Scheme If there is no feedback verification or checking. Disturbance ^ I T T I / T—I Output v Input ^(^X_^ Systen/ ™$y H (internal/ates)  Parameter / variation / Fig. If not. For a stable system.2. a system then operates in an openloop manner. its output starts to vary. In fact. In this mode. 5. 5.e. it is expected that the system will reach a new state and be at rest again.Control System of Robots 201 variables are called the state variables.3 The transient response of a system is the inputoutput relationship when the system is evolving from an initial state into a new state. To a great extent. the output when a system is at rest is called the steadystate response. Definition 5. the output oscillates or converges to infinity). a system directly responds to the system's input. the steadystate and transient responses of a system can be defined as follows: Definition 5. After a certain period of time. And the output when a system is making the transition from its initial state into a new state is called the transient response.
the steadystate response of a system can be analytically expressed as a function /(. it becomes clear that an openloop system does not have the ability to compensate for the variations due to disturbance d(t) and parameter variations Ac. This means that the control action by input u(t) requires knowledge about the system's description in terms of function /(. In other words. From Eq. (5.202 The Fundamentals of Robotics: Linking Perception to Action action is the system's input itself. d(t) the system's disturbance or noise. 5.c. the actual output will be equal to the desired output if the error . 5. And the control action to a system under control is determined by a function of the error signal. If we denote y(t) the system's output. the control objective is to minimize the error signal. the system's dynamic modelling must be known in advance.1. parameter vector and disturbance as follows: Ay(f) = % ' Ax(i) + %.) of input. An alternative scheme is the closedloop control. ' AW(<) + %'Ac+%* Ad{t) (5'2) Now.d(t)). Any uncertainty about a system's dynamic modelling will result in an unpredictable system output. On the other hand.). u(t) the system's input. state vector. These two notable drawbacks to an openloop system clearly demonstrate the inappropriateness of adopting an openloop control scheme to operate a complex engineering system. the control signal will continuously act on the system to alter its output so that the error signal will converge to zero (assuming that the system is stable).u(t). state vector.e. Fig. the variation of output depends on the term ^ • Au(t).2 illustrates the closedloop control scheme. When the error signal is defined to be the difference between desired output and the measurement of actual output. The basic philosophy behind the closedloop control scheme is to build a system which responds to an error signal instead of an input signal. As long as the error signal is not equal to zero. Under the closedloop control scheme. a vector of state variables). c the system's internal parameter vector and x(t) the system's internal state vector (i. parameter vector and noise as follows: y(t) = f(x(t).1) (NOTE: These notations are consistent with those used in control engineering textbooks). we can express the variation of output as a function of the variations of input. The working principle behind a closedloop control system is very simple.
an error signal is simply the difference between desired output and actual output. the control action of minimizing the error signal does not require any precise knowledge about the system's dynamics. Hence.2 Illustration of closedloop control scheme. the PID control law and its variations are the most popular control methods employed in industry. For example. In general. Normally. fast. one must be clear about what error signal the system is to minimize. the purpose of designing a control law is to find out a suitable control function which determines the appropriate control action (or statefeedback if working in state space) in order to meet the predefined control performance. stable. one must know how to measure actual output by using a set of sensing elements. signal vanishes. one must consider the following issues: • Specification of the Desired Output: A system is normally dedicated to achieving an outcome based on a predefined specification. 5.g. accurate etc). and parameter variation. disturbance.. Therefore.t™. • Sensory Feedback: Before a control action can be determined. • Design of Control Algorithms or Laws: The ultimate goal in studying the feedback control system is to design a control algorithm or law which will make the steadystate and transient responses of the system meet certain performance specifications (e. a closedloop control system is robust in the presence of model uncertainty.t ^~^ E r r o r l 1 In P ut /<+> / Systen/ (intemal/tes) / Output gHS%g^ Controller ^g)»> —[—» Parameter variation Measurement or Feedback ^ / / Fig. In order to construct a closedloop control system. For a stable closedloop control system. • Conversion of Control Signals to Power Signals: . one must know how to specify the desired output that a system has to produce. Thus.Control System of Robots 203 Disturbance ~ 1 Desired o.
dt + Td. in terms of accuracy (how small is the error between the desired output and the steadystate output?).1 Let us consider a motion controller implementing a PID control algorithm. ^ 1 L Ji Jo at j (5. For example. kp/Ti integral control gain and kp»Td derivative control gain. As a result. a robot's electromechanical system is a plant which is controlled for the purpose of achieving desired motion output. The input to a plant is normally related to a certain form of energy.2. \e(t) + 1 . And a system's dynamics means the relationship between its input and output during the period of its transient response. we look at three important performance indicators: a) the absolute stability of the system (is the system stable or not?). the dynamics underlying the PID control algorithm is described by c{t) = kp . it is desirable to analyze its performance with respect to its steadystate and transient responses. . Normally. b) the relative stability of the system in terms of response time (how fast does the system's output converge to the desired output?). 5. A system is said to be a continuoustime system if all variables inside the system are continuoustime variables.204 The Fundamentals of Robotics: Linking Perception to Action A system under control is usually known as a plant.3 System Dynamics By definition. it is necessary to convert a control signal to a corresponding power signal which is the actual input to a plant under control. all systems will exhibit certain types of dynamic behaviors caused by the interaction among the elements inside the systems. Thus. The input to the controller is error signal e(t) and the output from the controller is control signal c(t). / e(t) .3) where kp is proportional control gain. • Analysis of a Control System's Performance: Given a closedloop control system. And the control signal is usually calculated by a computer and is in the form of electrical signal without any associated electrical power. This conversion is normally done by a set of control elements. Example 5. the dynamics of a continuoustime system can be conveniently described by a differential equation. and c) the relative stability of the system. In the time domain.
the dynamics of a discretetime system is usually described by a difference equation.3 with respect to time gives dc(t) .with c(k)c{ki)^ d e r i v a t i v e M*) w i t h e(fc)e(fci) ^ a n d s e c o n d . Thus. We treat this digital computer as a discretetime system. 5. a statistical process control system is intrinsically a discretetime system. derivative ^^. m d2e(t)] .1) + a2 • e(k . o r d e r derivative 1^1 w i t h e(fc)2e(fcl)+e(fc2)j A g ft ^ ^ Eq_ ^ b e c o m e g c(k) = c(k .c(k .2 Assume that a PID control algorithm is implemented on a digital computer.1) + a0 • e(k) + a± • e(k . c(kT) .. Input to the system is error signal e(k) at discrete timeinstant k. A second reason for studying discretetime systems is because a control system may involve the sampling process in the sensoryfeedback channel. at timeinstant t = kT.at timeinstant t = kT is computed as follows: dc(kT) . \de{t) 1 .5) . By definition. any system having discretetime variables is called a discretetime system.. it is necessary to study discretetime systems. By definition. The difference equation relating output to input can be derived in the following way: Differentiating Eq.Control System of Robots 205 ••oooooooooooooooo As a result of the wide use of digital computers.1)T) . A. almost all closedloop control systems have both continuoustime variables and discretetime variables.1) where T is the time interval between two consecutive timeinstants.2) (5. As a result. Therefore.c((k . it is important to study discretetime systems. Now. and output from the system is control signal c(k) at discrete timeinstant k. we substitute derivative ^jp. A typical example is the statistical process control in which output is sampled and measured in regular discrete time intervals. In the time domain. Example 5. = lim dt T^o T or dc(k) _ c(k) .
5 is the difference equation describing the dynamics of a discretetime system which is also called a digital PID controller. (1 + ^ ) a2 = kp» ^ . A formal definition of Laplace Transform can be stated as follows: Definition 5. in terms of absolute and relative stabilities. Laplace Transform F{s) of function f(t) is /•OO F(s) = / f(t) • e~st • dt Jo where s is a complex variable.4 Transfer Functions It is not an easy task to solve a linear differential equation in the time domain. It becomes even more difficult if the equation in the time domain includes trigonometric and exponential functions. . a practical and powerful mathematical tool exists to deal with the issue of solving linear differential equations. can be graphically predicted in the splane. •••••••••••••••ooo 5. This tool is the wellknown Laplace Transform. • System performance. Fortunately. • The differentiation and integration operations in the time domain become algebraic operations in the complex s domain.2.4 For continuoustime function f(t) in the time domain such that f{t) = 0 if t < 0.206 The Fundamentals of Robotics: Linking Perception to Action with ' a0 = fcp«(l +  l + Ik) . • The trigonometric and exponential functions in the time domain become algebraic functions in the complex s domain. There are many notable advantages to using the Laplace Transform: • A linear differential equation in the time domain becomes an algebraic equation in the complex s domain (or splane). Eq. 5.ax = kp . • The inverse of the Laplace Transform contains both transient and steadystate responses to the system.
the ratio between the output's Laplace Transform and the input's Laplace Transform is called the transfer function. tk = k • T where T is the sampling period or interval). the corresponding discretetime function fk (t) will be oo fk(t) = ^2f(tk)m5(ttk) fc=0 (5.3 The differential equation describing a PID control algorithm (or law) is expressed in Eq.3 Illustration of transfer functions. if Y(s) is the Laplace Transform of output y(t) and U(s) is the Laplace Transform of input u(t). signals are in the form of a series of discrete values. 5. For a linear continuoustime system.Control System of Robots 207 ^ V(O • I G(s) I Y(s) • Fig. When U{s) = 1.3. As shown in Fig.7) where C(s) is the Laplace Transform of control signal c(i).6) If the input to a system is unitimpulse function S(t). This means that the transfer function of a linear continuoustime system is equal to the Laplace Transform of output in response to the unitimpulse as input.8) . the system's transfer function will be (5. Example 5. G(s) = Y(s). These discrete values are normally obtained by a sampling process over a continuoustime function.e.3. If f(t) is a continuoustime function and is sampled at a series of timeinstants tk (i. 5. the transfer function of a PID control algorithm is (5. and E(s) is the Laplace Transform of error signal e(t). 000000000000000000 For a discretetime system. 5. its Laplace Transform U{s) is 1. If we apply the Laplace Transform.
5. 5. the ZTransform of discretetime function /&(£) is oo if t = tk. Multiplying term z~k to both sides of Eq.5.208 The Fundamentals of Robotics: Linking Perception to Action where 5(.) is the unit impulse function and is expressed as follows: 6(ttk) = l [ liniATo ( s ^ J (NOTE:Ift<0.e.11) where k is the discretetime index.Transform.5 gives c(k)»z~k = c(kl)»z~k+a0»e(k)»z~k+ai»e(kl)»z~k+a2»e(k~2)»z~k. the ratio between the output's ZTransform and the input's ZTransform is called the Transfer Function. / fc (t)=0). for a discretetime system. 5. Example 5. The extension of the Laplace Transform to the discretetime domain is known as Z. By definition. Similarly.9 can also be expressed as oo (5. (5.9) where z is a complex variable and is related to the complex variable s in the following way: z = eTs. Since ZTransform is an extension of the Laplace Transform and the complex variable z is analytically related to the complex variable s (i. F{z) = YJf(tk)*zk fc=0 (5.12) .4 The difference equation describing a discrete PID control algorithm (or law) is expressed in Eq.10). 5.10.10) F(s) = £ / ( * ) • * . Eq. all properties of the Laplace Transform can be extended to the ZTransform through the application of Eq.* fc=0 (5. Eq. 5.
N(s) = 0).1 + a0 • E(z) + ai • E(z) • z~x + a2 • £"(z) • z~2.2. 5. and H(s) the transfer function of the feedback sensor. If there is no disturbance (i. K the control gain of the proportional control law.12 becomes C(z) = C(z) • ^ .2) • z~{k~2) • z"2 = £ W * z~2' Eq.Zo<k)*zk (5.1) • zVtU • z~l = C(z) • z1 < E(z)=T.Gp(s) Y(s) = R{s) l + K»Gp(8)mH(8)' (5. 5. the transfer function of the discrete PID control algorithm (or law) is g ^ = i = fl0+Oll"I+iaa""a (514) oooooooooooooooooo 5. the transfer function of Y(s) to R(s) will be K.13) Er=o e(k . D(s) = 0) and no sensor noise (i.5 System Performance Based on the transfer function.4 shows a simple closedloop control system with a proportional control law.e.1) • z~{k~1] # ^ = E(z) • 2" 1 . Fig. it is easy to analyze the performance of a closedloop control system.15) { ' . Tracking of the Desired Output Assume that the Laplace Transform of the desired output is R(s).Control System of Robots 209 By applying the following relations n°=o c(k . As a result. Er=o e(fc . We denote Gp(s) the transfer function of the plant.e.
When K = 1 (see the left subplot).E(s) U(s) A^ I 1 Y<S) I H(s) 1 < /<^>s N(S) Qjy* Fig. When the proportional control gain is increased to 100.4 Block diagram of a closedloop control system with a proportional control law. we only examine the output caused by disturbance D(s). the steadystate response of the closedloop control system is 0. Now. When K —• oo. The two corresponding responses are shown in Fig.210 The Fundamentals of Robotics: Linking Perception to Action D(s) R(s) ^.5 Refer to Fig. The plant's transfer function is Assume that the feedback sensor's transfer function is 1 (i.e. Eq. almost no error). we use the Simulink of MATLAB software to simulate this closedloop control system and choose two values for the proportional control gain: 1 and 100.e. the steadystate response is almost 1 (i. R(s) = i). the Laplace Transform of a feedback sensor is 1. H(s) = 1).5.15 becomes Y{s) 1 i?(s) ~ H(sY Normally. In this case. 5. 5. ••oooooooooooooooo Disturbance Cancellation Assume that the system under control is at rest and there is no sensor noise.e. and input to the closedloop control system is a unit step function (i. From . 5.5. Example 5. 5. Gp(s)) is intrinsically stable. It is clear that a large control gain K will result in a system which closely follows desired output R(s) if the plant itself (i.4.e.
. Eq.8 + 3 and the feedback sensor's transfer function is 1 (i. 5. 211 r— j M J 0 1. 5. 5 . Assume that there is no input (i. Fig.16) { ' . .6 Refer to Fig. 5. H(s) = 1).e. . Example 5. 5. The horizontal axis is the time axis with the unit in seconds.8. 10 .4. 15 I 20 J 0 .4.7.5 Example of tracking the desired output with different proportional control gains: The left subplot is the result when K = 1 and the right subplot is the result when K = 100. 5 . y(s) ~o D(s) ~ °This means that a large proportional control gain will reduce disturbance. 10 • 15 20 Fig. The plant's transfer function is Gpi8) = *+ 2. 1. 1 2r .Control System of Robots 0.6 0.16 becomes > (5.e.2 . the transfer function of Y(s) to D(s) is r(*) = Gp(s) D(s) l + K*Gp{s)*H(s)' When K — oo. R(s) — 0) and the system is subject to a periodic disturbance described by a sinusoidal function as follows: d(t) = sin(t) or sz + 1 .
we use the Simulink of MATLAB software to simulate this closedloop control system and choose two values for the proportional control gain: 1 and 100. 5. When K = 1 (see the left subplot). 5.e. Eq.01. When the proportional gain is increased to 100.31 i 1 1 1 0.oosM I / / [ l I I II II 0.Gp(s)*H(sY When K — oo.011—r 1 1 i \ \ \ o. Now. the amplitude of output is about 0. 5.Gp(s). let us examine what happens if the feedback sensor introduces noise (i.6 Example of disturbance cancellation with different proportional control gains: The left subplot is the result when K = 1 and the right subplot is the result when K = 100.H(s) D(s) l + K. the transfer function of Y(s) to N(s) is Y(s) = K.01' 0 Fig. 5.006 0.0081 II ' 5 I ' II > 10 ' 15 II20 04' • 0 ' 5 ' 10 ' 15 ' 20 O.212 The Fundamentals of Robotics: Linking Perception to Action Now.002 • 1/ 1/ V '0. From Fig.6.28. oooooooooooooooooo Effect of FeedbackSensor's Noise Assume that the system under control is at rest and there is no disturbance. The horizontal axis is the time axis with the unit in seconds.4.17) { ' This means that 100% of feedback sensor's noise adds to the system's .17 becomes > N(s) • (5. the output's amplitude is diminished to about 0. The two corresponding responses are shown in Fig. N{s) ^ 0).006^1 I Ij I I [ II 0. 0.
Without a guarantee of absolute stability. the big challenge is how to design a closedloop control system when the dynamic model of a plant is not exactly known. In other words. 5.6. If the dynamic model of a system under control is known. if e(t) — oo when t — oo.1 RootLocus Method Let us consider a linear continuoustime system. the roots of the numerator in Eq. the primary concern in the analysis of a closedloop control system is absolute stability.. all discussions about timely and smooth transient response are useless.(s+Pn) . the Laplace Transform of the output.6 Analysis of Absolute Stability Refer to Fig. If G(s) is the transfer function of a closedloop control system.18) Normally. Thus. and the roots of the denominator in Eq. By definition.Control System of Robots 213 output if a large proportional control gain is chosen.2. > > Intuitively.19 are called the zeros of a closedloop control system. However. In robotics. In control engineering. Ideally. The exact dynamic model of a robot's electromechanical system is rarely known in advance. it is easy to analyze the stability of the closedloop control system. 5. in response to the unit impulse as input. Assume that G(s) can be expressed as C( ^ U \ = •A(s) ^ ^ ' ( s + a i X s + fl2)(s + Qm) B{s) (a+Pi)(*+P2). iq x [ ' where m < n and KQ is a gain coefficient. there are many powerful mathematical tools for studying a system's stability. we normally face this situation. the performance of a closedloop control system depends on the quality of the feedback sensor. The system is stable but the transient response is oscillatory.5. a closedloop control system is definitely unstable.2. Since a closedloop control system is a system that determines control action based on the error signal.19 are known as the poles of . (5. . 5. 5. 5. the dynamics of error signal e(t) not only influences the system's transient response but also its absolute stability. the design of a control law should make a system not only stable but also have a timely and smooth transientresponse. will be Y(s) = G{s).. G(s) is in the form of the ratio of two polynomial functions in terms of the complex variable s.
ViG[l.e. e~Pit —• oo when t — oo. When pi is positive. + bn • e"^*..* 3 _ + . The partialfraction expansion of Eq.2 and S2 — —3. Therefore. R(s) = j) of this system.19 yields G(s) = bS+pi where 6* + . This method of analyzing a system's stability is known as the RootLocus method.21) If we compute the inverse Laplace Transform of Eq.n]. a closedloop control system will be stable if and only if all the poles of the closedloop control system's transfer function are located in the left halfplane of the complex s domain. (5. When the impulse response of a system goes to infinity.7 The transfer function of a closedloop control system is _ U[S)~ 6 . the system is stable because the two poles are in the left halfplane of the complex s domain. 5.22) It becomes clear that e~Pit — 0 when t — oo if pi is positive. > * » if pi is negative. the output of a closedloop control system responding to the unit impulse input will be y(t) = h . 5..A _ S +p2 S+pn (5. ( S + 1) (S + 2)(s + 3) The two poles of the transfer functions are: s\ = . Fig. it simply means that the system is not stable.8 Refer to the above example.20. Example 5.7 shows the unitstep response (i. ••oooooooooooooooo Example 5. Otherwise. (5. In other words.2 ) ( S + 3) .20) = {(«+?*)• ! ^ y } »=*. e~Plt + b2* e~P2t + .214 The Fundamentals of Robotics: Linking Perception to Action a closedloop control system.. + . Assume that the transfer function of the closedloop control system becomes G(s) G()  6'(* +D ~ ( * . the root s» = — pi is located at the left halfplane of the complex s domain.. 5.
5. 5. 1 J 3 2.Control System of Robots 215 0. .e. «p^ 4 . Fig. .8 Unit step response of an unstable system.g.5 2  I 1.8 shows the unit step response (i. 5. we frequently face the situation in which the transfer function of a closedloop control system is not exactly known for various reasons (e. Pole si = 2 is now located at the right halfplane of the complex s domain. .5  /  °0 2 4 6 8 10 12 14 16 18 20 Fig. with the unit in seconds. The horizontal axis is the time axis. The system becomes unstable. oooooooooooooooooo 5. . The horizontal axis is the time axis.2 Oi i 1 i 1 [ < 1 1 ' Fig. . . .5  I  0.2.2 Lyapunov's Method In practice.7 Unit step response of a stable system. . R(s) = i) of this system.6. with the unit in seconds.
The basic idea behind Lyapunov's method(s) can be explained as follows: Lyapunov Function When we have scalar function V(x) which is a function of vector x. when V(x) < 0. scalar function V(x) is converging to zero. For example. there are many scalar functions which are qualified to be Lyapunov Functions. difficulty in obtaining an exact dynamic model. V(x) exists for all t > 0). V(x) = \xt(t) • x(t) is a Lyapunov Function. this function is called a Lyapunov Function if and only if the following conditions hold: ( V(x) > 0 \V(a. In this case.e. we can apply the Lyapunov's method to test the asymptotical stability of a closedloop control system. a Lyapunov Function is also known as a control objective function because the objective of control action is to make this function converge to zero. In fact. (5. Lyaponuv's Stability Theorem Let x(t) be the state vector of a system. unmodelled dynamics etc). Control System Design Using Lyapunov's Stability Theorem Lyapunov's Stability Theorem does not help much with the stability analysis of an existing closedloop control system but is useful in the design of an asymptotically stable system. In fact. In practice.) = 0 if x =fi 0 and for alii > 0 if x = 0 and for alii > 0 l ' and V(x) has a continuoustime derivative (i. the system will reach a state of equilibrium. vector x will also be converging to zero if V(x) goes to zero. we can make use of the . In other words.24) the system is said to be asymptotical stable at the state when x(t) = 0 for all t>0. As V(x) is a Lyapunov Function. if x(t) is a continuoustime vector. This is an advantage.216 The Fundamentals of Robotics: Linking Perception to Action nonlinearity. As a result. If we are able to specify Lyapunov Function V(x) and if its firstorder derivative V(x) satisfies the following condition: V{x) < 0 V* > 0. when x(t) = 0 for all t > 0.
many practical control systems (including robots) rely on the popular PID control law (or its variations) to achieve satisfactory control performance. integral time constant Ti and derivative time constant Td so that the overall dynamic response of a closedloop control system is empirically judged to be satisfactory. The control law has a great influence over the dynamic response of a closedloop control system. ZieglerNichols methods are popular in control engineering. 5. The second . By definition.3. However. when e(t) = 0 for all t > 0.25 are sufficient conditions. but not necessary for a system to be stable.2. When designing a control algorithm or law. Because of the sensor feedback. e(t) will also converge to zero. output from a control law is called the control signal (or action) denoted by u(t). Note that the conditions expressed in Eq.e. then this method is applicable.7 Tuning of PID Control Algorithms In real life. ^' ' If such a control law exists. The tuning of a PID control law is to empirically find suitable values for proportional control gain kp. If the response looks like an Sshape curve. 5. A closedloop control system is a system that responds to the error signal denoted by e(t). the system is asymptotically stable and will remain at rest. Refer to Eq. V(e(t)) will shrink to zero with time. When V(e(t)) shrinks to zero. Refer to Fig. there are many systems. Therefore. it is important to know how to empirically tune a PID control law. the dynamics of which is not exactly known. the objective becomes to find a Lyapunov Function V(e(t)) and a control law in which control signal u(t) guarantees the following conditions: ( V(e(t)) > 0 \ V"(e(t)) < 0 for all u(t) and t > 0 for all u{t) and t > 0. system under control) in response to a unitstep function. Consequently. The first ZieglerNichols method requires recording the output of a plant (i. 5. This is an expected and desirable behavior for a closedloop control system.Control System of Robots 217 Lyapunov Stability Theorem to design a control law which will make the system under control asymptotically stable. The process of adjusting the parameters of a control law is called controller tuning. 5.2. error signal e{t) is indirectly a function of control signal u(t) as well. There are many empirical methods for tuning a PID control law. For example.
218 The Fundamentals of Robotics: Linking Perception to Action ZieglerNichols method requires the closedloop control system to exhibit a continuous oscillatory behavior when a large proportional control gain is chosen for a pure proportional control law. let us use a PID control law to improve the dynamic response of plant Gp in Eq. a proportional gain of 6 is added to Gp in order to bring the steadystate response to the final value 1. The first objective in tuning is to determine a proportional control gain which makes the closedloop control system stable. Now.9 Response of an openloop system to a unitstep function with a proportional gain of 6 added to Gv.3 • / 0.8  / 0. 5. 5. to a unitstep function. 5. An alternative method is to manually tune a PID control law in three simple steps. 5. described by Eq.2  / °0 1 2 3 4 5 6 7 8 9 10 Fig. For the sake of comparison with the closedloop control system. Fig. In practice. 5. we choose a plant which the transfer function is G '(. Fig.9 shows the response of the system.+») (5'26) 0.5 / 0. + a)'(.10 shows a block diagram of the closedloop control system with a PID control law.26.6 • / 0. these two methods are not convenient for tuning the PID control laws inside a robot's control system. Tuning the parameters of the PID control law can be accomplished in the following steps: Tuning of Proportional Gain kp The proportional control gain has an overall influence on the performance of a closedloop control system. . reasonably smooth and accurate. For the purpose of illustration.4  / 0.26.
As a result.e.2  J/KP20—' 06 . . We can see that the proportional control gain at kp = 30 permits the unitstep response to slightly overshoot value 1. This timeinstant is known as the rising time of a closedloop control system. 1. Tj — oo and Td = 0. Thus. Fig. we set kp = 0. We gradually increase the proportional control gain until the unitstep response of the closedloop control system reaches value 1 at a certain timeinstant.11 shows the results of tuning kp for the example in Fig.11 Effect of proportional gain.10 Block diagram of the closedloop control system with a PID control law. At this step.Control System of Robots 219 R(s) /r~\ E(s) I i p(^)—• kp(U^ + TdS) I • I GP(S) I Y(s) p^> H(s) = l 4 Fig. but the steadystate error is quite large.2 j / 0 1 2 3 4 5 6 7 8 9 10 Fig. 1). 5. 5. Tuning of Integral TimeConstant Ti When the proportional gain is set as kp = 30. The horizontal axis is the time axis with the unit in seconds. it will make the steadystate response converge to the desired output (i.10. 5. 5. It is natural to turn "on" the control action from the integral part of the PID control law. A control action proportional to the integral of the error signal in a PID control law allows to reduce accumulated errors over time. the closedloop control system is stable.II / Kp=10 0.
5. the transient response is oscillatory). . Comparing Fig. the system lacks a good transient response (i. In other words. 5. Thus. Fig. 5.220 The Fundamentals of Robotics: Linking Perception to Action the second step of tuning is to adjust the integral timeconstant Tt while turning "off" the derivative timeconstant (i. The horizontal axis is the time axis with the unit in seconds. However. the last step in tuning is to adjust the derivative timeconstant. 06  0.10. When the integral timeconstant is set as T* = 1.21 0 2 4 6 8 10 12 14 16 18 20 Fig. the accuracy of the closedloop control system is achieved with a large integral timeconstant. 5.0. In order to compensate for the loss of stiffness.13. 5. 5. Fig. the unitstep response quickly converges to its steadystate value while the overshoot is reasonable and quite small.12. When the derivative timeconstant is set as Td = 0.1. The timeinstant when the actual output converges to the desired output is known as the settling time.e Td = 0). it is natural to turn "on" the derivative part of the PID control law.e. the stiffness of the closedloop control system will be reduced with a large integral timeconstant.12 shows the results of tuning T» for the example in Fig.12 Effect of the integral timeconstant.9 and Fig. we can see that the settling time of the closedloop control system is less than two seconds while for the openloop system it is more than three seconds. the unitstep response will quickly converge to its steadystate value. 5.10.13 shows the results of tuning Ta for the example in Fig. 5. Tuning of Derivative TimeConstant T^ As we can see in Fig.
Control System of Robots 221 "I . the formation of a closedloop control system must address the following issues: • How do we appropriately define the error signal? • How do we efficiently determine the control signal based on the error signal? • How do we act. On the other hand. However. decisionmaking.os / TdO. 5. . And.2 h ' rij=o. in order to achieve a desired performance. The horizontal axis is the time axis with the unit in seconds. all of which a closedloop control system should have. ' ' ' ' ' ' ' ' 0. the purpose of the control algorithm(s) inside a robot's control system is to determine the control signal which will make the error signal converge to zero in order to produce the desired output.2W °0 2 4 6 8 10 12 14 16 18 20 Fig. and actiontaking devices. we understand that a closedloop control system is superior to an openloop system. in a timely and energetic manner. the control of a robot's motion execution must rely on a closedloop control scheme. in response to the control signal? These issues are closely related to sensing.13 Effect of the derivative timeconstant. As a result. Since robot control implies a closedloop control of constrained or unconstrained motions required for the performance of tasks or actions. the error signal of a robot's control system should be the difference between the desired motions and the actual motions. 5.3 Control Elements Our ultimate goal is for a robot to perform tasks by executing motions. These devices are also known as the control elements. A robot is a system in which the input is the tasks and the output is the executed motions.
v.14 A hierarchical view of a robot system.222 The Fundamentals of Robotics: Linking Perception to Action Nowadays. From a control point of view. 5. electromechanical. as shown in Fig. 5. Robot System s—\ \ / D sr d \ e ie Vmotions\) \ I • Control System Electrical power f^ _ I I Electromechanical System ^ Mechanical power Mechanical System w • . Motions ^ ^ T.. In Chapter 4. at this point. and control systems. we can say that a robot is a combination of mechanical. almost all control algorithms are executed inside digital computers.14. the device which alters motion direction is called the power . we studied the working principles of electric motors. For a motion control system. which includes the closedloop control system for motion execution. There must be other control elements handling the alteration of motion direction and regulating the input of electrical energy to the electric actuators inside a robot. a robot's control system is not simply a digital motioncontroller and its electromechanical system. also known as the digital motion controllers. The rotor and stator inside an electric motor constitute a dynamic pair in which the interaction is governed by electromagnetic force. Therefore. lasks I ^ P Sensing 'k 1 I f Actual \ V motions1/ Fig. there are two concerns regarding the appropriate operation of an electric motor: • How do we alter the direction of motion of an electric motor? • How do we regulate the input of electrical energy to an electric motor? Thus.
1 1 I . The wave generator is responsible for producing controllogic signals or pulse waveforms (pulsewidth modulated waveforms or PWM waveforms for short).Control System of Robots 223 switch. 5.15 illustrates one set of control elements for the motion control loop governing one degree of freedom (one actuator) inside a robot. Switch Circuit A \ ^ Electric ^ Inertial '  Motor  f Load ' / ' ' ! I „ . However. These units are the wave generator. a robot's control elements will necessarily include at least one digital motion controller.„ Control Elements Fig. 5. 5. We will study this in further detail in the later part of this section. Accordingly.1 Power Switches Here. and a power drive.15 ^ Power Generator J ] ' ' i / / Mechanical power Control elements necessary in a motioncontrol loop. Fig.15. here we simply highlight three basic units necessary for the construction of a power switch and power drive. interaction between the rotor and stator is caused by electromagnetic force. A product which incorporates both power switch and power drive is commonly referred to as the power amplifier. This is usually done in two ways: . we can alter the polarity or direction of the electromagnetic field. power generator and switch circuit. According to the working principles behind the electric motor. the power generator can be as simple as a power source or supplier.  i Command 1 \ i A Generator I 1 * i / 1 < 1 ' J i ! . Feedback *• Motion » Controller I  5 D/A I 1—. ( Electrical power Control signal Wave ' / .3. And the device which regulates the input of electrical power to an electric actuator is called the power drive. 5. as shown in Fig. In order to change the direction of motion. a wave generator may include two subunits: a) a logic wave generator and b) a PWM wave generator. we consider the case of rotary electric motors. We will study sensing elements in a separate section. On the other hand. Depending on the type of electric motor under control. a power switch.
Thus.6V). This can be done with an appropriate power switch. A unit which generates a set of controllogic signals (i. A typical active device is a transistor because it can amplify the power of an input electrical signal. resistors. (4) When no input signal is applied to the base. The voltage drop is about 0.16. 4. operational amplifiers etc). a transistor has three terminals. The input to a switch circuit is a set of controllogic signals which determine the sequence of commutation of the logic states for switching ("on" or "off"). The coefficient /? is between 50 to 250. CC and DD'.1 Power Switches for Stepper Motors Refer to Fig. 5. it is similar to a diode. VB VE = 0. capacitors. As a result. there are two types of devices: a) passive devices (e. The coils wound around the teeth of the stator are grouped into four phases: AA'. rectangular electrical pulses) is called a logicwave generator. then Ic = P • IB when an input signal is applied to the base. (2) From the collector to the base. also known as an inverter.3.g. The key characteristics of a npn transistor include: (1) From the base to the emitter. transistors. As shown in Fig. In order to make a stepwise motion.g. a power switch will consist of two units: a) a logicwave generator and b) a switch circuit.1. b) the collector.e. . and c) the emitter. 5.6 (volts) or 0.e.13. BB'. transformers etc) and b) active devices (e. A simple way to change the direction (or order) of currents is to use a switch circuit. it is similar to an inverted diode. and IQ is the current which flows to the collector (from the collector to the emitter). Switch Transistors In electronics. • Change the order of currents applied to the coils (alter the order in which we energize the coils). These controllogic signals are necessary for a switch circuit to function properly.224 The Fundamentals of Robotics: Linking Perception to Action • Change the direction of currents which flow into the coils inside an electric motor. these phases are sequentially energized.6V (i. no current will flow from the collector to the base. (NOTE: The product of a signal's current and voltage is its electrical power). IB — 0 and Ic = 0. namely: a) the base. (3) If IB is the current which flows to the base (from the base to the emitter).
Thus. Qo +va I AA' < J 1 BB'<. we electronically connect the collector to the emitter because a large current can flow from the collector to the emitter (very low impedance between the collector and the emitter). Ic = 0) when there is no input signal at the base. 5.17 Conceptual illustration of a switch circuit for a fourphase stepper motor. When switch QA is "on" and . we electronically disconnect the collector from the emitter (i. A conceptual solution is shown in Fig. When an input signal is applied to the base. 5. 4.Control System of Robots 225 *c I Collector Base l^J Base *c I Collector KJ y £ I Emitter y E I Emitter (a) npn transistor Fig. Switch Circuit For the stepper motor. DD'^J QA V) QB V) QC \) QD I) Fig. the coils are divided into four phases. ij 1 1 1 CC'^> . as illustrated in Fig.16 (b) pnp transistor Symbolic representations of transistors. 5.e. a transistor is a perfect electronic switch. Conversely.12. As a result of the third and fourth characteristics.17. Signal Qo serves as an enabling signal. it is necessary for a switch circuit to handle the order of energizing these four phases.
If we use "1" to represent the logic "high" and "0" the logic "low". If we combine all the bits of the controllogic signals together. QB. logic state 0100 means that phase BB' is energized. and low voltage (OV) corresponds to the logic "low" or "0". it is necessary to supply a set of controllogic signals to QA. and logic state 0001 means that phase DD' is energized. liooo !oioo !ooio oooi iiooo \ AA' BB' i \vm 'oooi looio 'pioo iooo I I BB' i i I . 5.18. high voltage (+5V) corresponds to the logic "high" or " 1 " . Usually. QC and QD. "on" or "off".. For the example shown in Fig.12. 4. phase A A' is energized.226 The Fundamentals of Robotics: Linking Perception to Action the other switches are "off" (assuming that QQ is always "on"). one bit or one digital signalline) to turn the current from a power supply. In order to control the direction of motion. i 1 i I i . logic state 1000 means that phase AA' is energized. 5. As shown in Fig. i .e.18 Pulse waveforms of the controllogic signals from a wave generator for a fourphase stepper motor: a) spinning in a clockwise direction and b) spinning in a counterclockwise direction. . 1 i c' c D> D : : I I : : i i i i i i cc DD' : : I I : : i i i i i i i i i ' (a) ' i ' ' ' ' ' (b) ' ' ' Fig. logic state 0010 means that phase CC is energized. 5. In a digital system. Logic Wave Generator Refer to Fig. Each phase needs one controllogic signal (i. we obtain a variable which is called the state of the logic wave generator. the coils are divided into four phases. the logic signal is in the form of rectangular pulse waveform. the following sequential order of logic state commutation 1000 > 0100 > 0010 > 0001 > 1000 > . The logic wave generator is responsible for generating the controllogic signals that control the order in which the coil phases are energized.15. . .
^ J Rotor's coils ] I Fig. 5. which is commonly used to alter the polarity of the direct voltage applied to the coils of a brushtype DC motor. if only switches B+ and B_ are "on". 4. 5. .19.2 Power Switches for Brushtype DC Motors Refer to Fig. Switch Circuit The switch circuit.e.1. emf current) to die off quickly when the switch is turned "off". Since a stepper motor makes stepwise rotation. the motor will change its direction of motion.12.14.19 Conceptual illustration of a switch circuit for a brushtype DC motor.3. Refer to Fig. Because of the mechanical commutator inside a brushtype DC motor. is the Hbridge circuit. Therefore. Alternatively. 5. the motor will rotate in one direction.Control System of Robots 227 will produce a clockwise rotation of the stepper motor shown in Fig.. 4. The purpose of the diode attached to each transistor is to allow the back electromotive force (i. It is easy to see that when only switches A+ and A. will produce a counterclockwise rotation of the stepper motor.. And the following sequential order of logic state commutation 1000 > 0001 > 0010 • 0100 * 1000 > . the direction of rotation depends only on the polarity of the direct voltage applied to the coils wound around the rotor of the motor. As a result. the power switch for a brushtype DC motor is very simple. the speed control in a stepper motor is simply achieved by regulating the frequency of commutating the logic states of its wave generator.are "on". its speed obviously depends on the frequency (or speed) of commutating the logic states.
5.20. I i I i I i B+\ I I I I B. The direction of rotation can only be altered through the electronic commutation.3. 4. The pulse waveforms of the controllogic signals are shown in Fig.e.3 Power Switches for Brushless DC Motors Refer to Fig.15. A brushless DC motor's coils are wound around the toothed structure of the stator. bining all the bits of the controllogic signals together (i. only two phases are energized. if we form a logic state variable by comA+AB+B~). 5.15. Similarly. 5. I i i i I i i i i i i Fig.228 The Fundamentals of Robotics: Linking Perception to Action LogicWave Generator Refer to Fig.20 Pulse waveforms of the controllogic signals from a wave generator for a brushtype DC motor. we know that the electronic commutation of the . W and WW. i A. i i i .'noo i loon i j i A+I i I i .1. If we examine the power switch for a brushtype DC motor. at any timeinstant. 5. the logic state of 1100 will make the brushtype DC motor rotate in one direction. and the commutation of this logic state to 0011 will alter the direction of motion. Power Switch According to the working principle of a brushless DC motor. The purpose of the logicwave generator (a subunit in the wave generator) is to produce the controllogic signals applied to the four electronic switches. These coils are divided into three phases: UU'. both order and direction of the currents which flow independently into these three phases.
if switches V+ and U. the number of switches is reduced to 6.21.. only one switch on the upper row and one switch on the lower row are turned "on". V.Control System of Robots 229 current's direction inside a single coil requires four switches.^ v ~V Fig. Y i^ . any pair of two energized coils will form a single circuit.are turned "on". if we connect one end of each phase's coil to a common point. f/_. 5.21. Alternatively. (NOTE: This switch circuit needs both positive and negative voltage supplies). LogicWave Generator Similarly. as shown in Fig. 5. it is necessary to have 12 switches. phases UU' and VV' are also energized but /„ < 0 and Iv > 0. The following sequential order of the logicstate commutation 100001 »• 100010 > 001010 > 001100 ^ 010100 ^ 010001 > 100001 »• .22.21 Conceptual illustration of a switch circuit for a brushless DC motor. JZ L_ u . In this way..and W. At any timeinstant. We can form a logic state variable by combining all the bits of the controllogic signals applied to switches U+. phases UU' and W are energized with Iu > 0 and Iv < 0. W+. will produce a counterclockwise rotation. However. Refer to Fig. .15 and Fig. 5.. The corresponding pulse waveforms of the 6 controllogic signals are shown in Fig. 5. the role of the logicwave generator here is also to produce a set of controllogic signals. if switches U+ and V_ are turned "on". If we independently control the commutation of the current for each of the three phases in a brushless DC motor. For example. H F 3.21. Fi "» >/ ^4 h w. Alternatively. 4. 5. V+. Refer to Fig. the following sequential order of the logicstate commu .
These signals must be directly. or indirectly.. As we mentioned before. almost all control laws are implemented on digital computers.230 The Fundamentals of Robotics: Linking Perception to Action i i i i i i i i ! 100001 1100010 '. 5.' i ! i i I i i Fig. i ! i i . 5.. i 1 I i 1 I i 1 I w. U. amplified into the corresponding power signals in order to drive the motors inside a robot or any machine. 5. And the corresponding pulse waveforms of the 6 controllogic signals are shown in Fig. the frequency (or speed) of the logic state commutation must be synchronized with the speed of the rotor. i i V_ i I I . • i ! i .22 Pulse waveforms from the wave generator for a brushless DC motor spinning in a counterclockwise direction. 1 . i  i I i w+.3. tation 100010 > 100001 > 010001 > 010100 > 001100 > 001010 > 100010 > . a device which performs this direct . It is important to note that for the proper operation of a brushless DC motor.23. However. This explains why the power amplifier for a brushless DC motor requires velocity feedback from the rotor. will produce a clockwise rotation. And the control signals calculated by digital computers do not carry any energy after the digitaltoanalog (D/A) signal conversion.2 Power Drives The primary concern of a control system is to design a control law for satisfactory control performance. 001010  001100  010100  010001  100001  i i i i i i i i U+ I I I ' 1 ' ' ' I I I I I I I v+\ i ! i . ' i i i i ] 1 i 1 I i I I I I i 1 I I I I ! 1 i i i .
i [ i  : i i I i i .27) Clearly. u.23 Pulse waveforms from the wave generator for a brushless DC motor spinning in a clockwise direction.010001 .  . i v+ .001100 . which is capable of supplying a sufficient amount of current. [ . as well. inductor etc).t/] will be We = / (Vam Ia) • dt.'100001 . Due to the low efficiency caused by heat dissipation. there are two basic working principles for amplifying a control signal into a corresponding power signal: • Direct Power Amplification: This method directly amplifies a control signal into a corresponding voltage signal. . the electrical energy applied to the motor within time interval [t s . capacitor. i I i w+ : . 5. i i 1 i  i i : i ( : i i i V. (5. or indirect signal amplification is called a power drive. resistor. ' I I I I I I I I 1 1 I I I I u+ ! . [ i .010100 .001010 ! 100010 .g. i i  i 1 J i 1  i i  i i w_ i i I i i i I i I i I i I i I i Fig.'100010 . we know that electrical power is equal to the product of voltage and current exerted on an electrical load (e. If we denote Va the voltage and Ia the current exerted on the coils inside an electric motor. i I  i I 1 . . we can manipulate either time interval [ts. i i I  i I .Control System of Robots i i i i i i i i 231 . this method is suitable for low power output (low power motors). £/] or voltage Va (or current Ia) to alter the amount of electrical energy applied to an electric motor. In physics. Accordingly.
• The output voltage is proportional to the difference between the two input voltages at the input terminals. Control Signal I L W a v e I I Generator Switch Circuit jk Mj : i I I Inertlal Load I j i i • Linear Power Drive Voltage ' Fig.24 shows a schematic illustration of a linear power drive together with power switch.1 Linear Power Drives Linear power drive directly amplifies a control signal (input). which is linearly proportional to the control signal. The key characteristics of an opamp include • An opamp's input impedance is very high while its output impedance is very low. . This method is easy to operate (it is easy to generate a regular pulse waveform) and efficient (there's less heat dissipation). and produces a voltage (output).2. Fig. the voltage applied to the coils directly comes from a power supply.e. the voltage is applied to a motor through a switch circuit. As shown in Fig. As for DC motors. 5. Therefore. 5. As a result. 5.3. it is commonly used to produce high power output (for high power motors). A linear power drive is basically built on top of an active device called the operational amplifier or opamp for short. Subsequently. As a result. 5. no current flows into the two input terminals of an opamp (i. the widelyemployed method of amplifying control signals is the pulsewidth modulation or pulsewidth modulated (PWM) method. an opamp is a device with two input terminals and one output terminal. In this way. an opamp has to be operated with a negative feedback.25.232 The Fundamentals of Robotics: Linking Perception to Action • Indirect Power Amplification: For an electric stepper motor. the"+" terminal and the "" terminal) .24 Schematic illustration of a linear power drive together with a power switch. The gain of an opamp is very high (in the order of 105 to 106). Its role is to regulate the velocity of an electric motor. the input voltages at the two input terminals can be treated as equal (the difference is practically zero).
the output voltage is equal to the input voltage.Control System of Robots 233 Control Signal ^ \ + .tf] will be We = J*' (l +  ^ . By applying the equalities: V\ = Vc and Va = V2. In other words. J. 5. 5.25 Conceptual illustration of a noninverting linear power amplifier .25 shows a conceptual solution for the design of a noninverting linear power amplifier. This is useful to isolate electrical signals. an opamp can serve as a perfect voltage follower. Since the current that flows through the resistor i?i is the voltage V2 will be V2 = IRl» or V2 = ^*(R1 + R2). With unity negative feedback. Fig.. the electrical energy released to the motor within time interval [ts. ^2 Fig.29) . (R1+R2) (NOTE: The current at resistor R2 is equal to the current at resistor i?i). the relationship between input and output voltages will be Va = i1 + § ) # V° (8 52) If Ia is the current from voltage Va applied to the coils of a motor. vc ^ ^ vi r>x v h. Vc • Ia • dt (5.
voltage V2 will be V2 = R2 • IRl or V2 = Rv£. The current that flows into resistor R\ is Since the current at R2 is equal to the current at Ri. 5. the relationship between the input and output voltages becomes Va = ^*Vc. 5.25. This is why the circuit. computed from a control algorithm or law.26 shows a conceptual solution for the design of an inverting linear power drive. From Eq. we can easily design an inverting linear power drive. 5. If an electric motor requires negative voltage to be applied to its coils. Fig._L_>\ V I\ Fig. we can see that the signs of the input and output voltages of the power drive are the same. Hi As Vi = Vc and Va = V2.26 Conceptual illustration of an inverting linear poweramplifier. til (5.30) Similarly. 5. electrical energy applied to the electric motor within time . is called the noninverting linear power drive. R2 Control Signal ^v _ p\ vc + \>n£_. directly regulates the electrical energy applied to an electric motor.28. shown in Fig.234 The Fundamentals of Robotics: Linking Perception to Action It becomes clear that control signal Vc.
Control System of Robots
235
interval [£«,£/] will be We= ff (^•VcmIaS\mdt. (5.31)
(NOTE: We can ignore the negative voltage sign). 5.3.2.2 PWM Power Drives
Refer to Eq. 5.27. Electrical energy applied to an electric motor depends on time interval [ts,tf], voltage Va, and current Ia. We know that all electronic components suffer from heat deterioration, which in turn, affects performance. As a result, the longer time interval [ts,tf] is, the more severe the heat deterioration effect. Due to thermal inertia, an electronic component will not heat up if it operates for short periods of time. Therefore, it is good to control the electrical energy by regulating time interval [ts,tf] while keeping term Va • Ia at its maximum value.
TciLincai PiwrrDuve
0
• 1
e
J
a
/„ I ; ^ K l ^ Coils
hur PWM I'Dwcr Dim
•
(a) Power Switch
J
^
(b) Logic waveforms
'i
Fig. 5.27 Illustration of the difference between a linear power drive and a PWM power drive.
Here, we use a single phase to illustrate the difference between a linear power drive and a PWM power drive. As shown in Fig. 5.27, in a linear power drive, controllogic signals Q+ and Q_ will be at the logic "high" for the whole period of time interval [tB,tf]. This means that when the two switches are turned "on" within time interval [£s,£/], continuous current Ia
236
The Fundamentals of Robotics: Linking Perception to Action
flows through these two switches. The control of the electrical energy is based on the control of input voltage Va (or current Ia). On the other hand, if we superimpose a periodic pulse waveform on controllogic signal Q, something which is easily done with an "AND" Boolean logic circuit, input voltage Va (or current Ia) will only be applied to the coils when the combined logic signal, applied to switch Q_, is at the logic "high". The duration of the logic high state at Q depends on width ton of the positive pulse. It is clear that we can fix input voltage Va. Normally, we choose it to be equal to the voltage of the power supply. And we can regulate pulsewidth ton. Assume that the periodic pulse waveform has a fixed time period (cycle). If time interval [t s ,i/] contains n cycles of periodic pulses, the electrical energy applied to a motor will be
We = Y,\r
°Va •/„)•*}•
(5.32)
From Fig. 5.27 and Eq. 5.32, we can see that switches Q+ and Q_ will carry current Ia only when the controllogic signal applied to Q is at the logic "high". Clearly, the PWM method has much less power dissipation caused by the heat deterioration of electronic components. Thus, a PWM power drive is more energy efficient than a linear power drive.
PWM Wave _ ^ Generator AND Logic
1 r
i j ^ i
{
i n r Motor • Inertial Load i I
]
^ Control Signal
Logic Wave Generator
Switch Circuit
n
~ Power Supply Fig. 5.28
I Voltage
s
Schematic illustration of a PWM power drive together with power switch.
Fig. 5.28 shows a schematic illustration of a PWM power drive together with a power switch. Since the control signal is not directly converted into any voltage or current signal, the PWM method falls into the category of indirect power amplification.
Control System of Robots
237
5.3.3
Digital Motion Controllers
Refer to Fig. 5.15. The most important control element inside a robot's motion control system is the digital motion controller. A digital motioncontroller is a decisionmaking device and should be treated as part of a robot's information system. In fact, a robot's information system, the details of which will be studied in the next chapter, is like a human brain. It primarily plays five important roles: • Computation: A typical computational load is the execution of a control algorithm in order to determine the control signal from the error signal inside a robot's closedloop control system. • Decisionmaking: A typical example is the determination of a control action exerted by an actuator (i.e. an action device) upon a joint. A control action can be determined by a simple computation or a complicated decisionmaking process. • Coordination: Since a robot has multiple degrees of freedom, one obvious coordination requirement is how to synchronize the executed motions at a robot's multiple joints. • Communication: At the lowest level, a digital computer can interface with the outside world through its input/output subsystems (I/O, for short). For example, the interface between a digital motion controller and a sensing device requires either a serial or a parallel communication channel. • Cognition: For highlevel control or decisionmaking, it is necessary for a humanoid robot to have the ability to incrementally build its own internal representations of the real world. In this way, a humanoid robot will be able to interact with humans through the conceptual symbols of a natural language. The cognitive ability of an artificial life or system is, intrinsically, computationdependent. The neural system of a human brain is a highly parallel neural architecture with about a 100 billion neurons. Obviously, the computing system of a humanoid robot must also be a parallel or distributed system with multiple centralprocessing units (CPUs). Due to the complexity of a hu
238
The Fundamentals of Robotics: Linking Perception to Action
CPU
4
Logic Clock
Memory
^ IT
(
U
T
I/O Sub systems >"•«.
^
)
Bus (Address lines. Data lines and Control signal lines)
Motion Controller
\ / Master Controller (Robot's Brain) T
I
Power Amplifier I ^F I ' 
Sfii.. Mowr
Inertia!
Load
(^)
Fig. 5.29 Schematic illustration of a digital motion controller together with a robot's master controller and a motor.
manoid robot (a large number of degrees of freedom), it is impossible to use a single supercomputer to handle all the computation, decisionmaking, coordination, communication, and cognition required. A cheap and affordable solution is to deploy an embedded and distributed computing system inside a humanoid robot. Fig. 5.29 illustrates a distributed computing system which includes a robot's master controller (at least one) and a digital motion controller. The basic building blocks of a digital computer include: a) CPU (Central Processing Unit), b) memory (e.g. registers, RAM, ROM, EPROM), c) I/O subsystems and d) bus (address lines, data lines, and control signals/strobe lines). All computationrelated tasks are executed by the CPU. All these tasks must be coded in the form of programs and stored inside the memory. It is important to note that the digital nature of a computer is dictated by the cyclebycycle operation of the CPU. In fact, all the operations of the CPU are synchronized by the regular pulse waveform of a logic clock. The frequency of the logic clock primarily determines a digital computer's speed or computational power. Consider the case of motion control of a single actuator shown in
Control System of Robots
239
Fig. 5.29. The digital motion controller will read the sensor's data through a I/O subsystem at discrete timeinstants. Based on the error signal, which is normally the difference between the desired motion and the actual motion measured by the sensing device, a control signal is calculated by a control program executed on the digital motion controller. The control signal is then sent to the power amplifier through an I/O subsystem at discrete timeinstants as well. The output of the power amplifier subsequently drives the actuator to accomplish the control action. If the closedloop motion control system is stable, then the control action will make the actual motion of the actuator converge to the desired motion. Refer to Eq. 5.5. If a PID control law is used to determine control signal Vc, the control program inside a digital motion controller simply implements the following recursive algorithm:
Vc(k) = Vc(k  1) + a0 • e{k) + ax* e(k  1) + a2 • e(k  2) (5.33)
with ' o 0 = kp • (1 + ^ + ^ ) < O l = kp • (1 + 2£t)
a2 = kp • ^
where kp is the proportional control gain of the PID control law, Ti the integral timeconstant, and Td the derivative timeconstant. These parameters can be experimentally tuned if the system dynamics are not exactly known. Parameter T, in the above equations, refers to the sampling interval between two consecutive readings of the sensor's data. Alternatively, T also refers to the control action interval between two consecutive outputs of control signal Vc to the actuator. 5.4 Sensing Elements
A human sensory system can be divided into five distinct subsystems, namely: visual, auditory, kinesthetic, gustatory and olfactory. These sensory subsystems provide input to the brain, so it can not only make decisions but also build personalized internal representations of the external real world. In order to make a humanoid robot demonstrate humanlike behaviors, its sensory system should be as complete and efficient as a hu
240
The Fundamentals of Robotics: Linking Perception to Action
man's. At the present, however, the robot's sensory system is far from comparable. We hope, however, that the emergence of the humanoid robot and MEMS (i.e. MicroElectroMechanical System) will inspire the development of smart sensors. One area for improvement is the optical encoder. It is a wellknown fact that almost all industrial robots use the incremental opticalencoders for position and velocity sensing. The primary reason behind this choice is that incremental optical encoders cost 1/10 the price of absolute opticalencoders. As a consequence, the robots using incremental opticalencoders must find the reference positions of the encoders before they can function properly. This is done by a process called homing. The homing procedure involves unpredictable movement done by a robot. Thus, it is not at all desirable for a humanoid robot to do homing because this presents a potential danger outside the industrial environment. Society will not accept this potentiallydangerous behavior. On the positive side, as the result of research on human computer interaction (HCI), speech recognition is a relatively mature technology. In addition, continuous efforts to develop a robust speechrecognition system will certainly benefit the auditorysensory subsystem of future humanoid robots. In this book, we will study the visualsensory system of robots in a separate chapter. Here, we limit the discussion to the sensors which are closely related to a robot's motion execution. These sensors fall under the category of the kinesthetic sensory subsystem. 5.4.1 Generic Sensing Principle
For a general sensing instrument, the requirement of a sensor is that it somehow converts the variables of a system into corresponding signals suitable for display. But for a robot with automaticfeedback control using a digital computer, the requirement of a sensor is slightly different. The purpose of sensors is to convert the variables of a physical system under control into corresponding digital signals.
P i g i t a 7 W  A/D L 4 \Signaly ^ Fig. 5.30 I 1 I~
Si S nal
4  1 Transducer U _ / T V a r i a b l e s of
~
)
Converter
\Physical System J
Generic sensing procedure in an automatic feedback control system.
Control System of Robots
241
Fig. 5.30 shows the generic sensing procedure. In robotics, the variables under consideration may include: position, velocity, acceleration, force, torque, sonar, and lighting. These physical variables can be converted into electrical signals such as voltage, current, resistance, capacitance and inductance by using a device called a transducer. An ideal transducer is a device which converts a signal from one domain into another without removing any energy from the system under measurement. For example, the speed sensor of a motor should not consume any mechanical energy from the motor itself. In physics, for the purpose of sensing, many working principles can be applied to the development of transducers. Depending on working principles, the sensors can be classified into the following typical categories: • Electric Sensor: This refers to a sensor which converts a physical variable directly into a corresponding electrical signal. A typical example is the potentiometer, which converts a position variable into a corresponding resistance value. A strain gauge sensor for the measurement of force and torque also falls under this category. • Electromagnetic Sensor: This refers to a sensor which makes use of the magnetic field to do the signal conversion. A typical example is the tachometer. A tachometer makes use of a rotational conducting loop to pick up the current/voltage, which is proportional to the rotational velocity of the conducting loop in a magnetic field. • Optical Sensor: This refers to a sensor which makes use of light as a medium to do the signal conversion. Optical encoders fall under this category. As shown in Fig. 5.30, it is necessary to use a signal converter if the output from a transducer is not in the form of voltage. Therefore, the role of a signal converter is to convert the output of a transducer into corresponding voltage which will then be sent to an analogtodigital (A/D) converter. If we denote y(t) the input to a sensor and s(t) its output, the key features of a sensor typically include: • Accuracy: This refers to the accurate relationship between the sensor's input and output. Ideally, we would like to have s(t) = ks • y(t) with ks = 1.
242
The Fundamentals of Robotics: Linking Perception to Action
• Precision: This refers to the statistical distribution of output s(t) when input y(t) is set at a fixed value. The smaller the statistical distribution, the more precise it will be. • Resolution or Sensitivity: This refers to the smallest amount of variation in input which will trigger variation in output from the sensor. If s(t) = ks • y(t), we have AyV2 = ks. The higher the value ks, the better the resolution. • Operating Range: This refers to the range within which input y(t) is allowed to vary. • Response Time: This refers to the dynamics of output s(t) in response to input y(t). Ideally, we would like to have a sensor which does nt exhibit any transient response. In other words, s(t) = ks*y{t) with ks being a constant gain. • Reliability: Reliability is equal to the inverse of the failure rate of a sensor. Obviously, the smaller the failure rate, the higher the reliability. 5.4.2 Safety Sensors
In the future, humanoid robots will not only operate in industry but also coexist with humans in society. Consequently, safety is a crucial issue as society will never accept a mechanized creature that is unpredictably dangerous. Ideally, a humanoid robot should not only obey humans but also be friendly towards them. Here, it is worthy to cite Asimov's laws published in 1950: • Law 1: A robot should not injure a human being. • Law 2: A robot should obey human commands as long as these commands are not in conflict with Law 1. • Law 3: A robot should protect itself as long as such protection is not in conflict with Law 1. In order to ensure people's safety, all of a robot's actions should only be enabled if they are judged to be safe to human beings and their environment. Thus, a humanoid robot should have an emergency stopsensor, a proximity sensor, and visual perception system. These sensors are necessary to ensure the protection of human beings, the environment, and the robot.
Control System of Robots
243
5.4.2.1 Emergency Stop Sensors In the event of an emergency, humans should be able to interrupt the actions of a humanoid robot. This can be easily achieved by providing one or more emergency stopbuttons which serve as simple "onoff" sensors. The output signal (the "on" signal) from an emergency stop sensor should have the highest priority to halt all the power drives inside a humanoid robot. Fig. 5.31 shows a prototype of a humanoid robot with two emergency stopsensors: one on the front part of the body and one at the back.
^ ^ ^ ^ ^ ^ ^ ^ ^ I H ^ ^ H Emergency Stop Sensor I ^^^^^HHVMVMVMVMVMVA19jjj^H^^^^^H
••i '
B» .
^b '* *
«W . fl i i''1 i i >• I v ^
•H^wPHIHIH
^^^^Mi^l Emergency Stop Sensor H n y stop sensor at the back of robot.
(a) Emergency stop sensor in the froni
Fig. 5.31 Example of two emergency stopsensors on a prototype of a humanoid robot.
5.4.2.2 Proximity Sensing Systems An emergencystop sensor is considered a passive safety sensor because it is manually activated by a human or other equipment. Alternatively, one may propose an active sensing system for safety. A typical example of an active safety sensor is the proximity system such as ultrasonic or infrared sensors. An ultrasonic sensor is a lowcost device which is able to measure the distance to an object within a typical range of 0.3m to 10m. Fig. 5.32a shows an example of ultrasonic sensor (i.e. the model Polaroid 6500) which is composed of two parts: a) the buzzer and b) the drive circuit. An ultrasonic sensor is an activesensing device because its buzzer not only transmits sonar waves but also receives the echoes. The working principle behind an ultrasonic sensor is based on the measurement of the flight time of the sonar wave because the speed of a sonar wave is constant (about 344m/s).
244
The Fundamentals of Robotics: Linking Perception to Action
(a) Buzzer and the '
• >) Single sensor with a miao coimoller.
(c) A ring of sensors with a micro controller.
Fig. 5.32 Example of proximity systems based on ultrasonic sensor(s): a) the sensor's buzzer and drive circuit, b) a proximity system with a single sensor and c) a proximity system with a ring of sensors.
INIT
1
^ _  1 6 PULSES AT 49.4 kHz
TRANSMIT
(INTERNAL) BLNK/BINH INTERNAL
BLANKING ECHO
L0W
III
, 1 2.38 ms
^ I P{

, 1 '
j 
[
;
if Time
•
ts
Fig. 5.33 Signal diagrams for the operation of an ultrasonic sensor.
For an ultrasonic sensor, there are two basic modes of operation: a) the singleecho mode and b) the multipleecho mode. Fig. 5.33 shows the signal diagrams corresponding to the singleecho mode. When an ultrasonic sensor (e.g. the model Polaroid 6500) is turned "on" by applying voltage to line Vcc, a minimum of 5ms must elapse before the
Control System of Robots
245
INIT signal is driven to the logic "high" in order to trigger the transmission of a sonar wave at timeinstant ts. When the INIT signal is driven to the logic "high", a series of 16 pulses at a frequency of 49.4kHz will excite the buzzer (the transducer), which generates and transmits a sonar wave. Since any mechanical device has a ringing effect due to vibration, an internal blanking signal must be driven to the logic "high" for about 2.38ms in order to avoid the detection of the buzzer's ringing. After the blanking period, the buzzer is ready to serve as a receiver and detect an echo. Once an echo is detected at timeinstant tf, the echo signal will be instantly amplified by the drive circuit and the logic "high" signal will be issued to the ECHO signal line, which is an output line of the ultrasonic sensor's drive circuit. By driving the INIT signal line to the logic "low" again, this will reset the ultrasonic sensor's drive circuit and the ECHO line will be driven to the logic "low". Then, the ultrasonic sensor is ready to start a new sensing cycle. Since the sonar wave travels at a constant speed in air, the distance to a detected object is linearly proportional to the time of flight tf — ts. This time of flight can be easily measured by the programmable timer I/O system of a microcontroller (e.g. M68HC11). Depending on the intended application, one can easily configure a proximity system with one ultrasonic sensor or a ring of ultrasonic sensors as shown in Fig. 5.32b and Fig. 5.32c.
5.4.2.3 Detection of a Human's Presence
There are two problems with the activesensing technique. First of all, a sonar wave can be reflected in different directions. This will result in multiple echoes, which in turn, will cause false detection of the presence of objects. Secondly, the response time is proportional to the spatial coverage and range of the active sensor. The larger the spatial coverage and range, the slower the response time will be. A human being primarily relies on the visual sensory system to gather information about his/her surroundings. Some notable advantages to a visualsensory system include the rich information, high resolution, fast response (parallel and spontaneous), and lack of interference with environment (passive sensing). Just as a visual sensory/perception system is important for a human being, a visual sensory/perception system is crucial in order for a humanoid robot to accomplish activities autonomously. We will study the visual perception system of a humanoid robot in Chapter 8.
246
The Fundamentals of Robotics: Linking Perception to Action
Here, we show an example of detecting a human's presence. The algorithm is based on a probabilistic RCE neural network. As shown in Fig. 5.34, the identification of pixels having the appearance of skin color is a simple process. It allows the robot to make a quick assessment of whether or not a human being is present. Obviously, a more elaborate approach is for the robot to be able to detect the face and hand gestures.
(a) Original (color) image.
(b) Blacked pixels of skin color.
Fig. 5.34 Detection of the presence of a human being through the identification of pixels having the appearance of skin color. (NOTE: The original image is in color).
5.4.3
Motion Sensors
A humanoid robot's mechanism is highly complex with many degrees of freedom. As each degree of freedom requires at least one motion sensor to provide sensory feedback, a humanoid robot will need a large number of motion sensors. And the performance of these motion sensors is crucial to the proper control of motions executed by a humanoid robot. As we mentioned before, motion sensors inside an industrial robot are typically the incrementalencoder type. Consequently, an industrial robot must perform a homing procedure before it is ready to work. For a humanoid robot, however, this homing procedure is not desirable because it involves unpredictable movements. Obviously, the absolutemotion sensor is the most appropriate choice for a humanoid robot. With the absolutemotion senor, the homing procedure is unnecessary. However, for the purpose of comparison, let us study the working principles behind both absolute and incremental motion sensors. Regardless of the type of motion sensor, one must be clear that the
Control System of Robots
247
purpose of a motion sensor is to provide motion feedback by measuring the motion parameters at a joint inside a robot's mechanism. As a result, there are two concerns: • Where do we place a motion sensor? • How do we measure the motion parameters? 5.4.3.1 Placement of Motion Sensors For the sake of clarity, let us consider the motion sensor for rotary motors. In Chapter 4, we studied the coupling of a dynamic pair with a kinematic pair. We know that a dynamic pair is normally coupled with a kinematic pair through a device called a speed reducer or torque amplifier. This means that the motion of a motor is not exactly equal to the motion of the corresponding joint driven by the motor. Now, the question is: Should a motion sensor be placed before the speed reducer or after the speed reducer?
Fig. 5.35 reducer.
Sectional view showing the placement of a motion sensor before the speed
The answer depends on the type of sensor and the application requirement. Ideally, we would prefer to place the motion sensor before the speed reducer, as shown in Fig. 5.35. In this way, we can simply measure the motion parameters of a motor. The output from a motion sensor is then divided by the reduction ratio of the speed reducer in order to obtain the motion parameters at the corresponding joint. Let us denote qt the angular position of joint i, qmi the angular position of corresponding motor i, and kri the reduction ratio of corresponding speed reducer i. If a motion sensor is placed before speed reducer i, the output from the sensor is qmi. But, the required output for motion feedback is qu which can be easily determined
248
The Fundamentals of Robotics: Linking Perception to Action
as follows: 9i = j—nri
(534)
The differentiation of Eq. 5.34 yields
dqi = j— •dqmi.
This simply means that the error caused by the sensor's output is reduced by a factor of kri. Clearly, the advantage of placing a motion sensor before a speed reducer is to obtain accurate motion feedback even if the motion sensor itself is not absolutely precise. If the working principle behind a motion sensor requires the physical contact between one fixed element and one moving element, the placement of a motion sensor before the speed reducer is not suitable because angular velocity qmi is kri times higher than angular velocity q\. The higher the velocity, the more severe the wearandtear due to contact friction.
Motor ^^^^^HH^^^^^^^^_
I^^^^^^^^B^^^^^^^^^^^I Joint
Fig. 5.36 Sectional view showing the placement of a motion sensor after the speed reducer.
Alternatively, one may choose to place a motion sensor after the speed reducer, as shown in Fig. 5.36. In this way, we can directly measure the motion parameters at a joint. In other words, the output from a motion sensor directly corresponds to the motion parameter(s) of a joint. 5.4.3.2 Potentiometers All of us should be familiar with the potentiometer. When we adjust the frequency of a radio, we rotate a rotary disk. And this rotary disk drives a potentiometer to alter its output resistance value inside the internal circuitry of the radio. Physically, the potentiometer of a radio is a motion
we have Vout _ r_ _ qi V or ~ R ~ 360° Vmt = ^ 5 . Therefore. resistancetovoltage signal converter and A/D converter). the axis of a potentiometer should be coupled with the shaft of a joint. If the circular resistor is uniform and the gap between points A and B is negligible.e. 1 ^^S^^^ Shaft of a joint 1 ® Fig.36) Clearly. As we discussed above.37 shows an example of a rotary potentiometer and its associated sensing elements (i. Assume that point B is the reference position of potentiometer i coupled with joint i. r the resistance value of arc CB. If the output voltage is picked up at the wiper. and qi the angular position of the wiper with respect to reference position B. the measurement of output voltage Vout allows us to exactly determine angular position q^. a potentiometer should be placed after a speed reducer because it is a contactbased sensing device.Control System of Robots 249 sensor because the input is the angular position of a rotary disk and the output is its resistance value.qi. A potentiometer is composed of two elements: one fixed circular resistor and one movable wiper. Fig. (5. we have r_ __ AB _ j^_ R~CB~360° [b6b) Assume that input voltage V is applied to the potentiometer. 5. . Let us denote R the resistance value of arc AB.37 Illustration of the sensing elements built on top of a rotary potentiometer. 5.
it is desirable to develop noncontact motion sensors.5V to 0. A diode is a nonlinear electric component. Working Principle Behind Optical Encoders The basic working principle behind an optical encoder is the conversion of a motion into a series of light pulses. From Fig. In order to generate light pulses.38. This is usually done with a photodetection device such as a phototransistor (or photodiode). Fig.3 Absolute Optical Encoders A contactbased motion sensor has some obvious drawbacks: It has short lifetime due to the wearandtear. When combined with an opamp circuit. Therefore. And. 5. an electrical pulse can be easily produced when there is a light pulse present at the base of a phototransistor. A phototransistor is a special transistor which will produce a current from the collector to the emitter when the base is exposed to a light source (or excited by a light pulse).e.3.8V. it is necessary to convert the light pulses into electrical pulses. it is easy to see that the logicsignal output will be high (i. or LED. No current can flow in reverse from the cathode to the anode of a diode. One good light emitting device is the lightemitting diode. It has two terminals: a) the anode and b) the cathode. "1") if light from the LED can .38 illustrates the working principle behind an optical encoder. An optical encoder consists of: a) a lightemitting device with an optical lens to produce focused parallel light rays. 5. b) a lightdetecting device with a logicsignal converter and c) a moving slotted (metal) sheet which can be interfaced with a rotary motor. Under normal working conditions. which are subsequently converted into a corresponding series of electrical pulses (digital logic signals). However. it has low accuracy of feedback output. After generating a series of light pulses. for short. it is necessary to have a light emitting device. The current can effortlessly flow from the anode to the cathode of a diode when the forward voltage drop is within the range of 0. Nowadays. As we mentioned above.4. one of the most widelyused noncontact motion sensors is called the optical encoder. A lightemitting diode continuously emits the light rays as long as there is a forward current which flows from the anode to the cathode. Accordingly. it is necessary to have a currentlimit resistor in order to protect an LED. currentflowingthrough an LED is within the range of 5mA to 20mA.250 The Fundamentals of Robotics: Linking Perception to Action 5. this is only possible for noncontact motion sensors. the ideal place to put a motion sensor is somewhere before the speed reducer.
e. 5. And a digital bit only carries one piece of information (i. In other words. If we put all the bits of an absolute optical encoder together to form a logic state variable. For a digital system.Control System of Robots 251 ' Light Emitting Device ' . either "on" or "off").38 Illustration of the working principle behind an optical encoder. an absolute optical encoder has to have at least 3600 logic states as output. Otherwise. ±£ 1 i ' . output '• i ' I ¥XtW . the number of logic states that an absolute optical encoder has will determine the number of absolute angular positions which can be measured. As one bit only allows the commutation between two logic states (i. "on" or "off"). if we would like to measure a full rotation of 360° at an accuracy of 0. the logicsignal output will below ("0"). and 12 bits will have 4096 logic states). Optical lens . Working Principle Behind Absolute Optical Encoders By definition. Therefore. 3600 logic states require at least 12 bits (NOTE: 11 bits correspond to 2048 logic states. an optical encoder is intrinsically a digital device.Logic Signal r s ^ i TLZ———r~\ + V~\x >^^V^. an absolute optical encoder should have at least 3600 commutating logic states. if light is blocked by the opaque (metal) sheet.1°. one important design parameter for an absolute optical en . pass through a slot which is aligned with the phototransistor. an absolute optical encoder is a device which is able to pinpoint the absolute angular positions of a rotary motor.e. one logicsignal line corresponds to one digital bit. . / ' Light Detecting Device \  Movable slotted sheet Fig.cSlgnai. As output from an optical encoder are logic signals. +5V. i ! ' _ ' ' • i+5Y Phototransistor i \ '• ' LE\ \ §*>':'\ I V ^ X v R ^ ' : K^ 4^. For example.
39 Illustration of a conceptual design of an absolute optical encoder with 16 logic states being engraved on a slotted.0111* ' 1001 1000 .. Light emitting devices Light detecting devices H"""'"'1" HI / j Motor SSSHSBB ™ I • ' • ^Kuiot '. the accuracy of an absolute optical encoder will be increased by kr times. Once the number of commutating logic states has been determined. SfiflBfiS Slotted (metal) Sheet of Disk shape ' Section * 0000 i 0001 jni^^^^jooio s noi i Q H H L ^ T r a c k _ y^^^^^^H^^Poioo jnnA(H)ii x IOIO T ' ' • " " . . diskshape sheet of metal. the option of engraving the commutating logic states onto a cylindrical tube sounds promising. the next issue is how to engrave these commutating logic states onto the movable slotted (metal) sheet.252 The Fundamentals of Robotics: Linking Perception to Action coder is the number of commutating logic states. The specification of this parameter depends on where the absolute encoder is placed on the motor. Fig. • If the absolute optical encoder is placed after the speed reducer of a motor (i. 5. for example: • If the absolute optical encoder is placed before the speed reducer of a motor. to measure the angular position of a joint). With the advent and maturity of MEMS technology. which can be in the shape of either a disk (the default option) or a cylindrical tube. the number of commutating logic states has to be specified directly according to the desired accuracy of the measured angle at a joint. ' iflllBffll Joint •) " 1 B H H S & Slotted (met^) sheet I Absolute Optical feeder v .e. where kr is the reduction ratio of the speed reducer.
The engraved disk of an absolute optical encoder is normally fixed on the shaft of a motor. 5. Fig. 5.39 shows a conceptual design of an absolute optical encoder with 16 logic states being engraving on a slotted.40 Illustration of a conceptual design of an absolute optical encoder with 16 logic states being engraved on a slotted. an absolute optical encoder allows us to obtain not only the absolute angular positions but also the direction of motion. From Fig. When the disk rotates with the motor. As a result.ck shape Section ' v 0(100 0010 olOO oilO 10O0 1010 1I0O 1110 oooi (ion oioi oin luoi ion noi nil Fig.39. 1 t^ Light deteciing devices ijyBiTi'iiii''B Light emi • f ' Motor '• illSB SMBHSH '""" Slotted (n • '[ i llsiHnfiB Absolute Opttcal Encoder \ vv Slotted (metal) Sheet of Tube Tr. or from 1111 to 0000.Control System of Robots 253 For the sake of clarity. tubeshape sheet of metal. Therefore. their dimensions are not uniform. the logic states will commutate either from 0000 to 1111. we illustrate the case of designing an absolute optical encoder with 16 logic states (4 bits). diskshape sheet of metal. The inner track will have the smallest circumference. diskshape sheet is divided into four tracks (corresponding to the number of logic bits in an absolute optical encoder) and 16 sections (corresponding to the number of logic states in an absolute encoder). Since there are four logic bits. it is necessary to have four pairs of lightemitting and lightdetecting devices in order to read out the logic states. When tracks are engraved on a disk. 5. we can see that the slotted. the density .
Instead of engraving the logic states onto a diskshape (metal) sheet. it becomes a rectangular sheet.40. it poses no challenge to manufacture the slotted. diskshape sheet of metal. Working Principle Behind Incremental Optical Encoders The idea behind the design solution of an incremental optical encoder is to reduce the number of tracks on the diskshape metal sheet to one.42. The output from an absolute optical encoder is the value of a logic state which indicates the absolute angular position. as shown in Fig. If the rotated angle exceeds 360°. Hence. a single track of periodic slots only requires one pair of lightemitting and lightdetecting devices. one simply engraves a periodic slotted pattern on a single track.41 illustrates a dismantled incremental optical encoder. it is easy to increase the number of tracks engraved on a tubeshape sheet for better accuracy. It has three parts: a) the slotted. When a tube is extended onto a planar surface. 5. A notable advantage to an incremental optical encoder is its compact size and low cost. a series of electrical pulses will be generated when the slotted disk of an incremental optical encoder rotates with a motor. The nonuniformity of the dimensions of the tracks also implies that the process of manufacturing slotted. diskshape sheets of metal may be costly.42. Furthermore. .254 The Fundamentals of Robotics: Linking Perception to Action of the sections (the number of sections per track) is limited by the inner track. b) the integrated portion of lightemitting and lightdetecting devices. In order to measure angular displacements. this drawback can easily be overcome by engraving the logic states on a tubeshape sheet of metal. 5. In principle. However.4 Incremental Optical Encoders Due to the concern about cost. tubeshape sheets of metal. The slotted metal disk has the periodic slotted pattern with eight cycles engraved on a single track. as shown in Fig. In summary. 5. an absolute optical encoder can either be placed before the speed reducer or after the speed reducer of a motor. All the tracks and sections engraved on it will have a uniform dimension and shape. Refer to Fig. 5. Fig. as in the case of an absolute optical encoder. one of the most widelyused optical encoders for industrial robots and other equipment like printers is the incremental optical encoder. In this way. or viceversa. 5. and c) the compact casing.4. it can be easily detected by counting the number of commutations from the smallest logic state value to the highest logic state value.3.
Because of its simple structure.Obviously. the accuracy of an incremental optical encoder is in the range of 1° to 0. eight (8) pulses indicate an angular displacement of 360° with an accuracy of ±45° (i.1°. the incremental optical encoder with a single track is only able to measure angular displacements without knowing the direction of motion. the more accurate an incremental optical encoder. one must address these two issues: • How do we register a reference position or home position? • How do we determine the direction of motion? . If n c denotes the number of cycles of the slotted pattern on the disk. an incremental optical encoder must be placed before the speed reducer of a motor.41 Picture of a dismantled incremental optical encoder. For the example shown in Fig.37) A larger value of nc indicates an increase in difficulty or cost for the manufacturing of the slotted disks.e.42.Control System of Robots 255 ijght emitting anil 1^^^BH Caung Slotttsd inrtdl *Mt of disk shape ^^^HPs&L' ^^^^Pr Fig. In practice. Consequently. will be A*=^!. what we need for the motion feedback in a closedloop control system is the measurement of an angular position or velocity in the absolute sense. 5. nc (5. But. one simply counts the number of pulses which is linearly proportional to angular displacements. In order to determine the absolute angular position from the measurement of angular displacements. denoted by A6. ^p). the accuracy of an incremental optical encoder. 5. the larger the number of cycles of the slotted pattern.
42. or home position. " ^ j B B L g J a k j . A motor will first rotate in one direction (either clockwise or counterclockwise) and its range of movement will be no more than 180°.42 Illustration of a conceptual design of an incremental optical encoder.42. In Fig. Determining the Home Position The first question is easy to answer. HHHBHi Slotted f metal) Sheet of Disk shape . as shown in Fig. This reference slot indicates a reference.* — Li phi detector B ~ ^•^^^H\ . detector H is to identify the home position. j • I——' —'—' ' ' ' ' ' ' '—•—' ' ' '— '—  fA' 'efl Clockwise Toiaiion . 5.^ ^ .' ' L J H H ^ ^ ^ ^ ^ B ^ H B r Light detector H I '.256 The Fundamentals of Robotics: Linking Perception to Action Light emitting devices Light detecting devices Hra& ? ^K^^^Bi I Motor ^B UH I . ^ ^ O n e cycle Light detector A • " v ^ ^ ^ T ^ " ' . a robot or any machine which uses the incremental optical encoders must undertake a homing procedure before it is ready for actions or operations. there must be a pair of lightemitting and lightdetecting devices to locate this home position. A homing procedure for one motor normally involves two rotations at most. 5. A common solution is to create a reference slot on the disk. 5. HI ^^^^^^B I ^ ^ ^ ^ ^ B Joint ' / Slotted (metal) s W  Incremental Optical Encoder ~ «. ' i > lB *A Counterclockwise rotation Fig. With an added reference slot. this homing procedure is not desirable for a humanoid robot to perform . As it is necessary to identify a home position. the motor will rotate in the opposite direction until a home position is found. As we mentioned earlier. If no home position is detected in the first rotation.
For any given cycle of the slotted pattern on a rotating disk. the logic signals from these two detectors carry the same information (they are logically equal). Obviously. the logic signals from these two detectors will not be logically equal. By default. it is impossible to determine the direction of motion from a single source of pulses. let tA denote the starting time of a generated electrical pulse from detector A. 5.42. In other words. and i s the starting time of a generated electrical pulse from detector B. the logic signal from detector B will be the complement of the logic signal from detector A.38) . However. Similarly. A phaselead of the waveform from detector A with respect to the waveform from detector B simply means tA < tB. Since the slotted pattern on a disk is composed of periodic cycles. In order to solve this issue. If the position of detector B has no phaseshift with respect to the position of detector A. detector B has a phaseshift (phaselag) of 90° behind detector A in a clockwise direction. if the position of detector B has a phaseshift of 180° with respect to the position of detector A. This issue is slightly more complicated. a cycle will correspond to a full phase of 360°. 5. the logic signal from detector B will be identical to the logic signal from detector A. a common solution is to introduce a second lightdetector (detector B). if the phaseshift between the two detectors is not equal to 0° (or the multiple of 360°) nor 180° (or the multiple of 180°). A single lightdetector (detector A) normally produces one series of pulses when the slotted disk of an incremental optical encoder rotates with a motor. (5. a common design solution is to choose a phaseshift of 90° between the two detectors. When the slotted disk rotates in a clockwise direction.42. If we use the term phase in signal processing. the cycle of the waveform from detector A will have the phaselead of 90° with respect to the cycle of the waveform from detector B. a new cycle of a periodic logic signal starts with the rising edge which marks the transition from the logic "low" to the logic "high". Determining the Direction of Motion The second issue is how to determine the direction of motion. For the example shown in Fig.Control System of Robots 257 because the movement of a robot during the homing is unpredictable. In practice. as shown in Fig. the output from a lightdetector will be a series of periodic pulses when the disk is rotating.
The output from an incremental optical encoder normally includes four signal lines: one for the pulse waveform from detector A. to regulate the electrical energy applied to a motor for the purpose of varying its velocity. we are ready to close the feedbackcontrol loop and concentrate on the design of control algorithms.4 Force/Torque Sensors So far. and direction of motion etc. the motion executed by a limb (e. Accordingly. if we choose detector B as a reference. A digital motion controller must interpret the received logic signals from these four signal lines in order to determine rotated angle or velocity. the direction of motion is easily determined by measuring t& and tsIn summary.4. it may be necessary to consider the force or torque feedback as well in order to produce the desired interaction force or torque between a robot and its environment. when a humanoid robot writes on paper with a pen.g. we studied ways to alter the direction of motion. its hand will experience contact force between the paper and the pen. an incremental optical encoder will impose a homing procedure on all robots or other equipment like printers which use it for motion feedback. the cycle of the waveform from detector B will have a phaselead of 90° with respect to the cycle of the waveform from detector A. and one for the common ground. This is because the accuracy of an incremental optical encoder itself is not very high due to the limitation on the number of cycles of the slotted pattern. 5. and to measure the output motion of a motor either before the speed reducer or after the speed reducer for the purpose of motion feedback. (5. for motion control. In other words. an incremental optical encoder is normally placed before the speed reducer of a motor. armhand or legfoot) may involve direct contact between its endeffector and the environment or workpiece. In other words. in order for the robot to perform certain tasks. Finally. we have tB < tA.39) Clearly. detector A has a phaselag of 90° behind detector B in a counterclockwise direction. However. . one for the pulse waveform from detector H. For example. one for the pulse waveform from detector B.258 The Fundamentals of Robotics: Linking Perception to Action Alternatively. if the disk rotates in a counterclockwise direction. Practically.
up view of force/torque sensor. The answer to this question depends on the following cases: • If the robot is operating in outer space where there is no gravitational force.43 Example of a force/torque sensor: a) the enlarged view of a strain gauge and b) the internal view of a force/torque sensor. Metallic (silicon) ' gnd elements ™ Elastic beams / Plastic sheet j L Wirings ^^^^^I^^^^^E^r ^^*8HB^^^ Electronic circuitry (a) Enlarged view of strain gauge.4. 5. . 5. Now.1 Indirect Measurement In Chapter 4. Jacobian matrix J is easy to evaluate if the angular positions of the joints are known.4. Fig. one may question whether it is possible to determine external force/torque £e exerted on the endeffector's frame of a limb from measured torque vector r at the limb's actuators. it is necessary to develop a sensor which is able to measure the force/torque externally exerted on the endeffector (hand or foot) of a robot's limb. it is possible to directly determine £e from measured torque vector r.2 Direct Measurement As most robots operate on Earth where there is gravitational force.4. we studied robot statics. if the robot is operating on Earth.4. it is not easy to determine £e based on the measurement of r because a portion of r is spent to balance gravitational forces acting on the links of a limb.Control System of Robots 259 5. external force/torque £e exerted on its endeffector's frame is related to torque vector r applied to the actuators of the robot's limb in the following way: where J is the Jacobian matrix of the robot's limb. When a robot's limb is at rest. • However. (b) Open .
5. the strain gauges placed on it will also deform. a strain gauge is a metallic (e. a tube). When a beam deforms in response to the stress caused by an applied force or torque. b) rigid body 2 (e. the beam deforms in response to the stress caused by the applied force or torque. In order to measure an applied force or torque. in order to design a force/torque sensor based on strain gauges.g. This deformation can be detected by placing one or two strain gauges onto one or two surfaces of the beam. A strain gauge is normally attached to a plastic sheet. Thus.43 shows an example of a force/torque sensor. one must address these issues: • How do we couple the applied forces and/or torques to a structure of elastic beams? • How do we convert (very small) variations in electrical resistances of the strain gauges to variations in corresponding voltage signals? • How do we map the voltage signals to the measurement of the applied forces and/or torques? Structure of Elastic Beams for Force/Torque Sensors A sensor should not interfere with the proper operation of a system under measurement. When an applied force or torque is coupled with an elastic beam. either increasing or decreasing. a tube) and c) a set of coupling elastic beams. In robotics. As a result. an element called an elastic beam is used.43.g. It is also known as a wrist sensor because a force/torque sensor is normally mounted between the endeffector of a robot's arm and its hand (or a tool).g. When body . a strain gauge behaves like a variable resistor.44. Accordingly. 5. Rigid body 1 is coupled to rigid body 2 through the supporting beams. the electrical resistance of which changes as a function of its deformation. 5. a force/torque sensor's structure consists of three parts: a) rigid body 1 (e. In other words. the electrical resistances of the metallic grid elements in the strain gauges will change in a specific way.260 The Fundamentals of Robotics: Linking Perception to Action In robotics. the structure of a force/torque sensor is commonly based on an elastic beam assembly. a force/torque sensor should structurally behave like a rigid body as long as the applied force & torque is within its working range. silicon) grid element of foil which can come in different shapes and sizes. As shown in Fig. a widelyused method of measuring forces and/or torques is with a strain gauge. Fig. As shown in Fig. This rule implies that a force/torque sensor should not cause any extra motion to a robot system.
If a beam is in a shape of a cube. Fig. 5. Y and Z axes. and b) three for the elementary torques about the X. foot or tool. force/torque vector £e acting on the endeffector of a robot's limb will have six elements: a) three for the elementary forces in X. force/torque vector £e experienced by the .44 Example of a threebeam structure for straingauge based force & torque sensors. we can have the endeffector's frame at the center of a force/torque sensor's beam structure. In practice. For the sake of convenience. there will be six strain gauges. which can almost be linearly mapped into a corresponding measurement of external force/torque vector £e with six elements. In total. body 1 (the outer ring or tube) of a force/torque sensor is attached to the endeffector's link of a robot's limb while body 2 (the inner ring or tube) is attached to a robot's hand. By placing some strain gauges on the surfaces of the beams. we will have six variable resistors denoted as R\. R5 and RQ. As a result. 2 is subject to the external forces and torques.Control System of Robots 261 (a) Three beam structure (b) Elementary forces/torques Fig. 5. it is able to capture the deformations in two directions if its two ends are fixed onto two rigid bodies respectively. In general. Each beam has two strain gauges placed on its two adjacent surfaces.44 illustrates an example of a threebeam structure for a force/torque sensor. R4. Since a strain gauge behaves like a variable resistor. these deformations will be converted into the variations in electrical resistances of the strain gauges. it is necessary to have a minimum number of three beams to produce six values of output. and Z directions. In a threedimensional space. the builtup stress between bodies 1 and 2 will cause the beams to deform. Y. R3. R2. when an external force & torque acts on body 2.
as shown in Fig. Assume that we convert the variation of resistor Ri (i = 1. A Wheatstone bridge circuit has two branches.. we will form a Wheatstone bridge circuit.And. Ideally.l + Ki.45a. K (a) Wheatstone bridge circuit (b) Signal amplifier Fig. where {fxifyifzY This force/torque vector £e will cause variations in electrical resistances of the strain gauges. AR4. By adding another three highquality resistors Riti.2. Rit2 and i?i.45 a) Wheatstone bridge circuit for measuring resistance variation and b) voltage signal amplifier. 5.. 5. Each branch has two resistors.Ty. AR3. 5.40) is the force vector and [jx.'Tz)t the torque vector.Ty.. 5. AR2.3. The input voltage to the Wheatstone bridge circuit is Vin and the output voltage is Vout. That is: ARi. AR5 and AR6.41) . From Fig. Resistance to Voltage Conversion In electronics.262 The Fundamentals of Robotics: Linking Perception to Action endeffector's frame will be Ze = (fXJyJz.45a. the output voltage is equal to V\ — V2..Tz)t (5.2 (5. a common way to convert the variation of electrical resistance to the corresponding variation of voltage is to use the Wheatstone bridge circuit.Tx. voltage V\ is Ki. these four resistors should have similar thermal characteristics in order to balance out noise caused by thermal effect.45a. 6) to a corresponding variation of the voltage signal. as shown in Fig.
43 with respect to time yields dVout = .45) A beam's deformation is very small. Vi) is Vi = ^*(V2V1) = ^*Vmt. The input voltages are V\ and Vi. (5. the relationship between the variation of the electrical resistance of strain gauge % and the corresponding variation of the voltage signal is (5..2.43 ) Assume that the electrical resistances of resistors Ri. voltage signal AVout needs to be amplified to an appropriate level before it can be further converted into a corresponding digital signal.45 into Eq.42) vout = I R2 . 5..e. substituting Eq..2 • ^ i  (544) Consequently. A typical circuit for voltage amplification is shown in Fig.46) By differentiating Eq.47 yields (5. 5.48 indicates that electrical resistance's variation ARi is linearly converted to the corresponding voltage signal AVi (i.6).. And.45b.48) Eq. (NOTE: Vin is usually within +5V). As Vout = Vi — V2.2 and i ? ^ are constant. As a result. .e. 5.ir—jr) # v < 5 . 5.i.47) (5. we can obtain AVi = ~*AVout. in digital form after the A/D converter). p ' p . 5. we have (5. so the corresponding variation ARi is very small as well. Finally. voltage V2 will be V2 = WJ^r*Vm.And the output voltage (i. Ri. AVi is the output from strain gauge i (i = l.Control System of Robots 263 Similarly. Differentiating Eq.46 with respect to time. 5.
we will have 36 equations which allow us to uniquely determine the 36 elements. tactile sensors are also the important feedback sensors. the direct output from the sensor is voltage vector AV = (AVi. Now. we know that AV has an approximate linear relationship with £e. In practice. it is difficult to establish an exact analytical relationship between £e and AV. AV3.5 Tactile Sensors When a humanoid robot operates. the question is how to compute £* based on the knowledge of AV. shown in Fig.49 provides six equations for the 36 elements in matrix W. When we have a pair of known £e and its corresponding AV. Vj e [1. its two feet are in contact with the floor/ground. Therefore. If there are six pairs of known £e and its corresponding AV.49) W is a 6 x 6 matrix with 36 elements. Let W = {u>ij. If more than six pairs of known £e and corresponding AV are available. 5. we have C = W • AV. the input to the sensor is the actual force/toruqe vector And. it is inevitable that it will touch objects in the vicinity. 5.4.6]} be the constant mapping matrix. AV6)(. when standing.44. AV4. AV2. AV5.6]. By definition. 5. however. a tactile sensor is a device which measures the physical quantities related to touch. 5. (5. Let £* denote the estimated force/toruqe vector. Eq. When manipulating objects with its hands. In theory. the robot's fingers are in contact with the objects.Vi & [1. Then.43 and Fig. This explains why the practical solution is to use a constant mapping matrix to convert measurement AV to an estimated force/torque vector £*. What we would like to obtain is an estimation of the actual force/torque vector. This procedure is known as the calibration. matrix W can be estimated by a least square method.264 The Fundamentals of Robotics: Linking Perception to Action Voltage to Force/Torque Mapping For the force/torque sensor. A touch normally involves force and/or torque . The purpose of this calibration is to determine matrix W which describes the voltage to force/torque mapping. For example.
^^^^^^^^^^^^^^^^^^^^^^v: '••l. a robot's hand) holds another object (e. Basically. However. there are many ways to design transducers for tactile sensors.46. 5.j'. As a result.or nanodetector. Of course. And. In robotics. an ideal transducer for a tactile sensor is a device consisting of an array of microforce/torque sensors as shown in Fig. Other common touch detectors are the capacitive transducer which converts contact into corresponding capacitive impedance.46 Illustration of the composition of a tactile sensor. 5. A tactile sensor is also built on top of a transducer and a signal converter. a tool). 5. In this way. a tactile sensor's transducer has three elements: a) a deformable substrate.g. 5. b) an array of touch detectors and c) a softcover tissue (like skin). .g.• * >*i " '""^s! inStil^^^^^^^^^^^B" ^^^^^^^^^^^^^0Bm»Vi^^F*^^^Pv'jr. we can obtain a complete measurement of contact. it is able to measure external force/torque exerted on its stitch. a tactile sensor does not require an accurate measurement of the force and/or torque caused by the contact.30. Each microforce/torque sensor is similar to the one illustrated in Fig. a touch detector can be as simple as an "onoff" switch which indicates the logic state of intouch or notintouch. the challenge here is how to fabricate the tactile sensor based on the micro. I Soft cover tissue Sic tt l ^ H C . Depending on the intended application.Control System of Robots 265 when one object (e. But it may also contain other information such as the rolling and slippage effects between two objects in contact. In fact. a tactile sensor is a kind of force/torque sensor.44. and the LVDT (Linear Variable Differential Transformer) which measures the linear displacement caused by contact. Refer to Fig. ^fS'k'^^^^HKdf ••• \ Substrate Detectoranayj^B^^^^^^^J^^H^^^^^^J Fig.
Accordingly.47. a closedloop control system is a system which responds to an error signal. Despite the presence of multiple closedloops inside a robot's controller. the design of a control algorithm is not a simple matter. ' ^ Sensor Feedback ^ ^ Fig. Then. 5. . we will study . the inertia! load of a robot is the robot mechanism itself including the tool or payload. The minimization of the above equation is a typical example of the control objective in a closedloop control system.1 Issues As the robot has multiple degrees of freedom.5. and qd its desired output. the immediate challenge is how to design a control algorithm which will make e(t) converge to zero after a certain period of time. its motion controller must be able to handle multiple closedloops of motion control.266 The Fundamentals of Robotics: Linking Perception to Action 5. if any. The error signal will be e(t) = <f(t)q(t).5 Control System Design If we know the sensor feedback. the basic structure of a closedloop control system is still similar to the one shown in Fig. * Electric Motors Speed Reducers Robot Mechanism .47 Illustration of the closedloop control system for a robot's motion execution. Strictly speaking. Although vast body of knowledge about control has been developed in control engineering. ' Kinetodynamic chain Desired output —• Feedback ^ Motion Controllers) Power Amplifiers ' ' . . 5. Now. In general. attached to its endeffector. As we discussed earlier. the next important issue becomes how to design the control algorithms. a robot's system under control is its kinetodynamic chain. Let q denote the actual output of a closedloop motion control system. In fact. 5. it is easy to close the loop of a motion control system. almost 90% of today's industrial control applications are still based on the popular PID control law.
a robot's motion control simply means the control of input motions at the joints of a robot's mechanism. a robot is designed to perform tasks which are usually specified in Cartesian space (task space). it is important to properly address the following issues relevant to the design of a robot's control system: • How do we specify the desired output of a closedloop control system in a robot? • How do we obtain the exact dynamic model of a robot's system under control? • What is the available sensory feedback information? And. the motion control inside a robot means the control of the input motions at the joints of a robot's mechanism. However. it is logical to specify the desired output of a robot's control system in task space. also known as the task/configuration space. And. there is strong reason to believe that the desired output of a robot's control system should be the desired values of some variables in joint space. the output motion from a robot's mechanism depends on its input motion. a task to be performed by a robot is normally specified in Cartesian space. On the other hand. We know that the motion of a robot's mechanism originates from the energy which drives the dynamic pairs. it is inevitable to encounter the situation in which the specification in task space has to be translated . the desired output of a robot's control system should be the desired values of the motion variables at the joints. we studied the statics and dynamics of an open kinetodynamic chain underlying a robot's mechanism. the desired output of a robot's control system can be specified in three spaces: • Planning of the Desired Output in Joint Space: In Chapter 4. Thus. Thus. Thus.5. the specification of the desired output of a robot's control system is not easy.Control System of Robots 267 some control schemes based on PID control law which are applicable to a robot's control system. Thus. how good is the sensory feedback? • What is the control law which guarantees a satisfactory control performance in terms of transient and steadystate responses ? 5. As the robot is a complex system.2 Planning of Desired Outputs In robotics. As a result. As we mentioned earlier. In fact.
it is not such a simple matter to do this. More appropriately. Let us consider an open kinetodynamic chain having n kinetodynamic pairs. 5. 4. The plant of a robot's closedloop control system is the kinetodynamic chain. the dynamics of speed reducers and the dynamics of electric motors. Despite this. We will study visual sensory and perception systems in Chapters 7 and 8.3.5. A control scheme in image space may be considered a special case of task space control.47. The general form of the dynamic model of an open kinemtodynamic chain with n kinetodynamic pairs is B(q). • Planning of the Desired Output in Task Space: Since a task is normally specified in Cartesian space (task space). Therefore. 5. we studied the dynamic model governing a robot's mechanism.1 Dynamics of a Robot's Mechanism In Chapter 4. it is meaningful to develop engineering approaches to imitate humanlike visuallyguided motion execution. This model is commonly called robot dynamics. But for motion variables.3 A Robot's System Dynamics Refer to Fig. it appears logical to directly specify the desired output of a robot's control system in task space. humans are skillful in performing visuallyguided activities.q)»q + G(q) = T (5.q + C(q.50) . We can say that the majority of our daily activities rely on our visual perception system. it is easy to adopt the control scheme in task space. like position.268 The Fundamentals of Robotics: Linking Perception to Action into the corresponding desired output in joint space. Strictly speaking. A hybridcontrol scheme in both joint and task spaces may be a good solution. velocity and acceleration. For some motionrelated variables (force/torque acting on an environment).137. • Planning of the Desired Output in Image Space: An image is a projection of a threedimensional (3D) space onto a twodimensional (2D) plane (image space). the dynamic model of a robot's system under control (the plant) should include the dynamics of a robot's mechanism. 5. Here. we simply point out the fact that a human vision system is qualitative (not metric). it should be called the dynamics of a robot's mechanism. Refer to Eq.5.
(NOTE: The bearings involved in a coupling can be treated as part of a speed reducer). Similarly. • T = (ri. . In other words. Thus. We need to further understand: a) the dynamics of the speed reducers and b) the dynamics of the electric motors. For the sake of simplicity. With regard to an electric motor coupled to a kinematic pair. is the torque exerted at joint i if it is a revolute joint. .3.T2. it refers to the force exerted at joint i. Otherwise. either voltage or current. rra)* is the vector of generalized forces and r.50 is not a complete dynamic model of a robot's system under control.. Otherwise. • T — (TI. QnY the vector of displacement variables at the joints. qi. A displacement with respect to a reference indicates the absolute position of a joint. we studied the couplings between dynamic and kinematic pairs.. 5.<}n)t is the vector of generalized coordinates and < is the angular displacement if joint i is a revolute joint.. qi is the linear displacement variable. we know that the controllable variables for altering a robot's motions are the variables. It accounts for the effect caused by gravitational force. T2.Control System of Robots 269 where: & • 1 = (<7i)<72> •••. This dynamic model relates joint variables q to the joint torques (or forces for prismatic joints). the joint variables will include angular (or linear) displacement and torque (or force) at the joint. •••.Tn)* the vector of torques (or forces) at the joints. The coupling devices are commonly called the speed reducers or torque amplifiers. 5.. directly related to the electrical energy applied to the electric motors. • B(q) is the inertial matrix of the open kinetodynamic chain. • G(q) is equal to a derivative vector of the potential energy of the open kinetodynamic chain.2 Dynamics of Speed Reducers In Chapter 4. From our study of a robot's control elements.. Eq.5.. q) is the Coriolis matrix which accounts for the Coriolis and Centrifugal effects. let us denote • Q = (<7i. the variables at the motor's shaft will also include angular displacement and torque.. • C(q. As a result. let us consider the case where the motion sensors are placed before the speed reducers coupled with the electric motors inside a robot. it only describes the dynamics of the open kinetodynamic chain underlying a mechanism.
.52 can be compactly written as { <? = K'1 • qm (5. . 0 = . Here. ... • Tm = (T m i..... 0 \ 0 ^ 2 ••• 0 K v = diag(k v l .kvl • q~i. kvn I Eq. •••! Imnf motor shafts.krn) V o o . .k v 2 . 0 \ 0 kr2 . . we have omitted the static frictional force of a speed reducer. ^ [T = (kri • rml • qm2. Kr can be called the reduction ratio matrix.kv2 • q2.<7m2. V 0 0 . Kr = diag(krl. the viscous friction coefficient)..fc r n y and /fc.Tm) is f q = (fc^ • 9mi.r m 2 ...53) r = Kr • rTO . •••.51) *T~i — Kri * 1~mi ^vi * Qi where kVi is the viscous damping coefficient of speed reducer i (i. .e.. krn • Tmn .. If the reduction ratio of speed reducer i coupled with electric motor i is kri.r) and (qm.Kv • q where ft'"1 is the inverse of Kr. 5... . the relationship between (q... This equation describes the dynamics of speed reducers.270 The Fundamentals of Robotics: Linking Perception to Action • 1m = (<7mi. Accordingly.. (5..kvn • <?„)*.Tmn)t the vector of angular displacements at the the vector of torques at the motor shafts.52) If we define (krl 0 . . then (5.kr2.k v n )= .. •••. j ^ • qmny .1 0 .kr2 • Tm2 .
48.Kmv • qm with (5.48 Illustration of the equivalent electronic circuit of an electric motor. q)9Qm + Gm (q) = Tm. substituting Eq. the equivalent electronic circuit describing an electric motor i looks like the one shown in Fig. 5.3 Dynamics of Electric Motors For an electric motor coupled with a kinematic pair.53 into Eq. 5. the relationship between the torque at its shaft and the voltage (or current) applied to the coils of its rotor or stator is not simple.54 describes the dynamic model of an open kinetodynamic chain in terms of the variables at the output shafts (before the speed reducers) of the electric motors.54) ' Bm(q) = K1 • B(q) • K1 Cm{q.K1 Kmv = K~l. Theoretically.5.50 yields Bm (?) •qm + Cm (q.Control System of Robots 271 Finally. electrical energy We applied to an electric motor should be equal to its mechanical energy Wm delivered at the output shaft.K\ Eq. In general. 5. The parameters relevant to an electric motor i include • Vai: input voltage applied to its coil(s). . 5. 5.C{q)q)»K^ (5.3. if the motor does not dissipate any energy.q) = Kl. v I Power Amplifier Iv* hi Rai 5=^  vtmf Rotor 4 " « 5 ^ B B ^ ^ M o t o r Shaft ^> «M[ij§fj/' HMUH1 Torque 10 Fig.55) Gm(q) = Kl.G(q).Kv. 5.
• qmi\ angular position or displacement at the motor's shaft. By definition. (5.56) Clearly. we have 7 mi = "'ti • lai .48: Vai = Rai • Iai + Lai —^ + vemf with Vemf = (5. Eq. it is not easy to predict the exact relationship between the voltage (or current) applied to an electric motor and the torque at its shaft. Rai\ resistance of its coil(s).output torque at the motor's shaft.tf] is • • • • we= [fvai* Jt. If ku is the torque constant of brushtype DC motor i. if we consider brushtype DC motor i driven by a linear power amplifier. vemj: back electromotive force (i. • Tmi'. dt. And. Lai: inductance of its coil(s). 4. By applying the principle of energy conservation. iai.57) nai • Qmi where kai is a constant. the corresponding mechanical energy output from motor i will be rtf Wm = Jt3 Tmi • qmi • dt. we can establish the following equation with regard to the equivalent electronic circuit as shown in Fig. we studied the relationship between current and torque in a brushtype DC motor (i. In Chapter 4. the induced voltage when a coil rotates inside a magnetic field). we have '*/ /•'/ / Vai • Iai dt= Jts Tmi • q'mi • dt.32).e.e. electrical energy applied to motor i within time interval [ts. 5.272 The Fundamentals of Robotics: Linking Perception to Action Iai: input current applied to its coil(s). However.
we have Hi where Vd is the control voltage signal applied to electric motor i. let us assume that the linear power amplifier for motor i is modelled by Eq. 5.50 shows the response of motor (without inertial load) to the input voltage signal of a unitsine function (i.59.59 is a simplified model which describes the dynamics of an electric motor.58 yields Vci = ^Jrmi kmi \ _ "" + Tmi.54 and Eq.58) Now.9 Refer to Eq. 5. The combination of Eq. Eq. The application of the above equation to Eq. And. Tmi is the timeconstant of electric motor i and kmi is its voltagetotorque gain. Then. 5. Let kmi = 1. This control voltage signal comes from a digital motion controller.Rl (5.59 fully describes the dynamics of a robot's system under control.^) dt ) feti»(fli+fl2) Ra. 5. 5. We can see that the motor responds to the input voltage signal in a timely manner.2 (Nm/volt) and Tmi = 5 (Ohms/Henrys). K"ii J™ Rai In fact. 5. we will obtain Vai = — • rmi + —.59) with { . 5.Control System of Robots 273 By applying the above equation to Eq. The (voltagetotorque) transfer function of the motor will be Tmijs) Vci(s) 1. Example 5.49 shows the response of a motor (without inertial load) to a unitstep voltage signal.2 l + 5s' Fig.28.. oooooooooooooooooo .e.• ——. 5. 5. Fig. sin(t)).57 and omitting back electromotive force vemf. (5.
if the position variable is the only variable which is measurable.^ 0. velocity. 5. On the other hand. The nature of the measured variables inside a closedloop control system determines the nature of the desired output.4 " Output torque ' / ... if the sensor's noise is negligible. if a sensor's output s(t) is the angular position at .1  : g! I I I I I I I I I 0 2 4 6 8 10 12 14 16 18 20 Fig. 1 1 — 1 :   • •: 0. it is always attached to the endeffector link. and acceleration are mathematically related to each other. a motion sensor can be placed either before or after a speed reducer.5  °.47..5. we can only specify the desired value for position.2 . one measurement may be sufficient. 5. As we discussed earlier.8 0. Since position.49 Transient response to the input voltage signal of a unit step function. For example. Sensory feedback is indispensable in a closedloop control system.: r ——i : 1 1 ./ : • • • : 0.7 0. velocity.4 Sensory Feedback Refer to Fig. For example. . 5.274 The Fundamentals of Robotics: Linking Perception to Action 1 I 1 1 1 JN : 0. if the velocity variable is also measurable.. both the desired values for position and velocity can be specified. acceleration and force/torque. As for the force/toque sensor.3  0. The horizontal axis is the time axis with the unit in seconds. For motion control. the feedback measurements may include position.6 Input voltage signal .
_0. the feedback sensors should not have any noise. 5. a motor's shaft. almost 100% of the noise caused by a feedback sensor will appear in the output.\ ^ / / I / \ _ / 1• I . The horizontal axis is the time axis with the unit in seconds. The transfer function of a feedback sensor will appear at the denominator of the closedloop transfer function. Ideally.50 Transient response to the input voltage signal of a unit sine function (i. Refer to Eq. a feedback sensor should not exhibit any dynamic behavior (i.15.2.e. This means that the output of a closedloop control is very sensitive to the noise of feedback sensor. O t u: up t torque \ . This means that the dynamics of a feedback sensor will slow down the response of the closedloop control system. 5. Vci(t) = sia(t). 5. • • • : \ \ _1 I 0 I 2 IV / 4 I 6 I 8 I k Z I 10 12 I 14 I 16 V/ I 18 20 Fig. In order to achieve better control performance.e.17. the corresponding angular velocity and acceleration will be f «(«) = ^ Refer to Eq.6  I • • • • / . . no transient response) and should behave like a unity gain channel. l\_/1 t I  0. If the control gain is very large.Control System of Robots 275 f\ Input voltage signal AA /A '.
the plant). b) the controller (i. it is called a MIMO system. there are many available control laws. Gc(s) the transfer function of the control law or algorithm and H(s) the transfer function of the sensing device. Sometimes. 5. Vel^citj 1 —W ' Forfe/tdrque *. the dynamics of a closedloop control system not only depends on the dynamics of the system under control but also the dynamics of the control law or algorithm. control law or algorithm). VcW d q Q) I j£>\ ^ ^ rtfy\} r ^—^ M °' i °" c °" tt °»< !r i System under Control :Gp(s) TV I *J Dynamics of p .e.' ' 1 ' Position . "cW . As shown in Fig.51. By properly designing control law Gc(s).60 should not render the final system unstable.5 Control Algorithms and Performances A closedloop control system consists of three parts: a) the system under control (i. a control system may just have a single input and a single output (SISO).60) where Gp(s) is the transfer function of the system under control. we will be able to build a closedloop control system with satisfactory performance in terms of transient and steadystate responses. In this case.e. •. ' I Electric Motors K Dyrfcsof — p Robot s Electromechanical  L System ' . it is called an SISO system. 5. a good control law is judged by its achievable absolute and relative stabilities: • Absolute Stability: The presence of the control law's dynamics Gc(s) in Eq. Some require knowledge about the exact dynamic model of a system under control and some do not. In control engineering. A convenient way to test a control . 1 ^ I Torque vector 1 h . Clearly. If a control system has multiple input and multiple output (MIMO).276 The Fundamentals of Robotics: Linking Perception to Action 5. the transfer function of a robot's closedloop control system can be generically written as (5. and c) the feedback sensor(s). "TT q(t) I 1 H(s) \* < Fig. 5. .5.51 Generic block diagram of a robot's closedloop control system. When a control objective is given.
5.61) (NOTE: Assume that the error signal e(t) is a vector). However.1 PID Control Laws (5. (5.Control System of Robots 277 system's absolute stability is by the Lyapunov stability criterion.64) Despite the large number of available control laws studied in control engineering. For example. A good control law should minimize the following cost function: L{t)= {et{t)*P»e(t)}»dt (5. the cost function to be minimized will be L(t)= f {et(t)*P*e(t) + V*(t)»Q*Vc(t)}»dt Jo where Q is another constant positivedefinite matrix. A system under control with a given control law will be asymptotically stable as long as the following condition holds: V(e(t)) = fi'(t) .5. the transient response depends on the dynamics of error signal e{t). e(t) < 0. 5. • Relative Stability: Relative stability refers to a control system's transient response.63) Jo where P is a constant positivedefinite matrix. The main reasons for this include: (1) It has a clearlydefined control objective as a PID control law's transfer function can be simply expressed as Gc(s) =kp. this is a sufficient. we can define a Lyapunov Function as follows: V{e{t)) = \et{t). In this case. (5. Vi > 0.e{t).65) . the most popular control law in industry is still the PID control law or algorithm. (5.(l + —L. If the control effort is directly measured by control signal Vc from a digital controller.+Td»s). a good control law should be able to achieve a desired outcome with minimum effort. but not necessary condition).62) (NOTE: For a stable system.
66) when t > ts. Based on experience. ts can be specified as the expected settling time of the system under control. 5.51.61 and automatically tune the parameters of a PID control law in order to guarantee the following conditions: \V(e(*))—0 (5 . (4) It is able to respond stably to the predictive signal (i. However.e. 5. (3) It is able to respond accurately and with a better steadystate (i.1 shows one set of empirical rules. and derivative timeconstant Td. Table 5. (7) It is easy to implement a PID control law on a microprocessor or microcontroller (see Eq. Table 5. (6) There are many methods of manually or automatically tuning proportional control gain kp. due to the integral law acting on past signals).e.1 Empirical rules for the tuning of a PID control law Parameters kp increases Ti decreases Td increases L(t) in Eq. (5) It does not require knowledge of the dynamic model of a system under control. Even though the PID control law is widelyused in industry. one may also come up with some empirical rules to support the automatic tuning of PID control laws. there is no theoretical proof to guarantee the stability of a system under its control. 5. this is not a critical drawback as a PID control law is usually implemented on a digital computer. the derivative law). integral timeconstant Tj. due to the proportional law acting on the present signal).278 The Fundamentals of Robotics: Linking Perception to Action (2) It is able to respond quickly (i.e. Thus.10 Refer to Fig. 5. It is not difficult to define a Lyapunov Function similar to Eq.5). The plant's transfer function is .63 decreases decreases increases Stability decreases decreases increases Example 5.
15 20 QL 0 .2___J^*e — ol 0 .52 shows the responses of the closedloop control system and the L function in Eq.e. 5. 15 1 20 ol 0 . We can see that the value of L decreases along with the stability (i.51 • . The horizontal axis is the time axis with the unit in seconds. 5 .63 when kp increases.6 „ 2 1. oscillation disappears).Control System of Robots 279 and a PID control law (i.4 1.6 0. 15 20 1. . 5. a robot is a typical example of the variableconfiguration system. We can see that the value of L increases along with the stability (i.^ La e vl u 0. = oo and Td = 0.63 when Td increases. 5.50.5. •••••••ooooooooooo 5. 5.63 when Ti decreases. 5.65) is used. Fig.6 0. 10 Fig.2 Kp=90 I L value _ _ .5. _ _ ^ .8 ^ ^_^~~~~"^ ^ ^ ^ ^ 0. more oscillation). Eq. Thus. 1. 5. . Fig.e. 1. nonlinear and timevarying system. 5 . 5.e.41 • • • 1 1. more oscillation). 10 . Clearly.4 o 2 0.4 f . Kp=6O . Kp=30 A .4.2 1 . 90 and 120 while T. 5 . 15 1 20 0 I Kp=120 L value L 0 . depending on q and Jacobian matrix J). a monolithic control law will not efficiently control a variableconfiguration system. A Output 08 IV^W . Fig.53 shows the responses of the closedloop control system and the L function in Eq. 10 . This .54 shows the responses of the closedloop control system and the L function in Eq. A robot is intrinsically a MIMO. 5 . 1 1. 10 . 5. The dynamics of a robot's mechanism is configuration dependent (i. 5.An alternative approach is the variablestructure control methodology. i 0. 60. 0. We can see that the value of L decreases along with the stability (i. .e.e.2 Variable Structure Control Refer to Eq.2 I 1 .52 Transient response when fep=30.
8 0. idea originated in Russia.6 04 Ti=5 ^ 1.68) . . The choice of a feedback control law for a specific type of system configurations is decided by a switching function s(t). We can extend the philosophy of variablestructure control to the case of multiple PID control scheme. 5 .8 0. 0. 5 .4i . 0 5 10 15 20 Fig. We can construct the following switching function: ft s{t)= Jo {a [et(t)*e(t)] + {la) [el(t) • e{t)]} • dt (5.4.8 0.2 ol 0 . • Ti=15 • • 1 1. and x(t) is a n x 1 state variable vector of the system under control.6 Lvalue 0 4 Lvaue 0. 15 1 20 ol 0 . In a sliding mode control scheme. 1. The philosophy behind the variablestructure control scheme is to use a set of feedback control laws to command a variable configuration system having unmodelled dynamics and uncertainty. 5. the dynamics of which is unknown or unmodelled. The horizontal axis is the time axis with the unit in seconds. switching function s(t) typically looks like s{t) = S • x{t) (5.8 0.2 1 1.53 Transient response when 7i=15. . 5 and 1 while kp = 30 and Td = 0. 10. _ 0 5 10 — 15 20 o. 10 .4 ( > • • 1. The switching function monitors the system's behaviors based on measurements related to the system's state variables.2 I . 10 .2 1 ™° ft " ' Outeut h ^ V 0.4 0.2 ^ 0.6 0.6 0.2 1 k 0. The wellknown sliding mode control is a typical example of variable structure control.67) where S is a m x n matrix (called a switching function matrix).280 The Fundamentals of Robotics: Linking Perception to Action 1. 15 20 1. .4 I Ti=1 Output o. Let e(t) be the error vector of a closedloop control system. and was proposed by Emelyanov and Barbashin in the early 1960s.4 r ' ' • 1 1.
Because of the nature of a robot which is programmed to perform repetitive tasks/actions.Control System of Robots 281 1f » 0.61 u ^ Lvalue ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ~ 04 0. a control law which tends to quickly converge the error signal to zero will be selected. a predefined PID control law which minimizes s(t) is selected as the current control law. Therefore.2 while kv = 30 and Tt = oo. the generalized switching function s(t) and the decision rules can be evaluated and finetuned through realtime interaction between a robot and its environment.1.^ ^ ^ . The horizontal axis is the time axis with the unit in seconds.54 Transient response when Td=0. By adjusting the coefficient a.8 ' Td=0. For example. 5 . 15 1 20 ol 0 .8 I 0.05 ' Output • 1 1 I ' Td=0.6 0.2 calf 0.15 0.15 and 0. we can balance the convergence to zero by both error signal e(t) and its derivative..4 0.2 ^ ^ ^ ^ ^ ^ Lvalue .6 0. ^ ^"^ gl 0 . But this will be at the expense of causing unacceptable oscillation and overshoot.2 ^ ^ Lvalue ^"^ . the idea of variablestructure control for .^ : Td=0.4 0.4 0. We call Eq. 15 20 1  • • ' 1 1 1 • ' ' 1 Td=0.2 0 5 10 15 20 0 5 10 15 20 Fig. when choosing a = 0. On the other hand. a decision rule can be established based on the output of the generalized switching function s(t).2 . a is a weighting coefficient which conditions the selection of a control law. 5. a control law which tends to minimize the derivative of the error signal will be selected. The PID control laws.8 r 0. like a robot. 10 .68 a generalized switching function. 5. ^ ^ . 10 .1 ' ' 1 0. where 0 < a < l . 5 . but the response time may be unacceptably long.05. 0. this interaction is intrinsically repetitive. 0. The decision rule will determine the appropriate control law when a variableconfiguration system enters a new type of configurations. When choosing a = 1.6 0. Given a set of possible PID control laws predefined for a variableconfiguration system.^ ^ L v a l u g ^ ^ ^ ^ 0. This will produce a very smooth output.
5. based on the output of the generalized switching function. will be PID2.56. Then.25 and a set of four PID control laws are predefined as follows: Control Law PID1 PID2 PID3 PID4  In the generalized switching function (i.25 0. s + 1 And the responses of the closedloop control system and the outputs of the s(t) function in Eq. we set a = 0. Let us further change the plant's transfer function to G P ^ = s2 + 4 .68) corresponding to the four predefined PID control laws.55 shows the responses of the closedloop control system and the outputs of function s(t) (i. Example 5. Fig. Now. 5. We set a = 4 and 6 = 1 .51. 5.25 0.e. 5.7. 5. s + 4 The responses of the closedloop control system.s + b kp Tj 4 3~~ 8 3 12 3 16 I 3  Td 0.68 corresponding to the four predefined PID control laws.25 0.68). The plant's transfer function becomes GP(S) = .2 + 4 . Eq. 5. based on generalized switching function s(t). let us modify the coefficients of the plant's transfer function to simulate the variation in the plant's dynamics. we can see that the selected control law.11 Refer to Fig. From Fig. .282 The Fundamentals of Robotics: Linking Perception to Action a humanoid robot matches the spirit of developmental principles underlying the mental and physical development (e. decisionmaking and actiontaking capabilities) of artificial systems or machines. We can see that the selected control law. 5. Eq.g. The plant's transfer function is G^s) = f + a. 5.55.e. will be PID1. and the outputs of the s(t) function in Eq. S + 1 and choose a unitstep function as input.68 corresponding to the four predefined PID control laws are shown in Fig. let us have the plant's transfer function be as follows: G P ^ = S2 + 2 .
5.2 W 0 : [ 0 . • . we can also specify . And the output motion of a robot's mechanism depends on its input motion (i. We can see that the selected control law. 5. 5. The horizontal axis is the time axis with the unit in seconds. the control scheme at the lowest level of a robot's motion control system is called jointspace control. The motion performed by a robot is the output motion of its mechanism.6. oooooooooooooooooo 5. Sometimes.21 o' 0 • 5 • 10 "' .6 JointSpace Control A robot is a machine which is skilled at executing motions. based on the output of the generalized switching function. 15 20 1.e.21 ol 0 : . 5 . 15 1 20 0.2 1. 4 0.55 Transient responses and the outputs of generalized switching function s(t) corresponding to the four PID control laws when a = 2 and 6 = 1 .57.4i • .8 ir^ • . Thus.5. will be PID3. This explains why a robot's motion control often takes place at the joint level. Therefore. 15 1 20 "f 0. the desired output should be desired joint variable vector qd and its derivative qd. 5.8 ir—i / 0. .2 Control law PID2 0. • 1 1. : • 1 1.21 0> 0 • 5 : • 15 20 10 Fig. 10 . are shown in Fig. 10 . the control of the output motion of a robot's mechanism is equivalent to the control of its input motion.41 • • • 1 Control law: PID3 » Control law: PID4 0. 5 • •» • . the motions at the joints).4j • • • 1 Control law: PID1 1.1 Planning of Desired Outputs In a jointspace control scheme.4 i .Control System of Robots 283 1.
.4j 1. Then. 5. . . studied in Chapter 3. • • • 0E . .2 tf : . desired acceleration vector qd at the joints.4 1. The solution to the second issue is the robot's motion kinematics. 20 1 0 1. .69) . 1 20 1 0' 0 1. one must address two issues: • How do we plan motion in terms of path and trajectory in task space? • How do we transform planned motions in task space into the corresponding ones in joint space? We will address the first issue in Chapter 9. in planning the desired output for a robot's control scheme in joint space. . A robot's input is the tasks/actions and the output is the executed motions by the robot. As a result.2 . . 10 . .r~~ 0. 5.Pi. a task/action is specified in Cartesian space (task space).4. : .8 / r~~~ : • •»(')  0.6 I r~~~~—'••— / • S(1) 1 0.56 Transient responses and the outputs of generalized switching function s(t) corresponding to the four PID control laws when a = 4 and 6 = 1 . Control law: PID4 ip——:— ^ 0. 15 1 20 0. . The horizontal axis is the time axis with the unit in seconds.2 0' 0 • 5 • 10 : 0 • 15 20 Fig. . . : . .2 . Control law: PID3 . 5 . Control law: PID2 1. Normally. 1 1. qd) of the joints is related to desired velocity vector Pf of the endeffector's frame as follows: qd = J^.6 < 0. 5 .4/ 0. Refer to Fig..14.284 The Fundamentals of Robotics: Linking Perception to Action 14. . .21 • • : • • • . .. Let J denote the Jacobian matrix of an open kinetodynamic chain and j t the pseudoinverse of J. . (5.2 Control law: PID1 . 15 . . 5 i 10 . : o.8 0. the desired velocity vector (i.2 1 0. 15 . • : • 1 1.e.. 10 .21 0! : . . .
S ( " 0.q) + Kmv] »g m + Gm(g) = r m . Thus. j 1i 0. = .4 I I 0...2J : Control law: PID3 . 15 1 20 0! . 15 20 0 0 1i O./<.. sm o.70 becomes Bm(q)»gm + \Cm(q.70) Vc = K1*(Tm+Tm.6 V r  —.. 5 10 . •• • : ...8 ^_j^__ =— . (5.tm) with Km = diag{kmi.km2. 15 20 Fig.71) In practice.Shf o.6 / _ '•• .2 :Control law: PID4 : 0' 0 5 • 10 ' 15 1 20 o' 0 • 5 • 10 . ^______J ... : o... .6 /*:^~~lm • / • . we can ignore the dynamics of electric motors and simply treat Vc = rm.. 10 . 5. 5.59.. let us consider that electric motors inside a robot can be operated in the torquecontrolled mode instead of the voltagecontrolled mode.2 H ! : :•• Control law: PID2 ol . this means that there is an unmodelled dynamics with regard to a robot's system under control. q) + Kmv\ • qm + Gm(q) = r m (5. .41 0... Then. 5.2 Control law: PID1 tf 0. The dynamics of a robot's system under control is { Bm{q) • Qm + [Cm(q. 5.. 0...4 0. 5 .. Tmn}. I 1. Eq. 5. 0. 080.6. .5.54 and Eq.Tm2.2 System Dynamics Refer to Eq.Control System of Robots 285 1 1 1 . kmn} and Tm = diag{Tmi.57 Transient responses and the outputs of generalized switching function s(t) corresponding to the four PID control laws when a = 4 and 6 = 4.4 I I 0. For the sake of simplicity.8.e L__J {• • • s'" • : 0. The horizontal axis is the time axis with the unit in seconds.
q^»{Bm2 By applying this result and Cm)*qm(im*Kmv*qm+(im»{TmGmKl*em) (5.6.3 Centralized Joint Control Scheme The purpose of a control law is to determine control signal r m which will act on the system under control in order to achieve the desired control objective and performance. the error vector of the robot's closedloop control system will be em(t) =qdmor. So. simply Cm = qm(t) (5.em) = 0 if and only if qm = 0 and em = 0.286 The Fundamentals of Robotics: Linking Perception to Action 5. Then. it is always a positive term. Differentiating Eq. 5. (5. (5. q) and Gm = Gm(q). em) > 0 Vgm ^ 0 and em ± 0. And. Kx • em. From the dynamic model of a robot's electromechanical system studied in Chapter 4.73) Here.73 describes the kinetic energy of the robot.5. denoted by g m . V(qm. Let the desired output be specified as the desired joint angularposition vector. Cm = Cm{q. substituting Eq. we obtain V =. 5. qm + 1 elm .74. If we choose K\ to be a positivedefinite matrix. The first term in Eq. 5. we can also prove that vt • N • v — 0 . we can prove that matrix Bm — 2 Cm is a skewsymmetric matrix. we define the following control objective function (Lyapunov Function): V = V(qm.72.75) where Bm = Bm(q). then we have V(qm. the control objective is to find a suitable control law which makes V converge to zero.74) From Eq. For skewsymmetric matrix N. em) = \<L* Bm(q) . we have em(t) = — q(t)m.5. Subsequently.72) Qm 9m where qm(t) is the measurement of the actual joint angle vector.71 into Eq.5.73 with respect to time yields V = qln • Bm(q) •qm + qtm» Bm(q) • qm + elm • Kx • em.
A good way to improve response time is to increase the norm of V. This implies that the convergence to zero by the system under control is quite slow because the norm of V in Eq. 5.76 is negative for all qm ^ 0. This can be achieved by introducing term q^ • K2 • qm —tfm• K2 • 5m. (5.75 becomes V = q^ • Kmv •qm + qim* (rm ~Gm~Kx» em). From Eq.72). which is equal to zero. (5. 5. 5. we can see that the second term is the proportional control law of a PID control scheme and the third term is the derivative control law.81. then the coefficients of the viscous frictions are quite small. The control law expressed in Eq. then we can guarantee the following conditions (V<0 Vqm^0 < [V = 0 Vgm = 0. the first term in Eq.81 is known as the PD control with gravity compensation. Consequently. 5. 5. If we choose the control law to be Tm = Gm + K1»em.77) According to Lyapunov's stability criterion. And Eq. If a robot's mechanical system is energy efficient. Moreover.qm if we choose the control law to be rm = Gm + Kx • em + K2 • em (5. This result implies that the first term in Eq.76 will become V = qtm.79) (578) (5. Then. 5. the first term in Eq.75 is equal to zero.Kmv*qm. Kmv is a positivedefinite matrix.78. 5.80) where K2 is a positivedefinite matrix and em = —qm (see Eq. Hence.77 will make the closedloop control system asymptotically stable because of the conditions guaranteed by Eq. we obtain V=qtm*(Kmv+K2). 5.81) (5. 5.76. the control law in Eq. into Eq.76) Kmv is a diagonal matrix with the elements being the coefficients of the viscous frictions. 5. In this way. Eq. If a robot .77 is to compensate for the gravitational force acting on the robot. 5.79 is small. 5.Control System of Robots 287 for any vector v.
5.71 yields Bmi •q'm + Nm(q. n] and Vj G [1. G{q) and Kv are divided by the square of the corresponding elements in reductionratio matrix Kr.81 is simply a PD control because Gm = 0.q). n] and Vj G [1. C(q. From Eq. q) = Bm2 • q'm + [Cm(q. we purposely split matrix B(q) into the sum of two matrices. the elements of matrices B(q). As a result.. q) + Kmv] • qm + Gm(q). q) = Tm with Nm(q. 5..83) [ Smite) = { ^ > VtG[l.55.84) .n]}. 5.5. we know that Bm(q) = K~x • B(q) • K~x.288 The Fundamentals of Robotics: Linking Perception to Action is operating in outer space where there is no gravity. Vi e [l.82) fBite) = {6«te). It can be represented as follows: B{q) = {hj(q).55. the reduction ratios) (5.50 is an n x n inertial matrix. Vi?j.. i = 1.n]} 1 B2{q) = {bijiq).6.. substituting Eq. the control law in Eq.4 Independent Joint Control Scheme We know that matrix B(q) in Eq. When the elements (i.2.e.83 into Eq. Vi / j . Vi 6 [1. n]} where {fcri. Now. n}}. as shown below: B(q) = B^q) + B2{q) with (5. Then.n]} [ Bm2(?) = { ^ . 5. 5. 5. 5. As we can see from Eq. Bm(q) can be expressed as Bm(q) = Bml(q) + Bm2{q) with (5.Vi£ [1. n} are the diagonal elements of reductionratio matrix Kr. Vte[l.n] and Vj e [l.
2. (5.86 indicates that the links of an open kinetodynamic chain are decoupled.88) Differentiating Eq. 5. 5. as ^Mqmi = rmi Vie[l.n). elements inside matrices Bm(q).87) Imi ~ Qmi where qmi(t) is the measurement of the actual angular position of motor i. we choose a proportional control law as follows: Tmi = kpi • emi. q) in Eq. we can omit matrix Nm(q. we propose an alternative solution. 5. Now. Assume that the desired output is q^..88 with respect to time gives V = Qmi • Qmi + emi • emi.84. 5. Based on this.87.89) From Eq. let us define the following Lyapunov Function V = V(qml.91) (5. As a result. Gm(q) and Kmv will be negligible.. Then. Let us consider the motioncontrol loop for joint variable qmi (i = 1. n). Thus.85) Since Bmi(q) is a diagonal matrix.. we obtain Ik2 \ V = qmi [—•Tmi~ emA .85 can be equivalently written Eq.q).n]. the error signal will be emi{t) = qL .Control System of Robots 289 in Kr are big (> 100).. (5. we can independently design a control law for the motioncontrol loop of each joint variable qmi (i = 1. Eq. . (5..eml) = i (<&• + e2mi). By default. we can use the PID control law for each control loop. Now. 5.86 into Eq.86) (5.89.2.. (5. we have emi = —qmi. Here. 5. a robot's system dynamics is simplified to Bml»q'mTm.90) .. Cm(q. 5. By applying this result and substituting Eq.qmi(t) or simply emi = (5.
5. 5.eml)>Q As a can see j3 > 0.kpilj. tasks or actions are specified in task space.92) Let us define the switching function s(. it is natural to consider whether the motion control loop can be formed from the sensory feedback in task space.91 satisfies the following conditions: (V <0 [V = 0 Vqmi ^ 0 and Vemi ^ 0 (5. Thus. One interesting advantage of this control scheme is that parameters {a. 5.. 5.94.90 yields I' k2 V = qmi»eml l^. Therefore. Clearly.9o) And we schedule the proportional control gain according to the following decision rules: { I a»f if s{qmi. 5. 5. Eq. 5. 5.) as follows s\Qmiy^mi) = Qmi • ^mi (5.2.91. for any set of admissible values of (a.n). this method is in line with the spirit of the developmental principles for the physical development of an artificial system. Eq.94 describe a variablestructure control (similar to the sliding mode control) which guarantees the asymptotical stability of the closedloop control system for joint variable qmi (t = l. the control law in Eq. like a humanoid robot. (3) can be learned during a robot's realtime interaction with its environment. \ (5.. .92 and Eq./3).94) (1/?)•££ iis{qmi.7 TaskSpace Control As we mentioned earlier. 5.95) Veml = 0.emt) < 0 (5.5.91 into Eq. where we> 1 and from Eq.290 The Fundamentals of Robotics: Linking Perception to Action Then the question is: What should proportional gain kpi be in order to make the closedloop control system asymptotically stable? Substituting Eq..93 and Eq..
let us define a Lyaponuv function as follows: V = V{qm.97) (596) where K\ is a positive definite matrix. When a robot performs an unconstrained motion. all vectors are referenced to the base frame (frame 0) of an open kinetodynamic chain.Control System of Robots 291 5. A robot's input is a task or action which is normally accomplished by a sequence of ordered motions. The linear and angular velocity vectors can be determined if the trajectory of motion in task space can be planned according to the specification of a task or action. the desired motion in task space can be denned by position and/or velocity vectors: (P^. £* consists of two vectors: a) desired force vector f£ and b) desired torque vector r^. Let Pe — {x.7.e] (5. where Pe{t) is the measurement of the actual posture of the endeffector's frame. When a robot is not in direct contact with its environment while performing a task or action. its motion is called unconstrained motion.5. In general. Pf). If a robot's motion is constrained by direct contact with its environment. By default.P. Pe is replaced by its differential variation APe.PS) or simply e = Pf . it is called constrained motion. Otherwise. Now. y. it is necessary to define one extra desired output which will be desired force/torque vector £f. z.7. uix^uiy^ujz)1 denote the angular velocity vector of the endeffector's frame in an open kinetodynamic chain. This can be obtained by the robot's vision system or from forward kinematics if joint angle vector qm is measured as feedback. If only path is planned. The first three elements of Pe form the linear velocity vector of the endeffector's frame while the last three elements of Pe form its angular velocity vector. . Then the error vector will be e(t) = Pf .1 Planning of Desired Outputs Refer to Fig. 5. 5. More detail on motion planning will be studied in Chapter 9.5.14.2 Posture Control in Task Space Let P£ denote the desired posture of the endeffector's frame in an open kinetodynamic chain.em)=l[qtm*Bm(q)*qm + et*K1.
we obtain V = $n* Kmv ^qm + qln [rm . the control law in Eq.J* • K~x • K x .Cm = Cm(q. 5.99.qm. we choose the control law to be Tm = Gm + Jt*K1»K1»eK2*qm where K2 is a positivedefinite matrix. q) and Gm(q) = Gm.2 Cm) • qm = 0 and substituting Eq. From our study of robot kinematics. Next. (5. 5.98) By applying the result e = —Pe (see Eq. if the robot is operating in outer space.100 into Eq. we obtain y= g <itm9(Bm2Cm2Kmv)*qm + qtm[TmGrn}pt*K1*e (5. By applying the result <4 • (Bm . 5.71 into Eq. e] .103) (5. q).100) (NOTE: Kr is a reductionratio matrix which is a diagonal matrix).101 becomes V = qtm» [Kmv + K2] • qm. (5. 5.292 The Fundamentals of Robotics: Linking Perception to Action Differentiating Eq. Then Eq.96) and substituting Eq.Gm . 5. then Gm = 0 and the .104) This simply means that the system under control is asymptotically stable.102) (5.101) Clearly. The transpose of the above equation is P^ = qU*Jt*K1.97 with respect to time gives V = qtm» Bm{q) •qm + ^tin* Bm{q) •qm + et»Kl. 5. 5. Accordingly. (5. 5. (5. we know that Pe = J*q=J*Kl.98. let J denote the Jacobian matrix of the open kinetodynamic chain. However.e.99) where Bm = Bm(q.102 guarantees the following conditions: (V<0 Vqm^0 \ [V = O Vgm = 0.
7.Ut) where £e is the measurement of the actual contact force/torque vector. From our study of robot statics. 5.106) (5. we know that torque vector r of the joints required to compensate for or maintain the contact force/torque at the endeffector is T = Jt»£e or rm = K1 • J* . from Eq. 5. Alternatively. the torques of the actuators will not only contribute to the dynamic behaviors for trajectory tracking but also to the desired contact . This is a proportional control law. 5. the control law for forcecompliance control can simply be Tm = K^1*Jt*& + K1»e{t) (5.3 Force Compliance Control Assume that the endeffector of an open kinetodynamic chain has a force/torque sensor to measure the actual contact force/torque between the endeffector's frame and its environment.5. In this case. If desired force/torque vector £f can be specified. (NOTE: r = Kr • r m ) . we may expect the endeffector's frame in an open kinetodynamic chain to follow a predefined trajectory and also to maintain a predefined contact force and torque with its environment. having position and velocity feedback). the error signal will be e(*) = Ce . If the control objective is purely to maintain desired contact force and torque (£.Control System of Robots 293 control law in Eq.102 is a PD control (i.105. one can also choose a PI or PID control scheme. 5.7.5. £e where J is the Jacobian matrix of the open kinetodynamic chain and Kr is the reductionratio matrix (a diagonal matrix).4 Hybrid Force and Trajectory Control For tasks to be performed by a robot.105) where K\ is a proportional control gain matrix which is a positivedefinite matrix.e.
{qdm .de (5.112) .£. 5.108) Since Bm is the inertialmatrix which has the inverse. one can simply suggest the following control law: Tm = Bm»yd + Nm + K. In this case.110 around the desired output with y = yd + Kx . qm. the dynamic model of the robot's system under control will become Bm • qm + [Cm + Kmv] •qm + Gm + Jtrn»£e = Tm (5.108 can be equivalently written as Tm = Bm*y with y = qm + B1. we make use of the inverse dynamics control scheme to derive a suitable control law. Cm = Cm(q. Gm = Gm{q) and Jm = J • K~l.Jm.111) and K is a proportional control gain matrix. Let us denote Nm = [Cm + Kmv) »qm + Gm.(ydy) with yd = qdm + B^. q). q^ and q'm. Accordingly. 5. (5. this control law does not consider desired output qm and qm.107 becomes Bm*qm + Nm + Jm*Ze = Tm. 5.110) + Nm (5. The idea underlying inversedynamics control is to linearize Eq. {qdm . we assume that the matrices in the dynamic model are known. (5.107) where Bm = Bm{q).qm) + K3 • ( £ . Assume that the desired output for hybrid force and trajectory control are: £f.294 The Fundamentals of Robotics: Linking Perception to Action force and torque. Eq.109) Under the inversedynamics control scheme. However. Then Eq.Ze.Jtm.qm) + K2 . Now.&) (5.
K2. £e is measured by the force/torque sensor mounted onto the endeffector (5.5 Impedance Control Let us examine. 5. 5.&) = 0 (5. to behave like a massspringdamper system. Dd.113) y = Vd + K1.112 and Eq. Sd) are the desired mass.5.115) . (£ed . the error or deviation of the actual posture of the endeffector's frame from its nominal or desired posture) • £e is the contact force/torque between the endeffector's frame and the environment. Jlm + K3). ( £ . These matrices should be positivedefinite matrices. In fact. In other words. Clearly.e.7. we may want the error dynamics of a control law applied to an open kinetodynamic chain. damping and stiffness matrices • ep — Pd — Pe (i. the control law under inversedynamics control scheme is { Tm = Bm • y + Nm (5.Control System of Robots 295 where yd = g^ . That is: (qd ~ qm) + K1*(qdmqm)+K2.£e). the error dynamics of a control law is expected to be governed by Md*ep + Dd»ep + Sd»ep=£e where: • (Md.K3) are the proportional control gain matrices. In this way.f S ^ 1 •.110. in the previous example. 5.(qi qm) + K2 . or simply (qdm qm) + ( B " 1 . the system under control will reach dynamic equilibrium when Qm = qL qm = qm and £e = £?./*•£? (the desired output of y) and (Ki. For some applications. the dynamics of error due to linearizing with Eq.qm) + K3 . {qdm .112. 5. error caused by linearizing is the difference between Eq.114) eq + K2»eq + K1»eq + ( S " 1 • J ^ + K3) • e( = 0 where eq = q^ — qm and e^ = (£ ~ £e • The system under control will be at the dynamic equilibrium when errors eq and e^ vanish. such as a robot's limb.
5.118) (5. In other words.108. (5. We can express torque vector r m as Tm = Bm*y with y = qmRefer to Eq. Eq. (qdm .121) Now. 5.116) And the desired output of y when the endeffector's frame is following a desired trajectory is yd = J^'{PiJm*qdm)(5.120 will result in the expression of Eq.122 and Eq. behave like a massspringdamper system while executing its own task (i. As a result. 5. the impedance control becomes a stiffness control. 5.296 The Fundamentals of Robotics: Linking Perception to Action A control scheme which can achieve this objective is called an impedance control. such as a robot's limb.qm)) (5.123) will make the open kinetodynamic chain.ep + Sd. Differentiating Eq. 5.122) The difference between Eq. following a trajectory). the control law { Tm = Bm • y + Nm + J^ • £e y = yd + J1 • { ( M ^ 1 • [Dd . Refer to Eq.117) + Nm + Jtm*£e (5. (5. By choosing Md = {0} and Dd = {0}. 5. 5.120) (5.115.ep + £e] + j m .ep + £e] + j m • {qdm .120 around desired output yd in the following way: V = Vd + tf • { ( M t 1 • [Dd *ep + Sd.qm)) .H9) (5. 5. .100. We have Pe = Jm»qm where Jm = J • K~x.117 becomes y = J^l*{PeJm*qm).e. we linearize Eq.118 with respect to time gives qm = J1 • (Pe ~Jm* qm).
8 ImageSpace Control It is easy to see that the majority of our daily activities are visually guided. a humanoid robot has two arms with hands. This is known as the eyeinhand configuration. Without a visualsensory system. it is necessary to have a solid understanding of how a robot's vision system works in order to understand the imagespace control scheme. another important configuration is the eyetohand. and two legs with feet. 5. However.Control System of Robots 297 5.5. Consequently. The behaviors from a headeye system are called headeye coordination If we model the mechanism underlying the headneckbody as an open kinetodynamic chain. we will study various schemes of limbeye coordination for a humanoid robot. we stated that the intelligence is a measurable characteristic of both human and artificial system. or eyetofeet) configuration which describes the typical setup of a robotic handeye coordination. it is interesting to study how a robot's vision system interacts with its motion control system. a robotic headeye coordination system can be formed by putting one or two cameras at the endeffector of an arm manipulator. Intelligence is quantitatively measurable and is inversely proportional to the effort spent on the achievement of a predefined goal/target. As a result.1 Scenarios A humanoid robot may look like a human being. In Chapter 1. let us briefly highlight some important issues related to it. Accordingly. For the purpose of cultivating interest in the study of imagespace control scheme.8. the combination of the binocular vision and the headneckbody mechanism will form a typical headeye system. we would have tremendous difficulty in adapting ourselves to the realworld for any physical activity. Naturally. it makes sense to save the study of imagespace control until the end of this book.5. Therefore. . On the other hand. In Chapter 9. The motion of these limbs can be guided by visual information in image space of a humanoid robot's binocular vision system. one way to measure the intelligence level of an artificial system is to test its ability to perform visuallyguided activities.
The second notable benefit is that the imagespace control scheme is also applicable to the case in which desired output (pf .8. Naturally. and design a control law which will command the motion of an open kinetodynamic chain so that the actual projections (Pe. velocity and acceleration onto an image plane are measurable. In this way.pf . its velocity Pe and its acceleration Pe. (NOTE: By default. Conceptually. . In order words. 5. we will understand that an image is the projection of a threedimensional space onto a twodimensional image plane.Pe. It is a crucial step towards autonomy of a humanoid robot.5.298 The Fundamentals of Robotics: Linking Perception to Action 5. one way of developing an imagespace control scheme is to imitate human vision.8. Obviously. all vectors are expressed in a common reference frame). This extension will result in interesting solutions to the problem of visuallyguided trajectory or target followed by a vehicle or humanoid robot. the desired output is specified in terms of the posture P e . the objective of the imagespace control scheme is to specify the desired output in image space. pe and pe the projections of an endeffector frame's posture. After we study the robot's visual sensory system in Chapter 8. not metric) with regard to geometric quantities such as shapes and dimensions.3 Methodologies It is not surprising that human vision is qualitative (i. Nonetheless.Pe) converge to the desired ones. we can consider that these projections of an endeffector frame's posture. the first benefit of the imagespace control scheme is the ability to automate the planning of desired output based on visual sensory information.e. velocity. human beings are skilled at performing visuallyguided activities. If we denote pe.pf) can be the projections of any object's posture.5. we can develop solutions which do not require any metric 3D reconstruction of the threedimensional objects or scene. and acceleration onto an image plane. velocity and acceleration in a scene.2 Objectives For a taskspace control scheme. we know that the desired output is the motion at the endeffector's frame in an open kinetodynamic chain.
However. we will be able to develop solutions which can outperform human beings in visuallyguided activities. It must have a set of elements which act together. As a result. .e. and d) the ability to withstand all kinds of uncertainty and variations. there is no proof about a PID control law's absolute stability. And a robot should be treated as a combination of mechanical.Control System of Robots 299 Alternatively. namely: a) the control elements (in the forward channel of a closed loop) and b) the sensing elements (in the feedback channel of a closed loop). Nevertheless. electromechanical. We leaned that the most important concerns regarding a closedloop control system are: a) absolute stability (i. Accordingly. It is almost impossible to precisely and completely model the dynamics of a robot system. a robot's motion control must be operated in a closedloop feedback manner. c) the ability to achieve fast responses. the steadystate response) and b) relative stability (i. By definition. An automaticfeedback control system has many advantages over an openloop control system. b) the resistance to noise and disturbance. We will study these methodologies further. the transient response). Therefore. We learned that the notable advantages include: a) the ability to achieve the desired output. And there are many techniques for the manual or automatic tuning of a PID control law.e. a powerful mathematical tool is the Lyapunov method which is useful for the design of asymptotically stable control systems. 5. There are many mathematical tools for the design and analysis of a closedloop control system. From a systems point of view. we can also explore the quantitative aspect of a robot's visualperception system. Although a vast body of knowledge about control has been developed. the most popular control scheme in industry is still the PID control law.6 Summary A robot is intrinsically a nonlinear. multiple variable system with a variableconfiguration. a system is a set of coherent elements which act together to achieve common objectives and outcomes. and control systems at this point. regardless of unmodelled dynamics of a system under control. in Chapter 9. we learned that a closedloop control system must have two sets of elements. it is easy to believe that a closedloop control system is not simply a control law or algorithm.
we emphasized the study of the popular optical encoders. While we briefly discussed safety sensors. force/torque sensors. As a result. Thus. we studied the issues of: a) how to plan the desired output. taskspace control and imagespace control. and d) how to design suitable control algorithms. If we use electric motors. The design of suitable control algorithms is not a simple matter. As for sensing elements. energy applied to a robot will come from the electrical domain.300 The Fundamentals of Robotics: Linking Perception to Action A robot is able to execute motion because of the energy applied to the joints of its underlying mechanism. we studied the generic sensing principle. c) how to obtain a timely sensor feedback. This is not desirable for a humanoid robot because it is dangerous to perform a homing procedure in a nonindustrial environment. Nevertheless. we learned that the PID control and variablestructure control are two popular control methodologies. There are naturally concerns about: a) how we regulate the amount of electrical energy released by an electric motor to its corresponding joint. Its primary role is to run the control algorithm(s) and interface with the sensors and actuators which are inside the closedloop control system. we advocate the use of absolute optical encoders in the design of a humanoid robot's control system. b) how to possibly establish a dynamic model of a robot's system under control. The most important control element is clearly the digital motioncontroller and the control algorithm(s) running on it. These require the robot to perform a homing procedure before it is ready to operate. A digital motioncontroller is part of a robot's information system. typical control schemes in robotics include jointspace control. and b) how we alter the direction of motion of an electric motor (or its corresponding joint). we also studied the working principles behind the power drive which regulates an electric motor's velocity. . Subsequently. we studied the working principles behind the power switch and logic sequence generator for the purpose of altering the electric motor's direction of motions. and tactile sensors. there are various control methodologies. Almost all the motion sensors found in industrial robots are incremental optical encoders. As a result of the complexity of a robot system. And. Regarding the design of a robot's control system.
how many logic states should it have? Explain why an incremental optical encoder allows us to measure the absolute angular position. How do you alter an electric motor's direction of motion? How do you regulate an electric motor's velocity? Explain why safety is an important issue in the development of a humanoid robot? What are the considerations for the appropriate placement of a motion sensor? If an absolute optical encoder has to measure 360° with an accuracy of 0. What are the control elements inside a robot's control system? What are the sensing elements inside a robot's control system? Explain the difference between linear power drive and PWM power drive. (2) What is the transient response of a system? And.7 Exercises (1) Explain why a closedloop control system is better than an openloop control system. what is the steadystate response of a system? (3) Explain the physical meaning of the absolute stability of a closedloop control system.Control System of Robots 301 5. (4) Explain the advantages of using Laplace Transform in the study of a closedloop control system.01°. Why should a force/torque sensor's mechanism be stiff? . (5) The transfer function of a closedloop control system is Y(s) R(s) = K (sl)(s + lY Is this a stable system? (6) Let us use a PID control law to control a system with a transfer function of Gp(s) = (s + l)(s + 2)(s + 3) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) Use Simulink of MATLAB to tune the parameters of the PID control law.
Sliding Mode Control: Theory and Applications. (24) Draw a block diagram of the closedloop control system which makes use of the control law in Eq. L.45b. Cambridge University Press. K. K. Why are they important to a robot's motion execution? (19) Explain the reasons behind the popularity of the PID control law used in the majority of automated equipment in industry. PID Controllers: Theory. 5. (6) Sciavicco. (2) Edwards.123. Design and Tuning. (1989) Fundamentals for Control of Robotic Manipulators. 5. P. Modelling and Control of Robot Manipulator. (3) Horowitz. North Carolina. John Wiley and Sons. (25) Draw a block diagram of the closedloop control system which makes use of the control law in Eq. Hagglund (1995). Siciliano (1996). 5.113. Instrument Society of America. and W. (26) Draw a block diagram of the closedloop control system which makes use of the control law in Eq. Prove that the input and output relationship is (18) Explain the difference between a force/torque sensor and a tactile sensor. and T. Taylor and Francis Ltd. 5. (1997). . Spurgeon (1998). (4) Koivo. Hill (1989). J. and S. K.8 Bibliography (1) Astrom.91. (5) Ogata.102. Research Triangle Park. Modern Control Engineering. McGrawHill. (20) What is the variablestructure control? (21) Explain the differences between the joint space control scheme and the task space control scheme? (22) Draw a block diagram of the closedloop control system which makes use of the control law in Eq. and B. A.5. PrenticeHall. J. The Art of Electronics. (23) Draw a block diagram of the closedloop control system which makes use of the control law in Eq.302 The Fundamentals of Robotics: Linking Perception to Action (17) Refer to Fig. C.81. 5. 5.
When a system is capable of performing realtime actions. Today's computer has not yet reached the ability level of the human brain. a humanoid robot's information system must be a realtime system. Without intelligence. When a system performs multiple actions. realtime also means the simultaneity of actions undertaken by the system. It is best to use a cluster of computers to form a distributed system. Thus. a body will have limited usage. Although it is an artificial system. This ability is an important function of a humanoid robot as well. an action performed by a system must meet a stipulated deadline. we can compare a humanoid robot's information system with the human brain. Thus. The predictability of a system's actions is very important as it is the only way to ensure the system's deterministic behaviors. intelligence will not develop. a humanoid robot's information system should be a distributed system supported by a realtime network in order to ensure predictable behaviors.1 Introduction Without a body. this means its actions are predictable. Artificial intelligence is synonymous with computational intelligence because all the mental and physical activities of a humanoid robot rely on dataprocessing carried out on digital computer(s). By definition.Chapter 6 Information System of Robots 6. Consequently. a humanoid robot's intelligence should develop. it is not realistic to rely on a single computer to handle the multiple functions of a humanoid robot. 303 . realtime means that an action is performed by a system within a specific time interval. The human brain is undeniably powerful enough to control and coordinate our physical and mental activities in realtime. Therefore. As a result.
e.2 Imitating the Brain Humans represent the perfect embodiment of mind and body. In general. sensing. nerve fibers). Fig. b) data storage. and c) the body's sensory systems. however. sensorymotor coordination.304 The Fundamentals of Robotics: Linking Perception to Action In this Chapter. the growth of a person's mental and physical capabilities is primarily influenced by interaction with the environment and society. and d) an innate neural system. sensing. and sensoryintegration. Therefore. an information system consists of four parts: a) data processing. From an engineering point of view. is that the cerebral structure is highly parallel with the predefined divisions of the brain's neural system. this is very similar to the formation of the cluster of basic feedback control loops: sensorcontrolleractuator. perception. and not a great deal about it is understood.1 is a simple illustration of the relationship between the human brain and human body. c) data interfacing and d) data communication. analysis. perception. From an engineering point of view. and decisionmaking for other chapters. in this chapter. we can consider the embodiment of mind and body as elaborate mapping among: a) the body's kinetodynamic systems. And. We will reserve discussion of the computational principles (or algorithms) underlying the dataprocessing related to control. and decisionmaking functions. An infant is born with: a) an innate biomechanical system. emotion. is undoubtedly a complicated process which physically resides in the brain. including the decisionmaking system (see Chapter 9). we will emphasize the fundamentals of these topics. b) the brain. which consists of consciousness and intelligence. Throughout the mental and physical development of the body and mind. we will study the basics of the information system underlying a humanoid robot's control. c) innate sensory systems. perception. The human brain is an extremely complex system. Because of the complex structure of the brain and body. A person's mind and intelligence are developed through the whole period of his/her growth. Now. What is certain. language. The mind. memorization. the divisions of the brain's neural system are mapped in a highly sophisticated manner to the sensory systems and motor systems along predefined pathways (i. we are able to perform all kinds of mental and physical activities. the question is: What type of innate information system should a . knowledge. 6. b) an innate biokinetodynamical system (the muscles and body's biomechanism). such as thought. 6.
we need to improve the available information and sensory systems. it is easy to form a cluster of networked computers to imitate the parallel structure of the human brain's neural system.r. . Today. ' ' Kg wU ^B ^^M H — ' "i ' • • * _i i i .Information System of Robots 305 Hi Body _ • ^^H ^H E • Hr. However. in order to support the multiple behaviors performed by an artificial body. However. humanoid robot have if we want its intelligence to grow through realtime interaction with the environment and society? It is not easy to give a clear answer to this question because today's information and sensory systems are not yet comparable to those of humans.i. . L ' "factor> _ BB ^s ' ' Sense ^m ^ Fig. These loops are called actors or agents.i Consciousness (^Representation"^) : i ! j B ' 9N SKB BG£ '_' s ' I^^B ^ H fl^B  H ^^B d Language ~~^ Division 1 of Neural System Division 2 of Neural System Division 2 of Neural System 1 ^^v _ ' . If we want future humanoid robots to have humanlike mental and physical capabilities. the challenge in robotics is whether it is possible to develop an innate "operating system" and "software architecture" that can automatically map sensory outputs to actuators' inputs in order to autonomously form sensorcontrolleractuator loops for specific behaviors. 6.  ^^H ^ I i b1^"'.ustator) _ "* n e s BK Upg ! Bst J53 9^< I! I I IJ ^H I .iin — [k > j _  H • T '" HI ! ~ ~ I i I J .1 A simple illustration of the relationship between the human brain and human body. it is necessary to have the "executive process" (or "operating system") which would harmoniously coordinate the multiple computational tasks running on a cluster of networked microprocessors. . i  .
2 A proposed actorbased framework for the embodiment of artificial mind and artificial body. 6.M.306 The Fundamentals of Robotics: Linking Perception to Action 6.S. The realtime interaction between a humanoid robot and its environment is crucial to the proper development of its physical and mental abilities. It is the embodiment of an artificial mind (a complex process for behavior development and learning). 'vriMif ^ v _£• 'VOulpul/' cw I L'Ml'J'^' Fig.2. as shown in Fig. \UII.3 Imitating the Mind A humanoid robot is not simply a combination of mechanics and control.1 Autonomous Actors or Agents As we mentioned already.. the human brain is a complex neural system consisting of about 100 billion neurons. . and an artificial body. 6. The architecture of the human brain's neural system is highly parallel with wellorganized partitions ded . Mind and Body &^k 0 \*\ i ii iin.3. 6.' •*  • j  1 . This embodiment enables a humanoid robot to interact with its environment in a timely manner.
a mental function refers to a specific type of mental activity which affects the internal representations of the real. we mean all actions carried out by the sensorymotor systems embedded in the body's mechanism (e. as shown in Fig. Basically. body.2. and skills. and enriches the language and internal representations. The outcome from a mental actor may serve three purposes: • To enrich language.3.2 Autonomous Behaviors An actor or agent inside an information system is an autonomous. explore. And. limbs. For a sensorguided actiontaking system. a distinct and indispensable component of an autonomous system. as shown in Fig. Naturally. we can consider a perceptiondecisionaction loop as a mental actor or agent. Accordingly. 6. a physical actor will mainly be guided by output from the belief system. By physical functions (or abilities). In this way.2. knowledge. acts upon the belief system. external world. or understand certain new things?) • Statedriven behavior. On the other hand. there is no doubt that the best control structure is the closedfeedback loop. (Is it rewarding to undertake certain actions?) • Curiositydriven behavior. (Is it interesting and exciting to do. Inside an information system. • To update belief system. It does not require reprogramming. (How do we achieve a goal or new activities?) . an actor or agent is an autonomous computational process or task. the basic mechanism that drives a mental activity is the wellknown perceptiondecisionaction loop. 6. Similarly.g. A mental activity is intrinsically conditioned by one or multiple sensory input(s). computational process/task because it requires no reprogramming. a mental actor responds to input from the sensory systems. we can treat a standalone sensor as a degenerated physical actor. we can consider a sensingcontrol loop as a physical actor or agent. 6.Information System of Robots 307 icated to various physical and mental functions. Note the following beliefguided behaviors: • Valuedriven behavior. (Is it different to perform the same tasks under different mental and physiological states?) • Expectationdriven behavior. we can treat a standalone perception as a degenerated mental actor. neck/head etc). Hence. • To enrich internal representation.
it will update the belief system to make "safety" the highest priority. serves as a catalyst to stimulate the birth. (NOTE: Whether or not a mental or physical actor is undesirable is determined by the belief system). experience. Obviously. . or to suppress undesirable mental and physical actors. so that the robot will quickly run away. we can treat the complex interaction inside a humanoid robot's information system as an artificial mind system. a behavior is a sequence of ordered actions performed by a group of interacting mental and/or physical actors. where knowledge is gained from the environment. Therefore.308 The Fundamentals of Robotics: Linking Perception to Action A physical actor's output will act on the artificial body. By definition. the creation of a new mental or physical actor will increase the number of possible configurations of interacting mental and physical actors.2 Behavior development and learning refer to activities dedicated to the formation. This in turn. and others. Accordingly. Just as the human mind describes the dynamic characteristics of the human brain. The formation. an artificial mind characterizes the dynamics of an information system. Interaction in the society. Based on the concept of mental and physical actors. the activation of a behavior depends on the configuration of the interacting mental and/or physical actors. we can attempt to define behavior development and learning in the following way: Definition 6. and suppression of mental and physical actors. if a mental actor's output is to escape a dangerous situation. enhancement and suppression of mental and physical actors cannot occur without input from the society and environment. Thus. enhancement. enhancement and suppression of mental and physical actors. For example. will trigger a set of physical actors which work together on the artificial body. We can attempt to define artificial mind as follows: Definition 6. one way to change a humanoid robot's behavior is to form new mental and physical actors to enhance existing mental and physical actors.1 An artificial mind is a complex process of behavior development as the result of interaction among the mental and physical actors in a body's information system. The mechanism underlying this complex process is behavior development and learning.
The large number of degrees of freedom implies the necessity to adopt a distributed sensing and control scheme for motion execution and coordination. motion control is critical to the usefulness of a humanoid robot. a robot's motion control involves sensing elements for feedback. Obviously. all principles and algorithms underlying visual perception are computationdependent. There are many computational principles and algorithms for motion planning. a humanoid robot's information system must support the following computational tasks: • Distributed Motion Control: The majority of a robot's activities are centered on motion as an action is a sequence of ordered motions. visual perception is indispensable. 6. Therefore. one critical issue in the design of a robot's control system is how to plan the desired motions. • Motion Perception: One important control scheme for a robot's motion execution is imagespace control. .2. From a dataprocessing point of view.Information System of Robots 309 6. • Motion Planning: As we studied in Chapter 5.g. And. a robot must be a realtime system with predictable behaviors. Therefore. which is in proximity to the sensors. this communication must rely on a network (e. The mental and physical actors of a robot's information system are intrinsically computationdependent because they process sensory feedback and act accordingly. As we discussed earlier. All the sensory feedback and control algorithms run on the digital motion controllers which are part of a robot's information system. it is critical for a robot's information system to support realtime communication among its internal motioncontrol subsystems. • Communication: A humanoid robot's body includes a complex mechanical system with many degrees of freedom. and controllers. As we studied in Chapter 5. There are two paradigms of imagespace control: a) visualservo control and b) visuallyguided planning & control.3. control elements for actiontaking. all of which must run on a digital computing platform. fieldbus). actuators. Intrinsically. and control algorithm(s) for decisionmaking.3 Computational Tasks Refer to Fig. a robot is a multipleinput system with a variable configuration. As a result.
.310 The Fundamentals of Robotics: Linking Perception to Action • Interrobot Communication: Obviously. • Behavior Development and Learning: A distinct feature of the humanoid robot. there can be no embodiment of artificial mind and humanlike mechanism (body). but also friendly. And. It is not inconceivable for humanoid robots to coexist with humans (like an automobile does). Ideally. 6. the principles and algorithms of behavior development and learning for a humanoid robot's physical and mental development are computationdependent. we know that the information system is crucial to the development of a humanoid robot. is its physical and mental selfdevelopment. a humanoid robot should adapt itself to the ways in which we communicate and interact. it is desirable to develop humanoid robots which can not only perform tasks as individuals. the humanoid robot has a wider range of applications outside the factory. a humanoid robot must not only be safe. a robot's information system must be able to communicate with other equipment as well as with other robots. A computer is a machine which can perform computations (dataprocessing) and can also handle data storage and communication. Without an information system. In fact. but also work as a group. we widely use digital computers for dataprocessing and communication. Just as a human experiences incremental development from infancy to adulthood. a humanoid robot's information system should be able to support the computations required for realtime manrobot interaction based on auditory and visual sensory data. Hence. Now.4 DataProcessing Hardware Nowadays. Obviously. as opposed to the industrial robot. Therefore. • Manrobot Interaction: A humanoid robot's activities should not be confined to a factory. It is interesting to note that the computer was first built during the Second World War in the 1940s. a humanoid robot should also be equipped with a selfdevelopment mechanism (yet to be discovered) so that it is able to physically and mentally grow through realtime interaction with its environment.
1 Digital Computers The first generation of computers were based on analog circuits. expensive. Often. If we plot the logic signal versus the time. 6. 0V).4. the term digital simply means "discrete state". As shown in Fig. a pulse signal includes two transitions: a) the transition from logic low to logic high. digital computers used vacuum tubes to construct the inner logic circuits. Here. a digital computer can be defined as follows: Definition 6. it is easy to understand why a circuit is called a logic circuit if it takes a pulse waveform as input and produces a different (or similar) pulse waveform as output. With the rapid development of solidstate circuit technology (especially with the invention of transistors at Bell Laboratory in 1948). ^ — One cycle —•> ' ' /! KJ LJ LJ —AH Rising Edge V= Falling Edge .3 A digital computer is a machine that undertakes pulsesynchronized calculations through a sequence of ordered instructions called a program. This explains why the old digital computers were bulky. +5V). 6. and fragile.g. while the "1" state can be represented by a high voltage (e.—• Time Fig.3. The simplest form of discrete state is the binary state which only has two values: 0 or 1. We also call a binary state a logic signal because we can associate the binary values "0" and "1" with the logic values "true" and "false". . Formally.3. we obtain a pulse waveform which is similar to the one shown in Fig. This also explains why the advent of a pulse circuit marked the birth of the digital computer. 6. microprocessors became a reality in the early 1970s. In electronics.g. In the early days. we call "1" the logic high and "0" the logic low. Now. Digital computer emerged in the early 1950s with the advent of digital circuits (circuits that manipulate pulses). the "0" state can be represented by a low voltage (e.Information System of Robots 311 6.3 Pulse waveform. and b) the transition from logic high .
• The second rising edge: This indicates the end of the current cycle and the start of a new cycle. The former is called the rising edge and the latter is called the falling edge. A formal definition of a microprocessor can be stated as follows: Definition 6.e.312 The Fundamentals of Robotics: Linking Perception to Action to logic low. a reference pulse waveform looks like the one shown in Fig. For one cycle. In order to make the results of a computation predictable. there are three events (logic transitions): • The first rising edge: This indicates the start of the current cycle. a digital computer can be made of microcircuits with transistors which is also called integrated circuit. Like a digital computer. To do this.4.2 Microprocessors With the advent of solidstate circuits and the invention of the transistor. it is necessary to use a reference pulse waveform. a computer is capable of performing all kinds of calculations.4 A microprocessor is a digital computer which consists of a single integrated circuit (IC. all the logic transitions inside a digital computer must be synchronized along the time axis. 6. the digital signals) in order to perform calculations. A device that generates a regular pulse waveform of reference is called the system clock. a new active device. • The falling edge: This indicates the halfcycle.2. has become a dominant product in the computer industry. all operations inside a digital computer are called cyclebycycle operations. 6. Because of these transitions. a computer can be treated as a machine that controls logic transitions for the performance of intended calculations. 6. In fact. As a result. all operations inside a digital computer are synchronized by these three events (logic transitions). a microprocessor is a dataprocessing device that manipulates pulse waveforms (i. called a microprocessor. Thus. In general.4. for short). Therefore.1 Memory If we look at the following simple expression: a=b+c .3.
Before this arithmetic operation can take place. the basic component in a memory circuit is a logic unit which only stores a bit. A bit is the basic unit in a memory and is also called a memory cell. the memory circuit is called a readonly memory. • If it is possible to perform both read and write operations while the memory circuit is "on". a dataprocessing device must have a circuit unit to store data. • If it is possible to perform the write operation only once while there is no limit for the read operation. If a memory is not divided into a set of sublists. In a digital computer. Memory is a special type of logic circuit. By putting eight such units together. the memory circuit is called a randomaccess memory. or ROM. and the data is simply a number.Information System of Robots 313 we easily understand that a is the result of the addition of the two operands b and c. while output from a memory circuit is called read. . a memory is composed of a list of (memory) locations and each location has a unique address. Memory is organized in the form of a list of bytes or memory locations. Hence. After this arithmetic operation. we must also know where to store the resulting value a. Clearly. • Input to a memory circuit is called write. Each memory location must have an index that indicates its address on the list. A logic variable that can hold the logic value of either "1" or "0" is called a bit. • If it is possible to perform the write operation repeatedly only with special equipment or procedures which normally undertake erasing and programming operations. In summary. the index of a memory location inside the (global) list is called a memory address. A memory's key features include: • Its logic output is equal to its logic input. the memory circuit is called an erasable and programmable readonly memory or EPROM. • The input and output of a memory circuit do not need to be synchronized. we form a memory location which can store a byte (8 bits). or RAM. we must know where the values of b and c are stored. this circuit unit is called memory.
65. As for exponent E. • The subsequent n\ bits for representing the exponent of base number 2.4. A byte can represent an integer number from 0 to 255.9255127 . In other words. (NOTE: 255 = 28 — 1. any real number with a decimal can be written as the ratio between two integer numbers. For any string of n bits. 2 8 — 1) and its minimum value will be 0. For a string of n bits. 64. If we denote F a floatingpoint number. the maximum value represented by a string of 8 bits).923. E is an integer representing the exponent by n\ bits. 63. or LSB. many physical quantities are represented with real numbers.2. or MSB.2 Number Representation A byte is a string of 8 bits that are normally labelled as 67. The maximum value for mantissa M will be 2 23 and its minimum value will be 0. the last bit (60) is called the leastsignificant bit. b2.e.314 The Fundamentals of Robotics: Linking Perception to Action 6. represented by a string of n bits. If we use MSB to represent the sign (0 for positive and 1 for negative). a string of n bits can represent an integer number within the interval: In real life. we have: F=(l)MSB*M*2E'Eo (6. Assume that an exponent is represented by 8 bits ( m = 8) after the MSB. 61 and b0. Accordingly. the maximum floatingpoint number will be: 71 7 _ / 1^0. Eo = 127 (27 — 1).1) where M is an integer representing the mantissa by 712 bits. and the mantissa is represented by the last 23 bits ( n 2 = 23).g. the number 2). the maximum value will be 255 (i. and Eo = 2™1~1 — 1. Example 6. Then. • The rest of n2 (ri2 = n — ni — 1) bits for representing the mantissa. A common way to represent a real number with a decimal is to use a floatingpoint number because any real number can be written as the product of a mantissa and the exponent of a base number (e.1 We represent a floatingpoint number with a string of 32 bits. a floatingpoint number is represented as follows: • The MSB for representing the sign. while the first bit (67) is called the mostsignificant bit. b§.
2 For a sine function y(x) = sin(x). addition. There is an obvious need to connect these two devices. This explains why the heart of a computer is made of the Arithmetic & Logic Unit or ALU. we know that the memory is a place to store data (i. an ALU can be treated as a logic device which has at least two input (the operands) and one output (the result).g. a set of wires that connects electronic logic devices is called a 6MS. In other . O 23 . multiplication. The ALU is a logic circuitry which is capable of performing the arithmetic and logic operations (e. 6. and Boolean logic). Example 6. The memory and the ALU are two distinct logic circuits in a digital computer. all the arithmetic operations of subtraction. subtraction. and multiplication can also be fulfilled with a sequence of ordered additions at certain precision. O255127 oooooooooooooooooo 6. division and multiplication) at certain precision.4. we know that any real and differentiable function can be expanded into a corresponding Taylor series at certain precision. In computer engineering.e. the only way to connect them is to use wires that can conduct pulsedelectrical signals.Information System of Robots 315 And the minimum floatingpoint number will be: P —( Ti 1 . the simplest way to design a dataprocessing unit is to construct a circuit that can perform addition on top of some Boolean logic operations. In general. (NOTE: For long distances. division. This fact explains why all calculations can be realized with a sequence of ordered arithmetic operations (addition.2. it is advantageous using optic fiber to carry pulsedsignals).4.2. In theory.4 Bus So far. its corresponding Taylor series is: x3 x5 (l)n*x2n+1 oooooooooooooooooo As a result.3 Arithmetic Logic Unit (ALU) In mathematics. Since these are electronic logic devices. the numbers) and the ALU is an engine that manipulates data to produce results. for short.
316 The Fundamentals of Robotics: Linking Perception to Action words. memory is a list of memory locations. 6. a digital computer performs calculations by executing a sequence of orderedinstructions called a program. we seldom refer to this bus because it is more convenient to directly refer to the controlsignal lines themselves. Step 3: Perform arithmetic and logic operations. for the sake of clarity. we need to transport the contents (data) as well as their addresses. Step 2: Fetch the operand(s). As we mentioned earlier. 6.5 Generic Architecture of Computers Now. The Von Neumann computer architecture. In practice.2. it is easy to understand that the procedure for an ALU to execute a program should look like the following: • • • • • Step 1: Fetch an instruction. In order to communicate or interface with a memory device. a bus is a vehicle for the transportation of logic signals (or numbers) across electronic logic devices. In order to work properly. A common way to do this is to have a bus dedicated to data (contents) and a bus dedicated to addresses. all digital computers are based on the famous architecture invented by the American mathematician. these are omitted in the illustration. The former is called a data bus and the latter is called an address bus. it is necessary to have controlsignal lines.4a. The next question is: What does a digital computer look like? Nowadays.4. each of which has an unique address. memory. Repeat Steps 1 to 4 until all the instructions of a program have been completed. Von Neumann. we know that a digital computer should have ALU. etc. One way to improve . computers based on the Von Neumann architecture must store the program and data on the same memory unit. Of course. Step 4: Store the result. Clearly. And. Since a computation involves at least a program and some data for processing. a digital computer requires other logic signals such as "read". it is not possible for the Von Neumannstyle computer to perform the operations of Step 1 and Step 2 in parallel. as shown in Fig. However. "write". is composed of a processor and a memory which are connected by data and address buses. and bus. As we studied earlier. The wires that carry logic signals for controlling operations inside a digital computer are grouped into a bus called a control bus.
the speed of computation is to use the Harvard architecture. b) perform arithmetic computation. The Harvard architecture is also known as the DSP architecture as all the DSPs are based on the Harvard architecture.4 The Von Neumann and Harvard computer architecture. CyclebyCycle Operations 6. 6. and c) write.6 As we discussed above. 6. the three basic actions of a processor are: a) read. developed at Harvard University. Step 3: Store the result. Since all operations in a digital computer must be synchronized in order to produce deterministic results.4b. Repeat Steps 1 to 3 until all the instructions of a program have been completed. 6. Step 2: Perform arithmetic and logic operations. a Harvardstyle computer will have at least two memories: one for programs and one for data. USA. As we can see from Fig. In fact. as shown in Fig.Information System of Robots 317 Address Processor Memory <  Data •  (a) Von Neumann Architecture Address Program Memory Data • processor Address • Data Data Memory (b) Harvard or DSP Architecture Fig.2. these three basic actions must also be synchronized. And.4b. the execution of a program will be performed in the following way: • • • • Step 1: Fetch an instruction and its operand(s). (NOTE: DSP stands for Digital Signal Processor). Thus.4. the Harvardstyle computer separates program storage from data storage. a processor's system clock is responsible for producing a regular .
Similarly.318 The Fundamentals of Robotics: Linking Perception to Action pulse waveform that serves as a reference for synchronization. the memory sends the data stored in the location indicated by the address to the data bus. • nw the number of cycles required to write the result in the memory. Let T be the cycle time of a processor's system clock. However. We denote: • nr the number of cycles required to read in one instruction and its operand(s) from the processor's memory. 6. if an instruction requires three cycles of the system clock. the processor only undertakes the computation of the instruction. • Step 3: When the second rising edge occurs. This number depends on the complexity of the instruction. • Step 3: When the second rising edge occurs. the memory latches in data and stores it in the memory location indicated by the address. (6. let us examine the timing involved in executing one instruction by a processor. In practice.2) . the processor sends the address to the address bus. the processor writes the data to the data bus. Now.4a. 6. • Step 2: When the falling edge occurs. It becomes clear that the total execution time for the completion of one instruction will be te = (nr+rii + nw)»T. During this period of time.3 and Fig. a common way to synchronize the write operation is as follows: • Step 1: When the first rising edge occurs. For example. Refer to Fig. special circuitry called a control logic unit automatically handles the allocation of cycles required by the execution of instructions supported by a processor. the processor sends the address to the address bus. • m the number of cycles for the processor's ALU to execute the instruction. then three cycles will be allocated to this instruction. the processor reads in the data from the data bus. a common way to perform arithmetic operations is to allocate a number of cycles of the reference pulse waveform to an instruction. • Step 2: When the falling edge occurs. A common way to synchronize the read operation is as follows: • Step 1: When the first rising edge occurs.
a processor requires the following registers: (1) Address register to hold the address. (2) Reduce nr and nw by increasing the number of bits that a data bus can carry at one time. This can be done in the following ways: (1) Shorten the cycle time of a processor's system clock (i. a data bus with 32 lines can carry four bytes at a time instant). 16bits. basically. the processor must have one register. (For example. For a read or write operation.Information System of Robots 319 For any given instruction. 6. a digital computer can be treated as a realtime system and can play the role of a robot's artificial brain. Therefore.2.7 Generic Architecture of a Microprocessor A processor is. As a result. called the address register. or more. (4) A programcounter register to indicate the address of a program's instruction to be subsequently executed.5: Register A register is simply a memory location having 8bits. as shown in Fig. to hold the address and one or more registers to hold the data. or a multiple of 8bits. Hence.e. a processor's responsiveness depends on the execution time te of an instruction. . (5) Index registers to hold the relative location of the sublists inside a memory if it is divided into different segments for the separate storage of the operation system. (3) One. When a processor is executing an instruction. for proper operation. programs. it is desirable to shorten the execution time of an instruction. a device that can repeatedly perform cyclebycycle operations. data and stacks etc. its corresponding execution time is a fixed number (i. When the ALU is performing computation. (2) Data registers (including accumulators). In order to improve time responsiveness.e.4. it cannot be interrupted or halted. In fact. increase the frequency of a processor's system clock). a processor should have the following basic elements. te is predictable). status registers to report the working conditions of the ALU. it is very likely that the processor will require registers to store the intermediate results. 6. (3) Reduce rii by incorporating more ALUs into a processor.
320
The Fundamentals of Robotics: Linking Perception to Action
i Processor ' < I Address Bus
I '
Memory I
> > !
j :^TF.;
j :
! , i '  i
i
,
  I , , i i
IX ;
R0M
I A ^ e s s Register (AR^J
SPRe^sler
 SyslemClock 
j I
I
i
i ! i 1 I ;
! r
1 I
^
I Control Logic Unit
j

EPROM
Operating System
I
 i i ' j I
P PC Register L _ _ ^ ^ _ _ = ^ _ _ J Index Registers  Data Registers ' ' I
i i ' ' i i
Instruction Decoder Program
1
i<^ I Data Bus
Instruction Registers
I jI
,

V>
Data

;
A L U Registers
I
'
i ;
!
\ \
\
ALU 1
/ /
/
i I
j r)
Stack
I/O registers
! ;

(NOTE: SP stands for stack pointer; PC stands for program counte
Fig. 6.5 The generic model of a microprocessor.
(6) ALU registers to hold the operands as the input to the ALU, (7) Instruction register(s) to hold the code of an instruction itself. Instruction Decoder Usually, a processor will support a set of predefined instructions which is also called an instruction set. All the instructions are compactly encoded in order to minimize storage space. When an instruction code is received by a processor after a read operation, it is necessary to have circuitry to decode it, in order to know the arithmetic or logic operation to be performed by the ALU. This circuitry is called instruction decoder. Control Logic Unit As we discussed earlier, for the proper operation of a processor, it must have a control bus to carry control signals. A typical control signal is a read or write signal which communicates between the processor and its memory.
Information System of Robots
321
On the other hand, a processor is a device which performs computations dictated by a program of ordered instructions. It is clear that a processor must handle the sequence logic of a program execution which is, in turn, synchronized by the processor's system clock. In fact, the circuitry inside a processor which generates the required controllogic signals is called a controllogic unit. Based on the above discussion, we can easily depict a generic model of a processor. Fig. 6.5 shows an example. 6.4.3 Microcontrollers
In robotics, we are interested in an information system which can serve as a robot's artificial brain. In order to do this, it is necessary for the information system to be able to interface with various other systems on a robot, or elsewhere. Consequently, a processor with a memory is not sufficient to construct a robot's artificial brain. A set of interfacing systems, commonly called I/O systems, must be incorporated into the microprocessor. For the sake of simplicity, let us consider the design solution of memorymapped I/O systems such as the processor families made by Motorola, Inc., as shown in Fig. 6.6.
I Address Bus I
•
Processor ^ Data Bus w Memory
^
= =
v
"
I/O Subsystems
A/D
D/A
Serial I/O
Parallel I/O
Programmable Timer
1
T T T T 11 T 1 '
Fig. 6.6 The generic model of a microcontroller.
In general, a microprocessor combined with a set of dedicated I/O systems is called a microcontroller. From Fig. 6.6, a microcontroller can be formally defined as follows: Definition 6.5 A microcontroller is a microprocessor with a set of I/O systems built on a single integratedcircuit or chip.
322
The Fundamentals of Robotics: Linking Perception to Action
The I/O systems commonly found on a microcontroller include: a) analogtodigital conversion system, b) digitaltoanalog system, c) serialcommunication system, d) parallelcommunication system, and e) programmabletimer system. We will study these, in further detail in the later part of this chapter. There are many types of microcontrollers on the market. Since a microcontroller is built on a single chip, products made from them are normally very cheap (less than C/S'$1OO). As microcontrollers are inexpensive, and DC motors for toys are also inexpensive, toytype robots are used more and more in places like high schools.
BBfetagyX
C  4 ''••Mii'U.V.l
Fig. 6.7 A toytype mobile manipulator developed with a microcontroller and a lowcost DC motor .
Fig. 6.7 shows an example of a mobile manipulator. This robot has an arm manipulator with four degrees of freedom (DOF). Its hand is a simple gripper with one DOF. Its mobile base has three wheels: two passive wheels and one steering/driving wheel (having two DOFs). All the motions of the joints are controlled by a microcontroller.
6.5
DataProcessing Software
Now, we know that a microprocessor is capable of executing instructions in a cyclebycycle manner. In order to make full use of the computational power of a microprocessor, it is necessary to know how to develop software which can effectively process data on a microprocessor.
Information System of Robots
323
6.5.1
Programming
The development of software requires both programming skill and knowledge of an application's domain. Here, we will study the basic concepts useful in cultivating good programming skills.
6.5.1.1 Programming Language
Usually, instructions to be executed by a microprocessor are grouped together to form an entity called a program. Different microprocessors from the same or different manufacturers like Intel, Motorola, Texas Instruments etc. usually have their own instruction sets with their own associated syntaxes. A set of instructions with an associated syntax is called the assembly language. A program written in an assembly language is known as an assembly program.
Since a microprocessor is a digital system, it can only recognize binary numbers. Therefore, each instruction must be encoded with a binary number in order for a microprocessor to interpret and execute it. As a result, an assembly program has to be translated into a series of binary numbers. This results in a series of binary numbers (codes) called a machine program. And, the set of binary codes corresponding to the instructions of an assembly language is called a machine language. It is not easy to use an assembly language because it is very different from natural language and is thus difficult to read and understand. Moreover, an assembly program written for one microprocessor cannot be recognized by another microprocessor if their instruction sets are different. This represents a serious drawback for assembly languages. Since all engineering concepts, principles and mathematical formulas can be expressed in a common natural language (English), one would expect that a microprocessor would be able to undertake and execute a program written in a natural language. However, until today, we have not yet reached that stage. One way to attempt to bridge the gap between the assembly language of a microprocessor and natural language is to use a special programming language. The obvious advantage to introducing such a special programming language is that it enables programs to operate despite different microprocessors. In engineering, the most popular and highlevel programming language is the C or C++ programming language. There are many good textbooks which describe the Cprogramming language in detail. Here, we will focus on some of the important aspects of the Cprogramming language from a
324
The Fundamentals of Robotics: Linking Perception to Action
programmer's viewpoint. 6.5.1.2 Programming Environment
Today's computers are all equipped with a special program which runs continuously when the power is "on". This special program primarily manages: a) the workload of the processor, b) the usage of memory, c) the data interfacing and communication with peripheral devices (including I/O systems), and d) the execution of application programs. This special program is commonly known as an operating system (or OS for short). Refer to Fig. 6.5. The OS of a microprocessor is normally loaded into the RAM after the power is switched on. With this addition to the computing hardware, the development of an application program becomes easy and productive. Fig. 6.8 illustrates an integratedcomputing platform for software development. With the support of an OS on a computing platform, many existing programs can be shared among users and programmers. There is no need to "reinvent the wheel" if a program exists already. Application Programs or Tools
^r
Peripheral Device Drivers
j
«
^ Operating System
j
Computing Hardware (Processor, I/O Systems, Storage System, etc)
Fig. 6.8 Integratedcomputing platform for software development.
For a programmer, the commonlyshared programming tools (i.e. existing proven programs) include: • An editor for writing and editing a program file, • A compiler for translating programs in highlevel programming languages into assembly programs specific to microprocessors, • A linker for incorporating shared resources (i.e. libraries) into the assembly program in order to produce an executable file, • A debugger for examining the execution of a developed program (e.g.
Information System of Robots
325
to check for errors and analyze the causes). 6.5.1.3 Program/Data Files
In computer engineering, a file is a record which stores a string of bits or bytes that may belong to a program or data. A formal definition can be stated as follows: Definition 6.6 A file is a physical entity which encapsulates the grouping of bits or bytes into a single record. The content (i.e. bits or bytes) of a file is usually encoded in a specific way to ease file exchange and content display. Since a file's contents is in the form of bits or bytes, it can be directly stored into a memory or any storage device. Inside a memory, the two basic parameters concerning a file are: a) its address (i.e. the address of the location storing the first byte of its content) and b) its length (i.e. the number of bytes it contains ). It is much more difficult to implement a file system on a computer. However, the usage of a file is a simple matter. This is because the basic functions for file manipulation are normally provided by a highlevel programming language. 6.5.1.4 Programming Procedure
A microprocessor is capable of performing computation through the execution of programs. However, a microprocessor or computing system that is capable of directly translating human intention into a corresponding program does not exist. In order to use a microprocessor to perform data processing, a user or programmer must manually do the programming. A formal definition of programming can be stated as follows: Definition 6.7 Programming is the process of translating a computational task or behavior from the mind of a user (or programmer) into a program. Programming is a highly creative activity. Fig. 6.9 shows a generic process of programming. Generally speaking, programming with a programming language involves the following steps: • Step 1: Translate a computational task from the mind of a programmer into a corresponding program written in a (highlevel) programming Ian
326
The Fundamentals of Robotics: Linking Perception to Action
Programmer
i
„ ,. Editor 1
I 1
/ •(
Program \ ) V^_j<iles^/
Compiler I
1
•(
Object
)
1—r
'
V^Files^y
( Library > V <i ) V_ files y ^1 Fig. 6.9
J * I
T
. . Linker
I 1
f Executable^ w ) V Ti \___Files_^//
A generic programming procedure.
guage. The outcome of this step is a developed program. For ease in maintenance and editing, it is wise to store a large program into a set of files. This is because it is easier to read and edit a file having a hundred lines than to read and edit a file having several hundred or thousand lines. • Step 2: Compile the program written in a (highlevel) programming language into a corresponding machine program encoded in the machine language supported by the microprocessor in use. The outcome of this step is the machine program stored in the object file. Usually, each program file will be compiled into a corresponding object file. • Step 3: Put the shareable computational functions into a file called a library. The outcome of this step is some library files. • Step 4: Produce the executable program by linking the machine program with the shared computational functions from the libraries. The outcome of this step is the final executable program encoded in machine language and stored in a file called the executable file. The use and sharing of libraries make programming a highly productive activity. A library is a record of the shareable computational functions encoded in a machine language. A formal definition of a library can be stated as follows: Definition 6.8 A library is a collection of shared program modules or
Information System of Robots
327
computational functions compiled in a machine language and stored in a file. While programming is a highly creative and productive activity, a good programmer must observe certain guidelines: • Readability of Developed Programs: As we mentioned earlier, a programming language is not as comprehensive as a natural language. Still, as much as possible, it is important to make a program readable by a human. For example, instead of naming "x" the variable of a motion sensor, one can name it "velocity" directly. • Reusability of Developed Programs: Programming is usually a team effort. Many generic computational functions, like the trigonometric functions, are shared among programmers. Therefore, one should always design a program by making it as modular as possible. In this way, generic modules can be shared with other programmers. • Interoperability of Developed Programs: Normally, a program written in a highlevel programming language can be operated across different computing platforms. However, if a program makes intense usage of shared libraries, one must take special care as the shared libraries may be different across different computing platforms.
6.5.1.5 The Basics of CProgramming Language
A good programming language must fulfill two objectives: • For the sake of simplicity and efficiency, it should be similar to the machine language. In other words, it should allow programmers to explore the characteristics of the microprocessor as directly as possible. • For ease in readability and programming, it should be similar to a natural language. In this way, a programmer will have less difficulty in translating computational tasks into corresponding programs. A programming language that satisfies the above two objectives does not exist. So far, the best programming language for engineering simulation and computation is the Cprogramming language. The Cprogramming language was invented in the late 1970s. In the early 1980s, this language incorporated classes, which implement the concept of objectoriented programming. This has resulted in the C++ programming language. The
328
The Fundamentals of Robotics: Linking Perception to Action
Cprogramming language is efficient because it is similar to the machine language. The popular operating system UNIX is written in the Cprogramming language. Data Types in C The Cprogramming language supports basic, primitive data types such as: • • • • • char: an 8bit integer, int: a 16bit integer, long: a 32bit integer, float: a 32bit floatingpoint number (i.e. a real number), double: a 64bit floatingpoint number (i.e. a real number).
If a variable always takes positive integer numbers, the keyword unsigned can be placed in front of a data type for integers. For example, unsigned char indicates an unsigned 8bit integer. An unsigned 8bit integer varies from 0 to 255. With the various data types, one can easily declare the variables in C. The following example shows the declaration of variables for the DH parameters of a link inside an openkinematic chain: float float float float offset_along_x_axis; offset_along_z_axis; angle_about_x_axis; angle_about_z_axis;
In engineering, it is inevitable to encounter many constants. In order to enhance the readability of a program in Cprogramming language, we can associate a constant with a name (i.e. a string of characters) by using the directive #def ine. For example, constant n can be defined as follows: #define PI 3.14; Data Structures in C To further enhance the readability of a program written in Cprogramming language, we can group the parameters, or variables, related to a same physical entity to form a data unit called a data structure. The directive typedef struct allows a programmer to define a data structure. For example, we can define a data structure, called LinkData, to house all the parameters or variables related to a link in an openkinematic chain:
Information System of Robots typedef float float float float struct { offset_along_x_axis; offset_along_z_axis; angle_about_x_axis; angle_about_z_axis;
329
float linkjnatrix[4][4]; float linkinertia[3] [3]; float link_mass; } LinkData; A data structure is treated as a new data type defined by a programmer. We can easily declare a variable of a data structure in the same manner as we declare a variable of an existing data type in C. For example, the following code declares variable linkl of the type LinkData:
LinkData linkl;
DataProcessing Loops in C Engineering computing may frequently involve repetitive processing of data with a block of instructions. The Cprogramming language provides two basic mechanisms to repeatedly loop over a block of instructions. This process is called looping. The following example shows the use of the directive for(expl; exp2; exp3) to implement a dataprocessing loop in C: LinkData left_arm_link[6] , right_arm_link[6]; void InitializeAllLinksO
{
int i; for (i=0; i < 6; i++)
{
left_arm_link[i] left_arm_link[i] left_arm_link[i] left_arm_link[i] .offset_along_x_axis .offset_along_z_axis . angle_about_x_axis . angle_about_z_axis = 0; = 0; = 0; = 0;
right_arm_link[i] . off set_along_x_axis = 0; right_arm_link[i] .offset_along_z_axis = 0; right_arm_link[i] . angle_about_x_axis = 0;
330
The Fundamentals of Robotics: Linking Perception to Action
right_arm_link[i] .angle_about_z_axis = 0; } ; } In the above example, expl sets the initial condition of the loop and exp2 sets the repetitive condition of the loop. If the condition is true, the loop continues until the condition becomes false. In addition, exp3 sets the default instructions to be executed at the end of each loop (i.e. iteration). The above example can also be implemented with the directive while inC: LinkData left_arm_link [6] , right_arm_link[6] ; void InitializeAllLinksO {
int i;
i = 0; while (i < 6)
{ left_arm_link[i] left_arm_link[i] left_arm_link[i] left_arm_link[i] . offset_along_x_axis = 0; .offset_along_z_axis = 0; .angle_about_x_axis = 0; . angle_about_z_axis = 0; . offset_along_x_axis = 0; .offset_along_z_axis = 0; . angle_about_x_axis = 0; .angle_about_z_axis = 0;
right_arm_link[i] right_arm_link[i] right_arm_link[i] right_arm_link[i] i++ ;
} ; }
As we can see, the expression inside the while directive is a condition to test whether or not the loop should be continued. If the condition is true, the loop will be repeated until the condition becomes false. In C, a programmer can prematurely quit the looping by using the instruction break. This is a very useful and efficient way to quickly exit a
Information System of Robots loop for whatever reason. DataProcessing Branches in C
331
Besides looping, engineering computing may also frequently involve selectively choosing a block of instructions to execute based on the current condition. The process of directing dataprocessing into different blocks of instructions is called branching. In C, the directive i f  e l s e provides a simple mechanism to branch between the two blocks of instructions. Obviously, we can nest the i f  e l s e directive to branch among multiple blocks of instructions. But, this will make a program less comprehensive. An elegant mechanism for branching among multiple blocks of instructions is supported by the directive switch inC. The following example shows how to use the switch directive to, selectively, initialize the data structures of the links belonging to a specified openkinematic chain: LinkData left_arm_link[6] , right_arm_link[6] ; void I n i t i a l i z e L i n k s ( i n t { int i; KinematicChainldentifier)
switch(KinematicChainldentifier)
{
case LeftArm: for (i=0; i < 6; i++)
{
left_arm_link[i] left_arm_link[i] left_arm_link[i] left_arm_link[i] .offset_along_x_axis .offset_along_z_axis . angle_about_x_axis .angle_about_z_axis = 0; = 0; = 0; = 0;
}
break; case RightArm: for (i=0; i < 6; i++) { right_arm_link[i] .offset_along_x_axis = 0;
332
The Fundamentals of Robotics: Linking Perception to Action
right_arm_link[i] .offset_along_z_axis = 0; right_arm_link[i] . angle_about_x_axis = 0; right_arm_link[i] . angle_about_z_axis = 0; } break; default: break; } } As we can see, the expression inside the directive switch must be an integer. This expression represents the current condition and is often compared to the predefined cases marked by the keyword case. If there is a match, the corresponding block of instructions is executed. Each block of instructions must ended with the break instruction. If there is no match for the current condition, the program will enter default, marked by the keyword default, and default instructions (if any) will be executed. Visibility of Data and Functions in C As we mentioned earlier, for the sake of easy maintenance and editing, it is wise to split a large program into multiple program files. It is now common for a single application program to have multiple program/header files. However, when a single application program is divided into a set of coherent functional modules and each module is stored in its own program file, it raises the issue of how to handle the visibility of data and functions across multiple program files in C. Fig. 6.10 illustrates an example of a simplified skeleton of a robotcontrol application. Functionally, we can create one module to handle the forward kinematics and another module to take care of the inverse kinematics. Each module is stored in its own program file (InverseKinematics.c and ForwardKinematics. c). And, the main body of the application is stored in another file called RobotControl.c. As a result, the visibility of data and functions across these files is a concern. In Cprogramming language, there are some welldefined rules governing the visibility of data and functions. Rule 1: All the variables declared inside a function are only accessible or visible by the instructions inside this function. These
Information System of Robots
333
Multiple Program Files of a Single Application RobotControl.c PostureData EndEffectorFrame ; float JointAngle[100]; void mainO { i RobotControl.h tffndef ROBOTCONTROL #define ROBOTCONTROL extern PostureData EndEffectorFrame ; extern float JointAngle[100]; extern void ForwardKinematics (int); extern void InverseKinematics (int); tendif
InverseKinematics.c include " RobotControl.h " static PostureData input; static float output[100] ; void InverseKinematics (int Chainld ) { int linkjd;
) • •
ForwardKinematics.c include " RobotControl.h " static PostureData output; static float input[100] ; void ForwardKinematics (int Chainld ) ( int link_id;
}
Fig. 6.10 Multiple program/header files of a single application, and the visibility of data (and functions) across the program files.
variables are called local variables. For example, the InverseKinematics () function has the variable name link_id. It is a local variable and only visible inside this function. If the ForwardKinematics () function also requires a local variable with the same name of link.id, it has to be declared independently inside the ForwardKinematics () function. In fact, these two variables share the same name but will be physically stored in different memory locations. Rule 2: By default, all the variables declared inside a program file (but not inside any function of the program file) are accessible or visible by all the instructions or functions across all the program files belonging to the same application. These variables are called
global variables.
For example, the EndEffectorFrame and JointAngle variables declared inside program file RobotControl. c are global variables and visible across all the program files.
6. memory management is no longer a critical issue in multitasking.c are only accessible by the instructions or functions inside this file. The insertion of the directives: #ifndef R0B0TC0NTR0L #define R0B0TC0NTR0L #endif is for the purpose of avoiding duplicate notification of a same group of global variables and functions. Thus.Tasking Refer to Fig. multitasking primarily involved: a) time management and b) memory management. Functionally. A digital computer is composed of a processor and memory which are interconnected by buses.10. time management is THE critical issue.h"). it can be treated as a system which is capable of undertaking dataprocessing and storage. #include "RobotControl.A global variable or function is notified with the directive extern.2. meaning it can be a string of any characters. Refer to Fig.334 The Fundamentals of Robotics: Linking Perception to Action Rule 3: The directive s t a t i c placed in front of a global variable will make this variable visible inside its program file only. These two functions are global functions which are visible everywhere across the program files of the application. the input and output variables inside program file InverseKinematics.g. . it is a common practice to notify the global variables and functions in some separate files called headerfiles.5.5. To enhance the readability of the developed programs. 6. In this example. Rules 2 and 3 are equally applicable to functions. 6. there is no s t a t i c directive in front of the InversekinematicsO and ForwardKinematicsQ functions. Nowadays. For example.1 The Basics of Multi. And.h file is a header file. As a result. For example. however. this string of characters along with the directive #includeserves as a tag to skip any repetitive notification (e. 6.Tasking In the past.2 Multi. the string of characters R0B0TC0NTR0L is userdefined.4. The RobotControl. memory is very cheap.
a computational task can be denned as follows: Definition 6. These tasks are different from the motioncentric ones performed by a robot. in a realtime system. each instruction of a program is executed by a processor's ALU within a fixed number of clock cycles. thus.11. For example. In other words. there is no issue of resourcesharing. it is clear that a digital computer can be treated as a system having two important resources: a) computing power and b) storage capacity. 6.9 A computational task. This memorystorage capacity is commonly measured in terms of bytes. It consumes not only the computing power of a digital computer but also memory storage. A repetitive task is a task which can be executed within a set of discrete time intervals. . But. MB stands for "megabytes" and GB stands for "gigabytes". very common to treat a digital computer as an MIMO system (multiple input/multiple output). As shown in Fig. a nonrepetitive task must be run within a single time interval. Computational Tasks The input to a digital computer is known as tasks. there is concern over resourcesharing. this measure is not so intuitive because different programs may have different numbers of instructions. In this case. In contrast. a processor's computing power can be measured in terms of the number of clock cycles per program. However. Now. A digital computer has a certain memory capability to store programs and processed data. is a dataprocessing job with a deadline. Under the context of an information system or computing system. there will be no issue of multitasking. a computational task can be repetitive or nonrepetitive. It is. today's computer is becoming more and more powerful in terms of computing power and storage capacity. And.Information System of Robots 335 Since the operations inside a processor are all synchronized by the system clock. Formally. it is important to understand how a digital computer handles multiple input and shares the limited resources. Two ways to measure computing power are by MIPS (million instructions per second) or MFLOPS (million of floatingpoint instructions per second). a task means a computational task related to dataprocessing. If a digital computer is further considered to be an SISO system (single input/single output). And.
we should avoid nonrepetitive tasks. or transform nonrepetitive tasks into repetitive tasks that satisfy the deadline constraint.g.336 The Fundamentals of Robotics: Linking Perception to Action SlaleS  Kuiinin. 6. ready for execution by the processor). Most importantly.e. a parallel computer).I'iiusdl KL\UI\ \hmloil ('oiiipkUil = l. a nonrepetitive task does not share a digital computer's resources with other tasks because it can neither be interrupted nor preempted.nlic Fig. there will be no issue of multitasking if the number of tasks is equal to or less than the number of processors. and d) completed or aborted. c) pause (either voluntarily or caused by preemption). a nonrepetitive task can only be in one of three states: a) ready (i.isk = Categories  NlllUqVllllM. This is because a repetitive task can be in one of these states: a) ready. b) running. The issue of multitasking arises only when the number of tasks is greater than the number . Time Management in MultiTasking For a complex computing platform with multiple processors (e. This provides the basic mechanism for the sharing of a processor's resources.e. Since a repetitive task can be run within a set of discrete time intervals. In other words. There are two types of repetitive tasks: a) periodic tasks and b) sporadic tasks. a repetitive task can be periodically switched from "running" to "pause".1 I KlpiMHlM' I IVIIDIIIL ' ^ Spor. it can participate in cooperative multitasking or preemptive multitasking. and c) completed or aborted. consuming the processor's resources alone).j I *.11 The states and categories related to computational tasks. b) running (i. Clearly. Whenever possible.
For the sake of simplicity. data and stacks. When a running task has to be stopped for whatever reason. there is an obvious need to manage the processor's time and memory so that the processor's resources can be shared. Both the program and data require space in the processor's memory. l'mcessnr(s) 1 ' Completed and aborted tasks • New task Paused task Fig. programs. 6.13. another special program will automatically release the processor's resources from the running task and launch a new task selected by task scheduler. as shown in Fig. Task • Scheduling T I Task Dispatching 1 . Since a processor can only execute one task at a timeinstant. we should also manage the sharing of a processor's memory if a large number of tasks are concurrently active. Memory Management in MultiTasking By definition. 6. In this case. the execution of its instructions requires an . as shown in Fig. A computational task is a computer program for dataprocessing. A special program which manages a task queue is called a task scheduler. let us consider the scenario of a single processor with multiple computational tasks. 6. A task queue needs to be managed so that the waiting tasks are ordered according to certain rules and priorities.12 Illustration of multitasking process when there is a single processor and multiple computational tasks. InterTask Communication Task Queue I \ " f p. all other tasks must wait in a place called a task queue. a computational task will not only consume a processor's time but also its memory.5. This special program is called a task dispatcher. When a task is running. Fig.12 illustrates the process of multitasking with a single processor. A common way to manage the sharing of this memory is to explicitly divide the memory space into distinct zones which are respectively assigned to the operating system (if any). 6.Information System of Robots 337 of processors. Thus.
5. Intertask Communication Another requirement of memory management is posed by intertask communication. there should be a mechanism to support this type of intertask communication. If task A wants to exchange information with task B. intense usage of the processor's registers. the contents (also called the execution context) of these registers must be saved so that they can be recovered once the task is resumed. When a running task is voluntarily or preemptively stopped.2 Cooperative MultiTasking In practice.338 The Fundamentals of Robotics: Linking Perception to Action DiSP'ay jfardDisk^ System_lJ ^ ^ 1 Memory i ^ Memory (Taskj)  I/O Systems  [°5D (^2) * * ^—^ "r A •(^ALU(J)^ I i cm • * * rjata m ' ""^"n ^n ( Task n ) registers V3^  Programs ' 'I  LJU Data/Stack Fig. . 6.2. index registers and accumulators. there are two strategies of sharing a processor's time among multiple tasks: a) cooperative multitasking and b) preemptive multitasking. This is commonly achieved by a shared memory if the tasks are active inside the same processor. Access is granted to all the tasks that use it for dataexchange. A shared memory is a reserved memory block in the data zone of a processor's memory. such as the program counter. 6.13 Illustration of memory management required in multitasking with a single processor. A stack is a memory block which holds such contents.
Under cooperative multitasking. The motion controller of the physical actor first reads the sensor feedback. • Task 2: Compute the control signal.14 An example of cooperative multitasking for a physical actor. ( Taskl ^ If aborted If done If aborted ^3iJ Task 3 ~~^ \ \ N . Since each task is treated as a state.Information System of Robots 339 The basic idea behind cooperative multitasking is to treat a set of active tasks as a state transition network or state machine. All the active tasks must be cooperative. The multitasking will not function properly if one task selfishly consumes all the resources (time and memory) of a processor. When one task voluntarily exits the running state. Physical Actor State 2 ( Task 2 N. a subsequent task is activated. • Task 3: Output the control signal to the power amplifier. the corresponding state machine . Fig. it calculates the control signal according to a control algorithm. As long as the motor is running. An active task can either be in the running mode or the sleeping mode. Subsequently. 6. 6. Any uncompleted task is an active task as long as it is not aborted. it will include these three tasks: • Task 1: Read the motion parameters from the motion sensor. Then. each task voluntarily commutes its running state to a paused (sleeping) state.14 shows an example of cooperative multitasking for a physical actor which controls an actuator with an inertial load. ReadMotionSensor L y^ " sleeping V \^ DriveActuator / State 3 1 State 1 Fig. If we treat the physical actor as a state machine. <( ComputeControlSignal \ If sleeping y ' / / i ^ > w If aborted \ «sleeping \ C^. it outputs the control signal to the power amplifier which drives the motor. these three operations are repeatedly performed by the physical actor.
G _J . r~^ . a time tick in Dynamic C is equal to 1/2048 second. j int flag_abort . ^ I if (ReadMotionSensor ()== false) flag_abort =true. It supports cooperative multitasking with a builtin directive costate. Rabbit 2000 microcontroller). completed motion. costate Task3 { if (DriveActuator ()== false) flag_abort =true.340 The Fundamentals of Robotics: Linking Perception to Action will have four states.g. these three tasks of the physical actor will voluntarily and periodically sleep for 20 timeticks. } if (flag_abort == true) exit. a task can voluntarily commutate into the sleeping state. This example can be implemented with a realtime programming language. waitfor(DelayTicks(20)). .14. The function DelayTicksO is a builtin function of Dynamic C.g. and the fourth state will describe the situation of exit (e.zworld. which implements the state machine described in Fig. 6. costate Taskl { CooperativePhysicalActor. sensor failure.c '. waitfor(DelayTicks(20)). actuator failure. Dynamic C from ZWorld (www.15 shows a sample program in Dynamic C. For example. in Dynamic C). etc). waitfor(DelayTicks(20)).com) is a Cprogramming language with multitasking. we can easily implement multiple tasks. With the builtin function waitforO in Dynamic C. By using the directive costate. 6.15 Sample program of cooperative multitasking in Dynamic C. } if (flag_abort == true) exit. void main() { flag_abort = false. which run concurrently on the microcontroller supported by Dynamic C (e. 6. if (flag_abort == true) exit. As shown in the sample program. Fig. (called costatements. / Fig. For example. costate Task2 { if (ComputeControlSignal ()== false) flag_abort =true.
This process will be repeated until all the tasks are completed or aborted. and reenters the task queue to wait until it becomes the running task again. Otherwise. in order to properly coordinate its behaviors in various scenarios of task execution.i •*. . If the level of priority is the same for all tasks. it exits. • Mental Actor 2: Coordinate visuallyguided hand gestures or manipulations. the behaviors of the humanoid robot should include: • Mental Actor 1: Coordinate visuallyguided biped walking. For example.12.16. it is necessary to use preemptive multitasking. it is not able to dynamically change the priority or order of task execution. the order of task execution is determined by a queue which is also known as a task queue. it will be preempted and put back in the queue as a paused task. Under preemptive multitasking. when task i becomes the running task. active task i will have two important parameters: • tStii the assigned quantum or time slice which is dynamically changeable. In this way.i = £P. for dancing or playing a sport. we simply assign a priority level to each task. If this task is still active after one quantum. Under preemptive multitasking. the assigned priority level which is also dynamically changeable. as shown in Fig.3) If task i is completed before or within this amount of time.5. the amount of the processor's time consumed by this task will be: te.2. A queue is a FirstInFirstOut (FIFO) buffer in the memory. A simple way to schedule the execution of tasks having the same level of priority is to assign an equal. • LPti'. This time interval is called a quantum or time slice. 6.3 Preemptive MultiTasking Cooperative multitasking does not impose a rigid time constraint on concurrently active tasks. For applications which require stricter time management.. This process is known as the roundrobin or timeslicing scheduling method. The running task will consume one quantum of the processor's time. a new task from the queue will be selected as the running task. 6. it is preempted after the elapse of time te^. task scheduling is based on priority and order. Subsequently. Refer to Fig. Assume that a humanoid robot has a builtin distributed information system.!• (6. predefined time interval to each task. In order to prioritize task execution.Information System of Robots 341 6. Moreover.
it is necessary to adopt preemptive multitasking in order to guarantee the apparent simultaneity of these actors. Fig. time. Htysica) Actor 1 ' Motion Control of Biped Walking < ~ I J r r n J V. Dynamic C from ZWorld supports preemptive multitasking with the language construct slice (buffer.^3  LegEye Coordination • — k'ysiWAJ*it. • Mental Actor 3: Coordinate visuallyguided head movements. and provides efficient time.3 RealTime Operating Systems To efficiently handle multitasking. A realtime operating system is usually small.and memorymanagement services such as task scheduling.1 I > .. The second argument is time interval te^.1 I • ' ••• •••• •• ii J Computer 2 MemniAc.17 shows a sample program in Dynamic C that implements these behavior coordinations in preemptive multitasking.  Visual Perception ^ ^ Computer 3 <^Jr .16 Example of a distributed information system for a humanoid robot.4Yy}:'.5.342 The Fundamentals of Robotics: Linking Perception to Action Computer 1 Mtflldl A i t o r I II .I ~ MinrjJ Aclur 2 1 1 ''. Physical Aclnr 2 Motion of Dual Hand Arm Manipulation Phyiicd Aulor 3 Motion Control of Active Stereovision Head Actuators \V Sensors Actuators it Sensors Actuators it Sensors Fig. The first argument buffer is a parameter which specifies the size of the stack to save the execution context when actor i is preempted. it is best to use a realtime operating system. 6.interval). which will be consumed by actori when it becomes the running actor. . If these mental actors run on a single computer.. 6. 6. intertask communication and realtime interruptions. task dispatching.
17 ) Sample program of preemptive multitasking in Dynamic C. priority_actorl*time_slice) MentalActorl { LegEyeCoordination (). 6. RTLinux. QNX. in robotics. int priority_actorl. —^— G)>> Fig. void main() { time_slice= 10.Information System of Robots 343 x j PreemptiveBehaviorCoordination.6 Data Storage and Retrieval Data storage and retrieval is an important part of any information system. Therefore.c I int time_slice. . it is obvious that a humanoid robot needs a memory to store all of the acquired internal representations of perception and knowledge. many good textbooks provide a comprehensive description of the fundamentals of a realtime operating system. reusable. priority_actor2*time_slice) MentalActor2 { HandEyeCoordination (}. And. priority_actor3*time_slice) MentaIActor3 { HeadEyeCoordination (). Nowadays. slice(500. 6. the issue of how to store useful dataprocessing results which can be retrieved later for reuse is something that inevitably needs to be dealt with. This is because the dataprocessing results must be useful and therefore. data storage and retrieval are an indispensable part of a humanoid robot's information system. priority _actor3. there are many realtime operating systems on the market. Since a robot incorporates an information system. priority_actor2. If a humanoid robot have to develop its physical and mental abilities through realtime interaction with its environment. such as VxWorks. ) slice(300. Generally speaking. > slice(300. there lacks a standard regarding the storage and retrieval of reusable data (if any).
But due to the slow access speed. at a higher level. from the viewpoint of information. From the user's point of view. except for ROM and ERPROM. a memory or data storage device is a specifically organized file system. • It must support access (both read and write) to the data by multiple computing processes or tasks.6. any advanced computing platform will have an operating system. By definition. Usually. data is a list of ordered bytes. 6. The memorymanagement of an operating system includes management of both the processor's memory (e. is not a place to store reusable results or data. RAM) and the external storage device (e.1 Storage Devices As we mentioned earlier.g. a storage device is a system which takes data as input and produces data as output. harddisk. An operating system serves as an interface between the application programs and the computing platform's hardware components/devices. • It must maintain the integrity of the stored data. as shown in Fig. keyboard and printer. The common place to store useful and reusable data (including program files) is in an external storage device. have a corresponding file system which is an integral part of . It is true to say that all computing platforms. a CDROM is more appropriate for data backup than storage. and c) the peripheraldevice management module which handles input/output interfacing. 6. The processor's memory. having an external storage device. and communication between the processor and external devices such as the mouse. the DH parameters of an open kinematic chain can form one record.344 The Fundamentals of Robotics: Linking Perception to Action 6. the entity that stores a record in a storage device is called a file. Therefore.6.13). etc. data is treated as a record.g. a generalpurpose operating system consists of three major modules: a) the processmanagement module which manages the allocation of the processor's time for various computing processes or tasks. such as a harddisk or a rewritable CDROM. But.2 A File System At the lowest level of a memory. Some basic requirements for an externalstorage device are: • It must be able to store a large amount of data. For example. b) the memorymanagement module which manages the processor's memory and the storage device.
c" means a file which contains a C program. A realkernel is a minioperating system which supports.Information System of Robots 345 an operating system.18. • read: to retrieve data from an opened file. For example. InverseKinematic. With the help of these functions. becomes a realtime operating system for all personal computers (PCs).3 Data Storage Using Unformatted Files A computing platform having an external storage device is usually supported by an operating system and/or a realtime kernel. For example. all the bytes inside the record RobotData. RT Linux and Linux with Embedix. 6. referenced by variable robotdatabase. In this example. This data is stored into the RobotData structure. like C. During the execution of the application. The purpose of afilename's extension is to indicate the file's type. for a given application. a highlevel programming language. referenced by variable robotdatabase. For example. the task scheduling. a robot can define a data structure called RobotData to organize all the relevant information. Since the processor's RAM is not a place for the storage and retrieval of any reusable data. Windows NT (a generalpurpose operating system). in realtime. a simple way to store and retrieve data is to make use of a file. • write: to store data to an opened file. • close: to close a file. with the extension of the realtime kernel INTime. task dispatching.c". A file normally has a name and an extension. c is afilewith the name "InverseKinematics" and the extension ". With the support of a realtime operating system. normally provides a set of functions for the manipulation of a file. the data referenced by variable robotdatabase must be stored into a file. interruption services and intertask communication. VxWorks. so that it is ready to be accessed by other programs or users. Other examples include QNX. 6. as shown in Fig. the extension ". are stored in an opened file in a single shot . Typical functions include: • open: to open a file for read or write operations. at a certain point in time before the application terminates. This can be done with the sample codes.6. For example. all data referenced by variable robotdatabase is stored in the memory of the processor (RAM). parameters and values of the variables related to this application.
•filename) RetrieveRobotDatabase(char FILE'pf. sizeof ( RobotData ). the data must be homogeneous.18 Data storage to an unformatted file. 'filename) jfcj Retrieve Data from a File external RobotData robotdatabase . a set of related files can be grouped together and placed in the same folder. This division of a directory can go into any depth. pf). ""•""MM. all the data structures in the list must be identical. all the bytes of the data structure RobotData can be retrieved and placed back into the data structure RobotData. fclose(pf). if we have n robots). 6. . Conversely. that data stored in a file does not require any formatting of the file. For example. return true. However.6. we can simply set the third argument of function f read or f write to be n (In Fig. ") external RobotData int StoreRobolDatabase(char ( FILE «pf.. fclose(pf). 6.18. also known as a directory. This method is also applicable for the storage of a list of identical data structures. a file is a nondivisive entity. It is clear from this example. a directory can be divided into a set of subdirectories.zeof ( RobotData ). all the data stored in the file will be lost. it is set to 1). The two basic elements of afilesystem are: a) thefilesand b) the directories (or folders). "r")) == NULL) { printf( . But. 1.346 The Fundamentals of Robotics: Linking Perception to Action (J Store Data to a File robotdalabase . 1. pf). As a result. ta> C a n n 0 [ o p e n flIe for r e a ( r ) return false. (2) If a list of data structures is stored. if((pf= fopen(filename . In other words. in a single shot as well (with the function f read). / . } fread(robotdatabase . ) fwnte(robotdatabase . we call this procedure data storage using an unformatted file.^ Kill A7 ' Fig. ) (with the function fwrite). return true. referenced by variable robotdatabase.g.4 Data Storage Using a File System Under the file system of an operating system. Thus. int ( (X_ if((pf = fopen(filename . "w")) == null) I printff \n> Can not open file for write"). this method has several drawbacks: (1) If a file is corrupted for whatever reason. if there are n units of the data structure RobotData (e. s. At the operating system level. a directory can easily be configured into a hierarchical treestructure of subdirectories and files.6.
It is very similar to a file system itself. There are many advantages to this solution: • It is easy to develop. as a highlevel programming language usually provides a set of functions to manipulate the files (e. can read and write on the formatted file? • Should library functions be provided for reading and writing records if a new file format is proposed and accepted by a community of users or programmers? Fig 6. With the use of this function. write etc). what is the name of the file format (e. (NOTE: A record is a sequence of bytes grouped together). A formatted file usually has three types of entry: • Header Entry: This is a record that stores some generic and/or historic information about the file. In the Cprogramming language.g. open. one can make use of this type of hierarchy in a file system to create a base for data storage. PDF. a file can also be viewed as a hierarchical organization of records. Postscript. • It is safe to store large amounts of data. . TIFF etc)? Who invented the file format? Which version is the file format ? Is the data compressed? etc. the labels assigned to the records inside a formatted file) be standardized so that any user. the function f seek allows a program to store or retrieve a record at any location inside the memory space of a file. who follows the standard.e.6. as an operating system normally provides a powerful tool (e. In general programming. 6.19 shows a generic structure of a formatted file. as a corrupted file will not affect other files. A header entry will have at least one pointer to a directory entry. For example. Similarly.5 Data Storage Using Formatted Files If we view a file system to be a hierarchical organization offilesin a treelike structure of directories or folders. Here. a pointer means an address within the memory space of a file.g. the records inside a file can be arranged in a treelike structure of directories or folders. read. it is a simple matter to create a formatted file. Windows Explorer) for a user to navigate through the hierarchy of the file system.g.Information System of Robots 347 Naturally. • It is easily accessible by any user. When using a formatted file to store and retrieve data. it is important to consider several issues: • Should the naming of the records (i. close.
a pointer means an address within a specific memory space. However.. is the memory block where data is stored in a file. 6. and a pointer to the next directory entry. This helps to maintain the authenticity of the stored data. as shown in Fig.19. . For example. a data entry will also contain a pointer to the next data entry in that list. the number of data entries under this directory. . • Directory Entries: A directory entry is a record which indicates the grouping of related data entries under the same directory or folder. If the data entries in a directory are organized as a chained list. a directory entry can simply contain information about the name of the directory. If the directory entries are organized as a chained list. what is the standardized name of the data? What is the size of the data (in terms of bytes)? What is the pointer to the first byte of data? The value record.348 The Fundamentals of Robotics: Linking Perception to Action Generic Structure of Formatted File Header ZD 1 r Directory Entry 1 ^ Directory Entry 2 ^ H Directory Entry n  ^^— ~^~ Data ^ _r .19 A generic structure of a formatted file.o ~T~ Data Entry 2 ^ 1 ^ ^ Value I Entry 2 ^ _ ^ H Va ' Ue I  Btty 2 H ^ ^ Va ' Ue I Fig. at least one pointer to the first data entry. • Data Entries: A data entry is a record which stores the associated information to a piece of data. In addition. 6. some confidential or proprietary information ..i. ^Y— T" fc. information written into a formatted file cannot easily be modified by an ordinary user. It is easy to share a formatted file among a team of users or programmers. Data ^^ —fc> v<.
6. 6. In this way.7. One of the advantages of using I/O registers is the ease of programming (this will become clearer later). a robot's information system must have a comprehensive set of data interfacing and communication subsystems. These subsystems are also called the I/O communication systems. For example. an I/O system can be configured for input. or devices. The primary role of an I/O system. analysis and control of a robot. 6. 6.1 I/O Registers A common way to operate an I/O system is to make use of the I/O registers. Hence. By default. a motion controller is a computing system equipped with some specific I/O systems.1 Basic Concepts For the sake of simplicity. a motion controller can receive data from sensors and send data to actuators through its I/O systems. an understanding of the working principles of the common I/O systems is important for the design. Obviously. And. A register with 8 bits is comparable to a set of eight electronic switches. especially a robot. You can switch each bit "on" or "off". it is possible to set up an access control using a password with a formatted file. To a certain extent. output. as shown in Fig. send and communicate data.7 Data Interfacing and Communication A computing system without a data interfacing or communication module can hardly be useful for the control of any machine.5. I/O registers are divided into three groups: (1) Control registers: These registers are for programs to configure an I/O. let us consider memorymapped I/O systems.Information System of Robots 349 can be encoded into a formatted file. the performance of which depends on the realtime performance of its I/O systems. an I/O register is a memory location consisting of 8 bits. . For example. or device.7.1. or both. A robot is a realtime system. is to receive. each bit carries one piece of information related to the configuration or status of the I/O system.
However. Usually. Sometimes. Therefore. .1.20. We know that an I/O system serves as an interface between two devices or information systems for data exchange. as shown in Fig. most of the time. an I/O port is directly mapped to its corresponding I/O data register. (3) Data registers: These registers serve as a buffer to hold data which is yet to be transmitted or received. For a memorymapped I/O system. For example. a single register will serve as both the control and status register. 6.350 The Fundamentals of Robotics: Linking Perception to Action (2) Status registers: These registers allow programs to know the status of an I/O system. 6. depending on the complexity of an I/O system. when a program examines the status register.20 Conceptual illustration of the mapping between an I/O port and its corresponding I/O data register. the pins or lines of an I/O system are grouped into units called I/O ports.7. there will be a separate register for the data register.2 I/O Ports Memory I/O Port I/O Data Register '"•' '" '" * LLLt Fig. capable of holding one byte at one timeinstant. there must be some connection pins or lines to physically link these two devices or systems together. it will know whether the data communication has been completed successfully or not. 6. An I/O port has 8 pins which correspond to eight logic bits (or one byte).
1. Fig. the time responsiveness of the network is as crucial as the time responsiveness of each computational unit inside.7. which include: • Pointtopoint Topology: In this case.Information System of Robots 351 6. it is necessary to consider the most appropriate topology.e. 6.21 Typical network topologies. the network will become complicated if more devices or systems are hooked onto it. which are commonly called nodes. a network may involve the connection of more than two devices or information systems.3 Communication Networks Except in special cases such as sensory data input and data output to actuation devices. The wires which physically connect these two devices together are called the network. System 1 \i •] System2  1 System » 1 1System2 System4 4 • System3  j System4 [  System3 (a) Point topoint topology System 1 r _——1 V "^~~~~~~1 * System 4 (c) bus topology System 2 r V * System 3 r _______^ L"""^ i (b) Token ring topology 1 System 1 ^y^Jf (Router) System 4 (d) Star topology System 3 i System 2 Fig. any pair of devices or systems has a dedicated connection line. A network with a predictable time responsiveness is called a deterministic network. For a distributed realtime controller (i. However.21 shows some typical network topologies. an I / O system generally acts as a communication controller for data communication between two devices. When this happens. 6. a controller with many computational units connected together by a network). . It is easy to meet the realtime requirement for data communication. In a complex system.
For this type of network. bus topology means that all the devices or systems in a network are electrically connected to a common point if we ignore the impedance of the network's wiring. the next issue is how to initiate and synchronize data communication.7. The router allows the communication system to intelligently dispatch data to the proper destination while masking out malfunctioning devices in the network. the data must carry address information so that it is relayed to the proper destination. Depending on the protocol of communication.4 Common Issues Data communication over a network of a given topology simply means the exchange of a string of bits (serial communication) or bytes (parallel communication) between two devices or systems. a tokenring topology has a direct connection between two adjacent devices or systems. the first issue of data communication is how to pack data in the form of a string of bits or bytes. In this way. • Bus Topology: A further simplification of pointtopoint topology yields the bus topology. Once the data has been framed. two common techniques are available: • Polling Technique (also known as Polled I/O): When using this technique. The framing process may involve the addition of extra information such as the destination's address and error checking.g. If a slave device wants to communicate data.1. Obviously. fieldbus). To do this. • Star Topology: An improvement on bus topology is the star topology with a router to serve as the common connection point. Depending on the protocol of communication. The process of packing data for communication is known as data framing. instead of having one connection between any pair of devices. a network of bus topology may be deterministic (e. The role of a master device is to periodically check whether a slave device . But.352 The Fundamentals of Robotics: Linking Perception to Action • Tokenring Topology: This is a simplified version of pointtopoint topology. 6. a network of tokenring topology is usually deterministic. it must set a flag bit in the status register of its I/O system. the slave device signals its intention to communicate data. there is one master device on a network and the rest are slave devices. From an electrical point of view.
The purpose of error checking is to ensure that the received data is equal to the transmitted data. data communication occurs immediately.g. a byte). a byte) has been received. and how to perform data framing (if any) and error checking (if any). After data communication. it . 6. the purpose of acknowledgement of data reception is for the transmitting device to check whether the data was successfully received by the receiving device. any device on the network can signal its intention to communicate data with another device by sending out an interrupt signal and taking control of the communication line (if it is free). programming an I/O system involves the following steps: • Step 1: Configure a chosen I/O system by properly setting up the bits inside the control and/or status registers. how to initiate and synchronize data communication. For a receiving device. • Interruptdriven Technique (also known as Interruptdriven I/O): When using this technique. a programmer usually does not need to directly solve these issues because the I/O system's hardware will do it. We will examine the details of the above issues when studying serial and parallel I/O systems.7.1. a third issue is error checking and acknowledgement of data reception. • Step 2: For a transmitting device.Information System of Robots 353 wants to undertake data communication by detecting the flag bit in the status register of the slave device's I/O system. And.5 Procedure of Programming I/O Systems Although there are many technical issues related to successful data communication. it detects a flag bit on the I/O system's status register in order to know whether a data unit (e. In general. If the detection is positive. For example. it detects aflagbit of the I/O system's status register in order to know whether a data unit has been automatically transmitted by the I/O system's hardware. it transmits a data unit (e. For a receiving device. • Step 3: For a transmitting device.g. The job of a programmer is to know how to operate the I/O system by programming the associated registers. a device must indicate whether to transmit or receive data.
speed. there are many solutions for D/A conversion. = 1. 6. The input to this circuit is a digital number.) are intrinsically analog. pressure. it is often necessary to exchange data between an analog system and a digital system. if it takes any value encoded in a real decimal number.354 The Fundamentals of Robotics: Linking Perception to Action reads in the data unit and clear the flag bit on the I/O system's status register. This means that the corresponding logic switch is "off". it is called an analog system or device. However. dimension etc.7]. it is called a digital variable. most of the values for physical quantities (e...Depending on the circuit.22. Rbi = 0.4) It is easy to see that Rbi = 00 when 6j = 0. perception and decisionmaking. parameter or signal. a digital controller gets motion feedback from a potentiometer)? In electronics.1. parameter or signal. On the other hand.7. However. when 6. it is called an analog variable.g. with the advent of the digital computer. • Repeat Steps 2 and 3 until data communication is completed. This raises two issues: • Digital to Analog Conversion (D/A Converter): How do we feed the output of a digital system to the input of an analog system (e. 7).22 shows one example of a D/A conversion circuit. Fig.h Vie [0. Electronically.g.. the output of the D/A converter is a voltage signal Vout. 6. a system or device which manipulates digital values is called a digital system or device.. for example. control. Similarly. this means that the corresponding logic switch is "on". a byte: Din — 6766656463626160 with 6^=0 or 1 (i = 0. (6. In nature. Otherwise. a digital controller sends control signals to the power amplifier of an actuator)? • Analog to Digital Conversion (A/D Converter): How does a digital system read the output of an analog system (e. if a variable. For the example shown in Fig. 6. temperature. parameter or signal only takes discrete values encoded in binary numbers.g.2 D/A Converters By definition. and the use of it for sensing. bit 6j of the input digital number Din can be treated as a logic switch with the resistance value of Rbi = l . the following relationship .
.3 A/D Converters In many real applications. it will become linear with a specifically selected set of resistor values i?.2. As a result. we can see that D/A conversion occurs simultaneously. 6.. 7). the time response of a D/A converter is normally deterministic.7.. or vice versa? From the circuit as shown in Fig. 6. Logic switch Fig. CCD (CoupledCharge Device) imaging sensor. 6. two important criteria for evaluating a D/A or A/D converter are: • Linearity: Can the input and output relationship of a D/A or A/D converter be mathematically described by a straight line? • Sensitivity: What is the relationship between variation ADin and variation AVout. However. holds: (6. (i = 0.. . In fact.5) This relationship appears to be nonlinear.22. it is necessary for a controller to read in the sensory data from an analog device such as a potentiometer. Q .1.22 Example of a D/A conversion circuit.Information System of Robots 355 y. strain gauge. and microphone etc.
If the original signal is not in voltage. The basic procedure is as follows: • Prediction: A counter automatically predicts possible output Dout from the smallest to the largest number.356 The Fundamentals of Robotics: Linking Perception to Action For an A/D converter.. • Verification: For each predicted value of output Dout. The advantage of this A/D conversion method is its low cost because it is relatively easy to design. denoted by Dout. I •Di«ital 11""""111 r i A DILMLII (ounkr I output M\ pi Mb I X^ .23 Example of an A/D conversion circuit. denoted by Vin. the control logic stops the counter. the cycle of prediction and verification will continue until Vpre(uct — Vin• Result Output: When Vpredict — Vin. the output from an A/D converter is a binary number. If the counter has eight bits (one byte). I . the input is an analog signal. The drawback of this method is the variable . a D/A converter generates a corresponding predicted voltage Vprecnct. * ('miUiil I uiik r " n. 6.. the analog input signal can be represented in the form of a voltage signal.«. 6. And.. and the actual number of the counter is the output of the A/D converter. Its working principle is very similar to that of a closedfeedback loop. In many circumstances. There are many solutions for the design of an A/D conversion circuit.•M ip u iM^> .23 shows a popular one. ' ' \ « 1 !). RLLMSK. Fig. known as the progressiveapproximation method. *\sS A/I) Converter Fig. the smallest value is 0 and the largest value is 255. As long as the predicted voltage is smaller than the input voltage V%n. we can easily convert it to a corresponding voltage signal.
10 bits 12 bits etc).1 Device Control with Parallel I/O Systems In robotics. This display device has seven segments. the LDP is "on" while the RDP is "off". C. E. For example. f.24 shows the use of a parallel I/O system to control a sevensegment LED (Light Emitting Diode) display. c. In other words. it is necessary for a robot's controller to periodically check the status of the keyboard's pushbuttons In short. bit 67 controls the "on" or "off" switch for the LDP while the complement of bit 67 controls the "on" or "off" switch for the RDP. as well as some simple characters (e. if a joint angle varies within the range of [—45°. the teach pendant).1. there are many buttons. 6. e. 6. 0V or +5V. A.4 Parallel I/O An input and output system that has at least eight signal lines is called a parallel I/O system because it can input or output at least one byte at a time.) can be easily controlled by a parallel I/O system. labelled as a. 0 or 1. assume that the connection between the pins of the I/O port and the segments of the LED display is indicated by Table 6.g.7. In this example. However. Since a parallel I/O system can simultaneously hold at least eight logic signals (bits). An alphanumerical display is the simplest way for a robot to express its internal states. For example.g.Information System of Robots 357 response time which depends on the length of the counter (e. and g. each joint has at least two limitswitches due to the mechanical constraint of a kinematic pair. switches and displays which require the attention of a robot's controller.g. A sevensegment display device can show numeric digits from 0 to 9. the time response of an A/D converter is still predictable because the conversion time for the largest input value is deterministic. When b7 = 1.e. F.4. On the other hand. 8 bits. b. etc. Fig. all devices which have input and output depending on a set of simultaneous logic signals (i. but also for the control of external devices such as buttons or switches etc. 135°]. d. . 6. a limitswitch for the position —45° and another limitswitch for the position 135° have to be incorporated into a robot's body.). if we want to dialogue with a robot equipped with a dedicated keyboard (e. "on" or "off' etc.7. it can be used not only for data communication. It also has two extra segments for decimal points: LDP (Left Decimal Point) and RDP (Right Decimal Point).
•__^ l ^ * m I ldp • rdp I Fig. for handeye coordination behavior. computer 1 is a receiving device while computer 2 is a transmitting device. In parallel. computer 1 will receive visual sensory information from computer 2. a parallel I/O system is capable of reading in or writing out at least one byte at a time. this number is 0x9B.358 The Fundamentals of Robotics: Linking Perception to Action Micro controller External Device I Processor I I/O Port ^B LED Display ^j yf % W H ^M *• b5 * ^3 f e b c Memory A Parallel I/O H — • • b. the output binary number from the parallel I/O port should be 01001011. we can use it for data communication because a stream of bytes of any length can be communicated through a parallel I/O system.7.24.1 Pin Connection. Table 6. computer 1 will send commands and data to computer 4. 6. Therefore. For the example shown in Fig. if we want to display digit "5" and the RDP. In this case. the processor simply writes byte 0x9B to the data register of the parallel I/O system.4. In order to light up digit "5" and the RDP.2 Interfacing for Data Communication By definition. In hexadecimal. 6.24 Example of a sevensegment LEDdisplay control by a parallel I/O system. 6. data communication involves two devices: a transmitting device and a receiving device. Normally.16.3 For the display control as shown in Fig. oooooooooooooooooo 6. Pins: br b7 b6 b5 b4 b3 b2 h b0 Segments: LDP RDP a b c d e f g~ Example 6. Obviously. computer 1 is a transmitting .
i 1 Memwy 1 : .. for longdistance communication. as shown in Fig. _i *T7^ '. : L _ l [^r '• i i ' Strobe A rr^LJ I : i Strobe A I '• . between a computer and a printer. 1 Micro controller 2 1 I/O Port . 6. 6.v . connecting the pins of the two I/O ports. For example.25 Interfacing for datacommunication with a parallel I/O system. j _ /„ > < _/J Processor .7..4. . the three important issues of data communication are data framing. and error checking. The digit indicates the number of wires inside the cable.3 Strobed Parallel I/O As we studied earlier. The slash with a digit placed on a data line. T Strobe B i  i ! • Parallel I/O M*f—r'' ' '. _ _  Fig. 1 . As a result. data communication using a parallel I/O system is only appropriate for shortdistance data communication (e. which represents a physical cable. ! l T Parallel I/O ! ' .25. the general connection looks like the one. is a graphic representation of the wiring in a computing and communication system. requires at least 10 signal lines to form a parallel link a) eight lines for the data bits and b) two lines for the control signals. the number "8" means that the cable. has eight wires or signal lines inside. . : • ' i i i : 1 Mem°^ 1 * y > — . between two devices inside a computing system etc). An integrated method which handles these issues is generally called a protocol of communication. 6. communication initiation & synchronization. this is a costly solution. . Micro controller 1 '. Parallel  I/O Port i 1 '  Processor '. data communication. The process of connecting two devices for data communication is called interfacing. Many standards exist in literature. In general.g. When two devices communicate with each other using their parallel I/O systems. using a parallel I/O system. Obviously. "* „ i Strobe B .Information System of Robots 359 device while computer 4 is a receiving device.
4.4 Full Handshake Parallel I/O The strobed I/O is the simplest way to undertake data communication. the input operation at microcontroller 2 works as follows: • Step 1: Microcontroller 2 continuously monitors the strobe B line. In this case. Assume that microcontroller 1 initiates data communication. here we will only cover the working principles at the physical level behind how data communication is initiated and then synchronized for both parallel and serial I/O systems. Therefore. a stream of bytes has been exchanged). it latches in a byte from the data bus to the data register. the output operation works as follows: • Step 1: Microcontroller 1 writes a byte to the data register which will put one byte onto the data bus connecting the I/O ports of the two communicating devices or systems. Another way is to send a positive or negative pulse onto the strobe A line. Accordingly. Here. It will work properly under two conditions: . Under the concept of a strobed I/O. This can be done by driving the strobe A line to logic high or logic low. If data synchronization only relies on the initiating device ( microcontroller 1). a parallel I/O system must have a signal line to input the control signal from an initiating device or system. • Repeat Steps 1 and 2 until the session of data communication is completed (i. 6. By default.360 The Fundamentals of Robotics: Linking Perception to Action However. 6. • Step 2: If an incoming signal is detected at the strobe B line. a parallel I/O system must have a signal line to output a control signal that initiates a first session of data communication. it is called a strobed I/O. microcontroller 1 undertakes output operation. let us assume that this output signal line is called strobe A.25. Refer to Fig.7. On the other hand. Let us call this input signal line strobe B. • Repeat Steps 1 and 2. and microcontroller 2 is in the input operation mode. it is reasonable to assume that the initiator of data communication is the device or system which outputs a stream of data first.e. • Step 2: Microcontroller 1 sends a signal to the strobe A line to tell the receiving device or system that a byte is ready on the data bus.
• Step 3: Microcontroller 1 sends an acknowledgement signal to the strobe A line if no error is detected. 6. It also permits the difference in the speeds at which the transmitting and receiving devices or systems are handling the input and output data. The first condition is relatively easy to meet while the second condition depends on the working environment. • Step 2: Microcontroller 1 sends a signal to the strobe A line to tell the receiving device or system that a byte is ready on the data bus. (2) No noise exists on the data and signal lines. is the full handshake I/O. the .5 Serial I/O We know that the number of wires in the parallel link will be at least 10.25. • Repeat Steps 1. 6. Accordingly. The basic idea behind this is to notify the proper reception of each byte. If there are n systems or devices attached to a network.e. In practice. The output operation under the full handshake I/O mode works as follows: • Step 1: Microcontroller 1 writes a byte to the data register which will put one byte onto the data bus connecting the I/O ports of the two communicating devices or systems. • Repeat Steps 1.7. 2 and 3. the input operation under the full handshake I/O mode works as follows: • Step 1: Microcontroller 2 monitors the strobe B line continuously. This mechanism provides the opportunity to do all kinds of error checking. it is not realistic to assume that transmitted data will be errorfree regardless of the level of noise. a stream of bytes has been exchanged). which does not have this drawback. If we use a parallel link to form a network.Information System of Robots 361 (1) The two devices operate at the same speed for each cycle of input and output. An alternative solution to using the strobed I/O. it latches in a byte from the data bus and store it in the data register. Refer to Fig. • Step 3: Microcontroller 1 waits for an acknowledgement signal on the strobe B line. each system or device will have 10 wires. • Step 2: If an incoming signal is detected at the strobe B line. 2 and 3 until the session of data communication is completed (i.
g. Internet and MODEM). networked datacommunication is a good solution. we will study the basic working principles behind synchronous and asynchronous serial I/O systems. the electronic circuitry which implements the synchronous serial I/O mode is known as the Serial Peripheral Interface (SPI). the bits of the data have to be sequentially transmitted in series. Fieldbus (e. and b) synchronous serial I/O.362 The Fundamentals of Robotics: Linking Perception to Action number of wires (excluding the wires of the network itself) will be 10 • n. 6. The keyword "peripheral" implies that the synchronous serial I/O is more appropriate for data exchange within a short distance. Nowadays. in this case. But the actual operation of a serial I/O is more complicated than that of a parallel I/O. and the number of control signal lines to zero or one. CAN bus or DeviceNet) and MODEM. The parallel link has eight signal lines for data exchange and two signal lines for control signals. On the other hand. The electronic circuitry which implements the asynchronous I/O mode is known as the Serial Communication Interface (SCI). the serial I/O system and the corresponding serial link have become the standard approach to networked datacommunication.g. the number of wires inside a robot's body is a big concern as a robot is a complex system with many interconnected computational modules. In robotics. the asynchronous serial I/O is a costeffective solution for longdistance networked data communication (e. it is called a synchronous serial I/O. Similarly. Obviously. are built on top of these basic working principles. So. the data signal line(s) will classify the serial link into one of these three types. a parallel link is not a good solution to form a network. Certainly. it is called an asynchronous serial I/O. The control signal line classifies a serial I/O into two categories: a) asynchronous serial I/O. The evolution from a parallel link to a serial link sounds simple.1 Interfacing for Data Communication Refer to Fig. TCP/IP.7. 6. if a serial link requires one signal line for synchronization purposes. If we reduce the number of data signal lines to one or two. If a serial link does not have any signal line for synchronization purposes. And. 6.25. In this section.5. such as the Ethernet. obviously. In fact. all advanced communication protocols. the parallel link becomes a serial link because it can only carry one bit of data at a time. as shown in Fig.26: . In order to exchange data.
and if this data signal line is bidirectional. "^un '• Jr__ S. 6. ' Transmit s j '••'" ' 1 1 1 X ^ Receive . •••i.26 Serial Link Types. (2) Halfduplex Serial Link: Now. or systems. (b) Halfduplex Serial Link Transmit S'. —— sii llci ' s I 1 i —^—I i vmllu . the two devices or systems engaged in data exchange can simultaneously transmit and receive data.MIMI) I I • MMIIII (c) Simplex Serial Link Fig. the fullduplex serial link becomes a halfduplex serial I/O. For this type of serial link. I u Viiiiio I  I _ T v rial I/O j (a) FullDuplex Serial Link 1 v' Transmit Receive Transmit Receive [ j s i • 'M1 ' • I * . (1) Fullduplex Serial Link: For this type of serial Link. if we reduce the number of data signal lines to one. data can be alternately exchanged between two devices.I ^ i II 1 I Transmit ^ . or vice versa.26a.i* Receive • . 6. one device or system is a transmitter and the other is a receiver. as shown in Fig. This is because the serial link has two bidirectional data signal lines. During each session of data communication.Information System of Robots 363 . if we restrict the direction of data com .' V ^ _ _ Receive . (3) Simplex Serial Link: For the halfduplex serial link.
In order to communicate a set of bytes.364 The Fundamentals of Robotics: Linking Perception to Action munication to one way. a data frame is composed of: • Begin Flag: This usually takes one byte and is used to indicate the start of a data frame.1 111 j~4>ansmit (a) Data frame's Bits in Spatial Order Begin Type Flag Address Control Error Checksum End Data Field (LSB to MSB) fag nil 1 1111 h—H 1111—11 i i [—H 1111 — : (b) Data frame's Bits in Transmission Order Fig. End Flag Error Checksum Type Control Address BeSin Flag Data Field (MSB to LSB) ITTTF1111 1 11 11 [ — j  II  . the identification number) of the receiver into the data frame.. there will be only one transmitter at a time. 6.7. In other words.2 Bitoriented Data Framing A serial I/O can only communicate one bit at a time.e. In the spatial order from left to right. one must arrange them into a series of bits. • Address: A serial link is a common solution for networked datacommunication. Since a network is a bus. The result of a data framing process is called a data frame.27 illustrates the concept of a data frame. 6. This implies that each system or . The rest of the systems or devices hooked onto the network are the potential receivers. A simple way to identify the specific receiver is to encode the address (i.: Time ^ Illustration of the concept of a data frame. The process of packing a set of bytes (i. Fig. the transmitter and receiver are prespecified.27 lllil!. 6.5.e. then it becomes a simplex serial link. They cannot alter their roles. This means that more than two systems or devices can be attached to the same network. For this type of serial link. the data signal line sequentially carries the bits in one direction only. data) into a series of bits is called data framing.
5. the transmission order of a data frame (see Fig.3 Synchronous Serial I/O A serial I/O can only transmit a set of bytes in the bitbybit manner. In order to integrally communicate a set of bytes between two systems or devices. is called a synchronous serial I/O. Error Checksum: This may take a few bytes.Information System of Robots 365 • • • • device involved in a networked datacommunication will have its own address. End Flag: This simply indicates the end of a data frame. The types may include: ASCII. with a signal line for synchronization purposes. As a result. this can be done with a dedicated signal line. Uncompressed. 6. In a simple case. Fig. . Consider the case of a fullduplex serial link. Type Control: This usually takes one byte and is used to indicate the type of data to be transmitted.7. then there is no error in communication.27a). By default. In a serial I/O. 6. the data frame is transmitted by a shift register bitbybit.27b) is different from its spatial order (see Fig. 6. Before transmission.28 illustrates data exchange between two microcontrollers using a synchronous serial I/O. Otherwise. each computer hooked onto an Internet will have an IP address). the shift register always moves the bits towards the righthand side. At the reception side. Hence. this result (error checksum) is calculated and packed into the data frame. As we can see. Compressed. etc. A serial I/O. the receiver may request the transmitter to resend the data. If the two error checksums are the same. a serial I/O's hardware operates two cascaded loops: a) the outer loop on the set of bytes and b) the inner loop on the bits of a byte. the network will have at least three wires: two for data bits and one for a synchronization signal. synchronization is an important issue. Compression Format. Data Field: This refers to the set of bytes belonging to the data. the error checksum is recalculated again by the receiver. The calculated result is compared with the received one. 6. (For example. It contains the results of an errorchecking algorithm. Data is a set of bytes. Binary.
: ^ i \ i i '  ' Receive cu feao^^am^BM«M^^^'^mm^aatara^g. • i Clock signal I I 1 • Synchronous Serial I/O : . i Receive : ^ ^"^ Svnchmnoiis Scnal f'O '• Fig. j " r nn.366 The Fundamentals of Robotics: Linking Perception to Action Microcontroller 2 Microcontroller 1 i . write a byte to the serial I/O's data register . • Repeat Steps 2 and 3 until all bytes have been transmitted. 6. • Step 2: Send out a synchronization pulse to the synchronization signal line. r H B K ^ E E l Ta s . I ' ' _. The outer loop of the transmitter's synchronous serial I/O may work as follows: • Step 1: Configure the synchronous serial I/O for transmission. • Step 2: When the status register's "transmission complete" flag is set.% %. Operational Procedure of Transmitters Refer to Fig. shift out one bit. the inner loop of the transmitter's synchronous serial I/O may include the following steps: • Step 1: Copy the byte from the data register to the shift register. Memory I A J i ' i ' < i : kj • t3 i A ^ ^ ^L ^ ^ A ^ ^ • 1 i . . . . And. Processor I I ! \ 1 Processor A A [ i ' i T • ' J Memory i ! 1 . 1 4 y Shift Register • i —'' ^ i ibJ!ibJ iL^M^oJ~i • I J . • Repeat Step 2 until all bits have been shifted out. At the same time. I ' i  I  Synchroaous Serial I/O T D a t a Reg'ster . • Step 3: Activate the inner loop. . 6. . 1 Clock signal ^ )  .28.28 Illustration of data communication with a synchronous serial I/O. Assume that microcontroller 1 is the transmitter and microcontroller 2 is the receiver.:m^iM^ ( srj n n n t l Tj Ti f i f l .
if we remove the synchronization signal line.5. Fig.e. and set the "transmission complete" flag in the status register of the serial I/O. Bitoriented Byte Framing When there is a synchronization signal line. (Remember. Step 4: Copy the byte from the shift register to the data register. or system. Repeat Steps 1 and 2 until all bits have been shifted in. A serial I/O without a synchronization signal line is known as an asynchronous serial I/O.4 A synchronization signal line in the synchronous serial I/O increases the cost (i. it is easy for a device.7. • Step 2: Activate the inner loop.29 shows data communication between two microcontrollers using an asynchronous serial I/O. • Repeat Steps 2 and 3 until all the bytes have been received. . So. read in a byte from the serial I/O's data register. logic circuitry to generate and detect synchronization pulses). to take control of a network and transmit data. 6. Asynchronous Serial I/O 6.Information System of Robots 367 • Step 4: Set the "transmission complete" flag in the serial I/O's status register to inform the outer loop that the transmitter is ready for transmitting another byte. • Step 3: When the status register's "transmission complete" flag is set. we need to find a way to resolve two issues: a) notification of taking control of a network and b) synchronization of the bitbybit transmission of a byte by the inner loop of a serial I/O. in networked datacommunication. This informs the outer loop that one byte has been received. Similarly. there is only one transmitter at a time). An alternative solution is to remove this line.e. Step 2: Shift in one bit for each incoming synchronization pulse. Operational Procedure of Receivers The outer loop of the receiver's synchronous serial I/O will work as follows: • Step 1: Configure the synchronous serial I/O for reception. the inner loop of the receiver's synchronous serial I/O will include the following steps: • • • • Step 1: Detect the incoming synchronization pulse. one additional signal line) and the complexity (i.
This state is called the idle state. 1 Memory ' : : .29 Illustration of data communication using an asynchronous serial I/O. One unit of time along the transmission time axis corresponds to one bit. it first identifies the status of a data line. the line becomes busy. when there is no transmission over a network (i. 6. (There will be two data lines for a fullduplex serial link).  Asynchronous Serial I/O *   ' :  i ^  > ' : Asynchronous Serial 1^0 ^ jL. : ^bMbMb^\b3 M Receive : • . i I • i ~ ~ . A common way of solving the first issue for an asynchronous serial I/O is to play with the logicsignal level of the data lines.e. Refer to Fig.m. No other device or system can become a transmitter as long as the data line is in the busy state. or system. 2 T • * § Q_> • * IN^kN^hKH : (fk^ftB. or system.ira_ns. has taken control of the data line. When a device or system has data to communicate. For example. or 0) to the bits of a byte.erl ' 1 :  j i ' ' JL  : 1 ^ ft X ^ I I i ' i ! Processor A J_  i ^^ T Memory . drives the data line to logic low (OV) for one unit of time to indicate that it is about to take control of the network.crocon t r o .£Sigfr ± : ' I : ..368 The Fundamentals of Robotics: Linking Perception to Action Microcontroller 2 M. The solution of driving a data line from the idle state to the busy state for one unit of time is viewed as the insertion of a bit (a bit of logic low.ttiit^aiaaiiiii^^ IT Data Register  A ' ' Pj\bMbMb2\bM I ! j^^^ftRegster  : i 4 V Shift Rcgiscer : i —:Receive llEJlWfeEK5n : ^ ! I . Once a device. This bit is called the start bit. a data line is always at logic high (+5V). ( Fig. If the data line is in the idle state for a certain unit of time. 6.Stt[iil H I . the device.N>riLh^:Hius.27.mt A. no activity). .
These extra bits are known as stop bits. If not. parity and stop bits to the bit series {bi. or systems. this means that there has been an error in communication. if we choose the "odd parity". it is necessary to consider the discrepancy in the predefined transmission speed of a same value (e. If the data byte becomes 0x89 (i.30 summarizes the process of bitoriented byte framing with the insertion of start. then the parity bit p is set to "1" because the number of Is in the byte itself is an even number. Due to the fact that the transmitter and receiver are two different devices. one kilobit is equal to 1024 bits. A typical way to perform error checking at the byte level is to introduce a parity bit p. 10001000). the addition of the extra units of time simply mean the addition of extra bits at the end of the bit series of a byte. corresponds to a bit.... The process of adding a start bit and some stop bits to the bit series of a byte can be called byte framing. of different performance. Parity checking is a simple way to detect error in data communication. 10001001). we can make the number of Is inside a byte be an odd or even number. or quality.1. The parity of a bit series must be preserved during transmission. In fact. In other words.e. 56 kbps). there is no need to have a physical synchronization signal line. Fig.e. we assume that one stop bit is added to the bit series of a byte.2. And. or kbps for short. Here.g. and if the data byte is 0x88 (i. For example. the hardware underlying an asynchronous I/O will also operate with two nested loops: a) the outer . the transmitter and the receiver will negotiate one predefined transmission speed during the initialization stage of data communication. Operational Procedure of Transmitters Similar to the operation of synchronous serial I/O. 7} of a byte.Information System of Robots 369 A common method for solving the second issue is to predefine a set of transmission speeds. the commonly known baud rate of a communication device refers to the transmission speed measured in terms of kilobitpersecond.. Since each unit of time. An easy way to accommodate this discrepancy is to introduce some extra units of time after the bit series corresponding to a byte. In a binary number system. 6. With the extra parity bit p. the parity bit p will be set to "0" because the number of Is in the byte itself is already an odd number. Another advantage is the possibility of simple error checking for each byte. (The RS232 or MODEM have a set of fixed transmission speeds). i = 0. With the use of byte framing. along the transmission time axis.
Refer to Fig. shift out bits from the shift register according to the chosen baud rate. 6. type of parity checking. Assume that microcontroller 1 is the transmitter and microcontroller 2 is the receiver. number of stop bits). write a byte to the serial I/O's data register. The outer loop of the asynchronous serial I/O at the transmitter side may work as follows: • Step 1: Configure the asynchronous serial I/O for transmission.29. loop for bytebybyte transmission of data bytes. • Step 2: Set the transmission speed (i. • Step 2: Copy the framed byte from the data register and status register where extra bits are stored to the shift register. baud rate) and byteframing configuration (i. • Step 3: Detect the data line's status. 6. • Step 3: When the status register's "transmission complete" flag is set.370 The Fundamentals of Robotics: Linking Perception to Action Start Bit Idle / Stop Bit(s) \ Idle 0 b0 bx b2 & b4 b5 b6 b.30 Byte framing for an asynchronous serial I/O. This informs the outer loop that the transmitter is ready for transmitting another byte. • Step 4: Activate the inner loop. (NOTE: In practice. The inner loop of the transmitter's asynchronous serial I/O may include the following steps: • Step 1: Do byte framing. and b) the inner loop for bitbybit transmission of the bit series of a byte. • Repeat Steps 3 and 4 until all bytes have been transmitted.e. • Step 5: Set the "transmission complete" flag in the status register of the serial I/O. P 1 3 / P^ 31 ' Time • Fig. • Step 4: If the data line is idle. the shift register will have more than 8 bits). .e.
they will very likely become part of society as well. Tomorrow. In addition. we can say that all internal operations inside a complex and distributed information system can be achieved in realtime. Today. In society. or systems. robots are used in industry.6 Programmable Timers As we mentioned earlier. • Step 3: Activate the inner loop. Naturally. a robot should also have a standard . it is possible to perform the data processing and data storage in realtime. all social activities are synchronized by a common clock (Greenwich Mean Time). As a result. • Step 4: When the status register's "transmission complete" flag is set. Consequently. shift in bits from the data line and perform error checking if any. This informs the outer loop that one byte has been received. 6. the outer loop of the asynchronous serial I/O will work in the following way: • Step 1: Configure the asynchronous serial I/O for reception. With a realtime programming language or operating system. • Step 3: Copy the byte from the shift register to the data register. • Step 2: Set the baud rate and byteframing configuration to be exactly the same as those used for the transmitter.7. • Step 4: Set the "transmission complete" flag in the status register of the serial I/O. a robot must be able to respond to unpredictable external realtime events in a timely manner. we know that data communication between two devices. the time required for the exchange of a set of bytes is also predictable once the two devices or systems have been engaged in data communication. Thus. • Repeat Steps 3 and 4 until all bytes have been received. read in a byte from the data register of the serial I/O. must be synchronized in one way or another. The inner loop of the receiver's asynchronous serial I/O will work as follows : • Step 1: Detect the start bit. • Step 2: If the receiver is awakened up by the start bit.Information System of Robots 371 Operational Procedure of Receivers On the receiver's side. This is because all operations inside a computing system are synchronized by a common clock. a robot's information system must be a realtime system in order to ensure the predictability of its actions or behaviors.
31: • Active Signal Output. • Active Signal Capture (input). From our study of the computing system's hardware.1 [~^ _w ^ Programmable Timer •< I » Real time Interrupt . This freerunning timer provides a reference time number tc to support timecritical actions. • _. 6. A counter is a register whose value increases at the rate of a clock signal. Micro controller i Processor i ^ P ^ tm ft I i i Memory . One way to overcome this restriction is to incorporate an additional clock system which is programmable. o. sensing or decisionmaking. The resulting circuitry is called a programmable timer. • SelfGenerating Realtime Interrupt.372 The Fundamentals of Robotics: Linking Perception to Action sense of time. this system clock is a freerunning clock which is not reprogrammable by any user or programmer. we know that a microprocessor has a system clock to synchronize its internal operations. When the maximum value of the counter is reached. . 6.31 Generic functions of a programmable timer I/O. (This clock system can simply be derived from the microprocessor's system clock itself). In principle. A programmable timer has its own freerunning timer in the form of a counter._i t External System ^. . . W Actuation Device ' Sensing Device   ^H Fig. A programmable timer is an I/O system because it can provide the following generic functions. it automatically resets to zero and starts over again at the rate of the clock signal. as shown in Fig. . ^ Active Signal Output Active Signal Capture I.
An active signal may be in one of these types: a) the transition from logic high to logic low (falling edge). Example 6. If these waves hit an obstacle.6.6. Fig. 6. its job is to record time number tc into an input capture data register. If the actual time number of the freerunning timer is tc. The active signal from a programmable timer is useful for many timecritical actions such as turning on a machine.Information System of Robots 373 6. 6. This triggers the sensor to transmit sonar waves towards the environment. c) always logic high and d) always logic low. an active signal's type is programmed by the user or programmer. 6. it instantly outputs an echo signal to the programmable timer. When there is an active signal detected by the programmable time. stopping an action.2 Active Signal Capture The second generic function of a programmable timer is the active signal capture. we can derive the distance between the ultrasonic sensor and the obstacle in the environment. Again.31. the echo signal line from the ultrasonic sensor is connected to the programmable timer's active signal capture. . an active signal will be sent out when tc = tc\.4 Consider the application of a microcontroller controlling an ultrasonic sensor for range sensing. b) the transition from logic low to logic high (rising edge). And. If the ultrasonic sensor detects an echo at timeinstant tC2. they will bounce back as echo. An output active signal's type is also programmable. etc. Assume that an ultrasonic sensor's enabling signal line is driven by the signal from the programmable timer's active signal output. The function of the activesignal output of a programmable timer is to output an active signal at a programmed time number or instant tc\. the programmable timer generates an active signal output to enable the ultrasonic sensor. The active signal capture function of the programmable timer stores the time number corresponding to timeinstant tC2From the time difference tc2 — tc\ and the speed of sound in the air. If we denote tC2 the time number held by the input capture data register. triggering a sensor. At timeinstant tci.1 Active Signal Output Refer to Fig.32 shows a microcontroller board and three sets of ultrasonic sensors.7. tC2 indicates the timeinstant when an external event occurs.7.
374 The Fundamentals of Robotics: Linking Perception to Action Fig.e. programmable time intervals. 6. 6. oooooooooooooooooo 6. it may be wise to dedicate one or more microcontrollers to perform diagnostic tasks. Some diagnostic tasks may be more appropriately programmed as interrupt service routines so that they can be activated at a regular time interval. and .32 Example of the use of a microcontroller to control ultrasonic sensors for range sensing. an interrupt signal will automatically halt the execution of the current computational task.6. Some typical diagnostic tasks for a robot include: a) check the level of power supply. c) check the working conditions or alarms of the sensors and d) check the working conditions or alarms of the computing modules.8 Summary It is not particularly important for industrial robots (i. In a digital computer. Due to the low cost of a microcontroller. b) check the working conditions or alarms of the actuators. arm manipulators) to have advanced information systems for dataprocessing.7. An interrupt's service routine is programmable by any user or programmer.3 Selfgenerating Realtime Interrupt A third useful generic function of a programmable timer is the generation of realtime interrupts at periodic. etc. and trigger the microprocessor to execute a service routine associated with this signal. storage.
it would be desirable to have a programming language. this is not the case with humanoid robots. offline programming to schedule their motions. it is not that straightforward.Information System of Robots 375 communication. which supports multiple behaviors (or multibehaving) exhibited by a single body. we learned the basic architecture of digital computers. However. we studied the working principles behind datainterfacing and communication. This supports the claim that an information system based on microprocessors is intrinsically a realtime system with simultaneous and predictable behaviors. The cyclebycycle operations inside the arithmetic and logic unit( ALU) reveal the fact that the program execution of a microprocessor is. or operating system. As for dataprocessing software. Here. predictable. It is easy to assume that a robot's information system will also be the foundation for an artificial brain. for the most part. Cooperative multitasking and preemptive multitasking are two basic solutions for handling the issue of running a pool of multiple. we first studied the philosophy which stresses the importance of developing a humanoid robot's information system. However. a humanoid robot's information system will most likely be formed by a cluster of microprocessors or microcontrollers. However. concurrent. that is: from autonomous mental and physical actors to autonomous behaviors. For dataprocessing hardware. we studied the basic concept of a realtime programming language and realtime operating system. Moreover. or body. A humanoid robot has a humanlike mechanism. an information system can easily interface with all kinds of analog sensors . we highlighted a possible pathway towards the development of an artificial mind. or a few. computational tasks on one. The research and development of humanoid robots will certainly be considered as marking a new era in the history of robotics. And the issue of how to develop the process underlying an artificial mind is still unresolved. Upon understanding the importance of a humanoid robot's information system. With the help of A/D conversion and D/A conversion. one of the challenging questions is: When will it become possible to program a digital computer with a natural language (linguistic programming)? Finally. Thus. we studied the basic working principles of today's computing hardware and software. The ultimate purpose of this robot is to have it perform humanlike behaviors so that it can be deployed in both industry and society. microprocessors. This is because industrial robots depend on manual. In this chapter. There is no pressing need for industrial robots to be equipped with artificial intelligence.
(17) What are the advantages of using a formatted file for data storage and retrieval? (18) Search the internet for the TIFF file format specification. and explain why a microprocessor can calculate such a function. the presence of a programmable timer will certainly ease the programming of the timecritical sensing and actuation tasks. (11) How is the read or write operation carried out between a processor and its memory? (12) Explain why the execution time of a program on a microprocessor is predictable. (15) Explain the working principle which allows a processor to concurrently handle the execution of multiple tasks. Explain why a TIFFlike file format allows for the storage and retrieval of heterogeneous data with a single file.9 Exercises (1) Explain why the information system is important to the development of a humanoid robot. (2) How can you imitate the artificial mind with an information system? (3) What is a behavior? (4) What is a mental actor? What is a physical actor? (5) Rodney Brooks had published an article called. "Intelligence without Representation." To what extent is the statement in the title true? (6) What is a digital computer? (7) What is a microprocessor? (8) What is a microcontroller? (9) Explain the difference between Von Neumann and Harvard architectures? (10) Write the Taylor series of the cosine function y(x) = cos(a:). . With the support of parallel I/O and serial I/O systems. 6. (16) Explain the importance of data storage and retrieval in robotics. For the control of external devices. (13) What is a machine language? What is an assembly language? And.376 The Fundamentals of Robotics: Linking Perception to Action and actuators. it is easy to build an information system on top of a cluster of networked microprocessors. what is a highlevel programming language? (14) Explain the difference between cooperative multitasking and preemptive multitasking.
World Scientific. (26) For networked data communication. C. .. Sacha (1992). (22) Prove that the relationship of Eq. (1991). G. Weiss. Intel Microprocessors. A. 6.5.5 is linear if we choose. M. Odrey (1986). M.. Programming. K. M. Realtime Systems.24. and Applications. (31) Search the internet to learn about the human brain's structure and neural system. Industrial robotics: Technology. 6. A. Programming and Projects. (30) Search the internet to know how the CAN bus works. N. Realtime Systems. M. Robotics: Introduction.2. R. J. G.l.. P. Ri = j v i = 0. C. (3) Groover.. (2) Gilmore. Nagel and N. Explain how to display character "F. (5) Krishna. Maxwell Macmillan. W. L. McGrawHill. Shin (1997).10 Bibliography (1) Fuller. and K. (29) Search the internet to learn how a pair of RS232 devices work. McGrawHill.7 where R is any positive resistance value. 6. Microprocessor: Principles and Applications. and K. Bhurchandi (2001). (4) Halang. M.Information System of Robots 377 (19) What are the typical topologies of a data communication network? (20) Explain the difference between the Polled I/O and the Interruptdriven I/O. (21) Prove Eq. (1995). and K.. M. how do you identify the receiver? (27) Why does a network only allow one transmitter at a time? (28) Search the internet to learn how a pair of MODEM devices work. (25) Explain the difference between the synchronous serial I/O and the asynchronous serial I/O. (6) Ray. (23) Are the conversion times of the D/A and A/D converters predictable? (24) Refer to Fig. McGrawHill. 6. McGrawHill. and explain why it is dedicated to networked industrial equipment control and networked invehicle device control.".
C. (9) (2002). PrenticeHall. (8) Tabak. McGrawHill. (1995). (1997). Using the M68HC11 Microcontrollers. Advanced Microprocessors. J.378 The Fundamentals of Robotics: Linking Perception to Action (7) Skroder. Proceedings of IEEE International Conference on Development and Learning. MIT. . D. USA.
In this chapter. A human being's visual sensory system can effortlessly measure the radiation of light rays from a scene. we will study the basic principles and concepts of digitalimage acquisition. such as an task execution and social interaction. the visual sensory system is the one that is able to produce the most accurate description of a scene that is able to "see".1 Introduction They say that. This sensation of colors is our physiological and psychological interpretation of light. feature extraction and perception of geometry in the next chapter. among the various sensory systems. undoubtedly depend on informationrich and accurate visual perception. They also say that. "Seeing is believing.Chapter 7 Visual Sensory System of Robots 7. Therefore. it is important to understand the critical issues underlying a robot's visual sensory and perception systems. the perception of motion (or action) in a dynamically changing environment is a necessary step towards the development of autonomous behaviors. and instantaneously derive the coherent representation of a scene from images (projections of light rays). the visual sensory system contains the richest information about a scene. Many physical activities." And. digitalimage modelling and digitalimage processing hardware. Light is an electromagnetic wave 379 . We will cover the fundamentals of a robot's visual perception system including image processing. In robotics. as while all sensory systems are complementary.2 The Basics of Light We live in a colorful world. "A picture is worth a thousand words. 7." This is true in robotics.
and to our robots.. is not the smallest physical entity of matter. Light Emission According to physics. the robot is able to visually perceive the stereo images of the scene.. The atom.1..2. also arranged in specific structures (atomic structures).•£ £*. k HHf II H 4 V • • * • " • . Accordingly.• S»fiS!^^l Po/ . however. 7. the world would be invisible to us. in turn. \ tf ^ ™ ! / • • • • • • • ^^^5 ^""^^^T^H I •••••• •«•••. and a pair of stereo images. all matters are composed of molecules arranged in specific structures.1 Physical Properties Without light. a robot has a pair of stereo cameras as part of its visual sensory system. Light carries radiant energy and exhibits both wavelike and (massless) particlelike behaviors in space._ . is composed of atoms. 7. In fact.• E^^^fcJL—••* A • tt • Fig. A molecule. 7. Look at the example shown in Fig. the atom is made up of a nucleus and a cluster of electrons orbiting around .380 The Fundamentals of Robotics: Linking Perception to Action that propagates in space and time. light has the properties of both physics and optics. Because of the illumination of the scene. known as molecular structures.1 An example of a robot's visual sensory system.
2b and Fig.Visual Sensory System of Robots 381 the nucleus. Instead. When the electrons make . an atom's internal energy is confined to a set of discrete levels. The latter mainly constitutes an atom's internal energy. } Ground State Level (c) Emission of light ' Ground State Level (b) Internal energy levels of atom Fig. an atom's internal energy will not make a continuous transition. An atom is an activelyvibrating entity because of the energy contained in it.2 Illustration of light emitting process. as shown in Fig. 7.2a. the total energy contained in an atom comes from: a) the thermal energy (thermal vibration) of the atom and b) the kinetic energy of its orbiting electrons. 7. A nucleus itself is composed of protons and neutrons. 7. A stable orbit is a trajectory along which an electron's energy is conserved. it will only transit from one permissible energy level to another permissible energy level. In other words. This discrete transition of internalenergy levels can be explained by the stable orbits of the electrons moving around the nucleus. 7. According to Danish physicist Bohr's theory (1913). the electrons make the transition across the stable orbits.„ . ElecaronV \^^ ^ ^ ^ ^ s~J Elegjrtta J (a) Atom's structure ' Energy Level n j Energy release * Energy Level n r~ Excited levels T* —__—i ' 1 J —__JL_i Energy absorption .2c. They should also contain gravitational and electric potential energies). as shown in Fig. Basically. When an atom absorbs or radiates energy. (NOTE: Electrons and protons have mass and electric charges.
denoted by j . 7. 7. A wave's wavelength is usually denoted by A. denoted by i. The transition of an atom's internal energy.3. the electromagnetic wave travels along a straight line. This process is called the stimulated emission of light. (7. The laser (i. An electromagnetic wave is a periodic wave.2c. When this transition occurs within a time interval in the order of 10~8s. AEemiSSiOn the energy radiating from the atom. the variation of an atom's internal energy is not continuous but a definite amount. A wave's spatial period is called a wavelength. also known as the ground state level. an atom's internal energy tends to stay at the lowest level.2b).Ei. AEthermai the variation in the atom's thermal energy. This process is known as the spontaneous emission of light. Additionally. as shown in Fig. Inside a homogeneous medium. a definite amount of energy is released in the form of electromagnetic waves that is the origin of light emission. 7. 7. All levels higher than the ground state level are called excited levels (see Fig. or level 1. (7. described by Eq. from a higher excited level to a lower excited level (or ground state level). When a certain amount of energy is applied to an atom. is manifested in the form of an electromagnetic wave.1) Refer to Fig. we have AEemission Electromagnetic Waves The energy radiated from an atom. coherent light of the same wavelength) is produced by a process of stimulated emission. 7. as illustrated by Fig. If there are n periods within a unit = Ej .2c.2. If we denote • • • • Eabsori> the energy applied to an atom. can also be triggered by an external cause such as an electric field. Since an atom's internal energy tends to remain at ground level.e. AEinternai the variation in the atom's internal energy. and absorbed. If Ei and Ej denote the internal energies of the atom at levels i and j .382 The Fundamentals of Robotics: Linking Perception to Action the transition.2) . the application of the principle of energy conservation yields Eabsorb — AEthermal + AEinternai + AEemission. its internal energy will jump to a higher level. this will cause the atom to instantaneously transit its internal energy to a lower level.
of time (one second).3) In a homogenous medium.f (7. This equation shows that a photon's energy only depends on its frequency.3.3 An illustration of a periodic wave in space and time. a wave's spatial velocity (or speed). ' '„ . Therefore. . A wave's spatial velocity is the travelled distance by the wave within a unit of time. this number n is known as the (temporal) frequency.3. the speed of an electromagnetic wave is a constant. Eq.4 i Number of periods per unit of time (frequency) i •• ' Fig. For example. (7.626 x 10~34 Newtonmetersecond (Nms) or Joulesecond (Js). Refer to Fig. denoted by v. As a result. Usually. denoted by / . By applying Eq. 7. is v = A • /. the frequency of an electromagnetic wave in a vacuum is determined by HPhotons (74) An electromagnetic wave is a carrier of energy. The amount of energy carried by a photon along an electromagnetic wave with frequency / and wavelength A is Ep = h.5) where h is Planck's Constant and is equal to 6.Visual Sensory System of Robots 383 i • Wavelength 'i . It is not a continuous stream of undulations but is formed by a train of undulating elements called quanta or photons.5 can also . the speed of an electromagnetic wave in a vacuum is approximately 3 x 108m/s. if we know the wavelength. A photon is an energy carrier with zero mass. 7. 7. 7. we denote c = 3 x 108m/s as the speed of light in a vacuum.
Eq. Visible Light According to Eq.6) Refer to Eq. This process is called interference. (7. we commonly call these electromagnetic waves light or visible light. will also depend on the characteristics of the atom.384 The Fundamentals of Robotics: Linking Perception to Action Table 7.1 lists typical colors and their corresponding wavelength ranges. this causes the superimposition of the two waves. will be determined by X = hAE1 • (77) *—^IJ emission Since Ai? em j SS i On depends on the characteristics of the atom. all materials excited by external energy will radiate electromagnetic wave(s).1 Colors and their corresponding wavelength ranges  Color Red Orange Yellow Green Blue Violet be expressed as Wavelength Range (nm) 622780 597622 577597 492577 455492 390455 EP = hj.1 and Eq. 7. 7. different materials will appear in different colors. 7. Because of the difference in wavelengths (or frequencies). our eyes are sensitive to electromagnetic waves having wavelengths which fall between 390 nanometers (nm) (or 10~ 9 m) and 780nm. The physiological and psychological responses of our eyes to these wavelengths produce the sensation of colors. Interference When two electromagnetic waves overlap while travelling in space. 7. Table 7. the wavelength of an electromagnetic wave.7. The interference process can enhance the amplitude of the electro . And. The wavelength of an electromagnetic wave depends on the property of the atoms in a material.2. radiated from an atom.6 shows that the wavelength of an electromagnetic wave. Interestingly enough. emitted by an atom.
our eyes can perceive visible light. there are three common ways to make an object radiate: • The Use of Thermal Energy (see Fig. 7.e. For an object to continuously radiates electromagnetic waves. Thus.Visual Sensory System of Robots 385 magnetic waves if the phases of the two interfering waves are the same (i. A typical example of this is the light from the Moon at night. more radiations will be emanated. the internal energy of the atoms inside the matter will actively transit between the ground state level and excited levels. the Sun is a natural light source that irradiates the Earth. For example. This phenomenon is called constructive interference. But. it can also attenuate the amplitude of the electromagnetic waves if the phases of the two interfering waves are different by 180°.4a): When heating an object. Light Sources Any object that irradiates a scene is called a light source. 7. inphase). This phenomenon is called destructive interference.4b): When the surface of an object is irradiated by a primary source of electromagnetic waves.4 Common effects of radiation. there must be a balance between energy absorption and energy emission.4. the surface of an object under illumination radiates electromagnetic waves which have wavelengths that primarily depend on the property of the atoms on the object's surface. 7. Radiation Illumination Reflection \ Radiation \ / /* 'Mllllcr ' I Heat I VI V 1 Mailer I Kv) V^ungsten/ Y 1 ' (a) Thermal energy (b) Electromagnetic energy 1I (c) Electrical energy ' Fig. 7. . the atoms on the surface will absorb and release electromagnetic energy. For example. Consequently. As shown in Fig. when a material is in flames. • The Use of Electromagnetic Energy (Fig.
a light's trajectory is actually an undulating wave centered around a straight line in a homogeneous medium. Depending on the smoothness of the surface separating the two media. this phenomenon is called refraction. or a vacuum. a straight line in a homogeneous medium).2. glass. However. Light Rays As we studied earlier. 7. Still. Another typical example is the semiconductor device. These are important in the understanding of a visual sensory system. the reflection may behave in either of the following ways: • Diffuse Reflection: If a surface is not polished properly. the .e. because they explain the origin of chromatic information. a tungsten lamp will emit light when current flows through it.4c): Certain materials will radiate when a current is applied to them. such as air. If this directionaltered light ray bounces back to the same medium. looks like a straight line. Reflection When a light ray strikes a surface between two different homogeneous media such as air and glass. 7. • The Use of Electrical Energy (see Fig.386 The Fundamentals of Robotics: Linking Perception to Action This explains why we can perceive the colors of surfaces illuminated by a light source of white color. which is a mixture of all the colors. LED (LightEmittingDiode). At the macroscopic level. as shown in Fig. its direction is altered. it is convenient to consider it as a ray (i.5. the phenomenon is called reflection. 7. for the sake of studying the geometric properties of light. and is not smooth enough. If the light ray passes through the surface and enters another medium. a trajectory of light in a homogeneous medium. light carries energy (photons) through space and time. For example. This is because light also has inherent geometric properties. That is why we often use the term light ray. a robot's visual sensory and perception systems can also infer geometric information about a scene with the help of light. In addition to chromatic information.2 Geometrical Properties The physical properties of light demonstrate its particlelike behaviors.
is equal to the angle of reflection. the following properties hold: • The incident light rays. If we denote 9i the angle of incidence and 9r the angle of reflection. then 6r = 8i. This type of reflection is called a specular reflection. In a vacuum. The ratio between c and v is called the index of refraction. This type of reflection is called a diffuse reflection. the reflected light rays. Refraction Refraction is the phenomenon where light rays pass from one medium into another. 7. the speed v of light is smaller than c. parallel light rays that strike it will be reflected back in all possible directions. In a homogeneous medium. light travels at a constant speed. which is the angle between the incident light rays and the surface's normal vector. In any other medium. the parallel light rays that strike it will be perfectly mirrored back as a bundle of parallel light rays. If we denote n the index . For a specular reflection. • The angle of incidence.iglil Ra>s V I Air ^4 Fig. and the normal vector of the surface are coplanar.5 Illustration of reflection and refraction of light rays. the speed of light is about 3 x 108m/s and is denoted by symbol c. • Specular Reflection: If a surface is smoothly polished and is like a mirror.Visual Sensory System of Robots 387 Incident Light Rays \ Reflected Light Rays Glass I \ Refracted I. which is the angle between the reflected light rays and the surface's normal vector.
5.5. n Ao (7'9) (7.2 lists the indexes of refraction for some medium with respect to yellow light. an important property of light is the direction change when a light passes through the surface separating two media. for a robot's visual sensory and perception systems.4 n=C. .10) Eq. According to Eq. Since v = A • / (i. Table 7. If we denote Ao the wavelength of light in a vacuum and A the wavelength of light in any other medium. However. which has a wavelength in a vacuum that is about 589nm. the index of refraction is always greater than or equal to 1.e. Alternatively.2 Index of refraction for yellow light  Medium Vacuum Air Water Glass Quartz Diamond of refraction.388 The Fundamentals of Robotics: Linking Perception to Action Table 7. we have X= f A or A= ^°. (7. 7. this means that a light's wavelength will change when it passes from one medium into another.0 1. Eq. 7.5 2. there is no loss of energy. there will be no light.10 indicates that a wavelength will be shortened when light goes from a vacuum into any other medium.0 ~ 1. This angle is also called the angle of refraction.3 1. This refraction parameter measures the speed change of a light ray when it passes from one medium into another.8) Clearly. 7. if the photons are absorbed by the medium. as shown in Fig.5 1. 7. a light's frequency / will remain the same if a photon continues travelling in the same medium. When light can continuously travel in the same medium.3). Let us denote 6r the angle between the refracted light rays and the surface's normal vector. we have Index of refraction  1.
After the light rays enter the glass.2. the light rays will converge or diverge. 7. For optical imaging.Visual Sensory System of Robots 389 Then.g. air). This property can easily be verified by Eq. glass) and the index of refraction n\ of the first medium (e. a new refraction will occur. we have sin(flj) _ n2 (? . This observation indicates that if the top and bottom surfaces of a transparent material are parallel to each other. it is not possible to converge or diverge the light rays.11 is useful to guide the design and fabrication of optical lenses widely found in cameras. l ' ' (712) Eq. and the normal vector of the surface separating two media are coplanar. In other words. which is shaped into two spherical surfaces. 7. However. as shown in Fig. 7. an optical lens system can be best explained using a convex thin lens. exiting from the glass's bottom surface.3 Refraction of Light Rays by a Thin Lens Refer to Fig. the light rays will enter the air again. the following geometric properties governing the refraction will hold: • The incident light rays. and other optical devices or equipment. 7. After the second refraction.6. a convex thin lens has the following useful properties: . Geometrically.5. the light rays. it is not possible to create images with scalable dimensions. Thus is born the optical lens system. will be parallel to the incident light rays. glass). In other words.11. if the top and bottom surfaces of a transparent material are not parallel to each other. sin(0r) ~ m or ni • sin(6i) — n2 • sin(9r).g. they will continue to travel along a straight line.g. If the top and bottom surfaces of the glass are parallel. the refracted light rays. The thickness of a thin lens is negligible. CDROM Readers/Writers. A convex thin lens is made of transparent material (e. • The ratio between the sine functions of the angles of incidence and refraction is equal to the ratio between the index of refraction n2 of the second medium (e. 7. CD players. When they reach the bottom surface of the glass. microscopes.
390 The Fundamentals of Robotics: Linking Perception to Action Sphere 1 \ . • Interestingly.7 Formation of an optical image with a convex thin lens.6 Sectional view of a convex thin lens.. V/ ! J p * y ^ . In other words. • The ray parallel to the optical axis (ray 1 or ray 2 in Fig.' ' Ray 1 •*^_ >». s Ray 2 / Fig. • The ray (ray 0 in Fig. / 2 or / i ) . 7. • The axis that passes through the optical center and perpendicular to the two spherical surfaces is called the optical axis.' Ray0 E^\& y ^ • ' Optical Axis 1  . 7.i \ *• Object  ^. »• "••••* Fig.k / Sphere 2 Optical Center . • The center of the lens is called the optical center.6) which coincides with the optical axis. . it will go through the lens without altering its direction. Lens A . — * ""••.~ d ^ .6) which strikes the lens at one side will be refracted and exit from the other side. the focal lengths at both sides of a convex thin lens are equal to each other regardless of the radii of the two spherical surfaces.5 ^&lge A . The refracted ray will intersect the optical axis at a point known as focal point (i.e. • The distance between a focal point and the optical center is called the focal length. will not be refracted by the lens. 7. 7. i :< do .
When an object is placed in front of a convex thin lens. we have % . as shown in Fig. point A will have a corresponding image A'. 7. point B will have a corresponding image B'.e. 7. it is easy to prove the following wellknown formula: i do + ii. the distance between the optical center and the focal point at either side of the lens). 7. and di the distance along the optical axis between the image and the optical center.g.di or hi = h0*^.Visual Sensory System of Robots 391 The refractive property of a convex thin lens is useful in the formation of optical images. the distance of the optical image relative to the thin lens will be Now. By applying Eq. and hi the height of the image. at 35mm) when the object moves away from the convex thin lens. The distance of the optical image relative to the thin lens is now The above results indicate that the image shifts toward the focal point (e. 7.1 Let us place an object at a distance of 1 meter (m) from a convex thin lens with a focal length of 35 millimeters (mm). As shown in Fig. di J (MS) Example 7.e. From the similarity between the two triangles OAB and OA'B'. as shown in Fig. From the two similar triangles OAB and OA'B'.15 indicates that a convex thin lens is able to form an image of scalable size.7. do <"<> (7. 7. let us move the object back another meter (i. 000000000000000000 Now.7.13. Let us denote / the focal length (i.7. Similarly. let us denote h0 the height of the object in the direction perpendicular to the optical axis. do — 2m). an optical image will appear at the other side of the lens. do the distance along the optical axis between an object and the optical center.15) Eq. The scaling factor depends on the distance of the optical lens .
3 The Basics of the Human Eye Human eyes (including most animal eyes) magically explore the refractive property of a convex thin lens as the basis for forming optical images. This is only achievable with an opticallens system. Example 7.1 Eyeballs In order to form a color image of a reduced scale. do = 2m). Now. the height of the image can be calculated as follows: hi = 0.3. Eq. we know that the image is formed at di = 0. let us move the object back another meter (i. small area. 7. human vision is intrinsically color vision. The above results illustrate that an image becomes smaller when the object is moved further away from the convex thin lens. small size. the eye or camera must have the following three elements: • OpticalLens System: The dimension of a scene can be infinite.15 is useful for inferring geometry from images. In addition.5 • °" 0 3 6 2 7 = 0. From the previous example. Therefore. By applying Eq. it is necessary to scale down the projection of this infinite scene into a definite. . In fact.7.018135 (m). It forms an important mathematical basis for a robot's visual perception system. Here.008905 (m).15.392 The Fundamentals of Robotics: Linking Perception to Action from the object and image (i.5 m is placed at a distance of lm from a convex thin lens with a focal length of 35mm. However. do and di). a sensing device has a definite.5 • = 0.e. oooooooooooooooooo 7. The height of the image is now di = 0.2 Assume that an object with a height of 0.03627 (m). 7. we will study some basic facts about imagesensing by human eyes.e.
the human eye has the three abovementioned elements. and to loosely focus them onto the large inner spherical area. The retina and macula are made of photonsensitive cells. More specifically.Visual Sensory System of Robots 393 • PhotonDetector Array: A light ray is the locus of a photon moving at a high speed (in the order of 3 x 10 8 m/s). This area . They convert the energy. 7.ovea / n Optic nerve fibers / ^^==^^ Blind Spot ^"^ " Brain's Neural U / Vitreous Chamber ' Computing Network s H Fig. retina. This focusing plane is also called image plane. Therefore. 7. iris. • Iris: All sensing elements have a working range. there must have a mechanism to regulate the amount of photons entering an eye or camera. crystalline lens. However.8 Simple illustration of a human eyeball. macula (including fovea) and opticnerve fibers. Comea /^* =====:a ^ : V Macula Optics \fkfm f V \\ / I AZj. Beyond the limit where a signal starts to saturate. The area where the opticnerve fibers are connected to the inner spherical surface of the eyeball is known as the blind spot. inner vitreous chamber.8. the vitreous substance inside the eyeball's chamber and the crystalline lens. into electrical signals which are then transmitted to the brain's neural computing network through opticnerve fibers. In order to capture photons and convert their energies into the corresponding electrical signals. called the retina. The device that controls the amount of incident photons is called an iris or diaphragm. carried by the captured photons. it is mainly composed of: a cornea. the human lens system consists of the cornea. This system's primary function is to refract incoming light rays to sharply focus them onto the macula. Of course. a sensing element will no more respond to any input. as shown in Fig. sectional view. an array of photondetectors must be placed at a focusing plane behind the opticallens system.
7. we can say that rod cells are only sensitive to light intensity. This explains why a cone cell is able to selectively respond to colors. there is no rod cells.394 The Fundamentals of Robotics: Linking Perception to Action is insensitive to incoming light rays. As we know. a cone cell is able to respond differently to the wavelengths of incoming light.3. a cone cell can only moderately respond to light intensity. the rod cells are scattered across the retina and macula. any spherical lens has its own optical axis. George Wald. Because of the low sensitivity. Because of the high sensitivity. forms the basis for human color vision. This selectivity. and his team at Harvard University. the eye's visual axis does not coincide with the optical axis of the lens. rod cells are not able to differentiate among various wavelengths of incoming light. insensitive to colors. a single photon is sufficient to trigger a response from a rod cell. This means that the center of the image at the macula is not the projection of the optical axis of the lens. and thus. 7. Interestingly enough. On the other hand. This phenomenon is also true for a manmade camera as any error in assembling a camera will make it impossible to project the optical axis exactly onto the image center.8. or visual pigment.2 Photosensitive Cells The human eye has two sets of photosensitive cells: a) cone cells and b) rod cells. the amount of rhodopsin inside a cone cell is much less than the amount of rhodopsin inside a rod cell. The line connecting the center of the crystalline lens and the center of the macula is called the visual axis. The distribution of the cone cells inside a human eye looks like a very sharp Gaussian distribution which is centered at the macula. he won the Nobel Prize in Medicine and Physiology for his work on human vision. at the narrow center of the macula. in response to colors. According to Wald's work. So. pioneered the understanding of the functional and molecular properties of rod and cone cells. Artificial color vision works . But. cone and rod cells are sensitive to visible light because of the presence of a protein called rhodopsin. To a certain extent. Refer to Fig. This high sensitivity of rod cells explains why rod cells will only respond to the variation of energy carried by the captured photons. However. As a result. a rod cell is almost a thousand times more sensitive to light than a cone cell. Structurally. a cone cell is much smaller than a rod cell. Since the amount of energy passing through a unit area is called light intensity. In 1967.
The following are key characteristics of human central vision: • Color Vision: As cone cells are sensitive to colors. a focused optical image will fall into a large inner spherical area inside an eye. As a result. which are only sensitive to blue. . the inner spherical surface of the eye is composed of: a) the retina. • Highresolution Vision: The human eye has around 67 million cone cells. Relatively speaking. Thus. • Bluesensitive cone cells. the central vision is intrinsically color vision. it is necessary to illuminate a scene so that the central vision can perform properly. However. However. On the other hand. Since the distribution of cone cells follows a very sharp Gaussian distribution centered at the fovea. By definition. which are only sensitive to red. b) the macula and c) the blind spot. the cone cells of the human eye are magically divided into three types: • Redsensitive cone cells. the center of the macula has the highest density of cone cells. According to the selective response to colors. which are only sensitive to green. As shown in Fig.Visual Sensory System of Robots 395 on a similar principle. the sensing and processing of images from the macula area is called the central vision. the cornea.3. the crystalline lens and the vitreous substance are responsible to refract and focus the incoming light rays to an eye. the light intensity has to be above a certain level in order to trigger any response from a cone cell. • Daylight Vision: Cone cells can only moderately respond to light intensity. • Greensensitive cone cells. while the sensing and processing of images from the retinal area is called the peripheral vision. there are about 120 million rod cells scattered across the retina and macula (except for the fovea).3 Central Vision Together. 7.8. 7. Human central vision makes use of both cone cells and rod cells to convert light into the corresponding electrical signals. the spherical angle of the macula is about ±20° while the spherical angle corresponding to the fovea is about ±2°. the combined density of both cone and rod cells at the macula is higher than at other areas of the inner spherical surface of the eye.
The human eye is composed of a lens system. In addition. In order for a robot to see a colorful world. 7. Therefore. From an engineering point of view.8. The output from the eye is a series of electrical signals which are transmitted to our brain's neuralcomputing network through opticnerve fibers. 7. all important visual information will come from the central vision of our eyes. Only a very small percentage of cone cells is present in peripheral vision. our eyes always focus the sharp optical image onto the macula. the eye can perceive the three primary colors (red. it is necessary to develop an artificial eye.3. commonly called an electronic camera (or camera for short).4 Peripheral Vision The visual attention of our eyes is mainly on the macula. is a good. In the imagesensing area. This is why visualsensing and processing for signals from the zone outside the macula is called peripheral vision. 7.396 The Fundamentals of Robotics: Linking Perception to Action Functionally. and fast motiondetector in image space. an iris. simple. Thus. The photosensitive cells of peripheral vision are mainly composed of rod cells. and an imagesensing area. . the key characteristics of human peripheral vision are: • Dimlight VisionDimlight vision: Since a rod cell is able to respond to a single photon. this can be explained by the fact that the brain's neural computing network performs very simple processing on signals from these cells because the main attention is devoted to signals from the macula's cone cells. called image substraction. One of the simplest imageprocessing operations. peripheral vision works even in a poorlyilluminated environment. It goes without saying that human vision is a very sophisticated system. We will study some basic imageprocessing techniques in the next chapter. green and blue) because the array of cone cells can further be divided into three subarrays which are sensitive to these colors. • Motion Detection: Signals from peripheral vision mainly contribute to motion detection. In other words. central vision is very sharp as well. there are two arrays of photosensitive cells: a) an array of cone cells and b) an array of rod cells.4 Digital Image Acquisition Refer to Fig.
1 Formation of Optical Images As we already mentioned. while a scene can be infinitely large. and a signalconversion circuit. it is easy to construct a visual sensory system composed of an optical lens. an iris. a camera is similar to a human eye. small size. image digitizer. there is no vision.9 The minimum set of hardware components necessary for a robot's visual sensory system.9 shows the minimum set of hardware components necessary for a robot's visual sensory system.4. This process is known as the formation of optical images. it is necessary to first focus the incoming light rays into a small area. light splitter & color filter. an imagesensing array. 7. a camera consists of a lens. signalconversion circuit. artificial iris. Functionally. Fig.' Fig. the most important item in a robot's visual sensory system is the electronic camera. For an electronic camera designed for colorimage sensing. light splitter & color filter. the human eyes have a definite. Just as in the human eye. and computing unit. 7.Visual Sensory System of Robots 397 * jMAgfa **) 2t\ • . Without a camera. imagesensing array(s). In order to perceive a large scene with a small sensing device. 7. Certainly. In this section. we will study these items in detail. the process of forming optical images will involve these steps: . Due to advances in semiconductor technology. similar to the macula in an eye.
1. To avoid any confusion. an object placed at different distances. we explicitly state the definition . This can be done with an optical lens system. In this way. For the sake of simplicity.4. 7. in practice. According to Example 7. di does not refer to the focal length / of an optical lens. an imagesensing device can be a planar device. However. The process of adjusting the position of the optical center is called image focusing. the planar surface of an imagesensing device is called an image plane. 7. let us consider an optical lens system to be a convex thin lens.1 OpticalImage Focusing The first step in forming an optical image is to focus incoming light rays into a small area. we should not fix the distance between an image plane and an optical lens. For the sake of convenience. all optical lenses have a mechanism which allows us to manually (or automatically) adjust the position of the optical center along the optical axis. If an object has an arbitrary shape which is not planar.10 Generic structure of an electronic camera.10. the focused image will not be planar. As shown in Fig.398 The Fundamentals of Robotics: Linking Perception to Action • Opticalimage focusing. we can consider that a focused image falls approximately into a planar surface. will have corresponding images at different distances relative to another side of the convex thin lens. Since a scene has a certain depth which is the permissible variation range in the direction parallel to the optical axis.1. • Light splitting & color filtering. di is commonly called a focal length. Camera casing ~^^ Image Sensing Array (Image Plane) / / / Ws Optical Lens Object W / / / 1_/ [ \ N \ Optical Axis Light Splitter and Color Filters Signal Conversion Circuit Fig. 7. relative to one side of a convex thin lens. Adjusting the position of the optical center is equivalent to adjusting distance di between an image plane and an optical lens. In fact. • Control of incoming light.
If point B is on the optical axis.\\/\do\\. 7.16 determines the dimension of an image spot. its image will be a point image. Eq. where the image at point B is in focus. 7. (7. Since an imagesensing array is composed of discrete sensing elements (or cells) which are arranged in a regular pattern. we call it the focal length of a camera and denote it with fc. 7. For the sake of clarity. £ > . the circle enveloping the image spot is called a circle of confusion. when a point on an object is out of focus.16 clearly illustrates the outoffocus phenomenon as well as the concept of the circle of confusion.11. 7.15. However. 7. In general.11 Illustration of an optical image focused onto a planar surface. When a point on an object is in focus. any change in the object's distance relative to the optical lens will cause the image to become a spot. the image spot will be a circle."t.1 Focal length in a robot's visual sensory system refers to the distance between the image plane and the optical lens.^ v '• *^~^v I I" Lens / Color filter Fig. the image of a point will appear to be in focus as long as the circle of confusion is not bigger than .Visual Sensory System of Robots 399 of focal length as follows: Definition 7. iris image plane j • circle of confusion M l ^_^^»— * ^ J T ''—>B ' A i"" C ® Focus Point » '^. Differentiating this equation with respect to time allows us to obtain the following relationship between the variation of hi in the image plane and the variation of the object's distance do relative to the optical lens: \\Ahz\\ = ho»^r. This is illustrated by Fig.16) Eq. Refer to Eq. the sharpness of the images formed on an image plane depends on distance do between an object and the optical lens. Given a value for fc.
the circle of confusion has a fixed permissible size as well. When an iris device is present in a camera. However.11. such as a photosensitive cell. (7. \\/\hz\\max. This clearly explains how we can perceive a large scene as wellfocused images. As shown in Fig. 7. all cameras have an iris device which can be manually or automatically controlled in order to regulate the amount of incoming light. the depth of field increases as well. described by A/ij max . = ^— . when do increases. the image spot on the image plane decreases. Therefore. This hole is called an aperture. We will study CMOSbased imaging sensors in further detail later in this chapter. Interestingly. it is easy to develop active imagesensing cells which have working ranges that can be dynamically reprogrammed. the largest permissible circle of confusion. its image spot on the image plane has a fixed size. which can regulate the amplitude of input. it is necessary to automatically adjust the iris device. an iris device will allow us not only to control the amount of incoming light but also to adjust the permissible dimension of the circle of confusion. Since the size of the imagesensing cells is fixed. the image spot of a point will appear to be blurry (out of focus). Therefore. this can be avoided if we are able to adaptively adjust the working range of the sensing elements.4. Eq. 7. With CMOS (Complementary Metal Oxide Semiconductor) technology. the amount of incoming light is controlled by adjusting the diameter of the circular hole of that iris. Interestingly enough. 7. If a scene's lighting is dynamically changing.17) no • di The value of Ado is called the depth of field of a visual sensory system. it is necessary to have a control mechanism. This second advantage can be explained as follows: If we have a point at a fixed location in front of an optical lens. However. Otherwise.16 allows us to determine the largest permissible variation of Ad0 as follows: Adomaa. When an . is fixed as well. When input to a passive sensing element goes beyond that working range. when the diameter of the iris decreases to a value less than the diameter of the optical lens. Since the size of the sensing elements is fixed. the output saturates.2 Control of Incoming Light A passive sensing element always has a limited working range.400 The Fundamentals of Robotics: Linking Perception to Action the size of the sensing element (or cell).1.
7. In this way. When the aperture of the iris is bigger than or equal to the dimension of the optical lens. 7. With a simple drawing. If we denote Ahi the variation of an image spot without an iris.19) (7. Human eyes can perceive colors because of the presence of three types of cone cells: • Cone cells which are sensitive to red light. we can easily verify this fact. 7.4. fc$ = 1. Now. Each beam is then channelled into a specific color filter (red.18 simply illustrates that Ah* is not bigger than Ahi. 7. then we have Ahi = ki»Ah*.Visual Sensory System of Robots 401 iris reduces the size of a point's image spot. and Ah* the variation of an image spot with an iris.12. When it becomes smaller than the diameter of the optical lens. A color filter is a transparent material which permits lights having a wavelength within a very narrow range to pass through. • Cone cells which are sensitive to blue light. as shown in Fig. Substituting Eq.18) Eq. 7.3 Light Splitting & Color Filtering Human vision is intrinsically color vision. green or blue). let us define ki as an iris effect factor. There are two common ways of performing light splitting and color filtering. • Cone cells which are sensitive to green light.12. . (7.\\Ah*\\max. Eq.19 indicates that a smaller aperture permits a larger depth of field because ki > 1.17 yields \\Ado\\max = ^—*ki. this means that the permissible variations of an image spot can be increased. 7.1. Independent Light Splitting and Color Filtering We normally place color filters behind an optical lens. This is shown in Fig. With color filters. ki > 1. we can selectively channel lights of a specific wavelength into an imagesensing area. A simple way to perceive colors with an electronic camera is to use a light splitter and color filter. 7.18 into Eq. we can split incoming light into three separate light beams. which filters the separated light beams.
Beam 3 — „ f Mirror Fig.402 The Fundamentals of Robotics: Linking Perception to Action Red Filter . Green Filter I Beam 1 ^ Beam 2 ^Splitter X y ^ » ytf Incoming Light Rays Green Beam A Blue Beam ^ '. and B filtering elements to form a filtering pattern. this . an obvious drawback is the cost and complexity. the camera output will be a set of three RGB component color images. the notable advantage of a filtering pattern is its simplicity and lowcost. for a robot's visual sensory system. as shown in Fig. As it is sufficient to use a single imagesensing array (chip) to convert incoming light into corresponding electrical signals.. 7. and is called the Bayer Pattern. Red Beam ^ . Unfortunately.13. He included two filtering elements for the green color because human eyes are more responsive to this color. Correlated Light Splitting and Color Filtering An alternative solution instead of splitting the incoming light into three separate beams is to design a special color filter which is composed of an array of filtering patterns. 7. G. Under the scheme of independent light splitting and color filtering.12 Illustration of independent light splitting and color filtering. mirror. we have the freedom to choose any combination of R. A color camera which does this is called a 3CCD camera. He proposed a filtering pattern formed by a 2 x 2 block of GR/BG filtering elements. 4 ^ ~Y Splitter . color niters and three CCD chips (including associated electronic circuits). because 3CCD camera consists of a light splitter. Since each beam after light splitting can be sensed by a dedicated imagesensing array (chip). it is necessary to use a set of three ChargeCoupled Device (CCD) chips to independently convert the three separated light beams into the red. This idea was first proposed by Bryce Bayer from Kodak Inc./ Jr M Blue Filter T • . However. However. green and blue component color images. One advantage of this method of light splitting and color filtering is that it preserves the spatial resolution of incoming light.
vi I Vg . G. + cb2. G.13 Illustration of correlated light splitting and color filtering. V'g + Cgi . In addition. and B values will be the weighted sum of the direct output. or decorrelation.Visual Sensory System of Robots 403 ^ Color F i l t e r . After color correction. ^ —  Mosaic Color Filter G B R G G B R G G B R G G B R G G B 3 •§ I . VI (7. in order to retrieve the three RGB component color images from the output of a single imagesensing chip. V'g and Vfc' are the direct output of R. and B values from the sensing elements placed behind a filtering pattern. .:' / >) BJ G . it is necessary to perform a colorcorrection procedure.Cgl • V. vg + cr3. vg + cb3. vb' where correction matrix C is ( Crl Cr2 Cr3 \ Cgl Cg2 Cg3 I .20) {vb = cbi • v. 7. The process of color correction can be explained as follows: Assume that V^. A simple way to compute weighted sums if there are four elements in a filtering pattern is as follows: r vr = crl • v. </) ^ ^ j ^ ! ' ^ I I I I ^ " 1 ^ "  £ G R G R G R G R G B G G R B G G R B G G R B G G R B G _ B _ J ^ B G B G B G B Filter Pattern / / i G R ' G R G R G R G . ' "^—'*'i I ' IBJ G I' B 1 ° IBJ G B IG B *sj4^'. + cr2. The actual output of R. Cbl Cb2 Cb3 ) Correction matrix C can be determined by a colorcalibration process. a color camera's output is still a set of three RGB component color images.'" / Fig. + Cg2 . advantage is achieved to the detriment of the spatial acuity of colors.
the internal energies of the atoms at the surface will jump to higher levels. such as video entertainment.4. . visual inspection and visual guidance. the orbiting electrons of the atoms will even jump out of the largest stable orbit. 7. This phenomenon is called the photoelectric effect and can be explained with Bohr's theory. As a result. 7. A common way to do this is to use photosensitive cells made of semiconductors.10. Heinrich Hertz accidentally discovered that the surface of certain materials will emit electrons when exposed to a bombardment of light rays. CMOS is the basic element behind all microprocessors. With the advent of semiconductors.4.2 Formation of Electronic Images Refer to Fig. When the CMOS is produced in large quantities. the photoelectric effect has become a practical way of developing imagesensing chips for electronic cameras. After the formation of an optical image on a planar surface (image plane). 7.2 CMOS Imaging Sensors As a result of the invention of active imagesensing cells at NASA's Jet Propulsion Laboratory in the mid1990s. For some specific materials.2. As a result. A camera which uses photosensitive cells is called an electronic camera or filmless camera. These electrons become free moving electrons in a material. its manufacturing cost goes down. the incoming light striking the surface will trigger the release of electrons. the CMOS imaging sensor is a commercially viable and attractive solution for electronic imaging applications. the CMOS imaging sensor has become increasingly popular. Let us take a look at two examples: the CMOS and CCD imaging sensors. as illustrated in Fig. 7. the critical issue is how to develop an imagesensing device which will convert optical images into corresponding electronic images.404 The Fundamentals of Robotics: Linking Perception to Action 7. memory chips and other digital devices.2.2.4. The electrons orbiting the atoms will transit to outer stable orbits.1 Photoelectric Effect In 1887. The photoelectric effect works as follows: When light rays strike the surface of a material.
The addition of one external electron to an atom of doping material will make that atom behave like an atom of pure semiconductor material. of Robots 405 Ptype and Ntype Semiconductors The term semiconductor is composed of two words: a) semi. A semiconductor's (electric) current conductivity is between a perfect conductor and a perfect isolator.Visual Sensory System. In other words. or doping material.3 In a material made of Germanium. • Case 2: An atom of doping material has one electron more than that orbiting the atom of pure semiconductor material. The process of adding a different material to pure semiconductor material is called doping. In this case. the pure semiconductor doped with an impurity material will readily act as a donor of electrons. Germanium's atom has 32 orbiting electrons. Two typical external causes are: a) an electric field. This type of manmade semiconductor is called an Ntype semiconductor. This type of manmade semiconductor is called a Ptype semiconductor. the pure semiconductor doped with an impurity material will readily act as a receptor of external electrons. an atom has 32 orbiting . we can choose Germanium as the pure semiconductor material. The material used for doping purposes is known as an impurity. to pure semiconductor material. the release of one electron from an atom of doping material will make that atom behave like an atom of pure semiconductor material. and b) exposure to lights. or doping material. the atoms of doping material will be surrounded by the atoms of pure semiconductor material. When the doping material is evenly diffused inside the pure semiconductor material. By adding an impurity. A special method of doping is to add a material which has an atom with one electron more or less than the semiconductor material's atom. Example 7. In other words. the conductivity can be controlled externally. For example. one of these two things may happen: • Case 1: An atom of doping material has one electron less than that orbiting the atom of pure semiconductor material. meaning "something that conducts electricity". If pure Germanium material is mixed with a small percentage of doping material. meaning "partial" and b) conductor.
The main body sits on a base which can be either metallic or a layer of isolator.4 Now. if we mix a pure semiconductor material. the result will be an Ntype semiconductor because an atom of Arsenic material has 33 orbiting electrons.14 Illustrations including a sectional view and symbolic representation of a photodiode and phototransistor. ••OOOOOOOOOOOOOOOO Photodiodes and Phototransistors The invention of Ptype and Ntype semiconductors in the late 1940s accelerated development of important electronic devices such as diodes and transistors. 7. P » Antireflection coating Anode ( t ) X^V^— ~~T «^~~Jx NIype Silicon \ j / \ [v^ / / Base (a) A photosensitive eel] L—^' Cathode ( ) n^_ (b) Symbol of photodiode ^f (c) Symbol of phototransistor Fig. 7. If the Germanium material is doped with a small percentage of material made of Arsenic. . with a small percentage of material made of Gallium. Light Rays WWUWWI Light Rays / 'p Light Rays B J. After the Ptype semiconductor is in place. we will obtain a Ptype semiconductor. 7.14a shows one design solution for photosensitive cells. oooooooooooooooooo Example 7. On the top surface of the main body. made of Germanium. This is because an atom of Gallium material has 31 orbiting electrons. For the example shown in Fig. If we specifically arrange Ptype and Ntype semiconductors together. we can develop an element called a photosensitive cell.14a. In fact.406 The Fundamentals of Robotics: Linking Perception to Action electrons. a thin layer of Ptype semiconductor is created through a diffusion process. today's microprocessors and solidstate digital devices are all made of diodes and transistors. we can see that the main body of a photosensitive cell is made of an Ntype semiconductor. Fig.
In general. Cj should be big. The direction of the current is opposite to the direction of the moving electrons. we can draw an equivalent electronic circuit. a photosensitive cell acts more like a current generator than a voltage generator. Cj can be big or small. For a ChargeCoupled Device (CCD) sensing element.14a. In general. as shown in Fig. The basic elements inside this circuit include: • Ip: A current generator with an output proportional to the amount of photons striking the surface of the photodiode. Because of the PN junction. as shown in Fig. an external closedcircuit will permit current to flow from the anode to the cathode. In electronics. as shown in Fig.14b) or phototransistor (see Fig. light absorption) material. After this. The device. 7.Series resistor of the PN junction. Clearly. the value of Rs is usually small. The Equivalent Circuit to a Photodiode In order to better understand the electrical properties of a photodiode. as shown in Fig. ActiveSensing Cells Refer to the circuit equivalent to a photodiode. a voltage generator or a . As the layer of Ptype semiconductor is very thin. • Cj: A capacitor of the PN junction. the value of Rj falls in the range of megaOhms. the Ptype semiconductor releases electrons as a result of the photoelectric effect. Ip falls in the range of milliAmperes or microAmperes. A photodiode may behave like a current generator. 7. the electric charges (the accumulation of electrons) build up. 7. 7. Depending on design and intended application.e. The PN junction was invented in 1949 and is the basic element in diodes and transistors. a photosensitive cell is also referred to as a photodiode (see Fig.Visual Sensory System of Robots 407 the surface is coated with a layer of antireflection (i. Depending on light intensity. is a photosensitive cell (or element). It works as follows: When light strikes the top surface of this device. If a pair of anode/cathode is placed on the device. • Rs'. Here. electric charges will flow into the retype semiconductor. 7.15.15.14a. 7. if a main body made of an Ntype semiconductor is deposited with a layer of Ptype semiconductor.14c). it is called a PN junction. • Rj: Shunt resistor of the PN junction. we call a photosensitive cell a photodiode.
(7.15 Circuit equivalent to a photodiode. (2) A CurrenttoVoltage Conversion Transistor: This transistor takes current from the photodiode as input.16a. . 7. This possibility led to the invention of active pixels in the mid1990s at the JetPropulsion Laboratory in California. one possible method is to increase the number of transistors in the photosensitive cell.408 The Fundamentals of Robotics: Linking Perception to Action I 1 1 O Fig. By adjusting the design parameters of the photodiode. In order to convert an optical image into its corresponding electronic image. If R is the electrical resistance across the collector and emitter of the transistor. an active pixel is composed of the following elements: (1) A Photodiode: This transistor is responsible for converting light intensity into a corresponding current. Collector and Emitter). the current (or voltage) produced by a photodiode is usually small. for short. it is necessary to have an array of imagesensing cells. Since the photodiode is made of the semiconductor. then we have Vp = (l + K)*R*Ip. 7. The former is the design objective for CMOS imagesensing cells while the latter is the design objective for CCD imagesensing cells. These cells are also called image elements or pixels. it is necessary to enhance the electrical signal of photodiode in one way or another. a photodiode can serve as a pixel. falling in the range of milliAmperes or microAmperes.21) (NOTE: A transistor has three terminals: Base. As shown in Fig. we can make it behave more like a current generator or more like a capacitor. Obviously. and K is the amplifying gain. capacitor. Output voltage Vp from the transistor is proportional to Ip. However. Thus.
16a. f «>"".2 . Assume that the active pixel in Fig. or active pixel. (3) A Reset Switch: This is a transistor which serves as an electronic switch.. " " CiirrcnriuVoluge Conversion IRUKISIUI (a) Active Pixel ' a''sdir" Ml 'oh Column Line Fig. Although the output voltage signal from an active pixel is still weak. this is not a drawback because the voltage signal can be easily amplified by a circuit.> •v—' .16a is located at the (i.16 Illustration of a conceptual design of an activesensing cell. From Fig. we can see that the power supply (+V) is connected to the ground when the reset switch is "on". The logic "AND" device has two input lines: a) row select and b) column select. 7. as . 7. i . This logic circuit consists of a logic "AND" device as well as a transistor which serves as an electronic switch. as shown in Fig. The above elements make an active pixel look like a controllable voltage generator. a logic circuit of cell selection can be added to an active pixel. i ™y^.\ ••. (4) A Logic Circuit for Cell Selection (optional): In order to individually select a cell from an array of sensing cells. Its purpose is to discharge the photodiode after the output has been read.16b.Visual Sensory System of Robots 409 R Active Pixel . In other words. If the voltage signal is amplified by the circuit. this switch resets the photodiode so that it can start a new sensing cycle. 7.j) position (ith row and jth column) of an imagesensing array. 7.
Hence. The time of exposure can automatically be determined. A better solution is to read out the voltage signals rowbyrow. there is only one voltage output from an active pixel. A more practical solution. Refer to Fig. In order to convert a twodimensional optical image into a corresponding electronic image. If an active pixel has a cellselection circuit. In other words. In this case.410 The Fundamentals of Robotics: Linking Perception to Action shown in Fig. therefore. this will be costly and impractical. is to read the electrical signals from an array of active pixels within a short period of time. if one image is read within a time interval of 40 milliseconds (ms). the amplified output will be V • = CMOS ImageSensing Array At any time instant. Fig. we can individually read the voltage signal from an active pixel at location (i. the critical issue is how to simultaneously read the electrical signals from these active pixels.16. it is necessary to make use of an array of active pixels.16b. based Rj2 *Rji »V (7 22) Rji • (Rj3 + Rji) . it is similar to a memory cell which is an integratedcircuit for holding the logic value "0" or " 1 " . 7. 7. Each active pixel can be treated as a memory location with a dedicated address (i. If we place active pixels together to form an array.j). 7. an imaging sensor will be able to output 25 images per second.17. is it necessary to read the electrical signals from an array at one time instant. or can it be done within a short time interval? If it is necessary to simultaneously read the electrical signals from an array of active pixels at one time instant. a single. For example. • Step 2: Expose active pixels to incoming light rays for a certain period of time. it is easy to form an imagesensing array with a set of active pixels. the number of active pixels in an imagesensing array may be on the order of 106 pixels. active pixel can only serve as an imaging sensor for a single point. 7. For example. In this way.17 shows a possible design solution for constructing a CMOS imaging sensor based on an array oinxm active pixels. The procedure of reading voltage signals and converting them into a continuous video format is as follows: • Step 1: Reset all active pixels.j) (zth row and jth column). This is because the size of an array may be prohibitive. it will be too slow. as shown in Fig. If the electrical signals are read out pixelbypixel.
• Step 5b: At the same time. An image can be either analogue or digital. • Repeat Steps 1. and 5 to continuously generate a video. Fig.17 Illustration of the generic architecture of a CMOS imagingarray having nxm active pixels. and set row index i to be 0. 7. and amplify the voltage signals from it. 7. 3. increase row index i {i = i + 1). as shown in Fig. 2. on light intensity and the characteristics of the active pixels.17. • Step 5a: Pack the amplified voltage signals from row i into the current image.Visual Sensory System of Robots 411 Video Output 1 •C^ — Analogue or Digital Video Conversion  wiP  wnj ~\MI1 [ \\irl j 1 AP  I AP h 1 AP  I AP h Addresses £ * g Row i ] AP \~ ] AP \ 1 AP   AP f _ _ _ _ ^ —I AMP ^Amplifier AP = Active Pixel I AP H I AP H I AP H Addrew Decoctar ™ ^Addresses I AP H I j Fig. 4. • Step 3: Select columns. and repeat Steps 4 and 5 until an image has been read. • Step 4: Select row i. which is an image inside a video. Each column will have its own dedicated voltage amplifier. 7. A CMOS imaging sensor's notable advantages include: .18 shows an example of a CMOS color camera.
As a result. a CMOS imaging sensor is energy efficient.18 Example of a CMOS Color Camera. A CMOS imaging sensor is similar to a memory chip. 7. 7. • Random Pixel Access: Refer to Fig. we can treat a reset action as an overflow. If V/j is the actual output of the voltage signal without considering . • Low Power Consumption: The voltage signal in a CMOS device is not higher than +5V. Because of the reset switch inside an active pixel. Let n^ denote the number of overflows which occur at the active pixel located at (i. This is a very important feature for embedded and/or mobile applications where power consumption is a major concern. As a result. the unit cost of a CMOS imaging sensor is very low. Due to the mass production. similar to the overflow of a numerical counter. In fact.17.412 The Fundamentals of Robotics: Linking Perception to Action CMOS Image Sensing Array Coated with a Mosaic Color Filter _ t nlmn' ^m'"1 pr»»" * **iiMP^ Optical Lens Fig. one can directly access an individual pixel if a fulladdress decoding circuit is present. it is easy to increase this working range.j). CMOS imaging sensors are manufactured in the same facilities as microprocessors and memory chips. • Wide Dynamic Range of Active Pixels: A passive sensing cell has a limited dynamic range (working range). • Low Cost: The elements inside a CMOS imaging sensor are transistors or transistorlike devices.
Because of this. the frequency of reset can be high if light intensity is strong.19 shows a conceptual design solution for a CCD pixel. For example. the first color CCD camera was announced in 1972. However. ChargeCoupled Devices (CCD) A CCD can be treated as a pair of coupled capacitors: one for light sensing and the other for charge storage. After this. The camera shown in Fig. The first capacitor is a photodiode that can serve as a photosensitive cell which will convert incoming light into corresponding electrical signals.4. 7.2. . 7. it is not necessary to have an automatic iris (or autoiris). however.n •• • Vmax where V^ax is the maximum voltage signal which triggers a reset action. Thus.3 CCD Imaging Sensors An important milestone for electronic imaging was the invention of the ChargeCoupled Device (CCD) ) by Boyle and Smith from Bell Labs in 1970. 7. 7. Most electronic cameras are based on CCD imaging sensors (or chips). • Automatic Compensation of Light Intensity: We can also make use of the reset switch to automatically adjust the response of an active pixel to lighting conditions in a scene. Both were from Bell Labs. a CCD can also be called a CCD imaging element or CCD pixel.• — V' 4. it is easy to incorporate signalprocessing hardware into an imaging sensor.Visual Sensory System of Robots 413 overflows. and low if the lighting is dim. The photodiode in Fig. Today. • Onchip Processing: Since CMOS imaging sensors use the same manufacturing process as microprocessors and memory chips. it is simple to develop smart cameras with builtin programmable functions.18 has an optical lens with an iris that is not automatic. the market share of electronicimaging products was made up of CMOS cameras. Fig. Because of this. CCD cameras still dominate the highend (high resolution and quality) electronicimaging products. the first blackandwhite CCD camera was announced in 1971. And. then the true output of the voltage signal will be V.19 behaves more like a capacitor than a current generator. since the late 1990s.
When the photodiode is exposed to incoming light for a certain period of time. a charge transfer occurs. there is a capacitor. the Ptype semiconductor will release electrons. A CCDpixel's function can better be described by its equivalent circuit. as shown in Fig.19b. as shown in Fig. 7. This is not the case. Since the layer of Ptype semiconductor is very thin. we can apply bias voltage Vb which will in turn increase the signaltonoise ratio. 7. Next to a CCDpixel's photodiode. 7.g.414 The Fundamentals of Robotics: Linking Perception to Action Light Rays Vg UUUIUU Rj Ami reflection coating I r+T: h ' : . Rj : Electrical resistance along the path connecting Cj to Cs. When incoming light strikes the surface of a CCD pixel.Electrical resistance when transferring electric charges out of Cs Refer to Fig.Capacitor for storage of electric charges from Cj. an active pixel made of CMOS). The basic elements inside a CCD pixel include: • • • • Cj: Junction capacitor of the photodiode. if the photodiode behaves like a current generator (e. . I ' i RS  _ _ To serial shift along a column Ntype Silicon / ^ ^ ^ Gate I y j ' ' ' • • • T • Electric Field (a) Charge Coupled Device (b) Equivalent Circuit Fig. 7. This capacitor serves as a reservoir to store electric charges from the photodiode. A charge transfer from Cj to Cs will have no signal loss as the thermal effect caused by Rj is negligible. Cs. the gate voltage Vg is turned "on" to generate an electric field. This electric field induces electric charges toflowfrom the photodiode into the capacitor. i .14. This explains why a CCD imaging sensor can produce highquality electronic images.19 Illustration of a CCD pixel. . Then. . Rs'. In order to enhance this accumulation. the freemoving electrons will flow into the main body (Ntype semiconductor).19b. however. This will result in an accumulation of electric charges. .
20 shows a conceptual design solution for a CCD imaging sensor made of an array of n x m CCD pixels. The CCD image sensing array works as follows: • Step 1: Discharge all CCD pixels. the manufacturing is even more expensive if we attempt to build an "active pixel" based on a CCD. 7. the electric charges of CCD pixels in a column can be sequentially shifted out to a signalamplification circuit for postprocessing. A common solution to read the electric charges is to treat the chargestorage capacitors within a column of a CCD imaging array as a shift register. This transfer is done horizontally (for all columns) at the same time. an existing CCD imaging sensor is not similar to a memory chip. Amplify the electric charges which are vertically shifted from row 1. Since a CCD pixel is more complicated than a transistor. The serial shift in each column must be synchronized by a serial shift clock. Refer to Fig. we can place a set of CCD pixels into the shape of an array. Fig. . 7. • Step 6: Pack the amplified electrical signals into the current image which is either analogue or digital. This can be done either by transferring the CCD pixels' electric charges out or by discharging the CCD pixels using a reset circuit. and is called parallel shifts. CCD imaging sensors are more expensive than CMOS imaging sensors. Each cycle of serial shifts will transfer a row of electric charges from row 1. manufacturing of CCD imaging sensors is not the same as it is for microprocessors and memory chips. In this way. • Step 2: Expose the CCD pixels to incoming light for a certain time interval. a CCD pixel consists of a pair of coupled capacitors: one for sensing light rays. In addition. we face the same issue of how to read the electric charges. • Step 5. and is called serial shifts. With an array of CCD pixels. This is done by a series of chargetovoltage amplifiers. This operation treats the chargestorage capacitors in each column as a single vertical shift register. As a result. • Step 3: Transfer electric charges from the photodiodes to the chargestorage capacitors.20. Thus. and the other for storing electric charges.Visual Sensory System of Robots 415 CCD Imaging Arrays Just as with a CMOS imaging sensor. and we cannot access individual CCD pixels. • Step 4: Transfer the electric charges vertically (rowbyrow). As we discussed earlier.
Photodiode S = Storage " Drive Circuit for Serial Shifts Fig... the drive circuits for CCD imaging sensors are complicated. 3. • Step 7: Repeat Steps 4. 6. The two cameras on top of the robot.20. I p[ s ]  P s  I p  s  Rown1 P S  P  s   p  s  Column m 2 Column m 1 Column m P S II P S   P  S Column 1 Column 2 Column 3 Rown AMP = Amplifier p .. and 6 until one image has been read. 7.416 The Fundamentals of Robotics: Linking Perception to Action Video Output <~^ Analogue or Digital Video Conversion n •••—.. because it is necessary to have high voltage to transfer charges. • Repeat Steps 1. In the case of a color .1.17 and Fig.. To perform parallel and serial shifts of electric charges.  p  S  PS P s  ~P~ s  PS Row 1 Row2 ... AMP' t k M TT ii. 5.. and 7 for a continuous video. 7. PS Row 3 ': : : 1 *• P S P S P S P S P S P S Rown2 ^ I ^ I p j S 11 P I S I I P I S I • . 7. tk .3 Formation of Digital Images Refer to Fig. 2. are CCDcolor cameras. 4. P S P S ^^V^V I p  S I F P T SJ [ P S : : : . 5.4. the CCD imaging sensor is not as energyefficient as the CMOS imaging sensor. : . Thus. 7. 7. Additionally.. as shown in Fig. ti . ~ £ J i ^ fp si I p si I p S ' : : • . it is necessary to make use of clock signals and high voltages. The output from a camera (either CMOS or CCD) is a continuous stream of images (video)... : ZTZTZV ..20 Illustration of the generic architecture of a CCDimage sensing array having n x m pixels. ' : . i i \MPJAMP AMP AMP AMP ii a JL it J k ^  * ^ P S  [ p  S  P S P s . and the cameras using CCD imaging sensors are bulky.
7. A TV program consists of a continuous stream of images encoded with the associated audio signals.21 Visual sensory system. the direct output from an imaging sensor consists of three streams of images: red component color images. blue component color images and green component color images. Output from a digital camera can be a stream of twodimensional digital images. image digitizer and computing hardware. If a camera's output has an analogue signal. then it is called a digital video camera.* Image Processmg <«= ^ • — Hardware '^= g I 1 pq < 1 Video Camera < j Digital Images . Thus. As shown in Fig.21). it is necessary to convert this into a stream of digital images. 7. If a digital camera outputs digital video. if a camera outputs analogue video.4. however. the camera is called an analogue video camera. called a digital video. This analogue signal is also called an analogue video.Visual Sensory System of Robots 417 camera.3. 7. images and news from a variety of TV programs.21. Since analogue video cameras are still popular.—I ^ Digital Camera <^ V Fig.1 Analogue Videos Everyday. broadcasting through air or over cable. representing a stream of twodimensional analogue images. composed of a camera. we receive a large amount of information. (See Fig. Another popular and important source of images are . 7. A I Digital Images T Video(s) . a robot's visual sensory system needs an image digitizer which converts analogue video into the corresponding digital video. the computing hardware in a robot's visual sensory system is digital. Output from an analogue video camera will be a continuous onedimensional signal. Digital cameras have been on the market since the late 1990s.
Only 484 of these lines are used to carry the signals of a video image. This time interval is called a line period. Hence. VCR etc. These lines are called signal lines. there is concern over how to standardize them.418 The Fundamentals of Robotics: Linking Perception to Action videos. guaranteeing its compatibility with display equipment such as TV receivers. while it is possible to specify the parameters. a standard creates a platform for product and service providers (manufacturers of a similar product) to compete against one another in order to better serve the endusers (consumers).S. let us stick to the term image. allowing an image to last for 33. video monitors. a frame usually refers to a coordinate system. The following are specifications of NTSC: • Image Rate: The image rate is 30 images per second. When an image is read rowbyrow. 7. it is packed into a video in the form of continuous lines (rows). So. Video Standards A standard is a set of specifications for a product or process. and b) PhaseAlternating Line (PAL) from Western Europe. it is important that video production equipment ensures a certain standard. This time interval is also known as an image period. it is dynamic data which can be specified (or described) by a set of parameters related to the color. in order to avoid confusion.33ms. In the TV industry. there are two dominant standards: a) the National Television Standards Committee (NTSC) from the U. Because of the huge consumer market for videos. 7. these lines will appear in the same order when displayed on the TV screen. • Number of Lines per Image: The number of lines per image is 525. an image is commonly called a frame. products or processes for a similar application (or function) can be interchanged. So. one signal line will last 63. In electronic imaging. The technical name for NTSC video is RS170A or Electronic Industry Association (EIA). If we pack the lines by following line index i which varies from 1 to n. In fact. an analogue video is a onedimensional signal. . In robotics. as shown in Fig. • Interlaced Image: As we mentioned earlier. When manufacturers follow a common standard. intensity and timing of images. A video is a continuous stream of images.20..17 and Fig. Since the image period lasts 33.33 milliseconds (ms).5 microseconds (/is).
This negative pulse is known as a horizontalsynchronization signal or. The technical name for PAL is the Consultative Committee for International Radio (CCIR). • HorizontalSynchronization Pulse: When packing a series of lines together. A common way to double the display rate is to split an image into two fields: a) the oddline field and b) the evenline field. The following are the specifications of PAL: • Image Rate: The image rate is 25 images per second. and n<z is the corresponding number of line periods taken up by VSync. or we can use a series of negative pulses which are the same as HSync. In order to do this. One way to do this is to use a negative pulse. • VerticalSynchronization Pulse: A video is a series of interlaced odd and even fields.4volt pulses which may last for a certain number of line periods. .Visual Sensory System of Robots 419 Our eyes' latency to the image changerate is 25 images per second. The NTSC adopts this latter solution to mark the end of a field.4 volts. HSync for short. the following condition must hold: 2 • (nx + n2) = 525.0 volt. • Black and White Signal Levels: The black signal level is 0. it is necessary to have a "marker" to indicate the beginning or end of a line. a higher displayrate is desirable in order to ease the stress on our eyes. at a rate of 30 images per second. However. it is necessary to have a "marker" to indicate the beginning or end of each field. The oddline field is composed of the odd lines of an image while the evenline field is composed of the even lines of an image. Therefore. This split image is called an interlaced image. or VSync. The field rate of NTSC video is 60 fields per second. For NTSC videos. Hsync is a negative pulse of 0.0 volts while the white signal level is 1. As a result. If n\ is the number of signal lines in a field. the imagebyimage display mode. is acceptable. consists of a series of negative 0. the verticalsynchronization pulse. NTSC is not the most popular video standard in the world but PhaseAlternating Line (PAL) is. we can use a negative pulse which is different from HSync. Thus. The corresponding image period is 40ms. for a long video.
the HSync and VSync are also known as timing signals (or sync signals). However. • HorizontalSynchronization Pulse: The HSync is a negative pulse of 0.0 volts while the white signal level is 0. the following discussions use PAL as a reference. These signals are encoded into an analogue video so that there is only a single.3volt pulses. it is necessary to use three cables. A onedimensional signal wave can be transmitted through a cable with two wires: a) one wire for signal.3 volts. and b) one wire for ground. it is sufficient to pack the timing signals into one component video. the following condition must hold: 2 • (m + n2) = 625. • Interlaced Image: A PAL video is also composed of a series of interlaced images. and ri2 is the corresponding number of line periods taken up by VSync. we can carry the time signals in a separate signal line. b) the video of green component images and c) the video of blue component images. For a color camera's three component videos. as shown in Fig. • Black and White Signal Levels: The black signal level is 0. 7. Each of these video streams is called a component video. It is common to use the green component video.420 The Fundamentals of Robotics: Linking Perception to Action • Number of Lines per Image: The number of lines per image is 625. Therefore.7 volts. The corresponding line period is 64/xs. If n\ is the number of signal lines in a field. If a color camera directly outputs a set of RGB component videos with the timing signals packed into the green channel. . • VerticalSynchronization Pulse: The verticalsynchronization pulse also consists of a series of negative 0. In NTSC or PAL . the field rate of a PAL video is 50 fields per second. For the sake of simplicity. onedimensional pulse waveform which represents the video. Alternatively. known as a sync signal line. the timing signals are the same. Component Videos A colorimaging sensor's direct output consists of three streams of video: a) the video of red component images.22.
.7V: : o. ^ ~ ^ ^ \ /Lastrow V   j First row \ : °^d V HSync 0... •.:inr:.ov.7V.o . Ni r 0. • • R I 1 I I .^ SVideo Electronic Camera Fig. Composite Videos As it is necessary to have at least three cables in order to transmit a set of RGB component videos.A. it is very costly to broadcast them over the air by modulating them into a band of radio frequencies (RF). It would also be costly for a TV receiver to directly receive RGB component videos.. the TV industry commonly uses a composite video for broadcasting and reception..3V . in order to reduce the cost. «»'"••«••'••' "• Composite ^ i d e ^  ] / v <: .. This color space can be projected onto any other threedimensional space..ovi 4* . / \ : : VSync Green component video i  :: '• ..22 Illustration of an electronic camera. 3 v ' ••• _ _ ^ : >••••\ Red component video •• : . Time '• .. S V. If the timing signals are in a separate signal line. o. 0. a[mLr::. ^L.^ . G and B define a threedimensional color space. • nmr\ i . then it is necessary to have four cables. !••••' : ^— :^— .—^^ : r ••••. Blue component video .. such as BNC cables.. to interface the camera with an image digitizer. : . including a set of RGB component videos.7V . 7.'.''• " "..  \ i. . A composite video is obtained by encoding the RGB component videos in a specific procedure which involves two steps: • Step 1: RGB to Luminance/Chrominance Conversion: The values of R. T7.Visual Sensory System of Robots 421 0. m A .— { : .." ^ . . Thus.:inn ! :•: . """ • • • • • • .n'...
3 + 0. (7. We can see that luminance Y is the weighted average of R.Y) where R. While this estimation results in some loss of color information.422 The Fundamentals of Robotics: Linking Perception to Action An interesting way of projecting the color space is to separate the luminance (intensityrelated information) from the chrominance (colorrelated information).43MHz. In theory. Y represents the luminance value. then I and Q are determined as follows: r / = . G and B values from a set of at least three consecutive C values.< 7 .cos(33°) \<2 = f7«cos(330) + V»sin(33°).493 • {B . In fact. and V represents the difference between red component color R and luminance Y. 7.23. As for NTSC videos. we can estimate the R. I and Q as follows: C = Y + / • sm(2nf2t) + Q* cos(2nf2t) where f2 = 3. In Eq. G.0). variables U and V represent the chromaticity of a color. For NTSC. G and B if the values of a onedimensional composite video C are known. If U and V are known. green and blue component colors at a pixel.11 • B U = 0.3 • R + 0. U and V as follows: C = Y + U • sm(2nfit) + V* cos(27r/ii) (7. the adopted conversion is called an RGBtoYIQ conversion. The plot U — V is also called a chromaticity diagram.58MHz. / and Q describe both the hue and saturation of a color. For PAL videos.11 = 1.23) V = 0.59 + 0.24) l ' • Step 2: Generation of a Composite Video: For PAL. so it is not a concern for the (7.YUVconversion. and B because the sum of the weighting coefficients is equal to 1 (0. our eyes can easily tolerate it. however. this adopted conversion is called an RGBto. U represents the difference between blue component color B and luminance Y.25) where / i = 4. G. The following equation describes an RGBtoYUV conversion: { Y = 0. In practice. a composite video is obtained by combining Y.59 • G + 0. and B are the values of red.877 • [R .26) .Y) (7. a composite video is obtained by combining Y. it is not possible to recover the original values of R. S i n ( 3 3 ° ) + F.
three component video streams will be sent to three A/D converters respectively. 7. Y and C signals are called an S.2 Digitization of Analogue Videos Most electronic cameras are used for news or videoentertainment industries. Y and C become two separate onedimensional signal waves. If the input is not a set of RGB component videos.Visual Sensory System of Robots 423 TV industry.3. R. An Svideo can reconstruct R. S. an image digitizer is still a necessary part of a robot's visual sensory system.21.26. Each A/D converter will turn a component video into a series of digital images.Videos One way to minimize the loss of color information is to remove luminance Y from Eq. And then.11 • B \ C = U • sin(27r/i«) + V» cos(2?r/ii) or (Y . it is widelyused in video products. an electronic camera's output is generally not in a digital video format.Video. However. Together. G.3 • R + 0.59 • G + 0. A video's timing signals can be packed into either signal Y or signal C. . As shown in Fig. As a result.0 . and B better than a composite video. or communicated to a microprocessor. the sync signals will be separated from the RGB component videos (or the green channel). 7. So far. 7. The purpose of an image digitizer is to convert analogue videos into corresponding digital videos which can be directly stored in the memory. such as camcorders.11 *B \ C = I • sin(27r/2£) + Q • COS(2TTf2t). Rather. 7. For the wired transmission of an Svideo. G.59»G + 0. however. * " ' (7. a cable must have two wires for signals Y and C. it is a serious drawback for a robot's visual sensory system because exact color information is very important in imagefeature detection and object identification.4.28) [ } Now. Fig. the Svideo format is not intended for broadcasting TV programs. and B will be projected onto Y and C as follows: ( Y = 0. Subsequently. As a result.23 shows the generic architecture of an image digitizer. 3 • R + 0. there will be a color decoder which will decode an Svideo or composite video into such a set.
. the frequency (l/i p ) of a pixel clock is the sampling rate when an effective signal line is digitized. the period of a pixel clock can be calculated as follows: 54 tp = — iis. the period of HSync is 10/xs. II i   I . The inner loop works as follows: • Step 1: Set the pixel clock. ..Video I I  1 .( •! j ^ N __ I I . • Step 4: Log onto the even field.. perform an A/D conversion with the inner loop. • Step 5: For line index ie which varies from 1 to ry/2 in the even field.f . .. The process of image digitization involves two cascaded loops.11. • Step 3: For line index io which varies from 1 to ry/2 in the odd field.li I Fig. the period of an effective signal line will be 54/zs (= 64 . The image resolution indicates the number of rows and columns that a digital image will have.424 The Fundamentals of Robotics: Linking Perception to Action S. we denote (rx.. • Step 2: Detect VSync pulse and log onto the oddfield. •—n I • = p H V r. Composite " ''! " j  =§3 1 • <•'' — i ••'• i K itiis o jnr vid» . . 7.29) v ' rx In fact.' " iI ^t:: ! l I. (See below).. 11 M i.10)."«2L RGB j i . And. perform the A/D conversion with the inner loop. (7. i : .23 Illustration of the generic architecture of an image digitizer.j . ' > L. We must set the resolution of the digital image. • :. Here. ry) as the number of columns and the number of rows. The outer loop works as follows: • Step 1: Configure the image digitizer. Therefore. (See below). For PAL.
The pixel value will be stored at location (i. 512). shown in Fig.Visual Sensory System of Robots 425 • Step 2: For pixel clock number j which varies from 1 to rx.24 shows one such set. Fig. From studying the process of image digitization. From the highlighted areas.17 or Fig. 7. or (1024. "••MMHMMHt ><ril Hl^^M^^Si^^^^M i Blue Color I m a g e of g r e e n c o m p o n e n t c o l o r Imaj ••! I In • "ii ". if digitizing the even field. etc.20. 7. oooooooooooooooooo . Example 7. green and blue component color images. In a robot's visual sensory system.ry) are (256.5 An image digitizer's output is a set of digitized red. 1024). 7. the typical values for (rx. _ j 2 • io — 1 \ 2 • ie if digitizing the odd field. (512.j) where index i is determined by .gm Red Color _____^^ Wn . perform an A/D conversion and save the results (pixel value) in the memory which stores digitized images. ry) of a digital image is not equal to resolution (n. m) of an imaging sensor. Image of red component color BF: . blue and orange. 7.itfii ••!••• Orange Fig. 256).24 Example of a set of digitized red. we can see that the input color image contains red. it is clear that resolution (rx. green and blue component color images.
5. IG = {g(i. one color space can be projected onto another color space.1 Representation in RGB Color Space We know that a colorimaging sensor's direct output is a set of three RGB component color images. After digitizing an analogue color image.j)]. we can represent a color image by { IR = {r{i. If a color coordinate is encoded .g(i.1 Chromatic Modelling Perceptually and numerically. any color can be represented by a point in a threedimensional color space. Interestingly enough. 7. Let IR. it is necessary to know how the color and geometry in the images are related to the color and geometry in the scene.j). the imaging sensor. Therefore. we have studied the physical properties and working principles behind the optical lens.5 Modelling of Digital Images So far.1. the component colors at this location will be [r(i. a pixel has a location and three integer values corresponding to the component colors: red. IG. 7. j). Then. An image is a twodimensional array and each of its elements is called a pixel. Therefore.b(i.j). j). green. 1 < i < ry and 1 < j < rx} where (rx. IQ and IB denote these three matrices respectively.24. are an example of the display of (In. the output will be a set of three matrices of pixels: one for the red component color.426 The Fundamentals of Robotics: Linking Perception to Action 7.30) = {b(i. Therefore. one for the green component color and one for the blue color. IB). green and blue. IB l<i<ry l<i<ry and 1 < j < rx} and 1 < j < rx} (7. colors will have different representations in different color spaces. j). Within this array.j). In order to infer the geometry of a scene from images. an image contains both chromatic and geometric information about a scene. 7. and the image digitizer. ry) is the image resolution. color is a combination of chrominance and luminance.5. a color image can be represented in an RGB color space. Given a pixel at location (i. and blue. which define the color coordinates of a point in RGB color space. The three component color images. Any color can be reproduced by a combination of three primary colors: red. as shown in Fig. Because of this.
j). j). AB = 150 . which in turn. However. 7.b(i. such as uniformity and discontinuity. When the numerical values of the three primary colors are the same. Thus. For example. the coefficient for green is larger than the coefficient for red. is encoded into luminance value Y. the weighting coefficients are chosen to be different.5. If we know the value of a set of RGB component color images.11 • b(i.100) is numerically equal to the difference {AR = 120 . Thus. Moreover. However. the sensation of red is stronger than that of blue. AG = 50 .Visual Sensory System of Robots 427 with 8 bits (one byte).j) + 0. j) = 0. where I(i. j) + 0. We will discuss this further in the next chapter. and compute the weighted average.j). This means that the same numerical differences in RGB values do not produce the same physiological sensation of color difference.80. . we can obtain the corresponding intensity image as follows: Ij = {I{i. there is no generic parametric representation of color distribution in an image.120). AB = 170 . If it is necessary to explicitly represent a color's chrominance and luminance. AG = 70 .59 • g{i. the perception of green is the strongest. the perceived difference in colors by the brain is not the same. RGB color space does not explicitly represent a color's chrominance and luminance. (7. a pixel's color coordinates [r(i.20. the only solution is to project the RGB color space onto another appropriate color space. 7. the difference (AR = 100 .j)] at location (i. It is worth noting that RGB color space is not perceptually uniform. by default. it is sufficient to process intensity images in order to derive geometric information about a scene. This phenomenon is called perceptual nonuniformity.31) As human brain interprets the three primary colors (red. Eq.j).j) will require 24 bits (three bytes) to store the RGB values.1.32) l<i<ry and 1 < j < rx} (7.100. a lot of useful visual information. In other words.2 Representation of Intensity Images The processing and computation of three twodimensional matrices will be expensive.g(i. green and blue) differently.30 indicates that we are only able to describe component color images in the form of twodimensional matrices. is larger than the coefficient for blue.3 • r{i.40.
. oooooooooooooooooo From this example. motion. we can see that an intensity image largely. 7.C ^ B ^ M B ^ B H Fig. 7. 7. 7. in the form of posture. The ultimate goal of a robot's visual perception system is to infer the geometry of a scene from digital images.5.2 Geometric Modelling The geometry of a scene. Therefore.32 yields the intensity image. Z) in a threedimensional space are related to the coordinates (u. let us consider the simplest case: a single point. .5. v) in a camera's image plane. is not explicitly measured by a visualsensory system. However. •^B' • . Eq. the application of Eq.15 indicates that the position of a point in a camera's image plane depends on the position of this point in a threedimensional space. In fact. We can see that there is almost no loss of visual information such as continuity and discontinuity. 7. shape and curved surface.25 Example of an intensity image. We will study this topic further in the next chapter. as shown in Fig. preserves the uniformity and discontinuity of the chrominance distribution across an image plane. if not totally.25. There is a deterministic relationship between the image coordinates of a point and its coordinates in a threedimensional space.6 For the color image in Example 7. it is important to know how the coordinates (X.428 The Fundamentals of Robotics: Linking Perception to Action Example 7. Y.
1 Pinhole Model and Perspective Projection Refer to Fig. 7.10. 7. The area in focus on the image plane can be treated as a planar surface.2. we can treat an optical lens as a small hole. including the assignment of coordinate systems. In this way. the formation of an optical image follows a perspective projection. Thus.26 Illustrations of pinhole camera model and perspective projection. In addition. the center of projection is the center of the optical lens. Mathematically. When a camera is described by a pinhole model.Visual Sensory System of Robots 429 7. as shown in Fig. Accordingly. perspective projection is a projection under which all points are projected onto a common plane by following straight lines which intersect at a common point known as the center of projection. the aperture of the iris is usually very small compared to the physical dimension of the objects in a scene. 7. . Actual Image Plane / / y (a) Pin hole camera model Actual Image Plane Equivalent Image Plane / 7 _— (b) Equivalence of perspective projection Fig.26a. the optical lens of a camera is described by a pinhole.5.
as shown in Fig. When we assign a frame to a camera. a frame is a righthanded coordinate system.430 The Fundamentals of Robotics: Linking Perception to Action 7. a coordinate system (or frame) can be assigned to it. and p is the projection of P onto the image plane.2 Assignment of Camera Coordinate Systems For the study of robot kinematics. Distance fc between the image plane and the center of projection is known as the focal length of a camera. the question is: . Since an image plane can be treated as a physical object.5. a camera will contain two frames: one frame for the camera. By default. as shown in Fig. and the other for its image plane. The x axis is parallel to the X axis of the camera frame. A common way to do this is to define the origin of the image frame at the intersection point between the image plane and the optical axis (the Z axis).2. the camera's posture is described by the posture of its frame. it is common to define the Z axis along the optical axis and the origin of the camera frame at the center of projection.2. Fig.26a. 7. we can choose the X axis to be in the direction parallel to the imaging sensor's rows. In a camera frame. Thus. the value of this focal length is negative because the image plane is on the negative Z axis.26a.5.26b.5. By default. as shown in Fig. 7. In this way.3 Assignment of Image Coordinate Systems Physically. the equivalent image plane is also called an image plane. it is necessary to assign a coordinate system to it. For the sake of convenience. The equivalent image plane and the actual image plane are symmetric about the XY plane of a camera frame.2. a camera's image plane is behind the center of projection. 7. 7.4 Determination of Image Coordinates As we discussed above. we can make use of the equivalent image plane on the positive Z axis. the geometric properties of the objects on these two planes are strictly equivalent except for the signs of the values. A camera is a physical entity (rigid body) as well. 7. and the y axis is parallel to the Y axis of the camera frame. 7.26b illustrates the assignment of an image frame. and the Y axis to be in the direction parallel to the imaging sensor's columns. If P is a point in the camera frame. However. Under a perspective projection. we adopted the strategy of assigning a coordinate system (frame) to a rigid body.
27. j/). Assume that P is located on the YZ plane of the camera frame. including the projection of coordinates from camera space onto an image plane. let us assume that point Q is located on the XY plane of the camera frame.} : .: .Visual Sensory System of Robots 431 What is the relationship between the coordinates of P and the coordinates of p? [jj  I  j I "JJ'[ E^^. From the fact that triangle PPiOc is similar to triangle poOc. Let us denote p = (0. we have =4cv c7 y Jc (733) (734) or y = fc»~ Similarly.: — . According to the perspective projection. CY.=::: : I — .. : v OJZSCJLL / Image Plane I "* > Q " i Fig. Under a perspective projection. 7. In this case. p will fall onto the y axis of the image frame. cZ)t.~f] j   j • « ""i"E::::::i2^:i: ". 7.27 Illustrations.:I:::::::::±:::J'"»«:::: i \*y . and an image array. image q of point Q will . P = (0. Refer to Fig.
y) can be defined as the inverse projectivemapping.35) or * = /C.38.36) In general. the coordinates (CX. 7. We will study this problem as well as some solutions in the next chapter. 7. cZ)t from (x. Since triangle QQ\OC is similar to triangle qoOc.37 is the mathematical description of a perspective projection from a threedimensional space into a twodimensional plane. 7. we can see that the coordinates (x. CZ. From Eq. 1)* the equivalent projective coordinates of a point in the camera frame. CY. This is because the scaling factor s is unknown. the following relation will hold: (x = fcc4 { (7. .38 as the forward projectivemapping of a robot's visual sensory system. cZy. 1)* the equivalent projective coordinates of a point in the image frame (see Chapter 2). CY. s) will be uniquely determined if a set of (CX. y. 7. CY.y. if the coordinates of point P in the camera frame are (CX. If we define Eq. CY. and the coordinates of its image p are (x. (x. (7.T.38) where s is a scaling factor. With a single camera and no a priori knowledge about an object or scene under consideration. cZ)t in the camera frame cannot be determined if the coordinates (x.y). However.37) Eq. we have cv cv (7. the recovery of (CX. CZ. CY.432 The Fundamentals of Robotics: Linking Perception to Action fall onto the x axis of the image frame. Eq. there will be no unique solution to inverse projectivemapping.37 can also be rewritten in matrix form as follows: /x\ * • \y // c ooo\ = r*\ • \i) Vo 0 1 0 / 0 /c0 0 [fj (7. y) in the image frame are known. and (CX. 1)* is given as input.
When a digital image is obtained from digitizing an analogue video. the size of a pixel.27. These indexes are called the index coordinates of a pixel. we know that a digital image can be obtained in two ways: • Direct Output from an Imaging Sensor: It is best to directly perform A/D conversion with the output from an imaging sensor as there will be no loss of chromatic information caused by the backandforth conversion from analogue images to video. 7. For example. Refer to Fig.5 Determination of Index Coordinates For the sake of programming. And.Cy) defines the dimension of an imagesensing cell in the x and y directions of the image plane. the question is: What is the relationship between image coordinates (x.2.5. it is still common to use an image digitizer to convert an analogue video into a stream of digital images. In the memory which stores digital images. the index coordinate system uv is assigned to an image array.1) (u = 1 and v = 1 in the MATLAB programming environment). and v the axis for the row index. represented by (Dx. 7. then . If (Sx.27. an imagesensing cell can be treated as a rectangular surface.Dy). The row and column indexes indicate that there is an index coordinate system associated with an image array. the row index of a twodimensional matrix is counted from top to bottom while the column index is counted from left to right. a pixel will correspond to an imagesensing cell. a pixel's size depends on the sampling steps along the columns and rows of an imaging sensor. Now. If we denote u the axis for the column index. and from video back to digital images.Visual Sensory System of Robots 433 7. the index coordinates for the upperleft pixel will be either (0. 0) (u = 0 and v = 0 in Cprogramming language) or (1. will be \n'=r' I uy = °y (739) • Direct Output from Image Digitizer: Today. y) and index coordinates (u. If a digital image is directly obtained from the A/D conversion of an imagingsensor's output. as shown in Fig. If (Cx. Sy) are the sampling steps for digitizing an analogue video. it is convenient to use the indexes to indicate a pixel's location in a twodimensional array. v)7 From our study of digitalimage formation.
Eq. (Sx.5. Dy) be the physical dimension of the pixels in the image array.40) As we studied earlier.Cy). Eq.6 Intrinsic Parameters of Cameras Consider a camera represented by the pinhole model.42 can also be rewritten in matrix form as follows: (u\ /£ 0 uo \ /x\ (7.43 describes the relationship between the image coordinates and the corresponding index coordinates.y) is (u. 7. In general.2. 7. I>O). Sy) is not equal to (CX. then we have { or \ §. If the index coordinates corresponding to image coordinates (x.43) 7.u0 (7. Let (Dx.38 describes the relationship between the coordinates in the image frame and the coordinates in the camera frame. the resolution of a digital image is different from the resolution of an imaging sensor.434 The Fundamentals of Robotics: Linking Perception to Action we have ( DX=SX I uv — av (7. with respect to the index coordinate system assigned to the imaging sensor. If we eliminate . And.are the number of digitizations which occurred along a row and column respectively. let us assume that the origin of an image frame is at (UQ.and ^.41) (742) where ^. 7.= u . Eq. Now.v).
However.Visual Sensory System of Robots 435 image coordinates (x. cz (7. cZy cannot be determined for any given (u. fy. This is because the parameter specifications of the camera and lens are not accurate. 1)*.6 Digital ImageProcessing Hardware "An image is worth a thousand words" implies that a large amount of information can be extracted from images. In general. a camera's intrinsic parameters can be estimated by a process known as camera calibration. cZ)t is given as input. However.44) with ffx 0 u o 0\ lPc = O/^oO V 0 0 1 0/ (7. 7.38 and Eq. if not impossible. it is difficult. 7. Interestingly enough. u0 and v0 are commonly called the intrinsic parameters of a camera. to know a camera's intrinsic parameters. We call such a matrix a projection matrix. If we know a camera's intrinsic parameters. the index coordinates (w.lv\= 'Pc.45) and Jx ~ Dx (7. v)* because the scaling factor s is unknown. 7.46) AAMatrix / P C describes the perspective projection of coordinates from the camera frame to the corresponding index coordinate system. fx and fy are two new variables. (CX. We will study camera calibration further in the next chapter. They can be interpreted as the focal lengths of the camera in the directions parallel to the x and y axes respectively.43 yields s. v)1 can be uniquely determined if (CX. In Eq. a . human brain's neural computing network is highly parallel and is able to process. in realtime. the combination of Eq. 7. CY. y. CY.45. The four parameters fx.
• A harddisk for nonvolatile storage of images and results. Now.6. • Mouse and keyboard for interfacing with the user or programmer. for interfacing with external devices or systems. today's PC's are capable of interfacing with all kinds of digital cameras and webcams through the Universal Serial Bus (USB). 7. will be 3 x 512 x 512 or 768 kilobytes (KB). Because of the speed limitations. a pixel's component color is encoded with one byte. . Refer to Fig. a robot's host computer can also serve as an interface between the robot and the human master. 7.28. as shown in Fig. SVGA or XGA. A host computer normally includes: • A microprocessor for all computational tasks.6. (NOTE: 1KB is equal to 1024 or 210 bytes). it is necessary to have an appropriate imageprocessing hardware to support this need.2 DSP Processors Normally. USB and Fire Wire are suitable for producing lowresolution digital videos. such as Peripheral Component Interconnect (PCI). 7. such as the videophone and netmeeting. such as VGA. For a color image of the size of 512 x 512. in terms of the number of bytes.28. 7. image digitizer and host computer. it is necessary to form a visualsensory system which consists of an electronic camera. the Pentium). it is quite a heavy computational load for a robot to process a sequence of color images in realtime. which is made from Intel's microprocessors (e.1 Host Computers Today's popular computing platform is the personal computer (PC). which packs digital data (including images) into a video of a specific graphic format. If an application requires highresolution digital video. • A monitor which displays analogue videos in a graphic format which is different from PAL or NTSC. or FireWire. • An external bus. Through the host computer. the data size.g. a human master can develop application programs for the robot to execute.436 The Fundamentals of Robotics: Linking Perception to Action large amount of information captured by all the five sensory systems of the body. Therefore. Due to the proliferation of webbased applications. if a robot's visual perception system must respond in realtime to streams of video images captured by a visual sensory system. such as an image digitizer. • A graphic card. In general.
image digitizer.28 Illustration of the generic imageprocessing hardware. DSP processor and pipelined imagecomputing modules. And. a DSP microprocessor is made up of five processors. known as the crossbar network. =H . As shown in Fig. I \< H I r1 y IVRAMI r PC LMonitorJ — — CM A I —  / ^  4  U <  T Cross Port Switch >^ >< j ' t 1 I MultiplePort RAM I CM: Computing Module Fig. A popular type of onboard microprocessors are the Digital Signal Processors (DSP). One of them is the master processor (MP). each of which has its own Arithmetic and Logic Unit (ALU). such as C80 from Texas Instruments. . An internal bus. 1 • I  RAM KimagePigitizer ]^camera Q .° P r °" S S O r 1<—»l I " U ^ * (Pentium) .Visual Sensory System of Robots 437 [pH [Kl H H 1UJ_T MpFPU Crossbar Network OnboardRAM"[ DSP Processor r^1^J^ C r .' Pipeline' I l\ PCI < "* * H D HvRAMh M ° du H 1~[ ''' . supports the fast exchange of large streams of bytes among the processors and onboard memory (RAM). 7. the other four are coprocessors. 7. Inc. ~ sp I. One common way to reduce the computational load of the host computer is to add a dedicated microprocessor on the board of the image digitizer. including the host computer. A DSP microprocessor usually has its own onboard memory (RAM) which can store a large amount of image data. which can only perform computations with integers.I 1—J Bus /' ''' / 1 CM A I (v^Tl fMonitor vT..28. with a floatingpoint processing unit (FPU). CM A I PVGA .
certain convolution filters for image featureenhancement or preprocessing). or algorithm. 7. > > For example.438 The Fundamentals of Robotics: Linking Perception to Action Example 7. A filter. we can form the subpipeline: CM1 — CM3 — CM2 or CM2 > CM3 > CM1 even if CM1.28). Since a filter's output can be another filter's input. 7. CM2. (See Fig. The configuration of an imageprocessing pipeline is userprogrammable. implemented on a chip is called a computing module (CM).29 shows an example of a color image digitizer with an onboard DSP microprocessor. Fig. 7.3 Pipelined Computing Modules Sometimes. This means it is easy to form a subpipeline for a given computational task.6.7 Fig. it is more efficient to implement some computational tasks with wellestablished algorithms using dedicated hardware instead of dedicated processors (e. a series of computing modules can be linked together to form a pipe. This pipe is known as an imageprocessing pipeline. This flexibility for reconfiguring an imageprocessing pipeline is achieved with a crossport switch which ties the pipeline's computing units together. CM3 and CM4 are all present in a pipelined imageprocessing hardware.g. This DSP microprocessor is programmable through the host computer. a crossport switch makes . In fact.29 Example of an image digitizer equipped with a DSP microprocessor. oooooooooooooooooo 7.
8 Fig. Example 7. A DSP microprocessor is a miniMISD computing platform. In general. it is necessary to use a cluster of microprocessors in order to supply the computational power required. an MISD computing platform has 16 or more processors. because its memory is shared among its five processors. In general. and conceptual symbollearning. This imageprocessing pipeline is programmable through the host computer.30 shows an example of a color image digitizer with pipelined imageprocessing hardware. however. Depending on the configuration of the processors and memories. 7. Image Dipili/ur "•^ PivlmcJ ( . a robot's visual sensory and perception systems should be on a parallel computing platform because a robot normally has two or more electronic eyes.'iiipuliml MKIIIIU!. 7.6. 000000000000000000 7.30 Example of an image digitizer equipped with an imagepro cess ing pipeline.4 Parallel Computing Platforms For advanced computations in realtime such as 3D reconstruction.Visual Sensory System of Robots 439 the pipelined imageprocessing hardware behave like the network in a startopology. face recognition. It . all processors have equal access to the shared memories. For an MISD platform. handgesture recognition. parallel computing platforms can be divided into two categories: a) Multiple Instructions Single Data (MISD). Fig. and b) Multiple Instructions Multiple Data (MIMD).
It is a universal standard for interconnecting all digital devices such as computers and digital cameras. is a standalone computing system which is very different from a personal computer. The USB is a serial communication standard. making it a good way to communicate digital images among an array of microprocessors. An MIMD computing platform is composed of a cluster of networked processors. As a result. it is possible to make use of the Internet because today's personal computers are all Internetenabled. including the host computer. 1 —T—'  RAM  C^^> . it is expensive and unpopular. image digitizer.31. 7.440 The Fundamentals of Robotics: Linking Perception to Action A. ——I microprocessor < I/O < .A <—> IVRAMI . And. . O * Computer 6 1 <"> < — > Computer 5 jUggilgrjj Z / I ImagrDigitiz~e7~< VRAM [  [camera O 1 < f Video Monitor Fig.31 Illustration of the generic imageprocessing hardware. . and a highspeed USB can transmit 12 Megabits per second (Mbits). as shown in Fig. . the basic working principle behind it is asynchronous serial communication (see Chapter 6). and an array of parallel processors. * . 7. it is easy to form an array of personal computers. This architecture is similar to a transputerbased parallel computing platform (no longer popular).^ (Pentium) . With the availability of USB and FireWire ports in today's personal computers. The transmission speed of the FireWire falls within the range of lOOMbits to lOOOMbits. $ $ * 1 i * 1 Computer 1 * * Computer 2 * > ^ '' y <—> Bus 4 *" H L 1 0 I •< >• Computer 3 1 I <> Computer 4 1 *> J I vr. IEEE1394 (or Firewire) is a new serial communication standard for computers and peripheral devices. Instead of using serial communication links such as the USB or FireWire.
Since 1990s. in the form of a cluster of networked microprocessors. The focused light's intensity is then converted into corresponding electrical signals by an imaging sensor. it is a common fact that the robotics. the biggest challenge now is how to develop a realtime programming language (or realtime operating system) which can effectively manage the distributed processing powers and memories. the optical lens of a camera focuses incoming light into a small planar area. it is necessary to control the light intensity with the help of a device called an iris. Subsequently. Without visual sensory input. A CCD imagesensing cell looks like a pair of coupled capacitors: one for sensing incoming light. we should not neglect the importance of the continuous improvement of electronic cameras. Rather. we learned that an electronic camera is a device which is part of a robot's visual sensory system and imitates the human eye. Therefore. the important perceptiondecisionaction loops cannot function properly at all. we should realize that the visual sensory system is of fundamental importance to the humanoid robot or other autonomous robots. Based on our studies in this chapter. and the other for storing electric charges. electronic imaging has been dominated by CCD imaging sensors. 7. It is necessary to form an array of CCD sensingcells in order to convert an optical image into a corresponding electronic image. Functionally. In this chapter. the output from an imaging sensor will be three streams of RGB component color images.7 Summary Although the electronic camera is one of the most important components in the robot. digital cameras.and visionresearch communities have made little effort in its development. A CMOS imagesensing cell looks like a current gener . In addition. known as an image plane. this has been fuelled by competition in consumer products (e. In this case. the CMOS imaging sensor has become the new device for electronic imaging. We also learned that when an optical lens focuses incoming light onto an image plane. the electrical signals are packed with the timing signals to form videos which can be either analogue or digital. camcorders) and the media industry. we learned that one way of sensing colors selectively is to make use of a lightsplitting and colorfiltering mechanism.Visual Sensory System of Robots 441 As we mentioned in Chapter 6.g. Since 1970s.
7. it is undesirable for a robot's visual sensory and perception systems. it is sufficient to process an intensity image which preserves most visual information and can easily be obtained by a weighted sum of the RGB component color images. the ultimate challenge is to recover this missing dimension. a dimension is lost. If a color camera's output is an image digitizer's input.13. And this broadens the horizon for the development of the smart camera on a chip.442 The Fundamentals of Robotics: Linking Perception to Action ator. they are more costefficient. the geometric aspect of an image formation can be treated approximately as a perspective projection. As electronic cameras cater to the video entertainment and media industry. electronic camera output must follow a certain video standard (e.8 (1) (2) (3) (4) (5) Exercises What is light? Does a photon's energy depend entirely on a light's wavelength? What is the index of refraction? Prove Eq.g. These digital images can be represented by three twodimensional matrices. an imaging sensor's direct output must undergo signal conversion. Before a computer can process an image. This will be a tremendous advantage for the future of embedded & visionguided applications such as the humanoid robot. As CMOS imaging sensors are produced with the same equipment as that used in the manufacturing of microprocessors and memory chips. Output from a CMOS imagesensing cell is proportional to the lights striking its surface. This can be done by using an image digitizer. In order to reduce the computational load caused by three twodimensional matrices. In order to conform to video standards. Explain the difference between the human eye's central vision and its . the coordinates in the camera frame can be uniquely projected onto the index coordinates in the image plane. in the next chapter. If we know a camera's intrinsic parameters. it is necessary to convert analogue video into a stream of digital images. However. when the coordinates are projected from a threedimensional camera space onto a twodimensional image space. This conversion will result in the loss of chromatic information. PAL and NTSC). the image digitizer's output will be three streams of RGB component color images. 7. Finally. and thus. We will discuss this challenge further. For a robot's visual perception system.
(18) Is a digital image's resolution equal to that of an imaging sensor's? (19) When digitizing a PAL video. (25) What issues are related to the use of a parallel computing platform? . (17) Explain why an image digitizer is necessary for a robot's visual sensory system.Visual Sensory System of Robots 443 peripheral vision. 7. (13) Explain the advantages and disadvantages of a CMOS imaging sensor vs. (24) What are the possible ways to form a parallel computing platform for imageandvision computing. What is the frequency of the pixel clock inside the image digitizer? (20) How is a color image represented inside a robot's "brain"? (21) How do you obtain the intensity image from the component color images? (22) Prove that the projection matrix from a camera frame's coordinates to the index coordinates with respect to the image array is: fix 0 u00\ Ipc=\ 0 /„ i>0 0 I . is it possible for a robot to see a colorful world? (9) Explain how to design an active pixel. (14) What is a video camera's output? (15) What are the typical types of video? (16) Explain why video camera's output is usually not digital. a CCD imaging sensor. the resolution of the digital images is set to be (512. (6) What is a circle of confusion? (7) Draw a graph to illustrate the fact that a smaller aperture will reduce the dimensions of the confusion circles. But. 512). (10) Prove Eq. (8) Human vision is intrinsically color vision. (11) Explain the working principle of a CMOS imaging sensor. (12) Explain the working principle of a CCD imaging sensor.22. \ o o i o/ (23) Explain why a robot should have dedicated computing hardware as part of the visual sensory system.
E. (1991). Complete Guide to Semiconductor Devices. E. Computer Vision and Image Processing. S. Jr. (5) Pratt. . John Willey and Sons. C. L. Woods (1992). AddisonWesley. (1998). (3) Gonzalez. (1987). PrenticeHall.9 Bibliography (1) Hecht. (1990). E. and R. Digital Image Processing. Optics. R. K. AddisonWesley. (4) Ng.444 The Fundamentals of Robotics: Linking Perception to Action 7. (2) Galbiati. K. Digital Image Processing. PrenticeHall. J. K. Machine Vision and Digital Image Processing Fundamentals. McGrawHill. (1995). (6) Umbaugh. W.
1 Introduction We constantly perform all kinds of activities and this dynamism depends on our ability to act and interact with our environment. We will place special emphasis on the computations of geometrical features which are useful for determining an object's posture and motion in a scene. we will discuss the basic computational principles and methodologies of image processing. people's activities are highly limited. mental and physical development is slowed. This autonomy and freedom of mobility in turn depends on our powerful vision. Image processing. and social interaction with humans. which are considered to be relevant to the development of a humanoid robot. and pattern recognition are three related but distinct subjects.Chapter 8 Visual Perception System of Robots 8. In this chapter. I do not intent to cover all this knowledge here. there can be no humanoid robot if there is no competent (artificial) vision. In fact. knowledge acquisition. Vision is an ability which enables autonomy in actiontaking. These three subjects form the important pillars of machine perception and we have accumulated a great deal of knowledge about these fields over the past 30 or 40 years. feature extraction and computational vision. This is also true for a humanoid robot or any intelligent artificial system. but rather to study those topics of primary importance to motion execution. 445 . computer vision. their knowledge acquisition is tremendously compromised. it is true that without vision. human interaction is difficult and. While blind people can live very rewarding lives. in general.
^i I n. Vision Process. ^i '.1 A Process of Visual Perception From a systems point of view. In computer graphics.in '• i Vision Applications 1 \ i>>ii.2 The Basics of Visual Perception From Chapter 7.••.I.2.il (•uul. we know that a visualsensory system's output is digital images.. . One may question why we are concerned about a visualperception system which takes digital images as input? 8.446 The Fundamentals of Robotics: Linking Perception to Action 8.J { I1 in j ^ M i l ^ M. attributes and applications of a visualperception . This inputoutput relationship is illustrated in Fig.1.iiki: I I Kimwk'Jpi 1 . Ohicil —1___  "*=^ Cruphics f —• Vision Attributes _'.VJI.I.IIKC ~. 8.' \. . 8. the goal is to produce realistic images or image sequences from descriptions of objects and characters in a (virtual) scene.•*" "'" "~ r * '. This output will be the visualperception system's input.1 system. artificial vision is an information process that takes digital images as input. Those who are familiar with computer graphics can easily see that the visualperception process is the inverse of the computergraphics process.\iiin>ilii'ii j Fig. „. ' I 'rv "< • /•< I \ision ! """*.n i j<^ I . ML IMIII'MICIII In&pixuoii ! IUcnuiiLuii(>]i [ Illustration of the process. and produces descriptions of the objects in a scene as output.' (^"IIKIM • (.
there is a fundamental difference between them: • In computer graphics. • Geometric properties such as shape (implicitly dimension). humans can perform visual perception effortlessly.2 Attributes of Visual Perception A physical object generally exhibits two types of properties: • Physical properties such as smell. We can formally define the term scene. as follows: Definition 8. • In a visualsensory system. it turns out that today's computers perform poorly when it comes to complex decisionmaking. etc. In fact.1 Visual perception is an information process which takes digital images as input. Whereas. input is a precise description of objects in a (virtual) scene. we can define the process of visual perception as follows: Definition 8. From our study of the robot's visualsensory system. 8. The inference of geometrical features from images is a decisionmaking process. outputs from these two processes are of the same nature (digital images or image sequences). . and thermal conductivity. the visualperception and visualsensory systems complement each other.Visual Perception System of Robots 447 Formally. stiffness. electrical conductivity. it is a challenging task for a machine (including a robot) to do it with comparable quality. Moreover. color. a scene means the environment in proximity to the robot. and therefore the visualperception system must infer them from digital images. We will study this topic further in Chapter 9. posture (position and orientation).2. However.2 A scene is a collection of objects and physical entities which are in proximity to an observer such as a robot. a robot's visualsensory system does not explicitly output descriptions of objects in a real scene. However. In robot vision (and computer vision). and produces descriptions of the objects in a scene as output. we know that the process underlying it is very similar to the process underlying computer graphics. input is not known in advance and the scene is real (not virtual). In this way. and motion (velocity and acceleration).
On the other hand. In a spatial domain. such as walking. luminance (intensity). a visualperception system always starts with image processing and feature extraction which computationally manipulate chrominance. a digital image is a 2D matrix of pixels. and texture. an image does not explicitly measure a scene's geometric properties. and texture (specific and regular distribution of chrominance and/or luminance). or texture information. which. This information combined with geometric features is useful for visual perception because an object's most important attributes are its geometric properties. As one spatial dimension is lost.448 The Fundamentals of Robotics: Linking Perception to Action As shown in Fig. in terms of posture. and grasping. manipulating. a scene is a threedimensional (3D) space. motion.3 Applications of Visual Perception A visualperception system is capable of recovering the geometry and appearance of a scene from images.1. Thus. and an image is a twodimensional (2D) plane. doesn't contain any geometric information. Since a digital image only preserves the appearance of objects in a scene. Refer to Fig. Visionenabled.1. as it is. it is necessary for its visual perception system to link tightly with the execution of activities. 8. An image is the projection of a scene onto a camera's image plane. (NOTE: Time adds another dimension). a robot's visualperception system will consider the following attributes: (1) Geometric properties. luminance. In summary.2. a digital image only preserves the appearance of an object in terms of chrominance (color). manipulating. luminance. luminance. For a humanoid robot. in terms of chrominance. autonomous behaviors can be classified into these categories: (1) Visual Guidance: Walking. The autonomy we gain from visual guidance gives us a sense of freedom in performing these activities. and shape. (2) Appearance. or texture information in order to obtain useful geometric information. 8. 8. From . their geometric properties must be inferred through the processing of chrominance. This capability is indispensable for a system such as the humanoid robot which should autonomously act and interact in a dynamically changing environment. and grasping are three types of daily human activities.
In terms of appearance. such as identity. smoothness of surface. an electronic camera can serve as . A subgroup of the product's parameters are usually related to the uniformity of geometry and appearance (i. Knowledge Acquisition: Our mental and physical abilities develop through realtime interaction with a dynamically changing environment. Thus. English). Visual Inspection: A successful vision application used in industries is visual inspection for quality control. Visual Measurement: It is worth noting that human vision is not metric. On the other hand. alignment.g. automatic visual inspection is a must. especially given the increase in the rate of production and the decrease in product size (e. and language symbols. it is easy for humans to read. In fact. the ability to decode meaning and emotion from body movements and gestures is very important in the development of a sociable humanoid robot. color perception in human vision is the physiological and psychological interpretation of lights by our brain. observe and associate meanings with conceptual symbols. With the help of a visualperception system. etc).Visual Perception System of Robots 449 (2) (3) (4) (5) the viewpoint of control. Two typical examples of visual identification in the domain of biometrics are fingerprint and face identifications. Moreover. A humanoid robot's visualperception system will also enable it to develop its ability to acquire knowledge through the use of a language. The purpose of quality control is to minimize variation in a product's parameters with respect to the reference values stipulated by the design (the mean and standard deviations). Visual Identification: Geometry and appearance may also provide other useful information. human vision does not metrically infer the geometry of a scene or object. In the area of manmachine interaction. emotion. The quality of a product is measured with respect to the product's design specifications. dimension.e. a humanoid robot should adopt as much as possible the imagespace control scheme for most of its motion executions. For example. This is not the case with a robot's visualperception system. color. color perception depends on the lighting conditions of a scene.g. shape. One important aspect of mental development is knowledge acquisition which results from the comprehension of natural languages (e. human vision is unable to guess the length of an object with a reasonable level of precision. semiconductors).
1 < v < ry and 1 < u < rx}. u). Green and Blue (RGB) component color images. A set of RGB component color images contain both the chrominance and luminance information about a scene being projected onto an image plane.2. 1 < v < ry and 1 < u < rx} IG = {g\v. followed by feature extraction and finally. 8. As shown in Fig. u). These represent the Red. u) + 0. u) = 0. And. (ry. it is necessary to find a way to infer this from the available information.2. but not its geometry.59 • g(v. u). luminance. In this chapter. Thus. geometry measurement. and texture of a scene.1. the number of columns and rows in an image array).3) As we already mentioned. 8. Since the most useful information relevant to motion execution is the geometry of a scene. 8. a digital image explicitly measures the chrominance. u) + 0. we will study the principles and methodologies relevant to the process of inferring geometry from digital images.2.3 • r(v. Let us consider intensity images as input to image processing. l<v<ry where and 1 < u < rx] (8. v) are the index coordinates.2) I(v. In Eq. (u. rx) is the image resolution (i.4 Information Processing in Visual Perception We know that a color image inside a robot's visualsensory or perception system is represented by three image arrays (2D matrices) as follows: { IR = {r(v. an intensity image can be obtained as follows: / / = {I(v. (8. The process of inferring geometry from digital images starts with image processing.450 The Fundamentals of Robotics: Linking Perception to Action a scientific instrument for the measurement of color (colorimeter) and geometry (photogrammetry or range finder). u).11 • b(v.e. an intensity image encodes the luminance of a scene being projected onto an image plane. u).1) IB = {b(v. the flow of information in a visualperception system generally involves the following modules: . Refer to Fig. 8. l<v<ry and 1 < u < rx} (8. By computing the weighted average of RGB component color images.
and color. This can be achieved with some specific linear and/or nonlinear filters. Image Processing A digital (color) image explicitly contains the chrominance. The corresponding geometric properties are implicitly encoded into the uniformity. For example. or texture. The presence of image features can be caused by various physical reasons. and discontinuity in a digital image's chrominance. Feature Extraction Image features are closely related to the uniformity. Therefore. the primary goal of imageprocessing. luminance. But.. and texture. Feature  Extraction y ^ [ Image  Processing ^  •  I Visual Sensors Recognition » . 8. with regard to the perception of geometry.2 Illustration of the flow of information in a visualperception system. _ Understanding 4 TT Ml M Mental 4 ! I I : I Real J 1 I Representation T *V World / : f~ • World / \r • Decisionmaking • Actiontaking Fig. depth. and texture. luminance. luminance. Therefore. continuity. luminance. the main causes are the object's geometry and color. is to detect the uniformity. and discontinuity of the chrominance.Visual Perception System of Robots 451 Geometry Measurement ir . imagefeature extraction is a first and necessary step towards geometry inference. and texture of a scene. continuity. continuity. . the discontinuity of chrominance or luminance may be due to the discontinuity of an object surface's normal vectors. and discontinuity in terms of chrominance. .
the image coordinates (x. it is possible to compute the actual dimensions of image features. it is still a question on how to reliably determine the threedimensional coordinates from images. Image Understanding Image understanding refers to the process of generating an ordered sequence of descriptions (or interpretations) of a scene. it is difficult for a robot to perform object recognition. each of which corresponds to only one physical object. Knowledge Representation The outcome of the linguistic description (or programming) of a scene is naturally a representation of knowledge. The success of image understanding depends on the robot's ability to perform object recognition and description at a semantic level. it is necessary to know the relationship among the index coordinates (u. these remain the critical issues for a visualperception system. v). where there are only a few types of products which are known in advance. This is relatively easy in a wellcontrolled and simple environment such as a production line. However. appearance. CZ) in a scene. . And. Thus. from images. and meaning. Therefore. In this way. Object Recognition Object recognition refers to the identification of objects in terms of geometry. or the content in an image. CY. the visualperception system is important for knowledge acquisition which is based on realtime interaction with the environment and others. However.y). This will be difficult unless the robot is capable of extracting image features and making a reliable estimation of the geometry of a threedimensional scene. there is a loss of one spatial dimension when threedimensional coordinates in a scene are projected onto an image plane.452 The Fundamentals of Robotics: Linking Perception to Action Geometry Measurement The spatial locations of image features are measured with respect to the index coordinate system uv. Therefore. The results of the linguistic description or interpretation of the external world will largely contribute to the formation of the robot's mental or virtual world. and the threedimensional coordinates (CX. Scientists around the world are actively researching object recognition. A crucial step towards object recognition is to group image features into distinct sets. which has no unit of measurement.
with respect to variables (u. 1 < v < ry and 1 < u < rx. v) and b) one temporal variable t (time). Instead. v. the two special types which are important to a robot's visualperception system are: a) image transformation and b) image filtering. If an input image at timeinstant t is represented by Ii(v.3. the original input image can be recovered exactly by an inverse transformation applied to the image transform. t). image processing will include all possible computational techniques for the manipulation of image function Ii(v. the exact representation of a digital image should be a function of three variables: a) two spatial variables (u. 8. In general. as shown in Fig. which the results can directly contribute to the perceptiondecisionaction loop.1 Frequencydomain Transforms One wellknown image transform is the Fourier Transform which is the representation of an image in a frequency domain. However. 8.t).u. The input to a robot's visualperception system is a sequence of digital images (or digital video.1 Image Transformation Image transformation refers to all operations which aim at obtaining alternative representations of an input image.2. we can denote a digital image by Ir(v.2. . t) and their neighborhood. for short). A systematic discussion of these topics is beyond the scope of this book. a robot's visual perception is a continuous dynamic process. 8. u. If there is no loss of information after image transformation. an image transform is an alternative representation of the input image. and knowledge representation. we will focus our study on the basics of image processing. By default. In other words. v). and geometry measurement. Accordingly. Of all the possible computational techniques. 8.1.3 Image Processing Refer to Eq. Thus.3. The result of an image transformation is called an image transform. a digital image means an intensity image and is a function of two spatial variables (w. feature extraction.u).Visual Perception System of Robots 453 image understanding. 8.
represented by Eq.0. u) • e~juJ™v • eju"u} u=l (8.52 . and blue component colors define a threedimensional color space. Vme [l. commonly known as the RGB color space. it is not wise to do the latter. ( n .n). green.4) where { Il{~V'u) = wm = ^ « ( m .l ) .^ i J l ^ 1J v=l u=1 Since the Fourier Transform is an alternative representation of an input image in a frequency domain.0.454 The Fundamentals of Robotics: Linking Perception to Action the corresponding discrete Fourier Transform will be ry rx F(m.0 .e^}. to the corresponding intensity image. the red.1 ) . which is Tr 1 4 IT*£ £ {F(m. 3 1 / \QJ = (8.1. 8.6) (. defines the transformation from RGB color space to YIQ color space: /R\ G \BJ /0.rJ.rv] (8.11 \ /Y\ 0. Example of ColorImage Transformation The result of the transformation from an RGB color image. \0.2 Timedomain Transforms If the result of an image transformation is of the same nature as the input image. Of all the possible 3 x 3 matrices.21 0. 8. Mathematically. for geometric feature extraction.5) wn = ^ . used by NTSC TV. the one below.32 ) • [ / ] .2.60 .30 0. Any 3 x 3 matrix will transform the RGB color space into another space. However.59 0.1.27 . The discrete Fourier Transform has an inverse. (8. we can perform image feature extraction either with the original image or with its Fourier Transform. n) = ^ v=l Y^ {Ti(v. Vne[l.7) . 8. it is called timedomain image transform. represented by Eq.ei—. is a timedomain transform.3.
rx] otherwise (8. Vv e [0.3a shows an image of a calibration rig. This is a vector that each element encodes a luminancevalue's frequency of occurrence.2922/ • I G . is the simplest and most useful transformation. 255]} (8. vo will fall within the range of 0 to 255. Before the calibration process can begin. This rig can be as simple as a box with a grid or an array of regular patterns painted on one or more of the box's facets. If we denote Ijn(v. image thresholding transformation divides an intensityimage's pixels into two groups: a) the foreground pixels and b) the background pixels. in the image plane. as seen by a robot's camera.Visual Perception System of Robots 455 Since the matrix in Eq.8) \B J Example of IntensityImage Transformation Image thresholding which acts on an intensityimage's individual pixels. In fact. a histogram is simply described by Hf = {h(v). 8. ry] and Vu e [1.4192 25.9) where VQ is a threshold value. it is necessary to precisely locate. 8. .8439 7. Fig.2742 9. If a pixel's luminance is encoded with one byte.6.3086 .u) the corresponding image transform.u)>vo Vv€ [1. u) = I [ 0 iU}n(v.2505 \ /R\ I ! } = \ 6.9. Example 8.u) the input image. the transformation from YIQ to RGB will be /Y\ \Qj /4.10) where v serves as the index from which to retrieve the frequency of occurrence h(v) of luminance value v. and Ijut(v. This process normally makes use of a special object called a calibration rig.9074 . the array of patterns on the calibration rig. Threshold value VQ can be manually set if the nature of the input images is known in advance. the mathematical description of the image thresholding transformation is (1 I°ut(v.8084 \14. One useful bit of prior knowledge is the intensity • histogram.1 A common process for estimating the camera's parameters in a robot's visual sensory system is calibration. If v denotes the luminance value which varies from 0 to 255.7 has an inverse.7504 16. (8.
we choose ^o = 80.4 Example of a histogram.. we first compute the intensity histogram. 8. 8.456 The Fundamentals of Robotics: Linking Perception to Action ^BK ^HrSag'' «••••» (a) Input Image (b) Image Transform Fig. Clearly. 8. Refer to Fig. So.3 Example of imagethresholding transformation: a) input image and b) image transform. 0 50 100 .4. Fig. (There is only one exception to the pattern at row 2. which are separable around the index value 80. . the array of patterns on the calibration rig has been isolated from the other objects. In this example. . — 250 Fig. 8..3b illustrates the result of imagethresholding transformation when v0 = 80. column 1).. 1500  1 ' ' " ' ' r ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ — • B E E . We can see that there are two dominant peaks in the histogram.— illllll.
This method evaluates all possible threshold values ranging from 0 to 255. Consequently.h. if (g. or outside a certain frequency. If we denote I}n(v.h.«).3. image filtering causes a loss of information. if (g.0. • Motion Transformation. h.1.i) is equal to (0.1). 8.2 Image Filtering (1) 82 In fact.11) ••(vltuHy8. if (g. i) is equal to (0. magnitudes and phases. a signal is the sum of harmonic functions.12 represents the following types of spatial transformation: • Projective Transformation. Vv e [!.3. 8. and chooses the one which minimizes the variances of luminance of both the foreground and background pixels. "filtering" means to remove a subset of (undesirable) signals from an input image.3 Spatial Transforms Mathematically. air. it does .rx] where /uA (a b c\ /u\ (8. In general. "Filtering" literally means to remove impurities or undesirable substances from something (e.Ui) the output image. it is possible to automatically determine the threshold value by analyzing the intensity distribution in an image. one may wish to remove the harmonic functions above. water. v) without altering the luminance value. The outcome of a transformation acting on spatial variables (u. Depending on the application. For image processing.g. d.Visual Perception System of Robots 457 oooooooooooooooooo As shown in the above example. e) form a rotation matrix. In a frequency domain. such as sinusoids and exponentials which have different frequencies.v) is called a spatial transform.»•„] and \/u e [l.1).1).u) the input image and Ifut(vi. or even signals). below. Eq.0. • Affine Transformation. a general spatial transformation can be described by / r * K « i ) = #*(«.i) is not equal to (0. we can also manipulate spatial variables (u. and the four elements (a. A simple and practical algorithm automatically determining the threshold value is Otsu's method. 0. b.
Y{s)=H{s)»I{s). 8. the Laplace transform of a differential equation is in algebraic form. Clearly. a system's output in the frequency domain is expressed as the product of the input's Laplace transform and the system's transfer function.13 is y(x) = h{x)*I{x) = / rx h{a)»I(xa) da = rx Jco I(a)»h(xa) da (8. it is possible to unify the philosophy underlying the design of a mechanical system. As shown in Fig. Y(s) is the filtered signal or image.2.13) If we treat I(s) as the Laplace transform of an input signal or image. And. From this point of view.5.458 The Fundamentals of Robotics: Linking Perception to Action not make sense to recover the original input image from a filtered image in the frequency domain. and a computational algorithm for image feature extraction. we learned that a differential equation describes a linear system's dynamics. a control system. 8. the ratio between the output's Laplace transform and the input's Laplace transform is called a transfer function.1 Convolutions In Chapter 5. the purpose of filtering is to alter the dynamics of a signal.14) «/ — oo . If the input to a system is an impulse. In the time domain. the inverse Laplace transform of the transfer function when the input is an impulse is called an impulse response.3. its Laplace transform is 1 and the Laplace transform of the output is exactly equal to the transfer function. it is easy to change the dynamics of an input signal or image by manipulating the filter H(s). (8.5 Dynamics of a linear system: a) the impulseresponse in the time domain. 8. Because of this. 8. the dynamics of which have been altered by the filter's transfer function H(s). and b) the transfer function in the frequency domain. the inverse Laplace transform of Eq. that is. For a linear system. l(x) •  h{x) 1 y(x) • I(s) •  H(s) 1 Y(s) • (a) Time Domain (b) Frequency Domain Fig. Just as the purpose of a control law is to alter the dynamics of the system under control.
n ) } . the operation in Eq. Eq.15) otherwise. By definition. \lm £ [l. and u — ui with n.u)*I}n{v.8.uO • i}n(vu m)}.Visual Perception System of Robots 459 where a is a dummy variable for integration. By applying Eq.u) (8.m .rx]. and the impulse response h(x) is known as the convolution kernel for signal or image filtering.u) = hk{v. the information in the first ky rows and kx columns is not considered in the convolution. To make full use of the information centered on an image .n).u .«i)}. an input digital image at timeinstant t is represented by jin = {/}«(„.«)= J2 E {M«"i. Indexes (v.r x ]} w and the output digital image is represented by jout = {/»*(„.fcy]and Vn G [1. « . If a discrete convolution kernel is {hk} = {hk{m. n ) » i } n ( w .r y ] and Vu G [1.(817) Vi=v — ky u\=u—kx If we substitute v — Vi with m. The above results are obtained in a continuous time domain.„).15. 8. 8. fcx] (8.17 will become ky kx /^ t (u ) u)= 2 H { M m .16 can be rewritten as v—1 u—1 /?"*(«.n) = 0 if Vm G [1. 8.vi. m=ln=l (8.ry] and V G [l. vv G [l.ky] and Vn G [l.16) = EJUi E" 1= i {^(^ . n) ^ 0 hk(m. In other words. the result of discrete convolution will be Ifat(v.ry] and [kx + l.^]}.u) in Eq.14 is called convolution.18) Consider an image's boundary condition. V« G [l. In a discrete image plane.««i)*// n («i.kx]} with { hk(m.18 must vary within the range of [ky + l.u). This indicates that the output is not centered on the original input image. 8. Eq.
20 can also be expressed as ky kx I°ut(v.6a shows an image array which the size is 10 X 10. 8. (NOTE: If kx = 3. ky/2 + 1) so that the output will be centered on the input image. (8. . it is still possible to use Eq.u + n. Eq. 8. 8.2 Fig.n)} = • 1 f110 ' . Assume that the convolution kernel is {h(m. a convolution kernel can be asymmetric. 110 [no.19) If a convolution kernel is symmetric. Eq.n) • l\n{v .10 . 8. Therefore. 100/6.rn + ky/2 + l.ky/2 ~l.460 The Fundamentals of Robotics: Linking Perception to Action plane.20) However.20. n) • I}n(v + m. the symmetricallyswapped version of the convolution kernel "1 0 . is shown in Fig.6b. The result of the convolution after applying Eq. Accordingly.18 becomes ky kx Ifut(v. In this case. u)=J2Yl m=ln=l {hk(m.1 . is 101" 101 101.6667 if working with floatingpoint numbers.20 to perform the convolution if the results of the convolution with either a convolution kernel or its symmetricallyswapped version are equally acceptable. and if ky = 3.1 " 101 . oooooooooooooooooo .n + kx/2 + 1)} . 7 7 1 = 1 Tl = l (8. 8. we should shift the input image using displacement vector {kx/2 + 1. kx/2 is equal to1. The convolution is computed using integers. For example. is 17.1)} . which would be 16. Example 8. 8. ky/2 is equal to 1).kx/2 .u) = Y^ J2 {hk(m.u .
This is a wellknown result in control engineering. it is possible to compute derivative yJ£' directly without invoking the input's derivative if response y(x) is already known.Visual Perception System of Robots 461 1 2 3 4 [b~ 0 IO ~O [0 10 O I [0 [0 1 [ O (T~ <T~ 0~ O~~ 100 0~~ <T~ <T~ 0~~ 6 ~ o~o~o~7ooToo7o6o~o~o~o~ (PO^IOOTOOTOOTOOTOOO^O^ 0~~ 10 2 3 4 J 0 0 0 I I 10 0 O~ O O O O~~ 0 ~ O~~ W W 50~ W 0~ O~~ 6"~ o~ o~~ W so" 83~ 83~ 50~ W o~ o~ 0 ~ 17" 50" 83" TOO 100 83" 50~ T T 0 ~ 5 6 7 8 9 o~ Too Too Too Too Too Too Too o~o~ o~o~TooTooTooTooTooo~o~o~ o~ o~ <T~ Too Too Too o~ o~ o~ o~ o"b~o~o~Tooo~o~o~o^"6~ cP6~6~(T"o~o~o~5~o~(r~ 2 4 6 8 (a) Input 10 o~ W W Too Too Too Too WT W o~ o~ W so" §3" Too Too 83~ 50~ W o~ 7 o~ o~ rf 50" 83" 83~ 50" W <T~ o~ 8 o ~ c P o ~ W5F50" Wo~o~o~ 9 o~ o~ o~ o~ W vf o~ o~ (T~ 6~ 5 6 10 o^6~o~o~o~o~o^o~o~6~ u\—I—I—I—I—I—I—I—I—I—I 10 o~b~o^o^"o~o~F"o~o~o^ u\—I—I—I—I—I—I—I—I—I—I 2 4 6 8 (b) Output 10 Fig.21) The computation of the inverse Laplace transform of Eq. let us multiply the complex variable s to both sides of Eq. . This yields smY(s) = s»H(s)»I(s).8. in which F(s) is the Laplace transform of function f(x). 8.21 can also be written as {s»Y(s)} = H{s)»{s*I(s)}. 8. then its derivative is exactly equal to the system's response to a unitimpulse input.22) (8. 8. Therefore.3.23) ax ax Eq.2. Eq. In control engineering. Now. the Laplace transform of its derivative *j£' will be s»F(s). 8.23 indicates that a linear system's response to an input's derivative is equal to the derivative of the system's response to the input itself.13. ~ (8.6 Example of convolution with a convolution kernel: a) the input image and b) the result of the convolution. (8. we call it the principle of dynamics conservation.22 results in *d£=h{x)**M. Here. we often use a unitstep function as a linear system's input. If we know a system's response to a unitstep input.2 Derivative of Convolutions If the initial conditions of linear function f(x) are all zeros. 8.
it is possible to interpret a feature as a specific type of dynamics exhibited by the signal itself.24) (8. Eq.24 yields (8.21 can alternatively be expressed as {s*Y(s)} = {s»H{s)}*I(s).n)}l_u _ 1 I <** J 6 [ ^ ^ If we apply Eq.7a shows an image array that the size is 10 x 10.20. We will discuss this in more detail in the section on imagefeature extraction. continuity. 8. we will obtain ( „. Clearly. input I(x) is able to resonate not only with a system's impulseresponse h(x). ^ . If we compute the derivative of the convolution kernel from Example 8. 8.7b. We call this phenomenon the principle of dynamics resonance. image and convolution kernel < ^ there is a resonance caused by the presence of vertical edges in the input image.* ' . and discontinuity. the result of the convolution between the input [ is shown in Fig. rior i Q (dh(m. but also with its derivative j .25) Eq.8. 8. From this point of view. Computation of the inverse Laplace transform of Eq. In other words. in terms of uniformity. 8. feature extraction from a signal can easily be achieved in two steps: a) resonance generation and b) resonance detection. If a filter is sensitive to a specific type of dynamics and the resonance stimulated by the input signal does not vanish.3 Fig.462 The Fundamentals of Robotics: Linking Perception to Action Interestingly enough. Example 8. oooooooooooooooooo . In fact.2 in the horizontal direction. With regard to feature extraction from signals.24 indicates that the response of a linear system (or filter) replicates the dynamics of the system (or filter) even though the input remains the same. there is a different interpretation to the above principle when dealing with image or signal filtering. 8. this indicates that the signal contains features corresponding to this type of dynamics.
1 / 1 / (8.Visual Perception System of Robots 463 1 2 3 [6 [5 lo lo lo [6 [6 lo [o lo I o"~ Too Too IOO Too Too Too Too Too <T~ <T~ Too Too Too Too Too Too Too Too <T~ 1 2 lo lo [6 lo lo lo 6 io lo [o~~ o~33~b~"o~o~o^cT~o~33~o~ 3 o"~ 50~ o~~ o~~ o~~ o"~ o ~ o"~ so" o~" 4 5 6 7 8 o~ Too Too Too Too Too Too Too Too o~~ 0~0~0~T50T50T50T50 0 ~ 0 ~ 0 " 0~0~0~T50T50T50T50b~0~0~~ 0~0^0~T50T56T50T50 0 ~ 0 ~ 0 " 0~0~0~T50T56T50T50 6~"6~0~~ 5 6 Ao~W25^25"o~'o~25"25'Wo~ 0~ VT 50" 50~ 0~ 0~ 50~ 50~ W 0~ 6~^ 0~ 75" 75" 0~ 0~ 75~ 75~ 0~ 0~ 7 0 " 0~ 75" 7lT 0~ 0~ 75" 75~ 0~ 0~" 8 0~0~75"75~6~(r~75"75~0~"0~~ 9 10 0~0~0~T56T50T50T560~0~0~" 9 0~0~50"5T6~0~50"50"0~0~~ b~o~o~6~o~o~(T~6~o~<r" i i I—I—I—I—I—I—I—'—I—I—I 2 4 6 8 (a) Input 10 10FFFFFFFo~FF .26 can also be expressed as ^ • y ( S ) j = ji. 8.13.27 yields r y(x) »dx = h(x) * f rI(x) •dx\.i I—i—'—'—'—'—'—'—'—>—' i 2 4 6 8 (b) Output 10 Fig.26 can also be rewritten as (8.8. 8.y( 5 )j = #(*).2. (8.28) This indicates that a linear system's response to an input's integral is equal to the integral of the system's response to that input. In order to verify this statement.3. 8. 8. Eq. Computation of the inverse Laplace transform of Eq.30) . s s Eq. Alternatively. The inverse Laplace transform of Eq.29 is fy(x)»dx=  /'h(x)»dx\*I{x). It becomes •Y(s) = »H(s)»I(s).26) ji./(s)j. 8.H( a )j •/(*). 8. let us multiply the inverse of complex variable s to both sides of Eq. ji.7 Example of a convolution with the derivative of a convolution kernel: a) the input image and b) the result of a convolution.29) (8.27) (8.3 Integral of Convolutions Both the principles of dynamics conservation and dynamics resonance are equally valid for the integral of a convolution.
The inverse Laplace transforms of Eq. 8. 8.8b.2.34) (8. it is interesting to know whether a filter itself has a degree of freedom to adjust feature localization. an important concern is the localization of detected features. Therefore. If we apply Eq.20 instead of Eq. In Example 8.8a shows an image array which the size is 10 x 10.31 and Eq.3. 8.464 The Fundamentals of Robotics: Linking Perception to Action This indicates that the dynamics of a system (or filter) is replicated by the output.13 yields {e~as • Y(s)} = H(s) • {e~as • I(s)} or {eas • Y(s)} = {e~as • H(s)} • I(s). This phenomenon can advantageously be exploited to design filters with the ability to adjust the location of detected features in a signal.6b.34 clearly illustrates the phenomenon of a signal feature's location shift which is caused by a filter.4 Spatial Displacement of Convolutions Interestingly enough. 8. 8. the above concepts are also valid for a convolution's spatial transform (displacement). we can see that there is a horizontal shift. 8. 8. For feature detection.• O i l . It is worth .33) (8.n — 1)} is shown in Fig. (8.a) and y(xa)=h(xa)*I(x).19 to compute the convolution. The shift is towards the left because we use Eq. The multiplication of the term ealPha^ to both sides of Eq.2.32) (8. if we shift the convolution kernel towards the right by 1 pixel. 8. the result of the convolution between the input image and convolution kernel {h(m.32 are y(x a) = h(x) * I(x . If we compare the output in Fig. we will obtain i roir {/i(m. even though the input remains the same. Example 8. 8.nl)} = .31) Eq.8b with the output in Fig. 8.0 1 1 . 8.20.4 Fig. 1. 8.
4 Image Feature Extraction An input image explicitly represents the chrominance and luminance of a scene (or object). For easy programming. the multiplication of complex variable s to a filter linearly enhances the highfrequency signals.4. 8. •ooooooooooooooooo 8. an image's geometric representation is the image's alternative representation. for manipulative and cognitive tasks. in a visualperception system. the most important attributes of a scene (or object) are the geometric features.1 The Basics of Feature Detection In the frequency domain. This is because the size of the convolution kernel is 3 x 3. this enhances a signal's discontinuity. it can also be called an image transform if we treat the complex process of imagefeature extraction as a nonlinear image transformation. However. Thus. Thus. Mathematically. Similarly. the multiplication of the . In the time domain. 10 2 3 4 5 6 7 8 9 10          I 0 0 0 0 0 0 0 0 0 10 2 3 4 5 6 7 8 9 10         0 0 0 0 0 0 0 0  0 6~O~6~b~Too6~6~(Pcr~(r~ O~O~O~TOOTOO1OOO~O~O~O~~ o~o~7ooTooiooTooTooo~o~o~~ o ~ Too Too Too Too Too Too ioo o ~ o ~ O~b~TOOTOOTOOTOOTOOO~O~O~~ 0~0~6^T06TOOTOOO~0~0~0~~ o~ o~ o~ o~ Too o~ o~ o~ <T~ 6 " o~ o~ o~ o~~ o~ o~ o ^ o~ o~ 6~~ 6~~ 0~~ VT 50" 50~ 17" 6~~ 6~~ 6~~ 0~~ 0~17~50~83~83~5(ri7^0~6~0~ o ~ s o 83 Too7oo83"5cTTTo^o~ o~67~TooTooTooToo67~T7~o~b~ 0~50"83^T06T60 83~50~T7~0~0~~ 6~~ W 50" 83" 83~ 50~ W 0~ 0~ 0~ o~ o~ TT so" so" TF o ^ o~ o~ b~ 6~ o~ o~ W W o~ o~~ o~ o~ o~ o~6~o~o~~6~~o~6~o~o™"(T~ 11'—'—'—'—'—'—'—'—'—'—' 2 4 6 8 (a) Input 10 o~ o~ o~ 6~ o ^ o ^ F " o~ 6~~ b~~ 11'—'—'—'—'—'—'—'—'—'— 2 4 6 8 (b) Output 10 Fig. pixels inside an input image's boundary are not processed. it is crucial to obtain the geometric representation of a scene (or object) from the chrominance and luminance information. 8.Visual Perception System of Robots 465 noting that the boundary of the input image array includes thefirstand last rows as well as the first and last columns.8 Example of a convolution with the spatial transform of a convolution kernel: a) the input image and b) the result of the convolution.
the uniformity of these characteristics are directly related to the characteristics of the geometric surfaces in a scene. and discontinuity. continuity.466 The Fundamentals of Robotics: Linking Perception to Action inverse of complex variable s linearly attenuates the highfrequency signals which has the effect of enhancing a signal's uniformity. it is clear that the presence of an object in image space is considered as the sum of an image's uniformity. occlusions. And.4. a shadow will form on the surface. Moreover. we can define an image feature either in image space or in the space of its transform. and (3) Discontinuity: f(x) is not differentiable at certain locations of x. However. an image should be treated as a spatiotemporal function. Similarly. 8. Since it is possible to transform an image into an alternative representation called image transform. a robot's visualperception system is a continuous dynamic process.1 Feature Definition Mathematically. the corresponding region in an image plane will be uniform. For example. or some derivatives of f(x) vanish at certain locations of x.1. luminance. the reflected light rays coming off the surface and entering the optical lens of a visual sensory system can be blocked by an opaque object as well. for the sake of simplicity. an image is the basic element of a video or image sequence. and texture of a scene (or object). Accordingly. let us consider an image to be a spatial function only. spatial function fix) in the domain of spatial variable x will exhibit three types of dynamic characteristics: (1) Uniformity: f(x) = constant within certain intervals of x. if light rays coming towards a surface are partly blocked by the presence of an opaque object. Shadows. if a surface is smooth and uniform in color. discontinuity of these characteristics in an image depends not only on the surfaces themselves but also on the optical and geometric interference among the surfaces. From the above discussions. As we mentioned earlier. This phenomenon is known as occlusion. Since an image explicitly represents the chrominance. For example. . However. (2) Continuity: f(x) is continuously differentiable within certain intervals of x. and surface boundaries are the common causes for discontinuity in an image.
1. 8. This is usually done with a specific filter. there is a saying about signal filters which sums up the problem." If a signal contains one type of feature. if there are multiple types of features plus noise. to reduce the other types of features including noise. which is meaningful and useful in an image (or its transform). image features must be meaningful and useful. imagefeature extraction becomes a very difficult task. In theory. it is easy to filter out the noise and enhance that feature. an image contains many types of features. Formally. In order to cope with the difficulty caused by noise and multiple features. as shown in Fig. However. By default. It is necessary to enhance the features of interest and. 8. garbage out.making) Fig. If the image quality is good. • Feature Enhancement: It is almost impossible to directly pick up desired features from an original input image if there are multiple features. 8. at the same time.9: • Noise Reduction: The purpose of this module is to reduce the noise level in an input image.3 Any uniformity or discontinuity. the process of analytically describing the detected image features is called image feature description.Visual Perception System of Robots 467 regardless of which space is used.2 A Generic Procedure for Feature Detection In general. a featureextraction process generally involves the following modules. image feature extraction is performed using intensity images. That is: "Garbage in. including noise. we can state the definition of an image feature as follows: Definition 8. this step is unnecessary and H^s) = 1. Based on this definition.4.9 A Generic featuredetection procedure. And. called a feature detector. it is impossible to design a filter which responds to only one type of feature and perfectly removes all the others. the process of detecting image features is called image feature extraction. Images Noise Reduction Feature Enhancement Feature Selection Features Hj(s) H2(s) (Decision . In electrical engineering. is an image feature. .
35) m.regardless of the type of noise and input. any filter which greatly reduces signal Hi (s) • y^y. possible to design a filter capable of responding differently to different features. a decisionmaking process is responsible for selecting the desired features. It is.468 The Fundamentals of Robotics: Linking Perception to Action • Feature Selection: Whether in theory or practice. . We can clearly explain this criterion using Laplace transform. We can generalize these as follows: Criterion 1: Resistance to Noise Canny's first criterion. Let 7(s) denote an input image's Laplace transform. John Canny in his work on edgedetection while studying at MIT as a graduate student first stipulated three criteria for the design of a stepedge detector.3 Criteria for Feature Detection "Garbage in. garbage out" generally implies that there is no ideal filter for a given type of feature. (m From Eq. After the feature enhancement. In practice.This type of feature detector will be resistant to the presence of noise. which if followed.is a good filter for noise reduction. Assume that input image I(s) is contaminated by white noise N(s). however.m. 8. The output from filter Hi(s) (the Laplace transform of discrete filter {hi(m. it is impossible to design a filter which can totally reduce undesired features and noise. N{s) or (8. n)}) will be Yx (s) = Hx (s) • I (a) + Hi (s) . called good response. And accordingly.1. it is more appropriate to call this criterion resistance to noise. there are some guidelines for general filter design.HM+aM. However. 8. we can see that an ideal filter is one which can totally cancel out term Hi(s) • jnx.36. In theory. can ensure a good feature detector. there is no such design of filter Hi(s) which is able to cancel out term Hi(s) • jffi. stipulated the necessity of maximizing the signaltonoise ratio.4.
As shown in Fig.9. Then.9. Canny believed that a good feature detector should minimize this shift despite the presence of noise. • If filter H2(s) incorporates multiplicative term e~ as . this criterion can be better explained using the Laplace transform. 8. let us substitute H2{s) with e~as • H2(s). Interestingly enough. Based on the above observations.38) The results shown in Eq. (8. a decisionmaking process normally identifies the features to be the local maxima from filter H2's output. 8. (837) (8. Again. The output from filter H2(s) will be Y2{s) = H2(s) • h(s) + H2{s) • JVi(»). it must have the ability to cancel out signal H2(s) • Ni(s).39) . This is because a complex function introduces the additive term: a • cos(w) + j[b • sin(u>)] or. I(s) { [N1(s) = H1(s)*N(s).32 and Eq. A phase change in the frequency domain means a location shift in the time domain.8.38 is a complex function having zero(s) and/or pole(s). this can be achieved by incorporating term e~as into filter H2{s). the multiplicative term e@s to output Y2(s). The presence of noise may slightly shift the local maxima away from their true locations. it will have one degree of freedom to compensate for the location shift in the time domain caused by signal H2(s) • N\(s). To prove this statement.34 indicate two interesting facts: • If H2(s)»Ni(s) in Eq. 8.38 becomes Y2(s) = e~as • H2{s) • h{s) + N2(s) where N2(s) = e~as • H2{s) • Ni(s).Visual Perception System of Robots 469 Criterion 2: Good Accuracy for Feature Localization Canny's second criterion is called good localization. Eq. it will cause the output shift in the time domain. 8. it is clear that in order for a filter to satisfy the criterion of good localization.8. Assume that the input to filter H2(s). as shown in Fig. is /i(s) + iVi(s) with ( h(s) = H^s) .
8. 8.010. Example 8. We choose the convolution kernel to be "0 1 0" {h(m. parameter a can manually be tuned and tested with sample images. the result of the convolution between the input image and convolution kernel {h(m.}. as . This is because a featuredetection filter should.40) n\ N3(s) = N2(s) + H2(s) . Criterion 3: Good Selectivity of Desired Features Canny's third criterion is called single response. 8.10b... a filter having good feature localization is one in which parameter a minimizes Eq. Clearly.39 becomes ^ 1 = H2(s) + N3(s) with (8. although the input image has no noise. By single response.. n)} is shown in Fig. he meant that a filter will only produce one response to a single (step) edge input if there is no noise.41. \^8 ^ 1! + . The following example illustrates this. In practice. + (^sn + . ) (8. 8..5 Fig. there are multiple responses for the horizontal edge in the middle. the singleresponse criterion should be called good selectivity of desired features.n)} = 0 0 0 .41) TV. oooooooooooooooooo In a discrete time domain.20. If we apply Eq. 8. 1.10a shows an image array which the size is 10 x 10. an output's redundancy may be advantageous as it allows us to achieve subpixel accuracy in localizing image features. In principle.470 The Fundamentals of Robotics: Linking Perception to Action If we substitute term e~as with its Taylor series: 1! Eq. Obviously. This criterion is applicable to the continuous stepsignal but not in the discrete time domain.
n)} is zero. If we apply Eq. Assume that we want to detect the type of corner at location (2. Example 8.4. Without edges. the corner at location (2. enhance the desired features and reduce all the others.1 . Here. 11 1 _ The response of this filter to a uniform region will be zero because the sum of elements in {h(m.3) has the strongest output (500). For this purpose.20. However.11a shows an image array which the size is 10 x 10. •••ooooooooooooooo 8.Visual Perception System of Robots 471 1 2 3 fo lo lo To lo lo jo lo lo lo I o ~ Too Too Too Too Too Too Too Too o~~ <T~ Too Too Too Too Too Too Too Too o~~ 1[o 2 3 [o lo lo lo o lo (o ( ( T i e r " o~~ Too Too Too Too Too Too Too Too o~~ o~o~o~6^o~o~o~"o~o~o~~ 4 5 6 o~~ Too Too Too Too Too Too Too Too 6~~ 0~0~0~T56i50T50T50 0 ~ 0 ~ 0 ~ ~ 0~0~0~T50T50T50T50 0 ~ 0 ~ 0 ~ ~ 4 5 6 o~TooToo50~56~50~50~TooToo6~ <T~ TOO TOO 50" 50~ 50~ 50~ TOO TOO 0~~ 0~~ 0~~ 0~~ 6~~ 0 ~ 0~~ 0~~ 0~~ 0 ~ 0~~ 7 8 0~0~0~T50T50T50T50 0~6~0~~ b~o~6~T5oT56T5oT5oo~cT<r~ 7 8 0~ 0~ 0~ 0~ 0~ 0~ 6~ 0~ 6~ 0~ o~o^6~o~o~o~o~"o~6~o~ 9 10 0~0~O^T50T50T56i56o~0~0~~ o^o~o^6~o~o~o~"o~o~6~~ 9 10 0~0~0~T50T50T56T50 0~0~0~ o~o~o~o~o~o~o~o~"o~o~ 111—'—'—!—i—!—I—!—!—!—' 2 4 6 8 10 (a) Input 111—'—'—'—'—'—'—'—'—I—' 2 4 6 8 10 (b) Output Fig.1 " 12 1 . To a certain extent. We can see that there are multiple responses to this filter.2 Edge Detection Edge is an image feature which characterizes the discontinuity of an image function in terms of chrominance or luminance. 8.11b. 8. including noise. much as possible. the result of the convolution between the input image and convolution kernel {h(m. any higherlevel description . we will not consider discontinuity caused by texture.6 Fig. 8.n)}= "1 . 8. we design a convolution kernel as follows: {h(m.3).10 Example of multiple responses in horizontaledge detection. it is possible to say that edges are an image's most important geometric features.n)} is shown in Fig.
the image's discontinuity can be measured by these firstorder derivatives: dl dl dl Iv = 1. and It = —.1 Definition of Edges An image is a spatiotemporal function which has two spatial variables (u.4. j ^ is the image gradient in the vertical direction (column) while ^ . Iu = J.472 The Fundamentals of Robotics: Linking Perception to Action 1 o 2 3 4 5 6 7 jo jo lo jo jo jo jo jo jo I 1 o 2 3 4 5 6 jo jo jo lo o lo jo jo jo cr~o~TooTooTo6o~<r~o"o~o~" o~o~TociT66To6cPcr~b~o~b~~ O~~6~To6T66T6o6~(r~6~0~<T~ o~b~TooT66T666~o~6~o~<r~ b~o~To6To6Toocf~cP(r~o~cr~ O ^ O ~ TOO TOO 100 TOO TOO TOO TOO O ~ 6~~2005003001002000"o~o~cT~ o~io63Ooo~Too3OOO~o~o~6~~ 0^"T00 300 0~T00 300 0 ~ 0 ~ 0 ~ 0 ^ <T~ Too3o6o~To63ooo~o~o~5~~ o~~ Too 300 6~~ 6~~ 100 Too Too b~~ o~~ 7 0 ~ T 0 0 300 0 ~ T 0 0 200 300 306T060~~ 8 9 o ~ o ~ Too Too Too Too Too Too Too 0 ^ o ~ o ~ Too Too Too Too Too Too Too o ~ 8 9 6 ~ T61) 306 o ~ b ~ cT~ 6 ~ 6 ~ Too (T~ o ~ b ~ Too Too Too Too Too Too Too o ~ o~o^o~o~o~o^o^b~o~o^ 111—I—I—I—I—'—1—I—I—I—I 10 10 o~o~o~o~"o~o~o~5~o~o~" 111—1—I—I—I—I—I—I—I—I—I 2 4 6 8 (b) Output 10 2 4 6 8 (a) Input 10 Fig. Mathematically.2. the local maxima of the above signals indicate the image function's discontinuity at these locations. these two derivatives can be computed independently. we can state the definition of edge as follows: Definition 8.4. Here. In fact. 8. Accordingly. let us consider spatial edges first.2.^) occur are called spatial edges. firstorder derivatives: ^ and ^ . And. Locations where the local maxima of {%. 8. An edge is a geometric feature because it has a specific location within an image array. locations where the local maxima of ^ occur are called temporal edges.11 Example of good selectivity in response to a desired feature. If we denote I(v. u. v) and one temporal variable t. of a scene's geometric features would be difficult to obtain.2 Types of Edges An image function can have two spatial. t) as an input image at timeinstant t. dv du at In fact.4 Edges are the locations in an image array where the image function's firstorder derivatives have the local maxima. 8.
edge M ^ (b) Ramp . For the sake of convenience. let us analyze and detect the image function's discontinuity separately in the vertical and horizontal directions.12: • Steptype Edge: This edge can be caused by two different colors on an object's surface. If fix) is an image function and represents the luminance (or chrominance) of a scene (or object). as shown in Fig.edge A fix) „ >x A. Steptype edges can also be caused by: a) the outline of a shadow on an object's surface.12 Typical types of edge. or c) discontinuity in the depths of two overlapping surfaces. b) impurities on the surface. the discontinuity of f(x) can typically be classified into these four categories. such as a bulb. . For example. will form this type of edge at the surface's boundary. • Rooftype Edge: This edge typically results from the discontinuity of the normal vectors of a single or two adjacent surfaces. A fix) * fix) —~Z1 (a) Step . and the other with spatial variable v.Visual Perception System of Robots 473 is the image gradient in the horizontal direction (row). 8. fix) ^^ (c) Roofedge •* L A (d) Ridgeedge Fig. we only need to deal with two singlevariable image functions: one with spatial variable u. In this way. • Ramptype Edge: A planar surface exposed to light rays from a single point. printed or handwritten characters exhibit this type of edge. 8. the edge separating two adjacent facets of a cube will form a rooftype edge in images. For example. Let fix) denote a onedimensional function with spatial variable x.
its analytical description will be the combination of /i(x).a • / ^ l(x . From Fig.13 Description of a ramptype edge. If we denote l{x — XQ) a unitstep function at location xo. If a ramptype edge has slope a within interval (xi. 8.474 The Fundamentals of Robotics: Linking Perception to Action • Ridgetype Edge: A curved surface normally exhibits a ridgetype edge. For example. ^ ( z ) and fz{x).43) ' h{x) = a • / ^ l(x .x2)dx (8.44) . a cylindrical or conical surface in certain orientations forms a ridgetype edge.x2). 8.12. 8. we can see that a rooftype edge is a combination of two ramptype edges placed backtoback. Refer to Fig.13.42)  /. And.x{)dx < f2{x) = b*l(xx2) h(x) = .W/ \/3U) Fig. we have (8. as follows: f(x) = fx(x)f2(x) with + f3(x) (8. a ridgetype edge can be treated as a combination of two ramptype edges and two steptype edges.
the noise reduction filter should be applied to the input image vertically. 8. an image is treated as a singlevariable signal. the general scheme for edge detection includes two parallel edgedetection processes. it is convenient to detect the edges in both the imageplane's vertical and horizontal directions. b) feature enhancement. • The discontinuity of the rampedge at location x\ can be detected with a filter which is the secondorder derivative of a continuous function. • VerticalEdge Detection: Similarly. the noisereduction filter should be applied to the input image horizontally. Since an image's spatial dimension is two. and b) horizontaledge enhancement with derivative filter H\2(s) (H12(s) = s). and b) verticaledge enhancement with derivative filter #22(5) (#22 (5) = s). 8. 8. meaning the removal of highfrequency signals. Based on the norms of a gradient. As a result. Clearly.9. To avoid the smoothing effect on the horizontal edges. To avoid the smoothing effect on the vertical edges. horizontally or uniformly (in both directions). and c) feature selection. as shown in Fig. one can choose to smooth the input image vertically.2. Since an image is a twodimensional signal. f2(x) is a steptype edge at location X2• This leads to the following interesting observations: • The steptype edge is the most important edge in an image.4. it is possible to invoke a decisionmaking process which selects the edges as final output.Visual Perception System of Robots 475 and b = a» (X2 — x\).3 A General Scheme for Edge Detection As shown in Fig. it is possible to compute the norms of the gradients at each pixel location. • Selection of Edges: If we know the image gradients in both the horizontal and vertical directions. the verticaledge detection process consists of two steps: a) noise reduction with filter H2i{s). feature detection generally consists of three steps: a) noise reduction. • A good detector of steptype edges is also a good detector of other types of edges. Noise reduction. .14: • HorizontalEdge Detection: The horizontaledge detection process consists of two steps: a) noise reduction with filter Hn(s). either vertical or horizontal. has a smoothing effect on the edges. For a given direction.
.46) (NOTE: H\2(s) = s.45. fc .1 M / / H I Fig.1 _ (8. {hn(m. i i I i.45) Looo.li // '11. _ ^ \ Ililim 11 l_\ .ra)}= " 1 2 1" 0 0 0 . which represents a derivative operator in the time domain).n)} the vertical direction yields with respect to spatial variable v in {/iii(m. i .4 Sobel Edge Detector A simple and useful edge detector is the Sobel edge detector.476 The Fundamentals of Robotics: Linking Perception to Action \ . If the numbers 2 in the convolution kernels are replaced by the numbers 1. we will only discuss the Sobel edge detector. •s^i 'I Arrays Images of Edges ( . 8. !:.III.II. _1 . HorizontalEdge Enhancement In Eq.2 ..IIK^ V . I.ra)*/ii 2 (m. • • •• fc II .•! ii.4. differentiating {hn(m.n)}= (8. 8. ' i ~ . the Sobel edge detector becomes the Prewitt edge detector.14 A general scheme for performing edge detection. Here..ll // '••>..• l"\ II I II i ••!.1'. Noise Reduction in the Horizontal Direction Sobel's convolution kernel for noise reduction in the horizontal direction is "1 2 1' 12 1 . .2.." — ^ .8.
u)>Iu(v~l. The kernel in Eq. _1 0 .u) (8.48. differentiating {h2i (m. which also represents a derivative operator in the time domain). The first inequality in Eq.n)} with respect to spatial variable u in the horizontal direction yields "1 0 . 8.Visual Perception System of Robots 477 The kernel in Eq. HorizontalEdge Selection Edge selection is a decisionmaking process which can be simple or complex. n) * h22(m. 8. (8.u).48) VerticalEdge Enhancement In Eq. u) as the horizontaledge is to test the following conditions: { Iu(v. 8.1 1 0 . Let us denote Iu(v. .n)}= (8. u) the gradient image of the enhanced horizontaledges.1 . {/i2i (m. This allows us to select edges of higher contrast and to remove the false edges caused by noise. 8.49) (NOTE: H22(s) = s.47 means that the firstorder derivative at an edge location must be higher than gradient value go.2 .47) Iu(v.1 " {h2i(m. Noise Reduction in the Vertical Direction Sobel's convolution kernel for noise reduction in the vertical direction is '1 10" 220 . A simple process for identifying location (i>. The second and third inequalities mean that a horizontaledge should be the local maximum in the vertical direction. n)} = 2 0 .u) > Iu(v + l.49 is Sobel's convolution kernel for the enhancement of an image's verticaledges. .u) > g0 Iu(v.46 is Sobel's convolution kernel for the enhancement of an image's horizontal edges.
u) > Iu(v. Ig(v. Let us apply the Sobel edge detector.50.u) > g0 Iv(v. edge gradient Ig(v.u). 8. that is.15 Example of edge detection using the Sobel edge detector: a) input of image array and b) output of edgemap.51) Fig.u 1) (8. The array of the . The second and third inequalities mean that a verticaledge should be the local maximum in the horizontal direction.u) as the verticaledge is to test the following conditions: { Iv(v. The intermediate results for both horizontal and vertical edges are shown in Fig.478 The Fundamentals of Robotics: Linking Perception to Action VerticalEdge Selection Let us denote Iv(v. E x a m p l e 8.16. In Eq. 8.u) will be the norm of the corresponding horizontal and vertical gradients. 8.15a shows an image array which t h e size is 10 x 10. the first inequality means that the firstorder derivative at an edge location must be higher than gradient value go. 1 2 3 fo lo [o [o [o [o 16 [o [o lo I 0~0~~ O~O~TOOO~O~O~(T~cr~ O~6~O~To6TO0Too6~O~O~(T~ [o [6 (o lo [o lo lo [o [o fo~ <T~ 6~~ 6~~ 424 400 424 6~~ 0~~ 0~~ 0~~ 0 ~ O " 424 424 6 ~ 424 424 0 ~ 0 ~ 0~~ 4 5 o~o~TooiooiooTooT66o~o~"o~~ o ~ Too Too Too Too Too Too Too c H o " 4 o~424424o~o~o~424424Toob~~ 5 o ~ 400 o ~ o ~ b ~ o ~ o ~ 400 o ~ 6 ~ 6 7 8 9 o~ o~~ Too Too Too Too Too o~ o~ o~ ¥~ o~~ o~ Too Too Too o~ o~ o~ o~~ 0~ 0~ 0~ 0 ^ T O 0~ 0~ 0~ 0~ 0 ~ O 0™ 0~ 0~ 0 ~ 0 ^ 0~ 0~ 0~ 0~ 0~ 6 o~ 4244240" o ~ o ~ 424 424 Too o~ 7 o~ o ~ 424 424 o~ 424 424 o~ o~ o~~ 8 9 0~ 0 " 0~ 424 400 424 0~ 0~ 0~ 0~~ 0~ 0~~ 0~~ T O 0~ T O 0~ 6~ 0~ 0~~ O O 10 o~o~o~o~o~o~o~o~o~o~ 11'—I—I—I—I—I—I—I—I—'—' 2 4 6 8 (a) Input Image 10 10 o"'^o~o~o~o~b~o~o~o~" 111—I—I—I—'—I—'—I—I—I—I 2 4 6 8 (b) Output Edgemap 10 Fig. For edge location {v. A simple process for identifying location (v.u) the gradientimage of enhanced vertical edges.u).u) > Iu(v.7 1 2 3 (8. u) = ^/Pu{v.u+1). Edge Selection for Final Output A simple way to combine the horizontal and vertical edges is to retain all the edge locations. 8.u) + Pv{v.50) Iv(v.
We can see that there are some artifacts. if we want to remove undesirable artifacts. where the results of the convolution make transitions across ze . 8. edges are identified at locations.15b. The idea behind this method is to perform edge tracing. In his work on edge detection.Visual Perception System of Robots 479 computed gradient norms is shown in Fig. Instead. Canny proposed a hysteresis thresholding method. 1 2 3 4 5 6 7 8 [6 lo lo [6 [6 [6 lo lo lo lo I 0~0~~ 0^300 400 306<T~ 0 ~ ( T ~ 0 ~ 6~~ 6 ~ 300 300 6~~ 300 300 6~~ 0~~ 0~~ 0~3OT3lwb~O~~0~3O7>3O0UJ0<r~ 0 " O~ 0 ~ 0 ^ 0 ~ 0 ~ 0 ^ 0 ~ 0 ^ 6~~ 0~3003006~~ 0~0~ 300 300 100 0 ~ 0~~"0~300 300 0~3063060~"0~0~^ 0 ^ 0 ~ 0^300 400 300 6 ~ 6 ~ O ~ O ~ 1 2 3 4 5 6 7 8 [6 [6 lo lo lo [o lo jo jo [o~ (T~ <T~ <T~ 300 <T~ 300 6~~ 0~~ 0~ 0~ 0 ~ 6~~ 300 300 <T~ 300 300 0 ~ 0~~ 0~ 0~ 300 300 0~~ 0~ 0 ~ 300 300 0~ 0~ O~ 400 O~ 0~^ 0 ~ 0 ~ 0 ~ 400 0 " 6 ~ 0~ 300 300 6 ~ 0 ~ O~ 300 300 0 ~ O~ 0 ~ 0 ~ 300 300 0 ~ 300 300 0 ~ 0 ~ 0 ~ 0 ~ 0~^ O~ 300 O~ 300 0~ 0~ 0~ 0~ o~ <T~ o~ o~ o~ o~ o~ o~ cT~ o~~ o^o^o"*o~o~o~o~o~o~o^ 11'—I—I—I—I—I—I—I—I—I—I 9 10 10 6~o"o~Tooo~Tooo~o~"o~~o~~ b~"o~o^"o^o~"o~o~"6~6~b~ I I I—I—I—I—I—I—I—I—I—I— 9 2 4 6 8 (a) HEdges 10 2 4 6 8 (b) VEdges 10 Fig. have been treated. The tracing of an edgechain stops when there is no more connected edge with gradientnorm greater than g\. with gradientnorms greater than g2. This process is repeated until all the local maxima.5 Gaussian Edge Detector with Tunable Response and Localization The first Gaussian edge detector was proposed by Marr and Hildreth. the gradientnorm of which is greater than g2. 8. Their idea did not follow the principle of dynamics resonance.The edgetracing process results in edge chains which are groupings of consecutive edges linked together. they advocated the use of secondorder derivatives from a Gaussian filter to perform the convolution with an input image. with two threshold values (91.32) (<7i < 92). which allows us to remove unwanted artifacts. which starts from a local maximum. In this way.4.2.16 Results of horizontal and vertical edges detected using the Sobel edge detector oooooooooooooooooo The above example illustrates that. the decisionmaking process for edge selection is not a simple task. 8.
14 and the principle of dynamics resonance.480 The Fundamentals of Robotics: Linking Perception to Action ros.u). • e 2«2. is invariant to the rotational transformation applied to function f(v.52) with 2 ^ # e " 4 ^ The Laplacian of Gaussian (or LoG for short). MarrHildreth's edge detector is based on the detection of zerocrossings of LoG. the continuous curve of a Gaussian function must be sampled in order to obtain a discrete convolution kernel. let us choose a to be 0. —4 (8. the stronger the smoothing effect. 8. Example 8. 8.6 and 0. To perform the sampling. the stronger the smoothing effect. For a Gaussian function.53) Parameter an controls the smoothing effect in the horizontal direction. a is the standard . a Gaussian edge detector with tunable response and localization can easily be designed as follows: Noise Reduction in the Horizontal Direction To smooth out the input image in the horizontal direction.17a.3. According to the general scheme for edge detection as shown in Fig.2 fix) = . we choose the following Gaussian function: hu{v. that is. The corresponding Gaussian curves are shown in Fig. u) is called Laplacian. smoothing effect for noise reduction. Now. 8. one has to specify the sampling range and the number of discrete values. Since an image is an array of pixels. This is why MarrHildreth's edge detector is also called the zerocrossing of LoG.52. 0. as shown in Eq.u) = ===• V27roii /(v'w) = 1 e ^Ti.8 Let us consider the following Gaussian function: 1 3.9. The sum of the secondorder derivatives of a Gaussian function with respect to spatial variables (v. A Gaussian function is a continuously differentiable function and has the advantage of a tunable. (8. We can see that the larger a. The larger the a.
2  f signma=0. 8. 8. The number of discrete values depends on the size of the convolution kernel. let us set this number to be 7.6 / \ \ 0.3 1. .9\\ \ \\ ' 1 °'2 ' o' I \«l \ u  Fig.54) V2?r • 0^2 Parameter a\2 controls the amplitude of the enhanced horizontaledges.e ~^T. 1.8  sigma=0. ^ ^*~2 0 2 4 (a) Gaussian ——' ' '—3 1 4 2 0 2 4 (b) Discrete Kernel •••ooooooooooooooo HorizontalEdge Enhancement After smoothing out the input image in the horizontal direction. such as: h12(v + av.6 0.5cr. Here. the corresponding discrete convolution kernels are plotted in Fig.8 sigma=0. which is very close to ±3<7. it is possible to enhance the horizontal edges with the derivative of a Gaussian function.Visual Perception System of Robots 481 deviation of Gaussian distribution.9 / / o. Based on these settings.17 Gaussian curves and the corresponding discrete kernels with different a values.2  / \ / I 0. Here.4 J 4  \y\ \ \ °4' I / x \ \ / sigma=0. (8.17b.6 \ f I / \ \ / \ / \ \ \ 0.u)= L7 v' .6 • sigma=0. and parameter av adjusts the vertical locations of horizontaledges.4r ll MiA signma=0. let us choose the sampling range to be ±2.3 \ 1.
5<r and a convolution kernel's size of 7. • e 2^.9 V ' 1  . In practice. we can set a^\ = On. (8.^Y\ \ 1 • \ ^ \r^ / j 2 2 .6 and 0.\ T _3I  4 1 1 1 2 0 2 (a) Gaussian Derivative 1 4 _3I  1 4 1 1 2 0 (b) Discrete Kernel 2 1 4 Fig.2 h2i(v.3 / \ sigma=0. We can see that the smaller the a. The corresponding convolution kernels.3.6 1 • / / \ \ y sigma=0.3 / 2 / 2 J V I 1 sigma=0.18b. . 0. 8. let us choose a as 0. Noise Reduction in the Vertical Direction To detect vertical edges. 3r 3 • • • A sigma=0.55) Parameter a^\ controls the smoothing effect. are plotted in Fig. The corresponding curves of the Gaussian derivative are shown in Fig.6 / —^A n^~ sigma=0. 8. the stronger the response of the edges.u) = •^/2TTO"21 ie 5 ^.9 Consider the following derivative of a Gaussian function: —x _ x2 q(x) = .18a. we first smooth out the input image in the vertical direction with a Gaussian function: 1 i.482 The Fundamentals of Robotics: Linking Perception to Action Example 8. Now. when we choose a sampling range of ±2.9. 8.18 Gaussian derivative curves and the corresponding discrete kernels with different a values.
19b. are plotted in Fig. and a to be 0. we can set a22 — <J\\Example 8. its inverse Laplace transform is: e~atcos{u>t). parameter a controls the localization of the enhanced edges.19a.6 Canny and Deriche Edge Detectors If a function is smooth and behaves like a lowpass filter.05 and 0. Let us set a to be 0. 8. 0. For example.5<r and a convolution kernel's size of 7. 8.0. 8. In other words. We can see that parameter a allows us to adjust the convolution kernel's shift. the inverse of the following Laplace transform: {s + a)2+u>2 . we can easily achieve a subpixel accuracy regarding localization of the enhanced edges. By controlling parameter a. The corresponding convolution kernels. the following Laplace transform: s+a {s + a)2+LU2 will behave like a lowpass filter.3. Not surprisingly.4. if we choose a sampling range of ±2.10 Consider the following derivative of a Gaussian function: (x + a) g(x) = —p= »e V 2?r • a6 (»+°)2 2»2 . The corresponding curves of the Gaussian derivative are shown in Fig.u + au)= (u + au) V 2TT • CT22 %H^~^ (gi56) ParameterCT22controls the amplitude of the enhanced vertical edges.1.2.Visual Perception System of Robots 483 VerticalEdge Enhancement Subsequently. while parameter au adjusts the horizontal locations of vertical edges. A lowpass filter in the frequency domain will have more poles than zeros. its derivative will enhance the edges in an image. the vertical edges can be enhanced using the following derivative of a Gaussian function: h22(v. In practice. Similarly.
o. Canny's Function for Edge Enhancement Canny's function for edge enhancement is as follows: (g{x) f{x)=lg{x) \ 0 where ifxe[a.zo] otherwise (8. Nowadays. we will not cover the details of these two methods. but instead. there are two widelyused edge detectors: a) the Canny method and b) the Deriche method.58) . simply explain their functions for edge enhancement.19 Shift effect on the curves of a Gaussian derivative and the corresponding discrete kernels with different values of a. is e~atsm(ut).O] ifzelO. 8.484 3r The Fundamentals of Robotics: Linking Perception to Action 3 31  2 1 1 1 1 0 1 (a) Gaussian Derivative 1 2 31  i 4 1 i 2 0 (b) Discrete Kernel 2 1 4 Fig. The above observation suggests that a smooth and differentiable function's derivative can be used to enhance edge features in an image.57) g(x) = [ai sin(wx) + a2 cos(ujx)]e~ax + [a3 s'm(ujx) + a 4 cos(wo.)]etrx — A (8. Here.
if we choose the convolution kernel's size as 7.20a shows the curves of Canny's function in interval [—4.1 / T \\ >igma=0. From this example.3\ //  0.20 Curves of Canny's function and the corresponding discrete kernels with different values of a. 8. Example 8. are plotted in Fig. 8.4] (i.5.5  \ I 0. xo = 4) when we set a as 0. 8.15. •••ooooooooooooooo .1 and 0. 2. sigma=0. 8. a3. function g(x) in Eq. 1.3 / M \ I /sigma0.58 is very similar to the derivative of the following function: a • e~ax sin(a.11 In Canny's function.20b. 02.3 respectively.Visual Perception System of Robots 485 and A = a2 + 04.5r f\ sigma=0.5'  4 '  ' ' 2 0 2 (a) Canny Function ' 4 2.5 ' o  / ' ' \ : 0S ' 0 " • 01 • • \ <y A • • 0.1 2.5'  ' 4 ' ' 2 0 (b) Discrete Kernel 2 ' 4 Fig. 0 . Fig. .x) + b • eax sin(wx). the stronger the response of the edges. a4) = (0.e. Conceptually.15.5  \ 9 2. 0. we set (ai.1 .5 c. The corresponding discrete kernels. we can see that the smaller the <x.0) and LO = 1.
0) and b± = — 03.4] when a equals 0.0 respectively. n)} computed from a Gaussian function. Deriche's function or the derivative of a Gaussian function. if we choose a convolution kernel's size as 7.n)} computed from a Gaussian function. we can see that the smaller the a. are plotted in Fig. The corresponding discrete kernels. oooooooooooooooooo 8. 8.21a illustrates the curves of Deriche's function in interval [—4.02. Fig.486 The Fundamentals of Robotics: Linking Perception to Action Deriche 's Function for Edge Enhancement Deriche's function for edge enhancement is as follows: '<*>{*.12 In Deriche's function. Again. u>) = (2. n)} computed from Canny's function.57 can be extended to infinity. Canny's function becomes Deriche's function.2.> . . The algorithm implementing the Gaussian. 8.21b.XQ] in Eq.4.0). the stronger the response of the edges.14. we can choose {h2i(m.n)y. Conceptually.60) = [b1sm{ujx)}eax.0. Since function f(x) in Eq. Canny or Deriche edge detector will consist of the following steps: • Step 1: Smooth the input image horizontally with convolution kernel {hu(m.i:>o where g(x) («9» (8.59 decays to zero when variable x approaches infinity. 1. • Step 2: Enhance the horizontal edges with convolution kernel {hi2(m.7 Summary of EdgeDetection Algorithm Refer to Fig. 8. by setting (01. 8. By default. 8. we set (bi. spatial domain [—XQ.04) = (0.0.5 and 1. • Step 3: Smooth the input image vertically with convolution kernel {h2i{m. Example 8.n)} = {hn(m.
51 4 1 1 1 2 0 2 (b) Discrete Kernel > 4 Fig. The convolution kernels for these three edge detectors are as follows: • Convolution Kernel for Noise Reduction: To reduce noise.0 \ V \\ / / / /p — 51 1  4 ' 1 ' 2 0 2 (a) Deriche Function ' 4 1.5  ^ sigma=1. 8. Example 8. Canny and Deriche edge detectors. let us compute the edge gradients using the Gaussian.21 Curves of Deriche's function and the corresponding discrete kernels with different values of a.5 . 8.a 1 " /"\ X 1 " J^ / 0. This step is also known as nonmaximum suppression. For the purpose of comparison. Deriche's function or the derivative of a Gaussian function.13 Fig. By default.n)} computed from Canny's function. • Step 4: Enhance the vertical edges with convolution kernel {h22{m.5r • • • • 1.0 I / \ / /7 / ° ""° 5 ' \ sigma=1. • Step 5: Compute the gradient norms at all pixel locations in order to obtain a gradient image.5  sigma=0.5 sigma=0.5 / \ 0. • Step 6: Select the edges as final output using hysteresis thresholding over the gradient image.n)}*. we can choose {h22(m.Visual Perception System of Robots 487 1.n)} = {/ii2(m.22a shows an input image.5  / • / \ \ °V y 0. we create a discrete convolution kernel with a Gaussian .
The hysteresis thresholding method is a better way of retaining the local maxima in a gradientimage. 8. and the kernel size to be 5.14 The gradientimages in the above example indicate the possible presence of edges.22c and Fig.3 and the kernel size to be 5. u> = 1.5368.6088.0.0584. By setting a = 0. A final step is to make a decision on whether a pixel location is an edge or not.4860.3. oooooooooooooooooo Example 8.6088.0.0. Fig.0584]. n)Y = {h22(m.n)} = [0.4376].0407].0407.4869].0028. n)} = [0.5. Fig. • Discrete Convolution Kernel of a Gaussian Derivative: To enhance the edges in both the horizontal and vertical directions. w = 1.5368.2. 8. By setting a = 0.4860.n)}* = [0.3298. we obtain {/iu(m.4376.22d respectively. the discrete convolution kernel computed from Deriche's function for the enhancement of the horizontal and vertical edges will be {h21(m. oooooooooooooooooo .2.488 The Fundamentals of Robotics: Linking Perception to Action function.1.n)} = [0. and the kernel size to be 5.0. By setting a — 0.0. and removing the nonmaxima.22b.4869. 2.2. We can see that there isn't any significant difference among the three methods.0. 8.0.ra)} = {/i2i (m. we obtain {h12(m. 0. we have {h12(m.0.n)}t = {h22(m. • Discrete Convolution Kernel of Canny's Function: We can also use the discrete convolution kernel computed from Canny's function to enhance the horizontal and vertical edges.3.0. The computed gradientimages are shown in Fig. we create a discrete convolution kernel from a Gaussian derivative function.23 shows the final output from a Deriche edge detector. • Discrete Convolution Kernel of Deriche's Function: By setting a = 1.0028. 8.0.n)Y = {h22{m.
4. a corner may not appear sharp. such as corners and junctions.14 is a geometric representation of the original input image shown in Fig. But. the discontinuities of curvature along a contour indicate particular points. A contour is a set of interconnected edges in an image Obviously. Canny and Deriche edge detectors.23). . Therefore. the term contour can be denned as follows: Definition 8. it is not easy to analyze an array of edges. Because of the smoothing effect caused by an edge detector.Visual Perception System of Robots 489 (a) Input Image (b) Gaussian Method j. 8.3 Corner Detection The set of edges in Example 8. When there is an array of edges.5 array. 8. (Refer to the results in Fig.22a. Theoretically. 8. the image feature known as a corner or junction.22 Results of edge gradients computed from Gaussian. ^ ^ > I (c) Canny's Method (d) Deriche's Method Fig. ^ I * > . We will discuss this issue further in the section on feature description. all highlevel descriptions of a scene (or object) can be obtained from lowlevel edge features. 8. in practice. it is interesting to know whether it is possible to detect directly from an input image.
Based on the concept of image surface.o=j' Fig.u). a corner can be defined as a location where the curvature of a contour is a local maximum.4. Here. we are interested in the question of how to detect corners from an intensity image directly without invoking any analytical description of contours. At a timeinstant. a corner can be defined as a location where the principal curvatures of the image surface are the local maxima. In a continuous space. Jl^~5>/ bDaaao j 8.u). In particular. . and along any direction. 8. According to this definition. an intensity image is a twodimensional spatial function. the detection of corners from edges should be considered as part of feature description. it is possible to compute the curvature at each location (v. corner detection directly depends on edge detection. If f(x) is a singlevariable function which is smooth and differentiable.1 Definition of Corners If we take the interconnected edges as input. If an image surface can be described by a smooth and differentiable function.3. the curvatures computed along two orthogonal directions (e.23 Final output of edges from a Deriche edge detector.490 The Fundamentals of Robotics: Linking Perception to Action h——/—•—r iy? =s^p±===h • a a D o F= I hanDDD / / _n • • a a o / / n_a_ci_Q_D.g. an image function can be treated as a surface in a spatial domain defined by the image coordinates (v. along the u and v axes) are called principal curvatures. Due to curvature computation.
• Condition 3: There is no edge in the vicinity of the zerogradient direction. And. This is a very stringent requirement because an image is digital and is not noisefree. We call this direction zerogradient direction.24 Illustration of corner definition. 8. u) is zero in at least one direction. An alternative definition of corner which does not require the computation of secondorder derivatives of an image function is as follows: Definition 8. • Condition 2: The gradient at (v.6 conditions hold: Location (v. it is necessary for the image function to be smooth and differentiable up to the secondorder.u) is an edge. . a curvaturebased corner detector is thus very sensitive to noise. THPW Background Background •»• Background A ^ Fig.u) is a corner if and only if the following • Condition 1: The location (v.61) K/i + (/») 2 ] 3 where / (x) = ^ and / (x) = J^ • In order to compute the principal curvatures.Visual Perception System of Robots 491 the curvature of f(x) at location x is «= »'»» (8.
5368 02. {h9o(m. 8.4869 0 0 0 0 0 .n)} to be: " 0 0 0 0 0 0 0 0 0 0 {hg(m. However. —. It works as follows: Implementation of Condition 1 As we stated in the above definition of corner. In order to measure about the edge at a pixel location. we can see that the corner at location A satisfies all the conditions for this definition. For example.24 illustrates the validity of this definition. the matrix's row 3 is a convolution kernel computed from a Gaussian derivative function. 8.n)}. If we rotate convolution kernel {ho(m.2 Zerogradient Corner Detector A cornerdetection algorithm depends on the definition of a corner.492 The Fundamentals of Robotics: Linking Perception to Action Fig. we can choose {ho(m. while all the edges satisfy Conditions 1 and 2. (8. Here. the first condition is that a corner must be an edge as well.24 does not meet Condition 3 because a smoothed contour locally looks like a line segment. If we examine the contour separating regions Ri and R2. 0 0 0 0 0 . 90° and 135° in order to obtain convolution kernels {h^{m. Let {hg(m.n)} and {hi35(m. such as the one at location B in Fig. we present a simple cornerdetection algorithm which uses the corner definition stated in the above section. n)} denote a convolution mask which enhances the edges in the direction of 6. a corresponding corner detector must compute the curvaturerelated measurement.53680. a noncorner edge. u) denote the array of enhanced edges in the direction of 9. Now.Pixel . we can choose 6 to be 45°.3. For example. let Ge(v. n)} with rotation angle 9. 8.n) \g=0} = 0.62) In Eq.4. If a corner is considered the location where the principal curvatures are the local maxima. 92. n)}. we obtain the corresponding convolution kernel for the enhancement of edges in the direction of 0. Assume that direction 9 takes a value from the set [9\. Ok]. 8.62. We call this algorithm zerogradient corner detector.4869 2. we simply filter an input image with a set of directional convolution masks.
we can determine the pixel locations corresponding to intervals [—12. Condition 3 can be interpreted as follows: Ge±(vi.ui) e £>_ and V(^i. it is not necessary to introduce threshold value SiImplementation of Condition 3 As we described earlier.u)>g0. One way to specify a reference value is to choose a default value g\. u) is considered zero. we have { D_ = {{v + i»sm9min. Let £>_ and D+ denote the sets of pixel locations corresponding to intervals [—12.i 2 ]}.On the array of an input image. Let 9min denote the direction in which directional gradient Ggmin (v. It is necessary to define a reference value below which a gradient value is judged to be low (or zero). In a discrete space..12} respectively. If we know the vicinity specified by D.66) .65) Vie[zi.A ] (863) where go is a threshold value on the directional gradient.64) Eq. if Qmin and (v. 0 2 . 8.Visual Perception System of Robots 493 location (v. Therefore. u + i»cos9min). D+ = {(v + i»sm9min.64 interprets a lowgradient value as anything that is not a highgradient value. V(ui. for any 0 in [ 0 i . 9k] except 0min. .u{) < g0. Then. (8. a zerogradient should be interpreted as a lowgradient value.—ii] and [11. h]} (8. the third condition states that there should be no edge in the vicinity of the zerogradient direction 0min. for all 9 in [9U <92. u + i»cos9min)..wi) e D+ (8. In this way. Implementation of Condition 2 Because of noise and the discrete nature of an input image. a vicinity in direction 9min can be specified by two intervals of indexes: [—12..u) will be declared an edge if the following condition holds: Ge(v. u) at (v. we can implement this condition as follows: Gom.u). it is rare to obtain the zero gradients perfectly across a uniform region of an image. —i\\ and [ii.Av^)<Ge(v. Vi e [i2. —ii] and [i\.and £>+. In practice. i 2 ]. £2] (*i < h).. u) are known.
determine the zerogradient direction (Condition 2). if we choose two pairs of orthogonal directions: (0°. accept the candidate as a corner. 135°). • Step 3: For each edge candidate.25. The major steps in the zerogradient corner detection algorithm are: • Step 1: Compute the directional gradients with a set of convolution kernels. Example 8.25 Illustration of the zerogradient corner detector. verify Condition 3. i2). such as is shown in Fig. 5). ii. 8. • Step 4: For each corner candidate. The response of the corners is shown in . • Repeat Steps 3 and 4. we only test the gradients computed in the direction perpendicular tO 0min Summary of Zerogradient Corner Detector The zerogradient corner detector does not involve any computation of curvaturerelated measurements. ^^^ Images ^^^fe Arrays of Comers Fig. the decisionmaking process for verifying the three conditions is simple. is illustrated in Fig. 8. we set (go. The zerogradient corner detector. This means that in the vicinity of a corner candidate.15 The zerogradient corner detector has three parameters: go and (ii. In addition. 2.494 The Fundamentals of Robotics: Linking Perception to Action where 0± = Omin + 90°. 90°) and (45°. If we have an input image. • Step 2: Scan the image array to locate the edges (Condition 1). i2) = (90. If confirmed. 8.26a.
We can see that the result is reasonably good. The response of the corners is shown in Fig.* _ •  Fig. 8. 8.27 illustrates another example of corner detection. i2) are set as (90. jft « • .27b.. 00000<>000^00<>^00^ . •ooooooooooooooooo Example 8. ii. * * * (a) Input Image (b) Corner Response Fig. they can easily be grouped together to form a single output. • • . At an actual corner. 8. A simple grouping by connectivity will cluster the multiple responses together to form a single output.iu*~ii.16 Fig. . 2. 8. 5).. 8. The parameters (g0.Visual Perception System of Robots 495 Fig. As for the multiple responses at an actual corner.• • (a) Input Image W"W"m ' • • (b) Corner Response * #.26 Results of corner detection.26b. The result is reasonably good. there may be multiple responses which are closely clustered together.•. pfjppjjMif'ji •'*'" •.27 Results of corner detection.
such as the exponential of the difference between two intensity values.4. (In other words. Testing for the similarity among intensity values is a complicated issue. luminance and texture. We will first study spatialuniformity detection.4 Spatial Uniformity Detection An image is a spatiotemporal function. 8. and then temporaluniformity detection. the intensity values of the pixels within the disk are compared with the intensity value underneath the center of the disk (also known as the nucleus). proposed independently by the teams HarrisStephens and ForstnerGulch. it is necessary for a corner detector from this category to invoke a noisereduction process. This makes it and all cornerdetection algorithms computationally expensive.496 The Fundamentals of Robotics: Linking Perception to Action 8. it has the smallest USAN. It not only exhibits discontinuity in both time and spatial domains but also uniformity. One interesting corner detector is the SUSAN algorithm. uniformity detection is an interesting and important issue in imagefeature extraction. A location is considered a corner if the USAN at that location is a local maximum. such as the difference between two intensity values. The output of this circular disk at this location is the number of pixels with intensity values similar to that of the nucleus. At each scanned location. rowbyrow and columnbycolumn. If a simple criterion is used. One representative algorithm from this category. This algorithm uses a circular disk to scan an input image.3 Other Corner Detectors There have been many studies on corner detection. proposed by the team SmithBrady.3.7 Locations where the curvatures of contours or intensity surfaces are the local maxima are considered corners. is the Plessey corner detector. Therefore. This category of algorithms uses the following definition of corner: Definition 8. USAN for short. the algorithm will be computationally expensive. .4. in terms of chrominance. the algorithm may not be resistant to noise. This number is called the Univalue Segment Assimilating Nucleus or. A large group of cornerdetection algorithms are based on the measurement related to the curvature of contours or intensity surfaces. If a sophisticated criterion is used. Since the curvature measurement requires the secondorder derivatives. also called SUSAN).
Vu.Ui). Accordingly. As noise in an image is random.4. Vvi G [1.Ui) I I(vuUi) = L.4.u) is equal to L.u).. uniform surfaces under uniform lighting conditions will create a uniform appearance in certain areas or regions on a camera's image plane. G [l.1 Definition of Image Regions In a scene. the simplest definition of an image region is the set of pixel locations {(i>j. (8.Visual Perception System of Robots 497 8. an image region can be defined as follows: Definition 8.2 A General Scheme for Uniformity Detection According to the above definition of an image region.. • Secondly. described by intensity distribution N{HL.r*]}.68) with k — 3 (or other value).} where image function I(v. a uniform surface does not have a uniform appearance in an image. 8. Pixels having similar intensity values may not correspond to a same surface in a scene.rx]} (8.ry]. e [1.. In other words. let us consider the case of the intensity image and the spatial domain defined by image coordinates (v.67) This way of denning an image region has two drawbacks: • First of all. uniformity detection is purely a decisionmaking process (or nonlinear filtering). If the input is a color image. V«4 G [l.aL. we can reasonably assume that random variable I(v. consists of those interconnected pixel locations where the following condition holds: RL = {{Vi. two separate surfaces in a scene may exhibit very similar appearances in a camera's image plane.4. Without any a priori knowledge.8 An image region.u) as a random variable.2.u%)  \\Hyi. Thus. This is because .Ui)nL\\ < k. Moreover. both mean \i and variance a1 are threedimensional vectors. the intensity distribution of a uniform imagearea should have a mean of yii£ and a variance of a\. because of variations in lighting and viewing directions in a scene. In this case. u) follows a Gaussian distribution. Vw.. an image is not noisefree.u) denotes an intensity image. For the sake of simplicity. then the image region refers to an area of a specific color. we should treat image function I(v. If I(v.4.ry]. i = 1.O\). image region RL having intensity value L will be RL = {{vr.
However. A feature detector generally consists of three steps: a) noise reduction. possible outcomes can be an action.e. there are two big challenges with using a decisionmaking process: • Uncertainty: Decisionmaking is a process which associates inputs with outcomes.28. sometimes there can be multiple solutions (or interpretations) for the same input. red) will appear numerically different in a camera's image plane. reducing all the other regions. On the other hand.e. When dealing with possibility (i. This indicates that there are multiple pathways for associating an input with an outcome. the feasibility of a solution or mapping) . In a robot's visualperception system. If there is uncertainty in the representation (or description) of the input or outcome. goals. This in turn makes decisionmaking a difficult job. or rejection. the likelihood of an observation). decisionmaking is a difficult task. 8. b) incomplete sensory data. Possible inputs include sensory data. under different lighting conditions. and d) ambiguities in the description of the desired outcome. statement. or goal. for uniformity detection. a uniformity detection algorithm involves the following modules: • Selection of Seed Regions: A decisionmaking process can be one or a combination of the following . we know that there are multiple solutions for inverse kinematics. In this case. and c) feature selection. For edge detection. feature enhancement is the most important step. For example. the most important step is the decisionmaking process of feature selection. hypothesis and contexts. Uncertainty can be caused by: a) noise in the sensory data. In general. Statistical inference is a powerful tool to cope with uncertainty because it effectively deals with probability (i. a similar outcome can be associated with a set of different inputs. redundancy occurs as well. context. it is necessary to adopt other useful tools such as a Neural Network or Fuzzy Logic. hypothesis. certain colored surfaces (e. confirmation. b) feature enhancement. • Redundancy: In our study of robot kinematics. And. As shown in Fig. c) ambiguities in the formulation of facts.g.498 The Fundamentals of Robotics: Linking Perception to Action no linear filter is able to enhance one region while at the same time. fact.
a2) can be done either manually (by expectation) or automatically (by experience) . or associative memory. Distribution N(fi. For example. the colored seed regions can be stored and reused. In other words. we will study a general solution for detecting colored regions.68 explicitly. • Grouping of Classified Pixels: As we already mentioned. or by using a classifier implicitly. orange. a2) of a region (or color) in a given input image. This knowledge can be stored and reused in the future. this is not the case with color. In general. Colors can be classified and codified using a generic representation or symbol (e.Visual Perception System of Robots 499 three: a) knowledgedriven. This can be done by testing the condition in Eq. yellow. brightness or dimness. will create ambiguity. the next step is to classify the pixels in an input image into groups corresponding to these seed regions. there is no generic representation to codify luminancerelated information. Among the partitions. for uniformity detection.g. any attempt at specifying a generic representation of luminance. pink. the conceptual symbols such as red. necessary to perform a simple process such as the grouping by connectivity to cluster the pixels belonging to the same region. It is. a user can manually delimit a region and use it to compute the corresponding N(n. The specification of N(fi. are treated as seed regions. a2) of a seed region defines a possible region to be detected from an input image. is a good mechanism for storing generic information or knowledge. An alternative way to specify the possible regions is to subdivide an input image into an array of regular partitions. therefore. such as darkness. Given an input image. etc). blue. cyan. b) experiencedriven and c) expectationdriven. in term of luminance (or chrominance). • Storage of Seed Regions: If a region has a generic meaning regardless of the input images. • Classification of Pixels: When we know a set of seed regions. an input image may contain multiple regions with the same distribution N(fj. there is no a priori knowledge about intensity (or chrominance) distribution N(fi. it can be treated as knowledge. 8. Generally speaking. a2). In the following section. those having the smallest variances.. However. green. a region corresponding to a red color has the generic meaning of "being red". A neural network. Obviously. By removing . a2).
4. u). The chrominance characterizes the property of "pure color" (i. a color is the measurement of a light's chrominance and luminance.~. Physically. the most widelyadopted color space for color measurement is the L*a*b color space. i • ' •'  I Recognition .28 A general scheme for uniformity detection. ^ i v II. display. ~\ ^i ( I'. which takes an intensity image as input. "I ^ 1 i < Lissillriilinn i>l'pixt'ls l T U. the triplet . and artistic visual effects. Output will be in the form of three twodimensional arrays of pixels which correspond to the Red. the amount of photons entering a unit area). At pixel location (v..e.. As a result. such as a camera. Let us consider a color camera and its associated image digitizer as an electronic device for color measurement. for color measurement.e. A common reference. we use a coordinate system as a reference. Just as when we measure an object's location.'.4. Fig.. L*a*b Color Space Color measurement by an electronic device. 8. we can obtain the luminance. However. • '  ' F i „ .I'. Green. a psychological sensation). Images .3 Color Detection Color is the brain's physiological and psychological interpretation of light. Ima8e Regions f"~ W ! I I I .. If we want an unbiased color measurement. it is necessary to define a color space which takes into account that reference. . is the "white" color. it is necessary to have a reference. the solution to detecting colored regions also applies to detecting uniformity. the chrominance from a color. and Blue component colors respectively.500 The Fundamentals of Robotics: Linking Perception to Action 1 j ^  •*•• I K •!•. So far. 8. depends on the device's characteristics. or the absolute intensity (i.immg . . And. while the luminance characterizes the property of energy. there isn't an ideal color space which satisfies all needs in terms of color measurement.
a(v.U).200\ I Y I = I 0.Visual Perception System of Robots (IR(V. L*a*b color space is both numerically and perceptually uniform.587 0. negative a axis measure the greenness of a color. Representation of Colored Seed Regions When we have a seed region in an RGB image as input.0 0. This means that a numerical difference between two colors at any location in L*a*b color space causes the same sensation of color difference. a(v.69 and Eq.16 ' «K«) = 5 0 0 . the a and b axes measure a color's relative chromaticity. In other words. u). More precisely: • • • • The The The The positive a axis measures the redness of a color. The negative L axis indicates the darkness while the positive L axis measures the whiteness.U)) 501 is called the RGB tristimulus values.b(v. If /xc and a\ denote the vectors of mean and vari .u)} are the L*a*b tristimulus values in L*a*b color space.607 0. u). 8. { [ y ^ ] 1 / 3 lz^}1/3} (8. Zmax) are the results obtained using white as reference color.u). the L value is proportional to a color's brightness with reference to white.174 0. we first convert it using Eq. u)}. 8.70) where (Xmax.u) = 25.299 0.114 • \Z) \ 0. In L*a*b color space.u) \lB{v. However.b(v.70 into the corresponding L*a*b image.IB(V. we can compute the means and variances corresponding to the triplet {L(v. positive b axis measures the yellowness of a color. Then. and triplet {L(v.u)\ IG(v.u) = 200 .u)) (8. negative b axis measures the blueness of a color.U). Ymax.116/ and /IR(v. If the RGB tristimulus values undergo the following transformation: /X\ / 0.69) ' L(v. the L axis measures the relative luminance of a color.IG(V.066 1. { [ ^ ] 1 / 3 _ _ [ ^ _ ] i / 3 j b(v.u). f f ^ } 1 / 3 .
an RCE neural network consists of three layers of neurons: • Input Layer: This layer contains neuron cells that store the feature vector of input data (or training samples). Moreover. As shown in Fig. 8. • Output Layer: This layer contains a number of cells. the input layer has three cells: one for mean value /j. developed by Reilly et al. The RCE (Restricted Coulomb Energy) neural network. an RCE neural network can serve as an activememory mechanism and is able to perform fast and incremental learning. we have /M Mc (ul\ a n d a\ = \ a\\ .a. is a specific design of the hyperspherical classifier. if necessary. and one for mean value fib• Prototype Layer: This layer contains a large number of neuron cells which are grouped into a set of clusters. a colored seed region is represented by a normal distribution N(fic> at) if the variations in its chrominance and luminance follow a Gaussian distribution. the next question is: How do we design a memory to store seed regions and a classifier capable of recognizing image regions similar to the seed regions? A simple answer is to use an appropriate neural network which can effectively deal with both the uncertainty of input data and the redundancy of solutions. an RCE neural network is a good tool for dealing with the redundancy of solutions. a neuron cell in a cluster represents one feasible way of mapping between cells in the input layer and corresponding cells in the output layer. (8. It is suitable not only for classifying separable classes. A RCE neural network can serve as a general purpose classifier for adaptive pattern recognition.502 The Fundamentals of Robotics: Linking Perception to Action ance.L. each of which represents a generic . Clearly. Architecture of a RCE Neural Network When we know the representation of the seed regions. one for mean value /j. In the case of colored seed regions.71) = \na\ W W/ Accordingly. in the early 1980s. but also for recognizing nonseparable classes. In fact.29. Each cluster is mapped to its corresponding output.
like a humanoid robot. there is plenty of freedom to allocate as many cells as possible in the prototype layer to represent or memorize a class. In the case of spatialuniformity detection in color images. Parameters in a RCE Neural Network In the original proposal for the RCE neural network.Visual Perception System of Robots 503 ~~TVfyf'l Mean a f 4 ^x:j—P : ^C^N ^ \ / / / Meanb f . still. and its environment. N f c ~ —J_^V \Xl/~~ \ " " ' ' " i \ Input Layer t V\'. the RCE neural network is indeed an incremental mechanism for the incremental representation of any meaningful entity. While a practical limit is imposed by the computing hardware.29 Architecture of an RCE neural network.y ^ \ \ w >*'•~^' ^ v Prototype Layer Output Layer Fig. One interesting feature of the RCE neural network is that there is no limit to the number of cells in the prototype layer. Since it is possible for a new cluster and a new cell in the prototype layer to be allocated at any time. prototype cells are described by a set of five parameters: . this incremental representation can gradually be built through realtime interaction between a physical agent. And. class. 8. a class refers to a region of a specific color.
N]) in the output layer.2:2. . . known as the class cell. _ J 1 ^ ~ \ 0 if u^ > 0 and ax < A^/3. prototype cell j of class i will respond to input feature vector X. parameter t will be increased by 1..UijW (8. These responses will then be the input to the class cell. prototype cell j of class i outputs value 1 to the corresponding outputlayer's cell.. When a prototype cell is fired.504 The Fundamentals of Robotics: Linking Perception to Action • c: Label indicating the class to which a prototype cell belongs. • t: Counter which indicates the number of times a prototype cell has responded to the input either during training or realtime activation. Finally. Response from an RCE Neural Network Assume that an RCE neural network has n cells in the input layer and N cells in the output layer. output . A prototype cell in the RCE neural network has two modes of response: (1) FastResponse Mode: In this mode. • a: Radial decaying coefficient of the hyperspherical influence field which can be set to zero if there is no decaying effect. prototype cell j of class i will use a radial basis function to determine activating signal u^ as follows: Uij = \ij . otherwise. • A: Dimension (i. • UJ: Position vector of a prototype cell in the feature space. If Uij > 0 and ax < A^/3.. [^'6) (2) ProbabilityResponse Mode: In this mode. the dimension of which is determined by the cells in the input layer.xn}. that is. prototype cell j of class i outputs a probability value based on the following calculation: \pij=e^^. or triggered.Wnx . radius) of the hyperspherical volume (also known as influence field) of a prototype cell in the feature space. In response to input data X = {2:1.e.72) where \ix is the mean value of input data X. (874) If we have class cell Cj (i £ [1. we can first compute all the responses of the prototype cells belonging to this class cell.
it is easy Refer to the architecture of an RCE neural network. it is necessary to have one training sample per class so that we can establish at least one pathway from input layer to output layer for each class. In order to train an RCE neural network. and the sample variance is a userdefined positive value. The dimension of the input layer depends on the size of the input data. N]). the input layer will have three cells if an RCE neural network is used to memorize seed regions represented by N(fic. When we use a set of training samples as input. (8.cr^). as shown in Fig. there will be no cell in the prototype layer.75) If input data triggers responses from all the class cells in the output layer. the training process for an RCE neural network operates in the following way: • Case 1: Creation of a New Prototype Cell: When we use training sample X. 8.29. This is because mean vector /xc has three elements. On the other hand. is determined as follows: P(Ci\X) = —p—. (8. Training of an RCE Neural Network (Vi G [1. If this training sample has only one observation (one vector). the number of cells in the output layer depends on the number of seed regions. belonging to class i. and can dynamically be increased or decreased.JV]. Vie[l. before an RCE neural network can be used for recognizing or classifying data. Therefore. When an RCE neural network is activated for the first time. The process of associating the cells in the input layer to the cells in the output layer is known as RCE neural network training. a set of training samples must be available. a conditional probability describing the likelihood that feature vector X belongs to class d.76) If we know the conditional probabilities P(d\X) to decide which class input data belongs to. the mappings between the cells in the input layer and the cells in the output layer must be established. At the very least. For example.Visual Perception System of Robots 505 from the class cell will be the weighted sum of inputs as follows: Ri= J2 ta'PiJi Vie[l. the sample mean is the feature vector itself.N}. if it does not . Let us assume that a training sample is represented by sample mean /i and sample varianceCT2.
etc. This makes it an ideal tool to incrementally represent any family of meaningful entities. this new cell will be connected to class cell Ci which represents class i. the incremental training process of an RCE neural network is very simple and has no issue of convergence. new prototype cell j is created with the following settings: Cij = Ci Uij = fJX • Xij =3»ax tij = 1 . Most importantly. if it triggers a response from prototype cell j of the same class Ci. 8. the conceptual symbols in a natural language. the radius of the hyperspherical influence field of prototype cell j must be decreased according to the following calculation: Afej = \\ukj ~ Hx\\ This indicates that the radius of the hyperspherical influence field of prototype cell j is now equal to the distance between the center of prototype cell j and the center of training sample X. if it triggers a response from prototype cell j which belongs to another class Cfc.<?ij =°~X And.72) from existing prototype cells (if any). belonging to class Ci. We can see that an RCE neural network has the innate mechanism to dynamically create new cells in the prototype layer. such as the regions in a color image. Summary of a Color Detection Algorithm The prototype cells in an RCE neural network perform dual roles: a) to memorize training samples and b) to classify input data with reference to . belonging to class Ci.506 The Fundamentals of Robotics: Linking Perception to Action trigger any response (see Eq. counter t of prototype cell j is increased by 1 as follows: • Case 3 (optional): Modification of the Influence Field of an Existing Prototype Cell: When we use training sample X. • Case 2: Increase in an Existing Prototype Cell's Counter t: When we use training sample X.
However. Thus. The classified and grouped pixels belonging to a hand's image are displayed in black.g. • Step 6: Classify the scanned pixel by testing the responses from the probabilistic RCE neural network. • Repeat Steps 4. in order to understand the meaning behind hand gestures. described by the L*a*b tristimulus values. Alternatively. oooooooooooooooooo Example 8. indoor or outdoor) with simple or complex backgrounds.30 shows the results of handimage segmentation based on skincolor recognition by a probabilistic RCE neural network. use the L*a*b tristimulus values as the mean value of input data and set the variance to zero. an RCE neural network can serve as both a memory and a classifier. it is necessary to reliably identify the hand's image which can be captured in different environments (e.18 In recent times. If a pixel or its subimage does not trigger a response from the probabilistic RCE neural network. If we treat seed regions as training samples. The algorithm implementing a probabilistic RCE neural network for color detection can be summarized as follows: • Step 1: Select seed regions to train the RCE neural network. • Step 4: Scan an input image rowbyrow and columnbycolumn. take a subimage centered at the pixel location. then an RCE neural network is an effective tool for detecting spatial uniformity in color images. regardless of differences in sex or race. it will be treated as belonging to an unknown class. and compute the mean and variance of the subimage.17 Humanrobot interaction is an important skill that a humanoid robot should possess. 8. described by the L*a*b tristimulus values. A useful visual cue for quick and reliable identification of hand's images is skin color. one of which is hand gestures. 5 and 6 until all pixels are classified. Fig. Example 8. We can see that the RCE neural network works well. • Step 5: At an unclassified pixel location. • Step 3: Train the probabilistic RCE neural network. there have been many ambitious initia . • Step 2: Compute the means and variances of the seed regions. There are different forms of interactive communication between a humanoid robot and a human master.Visual Perception System of Robots 507 stored samples.
tives at developing autonomous vehicles. The latter occurs when there is relative motion between a robot's visualsensory system and a scene.30 Examples of handimage segmentation based on skincolor identification.5 Temporal Uniformity Detection The property of uniformity not only exists in an imageplane's spatial domain. However.508 The Fundamentals of Robotics: Linking Perception to Action C. If the viewing . ••oooooooooooooooo 8. before that happens. but also in the time domain.i^i' I (b) Image 2 (c) Image 3 (d) Image 4 Fig. An alternative and relatively new approach is to combine a humanoid robot with an existing vehicle. it is necessary for the robot to be able to quickly and reliably recognize landmarks and road signs. The classified pixels are displayed in black. 8.ni Tm. One common way of attempting this is to develop a builtin intelligent system which is fully integrated with the vehicle.4. Interestingly enough. Fig. This makes it easy to identify landmarks and roadsigns from color cues. The day when a humanoid robot is capable of driving a vehicle on the road may not be far away. 8. all landmarks and roadsigns are painted with specific colors.31 shows an example of using a probabilistic RCE neural network to identify landmarks painted in red.
tk) denote the projection of an object onto the camera's image plane at timeinstant tk. Formally. it is necessary to detect the uniformity in the image plane's time domain.u. This is commonly known as image feature tracking.32. we can state the definition of an image template as follows: Definition 8. as shown in Fig. the projection of this object onto the image plane will exhibit temporal uniformity within a sequence of images. tk) appears to be constant within time interval [£i. 8.u. However.tk) can be treated as the timeinvariant representation of an object's image.4.tk) is the timeinvariant projection of an object onto the camera's image plane within a certain time .If Is(v. then Is(v. if we capture a sequence of images and the viewing angle with respect to this object remains relatively constant. the projection of this object onto the camera's image plane will appear uniform within a sequence of images captured by the camera. u. 8. Let Is(v. an object with a general geometry and color may not appear spatially uniform in the camera's image plane.£n]. it is important to be able to visually follow the temporal evolution of an object in a scene.1 Definition of Image Templates If we only have a single image. 8.u.Visual Perception System of Robots 509 (a) Road Image Fig.5. In a dynamically changing environment.31 colors. This image is commonly known as the image template of an object of interest. In order to be able to perform image feature tracking. (b) Red Landmarks Example of a landmark segmentation based on the identification of specific angle with respect to an object in a scene does not change too much within a certain time interval.9 An image template Is(v.
u.tk) /. the problem of temporaluniformity detection is finding the locus of an image template within the image sequence.u). 8. When we have image Ii(v. Vifc e [t2. Since Is(v.u).tn] (8. Since an image sequence is a discrete sequence.32 Illustration of spatiotemporal image function describing a sequence of images.tk) is timeinvariant.77) where gs specifies the minimum permissible variation of an image template. u.tk) (k € [l. and n > 1) and image template Is(v. the detection of temporal uniformity operates in two steps: .n]. In this way. a simple way to pinpoint the locus of the image template is to determine the imagetemplate's location in each image. 8.u.u. A more advanced method treats the locus as a trajectory with a specific kinematic property. this definition can be translated as AIs(t) = \\Ie(v. Mathematically.5.u. interval.4. tk) within an image sequence. the temporal evolution of the locus is predictable in order to cope with uncertainty due to noise and incomplete (or missing) data.(u.2 Template Matching When we have a sequence of n images Ij(v.ifci)!! <gs.510 The Fundamentals of Robotics: Linking Perception to Action U Fig. it can be written more concisely as Is(v.
tk) are predicted. . If we denote sub{Ii(v. all the possible locations of an image template in image Ii(v.uo)} the subimage centered at location (i>o.u. If no template image is available. 8. In practice. oooooooooooooooooo Example 8. If the template image of an object of interest is known in advance.u)~sub{Ii{v. In this way.tk) ("o.uo)}]\. taken from the image at timeinstant tk\.tk)  (vo.The detected locations of these image templates in the image at timeinstant tk are shown in Fig.Visual Perception System of Robots 511 • Prediction of Possible Locations: In this step.20 Feature correspondence in binocular vision is a crucial step towards the inference of threedimensional geometry of a scene (or object).u.u.19 In a visualperception system."o)  (vo. All the possible locations of the image template are within the window W : wx x wy. feature tracking is an important issue. The ability to follow an object of interest within an image sequence will enable a robot to lock its visual attention. for some distinct objects.33 shows the results of temporaluniformity detection within two consecutive images. 8. V(vo. The subimages centered at selected points of interest are used as image templates taken at timeinstant tki. (8. This is still difficult to do. the left image can be treated as an image taken at timeinstant tki and the right image as an image taken at timeinstant tk. In practice. as image templates. one can apply the template matching technique directly to detect the locus of the object's template image within an image sequence.«o). However. we can treat the problem of feature correspondence in binocular vision as the problem of temporal uniformity detection if the images captured by the two cameras in binocular vision do not differ too much. we can choose some subimages. if we know the location of the image template in the previous image Ij(v. we can define a small window W : wx x wy centered at this location.u. then the search for the image template's location is a minimization process described by min \\Is{v.78) Example 8.wO) £ W. Fig. we choose a location where the subimage is similar to the image template.33b.tk~i). • Search for the Image Template's Location: This step is to determine the most likely location of an image template within the window W : wx x wy.
8.34 Example of template matching between two images from binocular vision. Fig. We can see that the objects corresponding to the computer monitor and printer can be effectively matched within the pair of images. 8.512 The Fundamentals of Robotics: Linking Perception to Action (a) Image at tk1 (b) Image at tk Fig. (a) Left Image (b) Right Image Fig.33 Example of template matching between two consecutive images. 8. oooooooooooooooooo .34 shows an example of binocular correspondence.
8. In fact. Let I[(tki) and //(ifc) denote two consecutive images. Thinking about it from an engineering point of view.21 In robotics. the ability of a visualsensory system to detect a moving object in the field of vision has useful applications.5 GeometricFeature Description A digital image is a twodimensional array of pixels. If we consider the time interval between two consecutive timeinstants to be 1. it is possible that human peripheral vision makes use of the simple technique of temporaldiscontinuity detection to achieve the results of quick response.Visual Perception System of Robots 513 8. then the firstorder derivative of an image function. For example. the algorithm implementing temporaldiscontinuity detection will be extremely simple.79. (8. This implies that an image array is without any unit of measurement. there are two reasons which justify it. one may wonder whether temporal discontinuity within an image sequence is an important issue. while it seems too early to undertake a parametric description of geometric features in an image. Example 8.4. can be computed from the image difference as follows: AJ / (t) = J / (i fc )//(tfci)l. In our study of the visualsensory system. temporaldiscontinuity detection can help a humanoid robot to choose an object of interest so that the robot's visual attention focuses on it. with respect to time. .79) Therefore.35 illustrates a scene with a complex background. we learned that human vision consists of two parts: a) central vision and b) peripheral vision. because detection of temporal discontinuity is a quick and simple way to detect any moving object in a scene which has a stationary background. a pixel may correspond to any real dimension in a scene. It is. Fig. We can see that the intrusion of an arm can be quickly and easily detected using Eq. One important role of peripheral vision is that it enables humans to quickly respond to the intrusion of any moving object in their field of vision. oooooooooooooooooo 8. Therefore. 8.6 Temporal Discontinuity Detection When dealing with a sequence of images.
1 The Basics of Feature Description The ultimate goal of feature description is to derive a compact representation of pixels belonging to the same physical entity.5. In order to achieve this goal. First of all. it is necessary to first divide the feature points into a set of groups. 8. the contour of a straight line is a list of edges. Secondly. This process is commonly called feature grouping. as shown in Fig. As a result. For example.514 The Fundamentals of Robotics: Linking Perception to Action O^M u\ i^P IP* QH 'SIMP P I ^^^^^^^^^^^^^^^H Image at tk  Image at tk1 = Output Image Fig. a parametric description of raw data is a helpful way to infer the geometry of a scene.36: • Feature Grouping: An image contains a large amount of raw data. However. one must address the following issues. For example. the results are still in the form of individual pixels. a binocular correspondence of two circles is much simpler to handle than dealing with two lists of points. these feature points are unlikely to belong to a single physical entity. . the size of the data is still large. such as a contour or surface. in a digital image. After feature extraction. two end points. In general. such as: a*u+b»v+c=0 are sufficient and accurate enough to represent a straight contour. 8. called feature points. or a parametric equation.35 Example of temporaldiscontinuity detection. Feature description is an effective way to reduce raw data while preserving useful information. Even after the feature extraction. We will study the problem of binocular vision in the latter part of this chapter. 8. an image contains a large amount of information. because a scene usually has more than one object.
In the simplest case. Therefore. regarding the content shown in Fig. However.37. physical entity. This example indicates that feature grouping may be one or a combi .37. 8. the purpose of curve fitting is to determine a compact representation of it. Fig. there is no direct visualsensory data which computationally leads to this interpretation. Features ° Feature Grouping Contour Splitting ^ Curve Fitting Cl ^es Fig.37 Example of feature grouping with an invisible but expected. 8. a contour can always be approximated by a set of line segments. • Curve Fitting: When we have a simple contour.Visual Perception System of Robots 515 • Contour Splitting: A contour is a list of interconnected edges which may. or may not belong to a single object. Even if a contour belongs to the boundary of a surface. The psychological interpretation by the brain. as shown in Fig. is that there is a rectangle overlapping four circles.2 Feature Grouping Feature grouping turns out to be a very difficult process because it involves complex decisionmaking.36 General procedure for feature description. each of which can be described analytically by one equation. it is necessary to split a contour into a set of simple contours. 8.5. it may have a complicated shape which cannot be described by a single equation. 8. 8.
A pixel location is generally surrounded by a set of eight neighbors. 8. it is necessary to know . the brain is skillful at postulating a series of expectations.u). as shown in Fig. Therefore.ul) • 8neighborhood: An image is a regular twodimensional array. as shown in Fig. When we have pixel location (v. there are three typical types of neighborhoods. 8. This process of partitioning image features based on the property of neighborhood is commonly known as grouping by connectivity.u). decisionmaking regarding a pixel must take into account the results of its neighbors or its spatial causality. In order to achieve a deterministic outcome. this is one of the most difficult things to perform well. Here. when processing an image. Humans are very skillful at formulating expectations. However. An image is a twodimensional array. an image is a twodimensional spatial function.u+l).37. 8. south. the expectation of the presence of a rectangle is indirectly verified by the visualsensory data. Therefore.38b. For the example shown in Fig. the locations of its four neighbors are (vl.u). These expectations will eventually guide a featuregrouping process. • Causal Neighborhood: At any timeinstant.38: • 4neighborhood: Neighbors in the north. we will study the datadriven process of feature grouping. These eight neighbors are called the 8neighborhood. this expectation is confirmed as a valid outcome. even for today's robots.516 The Fundamentals of Robotics: Linking Perception to Action nation of the following: • ExpectationDriven: Based on our visual experience. • DataDriven: Two pixels which are close to each other are likely to belong to the same physical entity. 8. (v + l.2.u). east and west are called the 4neighborhood. Human vision makes use of this property to quickly and reliably partition visualsensory data into a set of groups. (v.5. (v. When we have pixel location (v.1 Types of Neighborhoods Grouping by connectivity is a datadriven process based on a neighborhood of image features.
u) (v+l.U+1) (vl. m)  VUJ € [u .u)  (v. For example. Of all the possible neighbors of a pixel.a.c). will have already been processed when the scanning process reaches pixel location (v. U) (vl. and the process will be deadlocked.v .38 Typical types of neighborhoods. the neighbors inside the neighborhood. those with known results are called causal neighborhoods. 8.ua) (v.U+1) (v+1. u .80) In the following sections.U) (vl. u+b) (vl.u+l) (V.u1) (V.1]} ii Vi e [v — l. If there is mutual dependency between a pixel and its neighbors. 8.u) (V. 8.u+b) (c) Causal Neighborhood Fig.a.u1) (V.38c.U1) (vl. u + b]} {(vi. For example. Ui) ] Mvn € [u . the results of decisionmaking with regard to the neighbors in the vicinity of a pixel. the neighbors of pixel location (v.u).u+1) (a) 4neighborhood (b) 8neighborhood  (vc. u) will be { {(vi. a decision cannot be made. u) (vc. ua) (vl.Visual Perception System of Robots 517 (vl. As shown in Fig. ua)  (vc.u1) (v+l.u) (v+l.b.c] if Vi = v. if a decisionmaking process is performed by scanning an input image rowbyrow (starting from the first row) and columnbycolumn (starting from the first column).38c.u) (v. . we will demonstrate the usefulness of the above neighborhoods. (8.H) (V. as shown in Fig. the size of a causal neighborhood depends on three integer parameters: (a.
Therefore. cr2). In this case.a2).2. • Repeat Steps 2 and 3 until all pixels belonging to N(fj. a2) may not correspond to a single physical entity. which characterize the cluster. check the results of its neighbors in the 8neighborhood. and increase the grouplabel variable by 1. we can directly compute some quantitative measurements.39b. If we know the pixels belonging to uniform distribution N(/j. If any one of them has been assigned with a group label.518 The Fundamentals of Robotics: Linking Perception to Action 8.39a shows a small image array which contains three distinct clusters of pixels. known as groups. • Step 3: At each unlabelled pixel which belongs to uniform region N(fi. assign the value of the grouplabel variable to the pixel.2 Pixel Clustering Assume that input to feature description comes from an algorithm of spatialuniformity detection. The results of applying the pixel clustering algorithm are shown in Fig.22 Figure 8.5.3 Boundary Tracing The pixelclustering algorithm outputs distinct regions. An algorithm for pixel clustering usually uses the 8neighborhood and includes the following steps : • Step l:Set the grouplabel variable to the value of 1. circumference. A boundarytracing algorithm normally makes use of the . If we know the pixels belonging to a labelled cluster (or region).. grouping by connectivity is a simple process which effectively groups the pixels into a set of distinct clusters. circularity and moments. • Step 2: Scan an input image. oooooooooooooooooo 8. assign this group label to the pixel. rectangularity. rowbyrow and columnbycolumn. such as area. 2 and 3 by the algorithm. it is necessary to partition the pixels in order to form distinct clusters. it is necessary to identify pixels belonging to the edges of a labelled cluster (or region). each of which has a label. 8.2. If none of its neighbors has been assigned with a group label.a2) have been labelled. We can see that these clusters are labelled as clusters 1.. we can choose to represent a labelled cluster or region by its boundary. Alternatively. In this case.5. pixels classified as belonging to uniform region N(fi. Example 8. A process which does this is called boundary tracing.
• Repeat Steps 1 and 2 until all pixels belonging to the labelled cluster (or region) have been marked. 4neighborhood and operates in the following way: • Step 1: Scan an input image rowbyrow and columnbycolumn. • Step 2: At each unmarked pixel which belongs to a labelled cluster (or region).40 Example of boundary tracing. 8.Visual Perception System of Robots 519 1 2 3 4 [o lo lo lo [o [6 [6 lo lo [o I 0^200 200 0 ^ 6 " O~O~~O~O~O~ 0~200(T~ 0~ 0^200 200 200 200 <T~ 0~~ 200 0 " 0 ~ 0 ~ 200 200 2002000"" 1o 20~T [o lo lo 6 [6 [6 6 [o lo I 1 0~ 0~ 0~ <T~ 0~ 0~ 0 3 0~ 1 0~~ 0~~ (T~ 2~ 2~ 2~ 2~ 0~ 4 <T~ 1 0~~ 0~~ (T~ 2~~ 2 ~ 2~~ 2~ 0 5 6 7 8 9 o~2OO2ooo~6~o^o~b~o^o~~ b~~ o~ o ^ o~ o~ 200 200 o~ 6~ b~~ 0~ 0~ 0~~ 0~ 0~ 200 200 0~ 6~ 0~ 0~0~ 0 ^ 0 " 0^0~0~ 2002000~~ 0 ~ 0 ~ 0 ^ 0~0~0~0~200 200 0~~ 5 6 o~T~T~o~(T~o~(r~o~o~cr~ 7 o~~ o~ 6~ 6 " o ^ 3~ 3~ 6~ o~ o " 0~ 0~ 0~ 0~ 0~ 3~ 3" 0~ 0~ 0~ 8 9 0^ 0~ 0~" 0~ 0~ 0~ 0~ 3~ T~ 0 ~ ~ 0~0~0~0~0^0^0~3~3^6~~ 10 5~(Po~o~o"o~o~o"o~o~~ 2 4 6 8 (a) Input 10 10 o^o~o~o^o^o~o^o~o~6" 2 4 6 8 (b) Output 10 11'—I—I—I—'—L—I—'—'—I—I 11'—'—'—'—'—'—'—'—'—'— Fig. . mark this as a boundary pixel. The result of applying the boundarytracing algorithm is shown in Fig. 8. Example 8. 8.39 Example of pixel clustering.40b. check the results of its neighbors in the 4neighborhood.23 Figure 8. If at least one pixel does not belong to the labelled cluster or region. l lo lo lo lo lo l lo lo I o o o~6~6~o~Tooo~o~o~o~o~~ 3 5~o~o~~T6o7ooTooo~o~o~cr~ 4 o~o~TooTooiooioo7o6o~6~o~~ 5 o~ Too Too 100106 Too Too Too o~o~~ 6 0~0~TOOTOOT06T06TOOO^O~0~~ 7 o^o^o^TooTboT6oo~o~o~o~~ 8 0~ 0~ 0^ 0~ T O 0~ 0~ 0~ 0~ 0~ O 9 0~ 0~ 0^ 0~ 0^ 0~ 0~ 0~ 0~ 0 " 10 0~0~0^ 0~ 0~ 6~ 0~ 0~ 0~ (T~ 111—'—I—'—'—'—I—I—I—I—I 2 1lo lo lo l l lo lo l lo l o o o o 6~0^o~"o~255o~o~o~o~0^ 3 o~o~o~25s0^255o~o~o~o~~ 4 o~ o~ 255o~ 0 ^ o~ 255o~ o~ o~~ 5 o ~ 2 5 5 0 ^ 0 " 0 6 0 2550" o~~ 6 6~0~255 0 ^ b " 0 ~ 2 5 5 0~0~0~~ 7 o~o~~o~2556^255o~o~o~6~~ 8 0~ 0~ 0^ 0~ 255 0~ 0~ 0^ 0~ 0~ 9 0~ 0~ 0~~ 0~ 0^ 0~" 0~ 0~ 0~ 0 ~ ~ 10 0~0~0~0~0^0~0~"0"0~0~ 11I—I—I—I—I—I—I—I—I—I—I 2 1o 2 4 6 8 (a) Input 10 2 4 6 8 (b) Output 10 Fig.40a shows a labelled region (label 100).
Then. . for the ease of curve fitting. check the results of its neighbors in the 8neighborhood. Since two successive scans are performed in this method. a twopass algorithm for edge linking includes the following steps: • Step 1: Set the contourlabel variable to the value of 1. Normally. this process does not guarantee that a contour will not have any intersection or bifurcation. edgebyedge. it is desirable to obtain a set of contours without any intersection nor bifurcation. At each edge under the scan. The output is a set of contours. rowbyrow and columnbycolumn. Thus. Edge linking can be treated as the process of clustering edges in an image. • Repeat Steps 2 and 3 until all edges have been labelled. In summary. The input to an edgelinking process is an array of edges. • Step 2: Scan an array of edges. If we put the eight logic values from the eight neighbors together in a clockwise (or counterclockwise) order.4 Edge Linking The purpose of edge linking is to partition edges in an image into a set of labelled groups. we will obtain a logic byte. If none of them has been assigned a contourlabel. This can be done using the 8neighborhood to scan the array of edges rowbyrow and columnbycolumn. it is necessary to perform a second scanning on the array of edges in order to detect the junctions where a contour is broken into a subset of simple contours. it is called the twopass algorithm for edge linking. However.2. this is not easy to do. If the transition from "1" to "0" in the logic byte occurs more than twice. assign the value of the contourlabel variable to the edge. its neighbors will be assigned the logic value "1" if they belong to the contour. check whether it is a junction point. and increase the contourlabel variable by 1. and the logic value "0" if they do not. In order to obtain contours without any intersection or bifurcation. assign this contourlabel to the edge. However.520 The Fundamentals of Robotics: Linking Perception to Action oooooooooooooooooo 8. known as contours. • Step 4: Scan a labelled contour. • Step 5: At each edge under the scan. the interconnected edges will be grouped into a single entity.5. If any one of them has been assigned a contour label. • Step 3: At each unlabelled edge. then the edge is a junction point. let us assume that the thickness of a contour is one pixel.
41 Illustration of a singlepass algorithm for edge linking. Thus. This method is called the singlepass algorithm for edge linking. a contour grows gradually. we can develop a more reliable edgelinking algorithm which can scan an array of edges in a single pass. The edge. and produce contours without any intersection or bifurcation.41): • Tail: The tail is the starting edge of a contour. 8. In fact. If we make use of the causal neighborhood. there are three clusters of interconnected edges.u2) cv / \ \ /* Head 1 ( \ Fig. The one on the lefthand side is a simple contour without any intersection or bifur . we scan an array from the top row until bottom. In Fig. the tail of a contour is the first scanned edge (or endpoint) of a contour. • Repeat Steps 4. is called the actual running head. • Head: When a contour stops growing. and from the first column until the last. the running head becomes the actual head of the contour. 8.Visual Perception System of Robots 521 • Step 6: Break the contour at the junction point if one exists. Normally.ui> Running Head IC4 / C3/ I / I c<f^ / / \& / /(V2. which is added to a contour during linking. and represents a contour with three parameters (see contour C2 in Fig. it is the last edge (or endpoint) of a contour. 5 and 6 until all edges on the contour are processed. • Running Head: During the process of edge linking. 8. Tail ^ * /C2 \ Cl\ \ / !V\(vi.41.
This will result in the priorityright rule. Subsequently. The one in the middle has one intersection at location (v2. 3 and 4 until all edges are processed.u). • Step 4a: If there is no neighbor. Refer to Fig. • Step 4d: If there is more than one running head inside Nc(v.41. c) to be (1. It is unlikely that all the contours extracted from an image will have a simple shape. Lengthen the contour which contains this running head to the edge. u). This decision rule is called the priorityleft rule. 8. thereby creating a new contour. a contour is a list of interconnected edges. assign the contourlabel variable to the edge. and assign the edge to be the new running head of the lengthened contour.1) and choose the priorityleft rule.41. 6. increase the contourlabel variable by 1. and assign the edge as the tail and running head of the new contour.522 The Fundamentals of Robotics: Linking Perception to Action cation. as shown in Fig. The merged contour will be assigned label i. 8.5. Assume that contourthickness in Fig. lengthen the contour which contains the running head to the edge. 8. • Step 4c: If the running head of contour i and the tail of contour j are both present inside Nc(v. and its running head will be the running head of contour j . lengthen contour i to the edge and merge these two contours at the tail of contour j . • Step 2: Scan an array of edges. we can choose the running head nearest to the limit on the righthand side of Nc(v. as follows: • Step 1: Set the contourlabel variable to the value of 1. u). Then. A singlepass algorithm which makes use of a causal neighborhood can be summarized. • Step 3: For an unlabelled edge at location (v.u). check the results of its neighbors in causal neighborhood Nc(v. u).3 Contour Splitting In a digital image. u).41 is one pixel. Alternatively. and assign the edge to be the new running head of the lengthened contour. 8. it is not difficult to verify that the singlepass algorithm for edge linking will produce five contours. If we set (a. • Step 4b: If there is only one running head inside Nc(v. rowbyrow and columnbycolumn. H3). • Repeat Steps 2. .1. The one on the righthand side has a bifurcation at location (113.u).u). disable the other running heads inside Nc(v. find the running head nearest to the limit on the lefthand side of Nc(v. U2).
In the subsequent iterations. Assume that the two endpoints of the contour are A and B.5. If the contour is not in a simple shape. If its distance to the approximation line is above threshold value do. In the first iteration. the equation describing an approximation line in the camera's image plane is: a»u + b»v + c = 0. the localmaximum edges at C\ and D\ will be selected to split the contour. contour D\B . split the contour at this edge.Visual Perception System of Robots 523 However.3. • Step 3: Compute the distance from each edge to the approximation line. . = \\a • Ui + b • Vi + c\\ . • Step 5a: Find the localmaximum edge nearest to one endpoint of the contour. We call this line the approximation line.24 Figure 8. If its distance to the approximation line is above threshold value do. Example 8. If the edge's location is (vi. split the contour at this edge. the distances of which to the approximation line are local maxima. Uj). then its distance to the approximation line is d. this is necessary before a curve can be fitted into a contour. • Step 6: If the contour has been split at least once. Mathematically. we must split it into a set of simple contours so that it can be approximated by a curve. If the intention is to obtain a set of successive linear curves (line segments). then a contour splitting algorithm can operate in the following way: • Step 1: Arrange the edges of a contour in sequential order.1 Successive Linear Curves The strategy for splitting contours depends on the expected outcome of the curve fitting. • Step 2: Compute line passing through the two endpoints of the contour. We call these edges the localmaximum edges. contour AC\ will be split at C2. repeat the algorithm until no splitting occurs. 8.42 shows an example which illustrates the procedure of splitting a contour into a set of successive linear curves. • Step 5b: Find the localmaximum edge nearest to the other endpoint of the contour. Va 2 + b2 • Step 4: Determine the edges.
oooooooooooooooooo 8. We also call these edges the localmaximum edges.Ui). . C2Ci. C2 __——Si ^^^^^ ' V Approximation Line Fig. We also call this line the approximation line. then its distance to the approximation line is \\a*Ui + b»Vi +c\\ Uj = . Its equation is a»u + b»v + c = Q. • Step 2: Compute line passing through the two endpoints of the contour.42 Generation of successive linear curves by contour splitting. a practical contoursplitting algorithm is as follows: • Step 1: Arrange the edges of a contour in sequential order. DXD2. C1C3.2 Successive Nonlinear Curves If the desired output is a set of successive nonlinear curves (circular. • Step 4: Determine the edges. parabolic or elliptic ). If the edge's location is (vi.3. the distances of which to the approximation line are local maxima. C3D3. • Step 3: Compute the distance from every edge to the approximation line. . The final output of the successive linear curves will be AC2.5. D3D1. and D2B.524 The Fundamentals of Robotics: Linking Perception to Action at Z>2. 8. and contour C\D\ at C3 and D3.
1 Approximation Line Fig. the input to a curvefitting process is a simple .25 Figure 8. contour E^B is further split at edge E3. E2E3 and E3B. the localmaximum edges at C\ and D\ are selected. In this example.43 shows an example which illustrates the procedure of splitting a contour into a set of successive nonlinear curves. repeat the algorithm until no splitting occurs. • Step 6: Between the two localmaximum edges. In the first iteration. C2 / ' 1 '/ ' ' E 3 1 \ .Visual Perception System of Robots 525 • Step 5: Find the localmaximum edge nearest to one endpoint of the contour. E1E2.43 Generation of successive conic curves by contour splitting. take no action (Optional: Depending on the intended application. there is no contour splitting at AE\ and E\E2. Assume that the two endpoints of a contour are A and B. 8.4 Curve Fitting When we have a simple contour which can be approximated by a linear or conic curve. find the localmaximum edge nearest to the other endpoint of the contour.5.But. The final output of successive nonlinear curves is AEi. two edges between C\ and D\ have a distance to the approximation line which is zero. • Step 7: If the contour has been split at least once. If these two edges are at the same location. In the second iteration. 000000000000000000 8. find the edge nearest to the approximation line and split the contour at this edge. splitting can occur at this location). These two points are E\ and E2 and the contour will be split here. Example 8. In general. it is easy to estimate the parameters of the equation describing the contour. Similarly.
8.. i = l. which minimizes Eq. 8.82) jsin0=7P=F I P ~ Va2+b2 ' (883) It is clear that parameters (6. And.. must satisfy dp Accordingly..Ui).2..n}. c) are three nonindependent parameters. i = 1. b..n} which can be approximated by a line. 8. we obtain p = —u cos 9 — v sin 6 (8.81 by term y/a2 + b2 yields u»cos6 + vsm0 with + p=O (8. .81) where (a. p) are independent and uniquely determine a line in the uv plane. When we have contour: C = {(vi..Ui). we define the following error function: n L = ^2(ui» cos6> + Vi sin 6 + pf .5. (8.4.2.526 The Fundamentals of Robotics: Linking Perception to Action contour in the form of a list of edges: C = {(vi.85) .1 LineSegment Approximation Let us consider image plane uv to be a continuous. Dividing Eq.84.84) The optimal solution for p. A line in the uv plane can be described by the equation: a»u + bmv + c = 0 (8. twodimensional space. the output are the parameters of an implicit (or explicit) equation describing the contour..
88) Eq.84 yields n (8.91) Finally. (8. Eq. we obtain the following equation: n ^^ {(iii cos8+ Vi sin 8) • (Ui sin 8 + ViCosO)} = 0 i=i (8.Visual Perception System of Robots 527 with { Substituting Eq. 8. the solution for 6 will be (8.8.87) Similarly.90) Since 2 • sintf • cos^ = sin(20) and cos2 0 . must satisfy §»• 86 As a result.89) < B = £?=!«? (8.86) L = ] T (Hi • cos 0 + Si sin 0)2 with uz — Ui — u and Vi = Vi — v.85 into Eq.89 becomes tan (2e) = z—^. with (8. (8. which minimizes Eq.92) . the optimal solution for 8. 8.88 can also be written as (B .sin2 8) = 0.87.sin2 6 = cos(20). 8.A) • sin 8 • cos 9 + C • (cos2 9 . 8.
.9i..95..vo. VQ). computing the difference. we obtain M sin 0 — VQ COS 9 = v cos 9 — u sin 9.coS0 I v = VQ + r • smv. (8.2.96 can be determined by a leastsquare estimation.93 can also be written in a parametric form as follows: (u = uo + r.93) where (VO.n. (8. n 1 n (8.94.97) »=i 8...96) Parameters (UQ.4.2.. i= l. This can be done by first multiplying sin 6 to the first equation and cos 9 to the second equation.n}.5. 8.v0)2 = r2 (8.2 CircularArc Approximation A circle (or circular arc) in the uv plane is described by the equation: (u . In this way. Eq..3 EllipticArc Approximation The implicit equation describing an ellipse or elliptic arc in the uv plane is (u — Un)2 K—^L (v — VQ)2 +^ — ^ =1  (898) . . VO) in Eq.uof + {v.5. As a result.0i).v0)2. 8.MO)2 + (vi . 8.95) From Eq.. i — 1.94) Now..528 The Fundamentals of Robotics: Linking Perception to Action 8. we can directly establish a linear system of equations: uosm9i .UQ) is the center of the circle while r is the circle's radius. (uo.2. and then. If we know (uo. we can estimate the tangential angle at each edge on the contour. o When we have contour: C = {(vi.J2 V(ui .Ui).. i = l. the raw data of a contour is represented in the form of a list of triplets as follows: C = {(vi. 8.vocos9i — Vicos9i—UiSiB. (8..ui.4. we eliminate parameter r from Eq.r) are three independent parameters which uniquely represent a circle. then the estimation of radius r is r = .n} which can be approximated by a circular arc or circle.
while feature grouping in the uv plane is very elementary and it is easy to partition feature points into a set of simple contours or regions.. Clearly. each of which only corresponds to a single physical object. When we have a contour in the form of a list of triplets: C = {{vh m. it is necessary to study the relationship between index coordinates (v. ru. In order to obtain metric measurements. Therefore. vo.. The index coordinates (v. the four parameters (UQ.99) { 8.100) Vi = vo + rv • sinSj. Without any knowledge about the real dimensions.6 Ui = uo + ru • cosOi.rv) can be determined by a leastsquare estimation.Visual Perception System of Robots 529 The four independent parameters of an ellipse are (uo. we can directly establish two linear equations from Eq. i = l. Feature grouping at a higher level is necessary because the final outcome must be a set of contours and/or regions. i = 1. this is insufficient for object recognition. This is because the projection of an object onto an image plane only creates a subset of simple contours and/or regions. and all measurements done in the uv plane are relative to a projective scale. i = 1. • Secondly.n. geometries or models of image features.. it is theoretically impossible to distinguish image .... 8.2. Y. Z) in a threedimensional space.n (8. a digital image is an array of pixels. As in the case of the circle. the parametric representation of an ellipse is (u = u0 + rucos9 [ v = vo + rv sine*.. ru.. not quantitative. n}..VQ. 9i). u) are without any unit of measurement..rv).99 as follows: (8. Geometry Measurement There are two problems with geometricfeature grouping and description in a camera's image plane: • First of all. u) and coordinates (X.2. feature description in image plane uv is qualitative.. .2.
Then we will look at the principle behind and challenges of binocular vision. A monocularvision system is normally composed of: a) a single electronic camera (either color or monochrome). which is referenced to frame r. rZ) of point Q which is referenced to frame r. If motion transformation cMr describes the transformation from reference frame r to camera frame c.44.44. Finally. it is necessary to know how the coordinates of image point q are related to the coordinates of corresponding object point Q. v) of its image point q in the image array. then the coordinates of point Q with respect to . The information transformation inside a monocularvision system is shown in Fig.530 The Fundamentals of Robotics: Linking Perception to Action features which have very similar geometries/appearances but belong to different objects. the image digitizer will convert analogue image into a corresponding digital image. and produces geometrical measurement as output. Then. as shown in Fig. we will first study what can be done with a single camera (monocular vision). If the camera's output is not digital. the electrical signals. and c) computing hardware. picked up by the imaging sensor of the camera. the question is: How do we determine the coordinates of point Q if the coordinates of its image q are known? This problem is called the problem of inverse projectivemapping.6. 8. Assume that we are able to detect image feature q in an image array. In order to derive an answer. Assume that frame c is assigned to the camera. rY.1 Forward ProjectiveMapping When we have coordinates (rX.6. referenced to frame r. the digital images from the image digitizer will be the input to the process of geometry measurement in the monocularvision system.1. which is referenced to index coordinate system uv. 8. 8. In this section. 8. This problem is called the problem of forward projectivemapping.1 Monocular Vision The monocularvision system takes images as input. b) an image digitizer (if the camera's output is analogue video). The image of point Q will form image point q. the problem of forward projectivemapping is how to determine coordinates (u. will be converted into an analogue image. is projected onto the image plane of the single camera. Point Q in a threedimensional space. Then.
ut . camera frame c are ( with cx\ rx\ (8. CY. we studied the relationship between index coordinates (u.103) . Ix^S^x v /^^y ^4 ^ ^ _ _ ^ _ _ _ _ _ _ _ _ _ _ N ' ' _ 1 ^ i—J Image Processing 1 ® 1 ^ A I Analogue " I image = / / r 1 2^^^.44 Illustration of geometric projection in a monocularvision system. with respect to camera frame c. \ . \ 0 0 10/ (8. 8.101) i / Vi / In Chapter 7.v) and the corresponding coordinates (CX. CJA (8.102) 7PC = / s i 0 «oO\ 0 ^ o 0 . CZ). I t A i . */V /r^' „ T Reference ^ Frame 7 / / . This relationship is described by projectivemapping matrix IPc as follows: /u\ (CX\ s» lv\ = 7 P C .101) I ry z cy cz = ^ r .• Fig. Feature Extraction J / / Digital image A  .Visual Perception System of Robots 531 u " * VaBg^V ' 1—=> ^ Inverse Camera _ Frame Image . . (8. Projection .
8.Dy) are not intuitive. if {rx. the size of the camera's image plane is \T = rx9nX ( 8 . 8.104) Eq.103.{cMr}.102 yields * • \v ={7Pc}. 8. The combination of Eq. . In fact.ry) are the resolution of the digital image.105) matrix H is known as the camera's calibration matrix.1. (8. the specification of parameters {fc. The coefficients inside are called the camera's intrinsic parameters. (Dx. 8.102. we have (8. parameter fc is the focal length of the camera. (8.532 The Fundamentals of Robotics: Linking Perception to Action In Eq.6. while the elements in cMr are called the camera's extrinsic parameters.107) tan(£) = &.Dy) the sampling steps of the image digitizer and (uo. The most intuitive way of indirectly specifying a camera's intrinsic parameters is to define focal length fc as well as the aperture angles in both the vertical and horizontal directions. s is an unknown scaling factor. and b) camera calibration for determining the camera's intrinsic and extrinsic parameters. Eq. Refer to Fig. 8. 8.106 > {hI = ry*Dy where wi and hi are the width and height of an image plane.l z .2 Simulation of Monocular Vision For construction of a virtual camera.104 has two practical applications: a) construction of a virtual camera for vision simulation. And. 8. H is a 3 x 4 matrix. If we denote 6U the horizontal apertureangle and 6V the vertical apertureangle. If we denote 1Pc H={IPc}*{cMr). 8. In Eq.101 and Eq.Dx.104 describes forward projectivemapping.t>o) the coordinates of the optical center in index coordinate system uv.44.
(NOTE: The initial orientation of the camera frame coincides with the reference frame).0 1. As a result.0.3337 292.0.Visual Perception System of Robots 533 If we know apertureangles (9u.6059 256. with respect to the reference frame.4160 . —100.0) (cm).108.0 0.0 \ / 365.0/ (NOTE: It is assumed that the optical center is at (256.1736 133.0cm.26 We are going to construct a virtual camera with a focal length of 1. 70°) and image resolution of (512.512).108) n .200) (cm) in a scene.200.Dy) can be computed as follows: x ~~ rx L d U V2 / (8.0 / and the camera's calibration matrix is Z2. sampling steps (Dx. OO^O^OOOOO^O^OOOOO Example 8.9166 \ 0 0.103.500.9848 179.0027) (cm).0 0. From Eq. B = (100.0 0.0.0 0. is obtained by rotating the camera frame —100° about the X axis. then the sampling steps (Dx.0 0.45b. as shown in Fig.27 We now position the camera we constructed in the above example at location (200. the extrinsic parameters of the virtual camera are cM r /1.0 0.0013 1.0.200.0 365.256)).0 256.5967 ~~ 0.3. and C = .2/c # tan(^ Example 8.8926 0.0027.34).0 .7446 1. projectivemapping matrix 'Pc of the virtual camera is 'Pc = 0.Dy) will be (0. aperture angles of (70°.0.0) (cm).8.0074 0.0.0 \ _ °° 0.0 0.0 .0v). If we apply Eq.400.0 / (NOTE: H is normalized by the element ft. let us choose three points in the reference frame: A = (200. 8.0 0.0 0.0366 748.9149X H=\ 0 1. \ 0.2104 \0.6059 0. The orientation of the camera frame. 8. Now.9848 .0.0 1.1736 0.
192) and c = (376. •ooooooooooooooooo 8. From Eq. v) and calibration matrix HI In Eq.1. it is necessary to have at least four constraints.0) (cm). iv \=H. let us rewrite Eq. in order to obtain a solution. 150 200 250 ' c 250. " \ .500. rZ). rY.332).109. given image point (u. 50. s • a 200. 8. 8.45 Example of a simulated monocular vision. the question is: Is it possible to determine coordinates (rX.^^^100 o o Fig.0. there will be an infinite number of solutions for the unknowns (rX. ^"~~^ \ ^^^\. I z . 8. the projections of these three points onto the image plane will be: a = (256.100.3 Inverse ProjectiveMapping Now. b = (194.104 as follows: s. 5 ^^^^^ N. 500 \ ^ ^ 2 4 0 ° ^ ^ I ^^^~400 3°° " ^ \ ^c^300 200 ^ \ y _ I _^J^^200 100 ^sv]r». \^**~^~^[ 100 200 300 400 500 °>. the scaling factor s is unknown.534 The Fundamentals of Robotics: Linking Perception to Action (400. If we apply calibration matrix H. (8.109. it is clear that one image point will impose only three constraints on the four unknowns. In order to have a unique solution.0. Otherwise.109) Then. (a) Image Plane (b) Camera and Scene 50 100 • 300^ b ' . Therefore.6. : . 8. there . rY. 253). 150. r Z) and s. ^^f^r' x 300 350 400 450 ioo.
For example. BC.C). the objects carried by a conveyor in a production line can be treated as twodimensional (2D). it is likely that the coordinates. Eq. C) with reference to frame r and {{ui.rzzy £31 = y/(rx3rJCi)2 + Cr 3 . we must solve the nonlinear equations in the three constraints expressed by Eq. 2 = 1.2. 8.109.rX3)2 + {rY2 . 2D Monocular Vision For some applications.3} the coordinates of points (A.110) W {?) In Eq. B and C provides three extra constraints. or b) to reduce the number of unknowns. and CA respectively. in a reference frame. the size of the triangle formed by points A. Assume that we have a set of three points (A. then ' £12 = s/{rXx .110. Modelbased Inverse ProjectiveMapping Refer to Fig.2.3. Interestingly enough.111) < £23 = V( r *2 . 8. and the relative distances between them are known. This is because their locations are uniquely determined by two coordinates .45. 3} the coordinates of the corresponding image points. we can establish the following system of equations: 8i»\vi\=H*\ * .111. If we apply Eq. This method of determining 3D coordinates is known as modelbased inverse projectivemapping. 8. If we denote £12. 2 = 1. B. 8.Visual Perception System of Robots 535 are two possible ways to proceed: a) to increase the number of constraints. and the nonlinear equations must be numerically solved by a computer program. rYt. Thus.Vi). it is necessary to have three more constraints in order to derive a solution.B.Y3)2 + {Z2 .2. a solution exists. are not all of equal importance. However. 8.rX2)2 + (»•*! . rZi). £23 and £31 as the lengths of AB.110 and Eq. Thus.Y^ + iz^ rZl)2. i = 1.T 2 ) 2 + (rZl rZ2y (8.111 contain twelve constraints for the twelves unknowns. 8. there are twelves unknowns and nine constraints. (8. Let us denote {{rXi.
536 The Fundamentals of Robotics: Linking Perception to Action with respect to a frame assigned to the conveyor's top surface. Eq.115. which has been normalized by its element /i 34 .112) \ 1 / If calibration matrix H.116) where p = l/s. If we set rZ = 0. l v \ = H . l r Y /u\ =£>• \ v \ (8.115) In Eq. As shown in Fig. Interestingly enough. 8. h3i hS2 1 / (8. it is clear that the backprojection from image point a has a unique intersection with place rZ = 0 at point A.45. This mapping can be called the geometric principle of 2D monocular vision (or 2D vision for short). This indicates that matrix H' has an inverse. 8. (8.114) s*\v\=H'»\ r Y\. is expressed as ( Eq. the equation becomes s . H' describes the mapping from a planar surface in a scene onto a camera's image plane. ^ ' Q . (8. 8. 8.116 describes the inverse. 8. In fact.115 becomes rx\ p . 8. If we denote D = (H')~l. Refer to Eq.112 can be rewritten as /in /112 hi3 hu\ h21 h22 h23 h24 1 / (8. inverse projectivemapping in .109. These two coordinates are sufficient for a robot to perform pickandplace actions.projectivemapping between a camera's image plane and the planar surface in a reference frame. Eq.113) fl31 ^32 /l33 and we define a 3 x 3 matrix H' as ( hu hi2 h14\ h2i h22 h24 .
0) 0 cos(k»0)J \ 0 / = where (lX(k). rY{k). a set of intersection lines can be obtained. remains unchanged. we will explore another interesting application of 2D vision: its use in acquiring threedimensional models of geometric objects. the coordinates {rX(k). 8. ••oooooooooooooooo . If we know rotation angle k • A9 which corresponds to the kth intersection line. with respect to the planar surface.116. rZ(k)} of a point on this intersection line with reference to frame r can be determined as follows: /rX(k)\ rY(k) \rZ{k)J / cos(k*9) 0 sin(k»0)\ (lX{k)\ 0 1 0 . We will study these applications in detail in the next chapter. If we add all the coordinates (rX(k). rZ{k)) together. the result is the set of 3D coordinates of the scanned points on an object's surface(s). The laser plane intersects with an object placed in front of it and this intersection creates a curved line on the object's surface. a reliable way of obtaining the 3D coordinates of the points on an object's surface is to use a laser scanning system.46b shows one view of the result. 8. as shown in Fig. commonly called a Laser RangeFinder or 3D Scanner. Fig.28 Nowadays. 8. rY(k).46a. If we know matrix D which describes the inverse projectivemapping from the image plane to the laser plane. 8. Here. Another interesting approach for 3D scanner is to use a laser projector which projects a laser plane onto an object placed on a rotating table. the X and Y coordinates of points on the intersection line can be computed by Eq. If we rotate the object being scanned. Example 8. The coordinates of the points on these lines can be computed by Eq. The geometric principle of 2D vision has practical applications in visionguided walking or locomotion by a humanoid or mobile robot.Visual Perception System of Robots 537 2D vision is unique as long as the posture of the camera. lY(k)) are the coordinates of the point with reference to the frame assigned to the laser plane. however. This line can easily be detected in the camera's image plane because a laser beam creates a very bright. curved line on an object.116. lY{k) \sin(&.
113 into Eq. Without the calibration matrix. If a camera's extrinsic parameters are unknown.112 yields ( f TV \ (8. 8. This means. _ rXhn+ rYhr2+ rZhlx+ fe14 " ~ '•X/131+ rYh32+ rZh33+ 1 „ _ rXh2i+ rYh^+ rZh2x+ h.4 Camera Calibration From our study of a camera's frame assignment in Chapter 7.117. it's necessary to estimate calibration matrix H using a process commonly known as camera calibration. Thus. and this center is not directly accessible.A " — rXh3i+ rYh32+ rZh33+ 1 • . The process of camera calibration can be explained as follows: Substituting Eq.. Therefore. calibration matrix H will also not be known even if its intrinsic parameters are available. 8.117) ry If we eliminate unknown s in Eq.46 Example of the application of 2D monocular vision for the acquisition of a threedimensional model of an object. 8. for a camera randomly placed in a scene. we will not know motiontransformation matrix cMr. we obtainr Z • (8117) x j (8.6. it is not possible for monocular vision to make any useful geometric measurements.538 The Fundamentals of Robotics: Linking Perception to Action S ^ ^ ^  Camera ^ ^ ^ ^ ^ ^ ^ ^ ^ H  B **£ * ^^^^^^ Rotating Table B j Laser Projector I (b) View of 3D coordinates (a) System setup Fig. we will also not know the camera's extrinsic parameters in advance. 8.1. 8.118) { . we know that the frame assigned to a camera is located at the center of the camera's optical lens.
rZ)} provides two constraints. rY.Orrzm). The camera under calibration will output three images. a 3 x 3 array of dot patterns are painted on a planar surface. After thresholding.v). 8.47. as follows: A»V = B. Then.v). (8. V and B.47. 8. (rX. The optimal solution which minimizes the squared error \\A • V — B\\2 is V={AtA)~1m{AtB). In each image. Assume that a set of n pairs of {(u. (rX. rY. rZ) of a point in a scene. we must have at least eleven constraints. rZ)} in order to calibrate a camera. 8.119 imposes two constraints on the eleven unknowns in vector V. rY. the center of gravity of each cluster indicates the location of . In this example.v) are known. rY. Eq. A calibration rig can be as simple as an array of dot patterns painted on a planar surface.0mm. 8. the dots of the calibration rig can be detected by a simple thresholding technique because the dots are in dark color on a rectangular white region.119) Eq.Visual Perception System of Robots 539 If we define \rX [o rY rZ 0 0 0 0 0 rX 0 0 (rX»u) rY rZ (rX»v) (rY»u) (rY»v) {rZ*u)] \rZ •v)\' [:] and V = [hn hi2 hi3 /li4 /l 2 l /l22 /l23 ^24 ^31 ^32 ^33] . 8. And. as shown in Fig. Since one pair of {(u. as shown in Fig.29 A common way to manually calibrate a camera is to use a calibration rig. rZ)} are available (n > 6). lOO. (rX. In order to derive a unique solution for the eleven unknowns in V.120) Example 8. v). (8. which are the input to the process for camera calibration. each dot produces a cluster of pixels.118 can be compactly written in a matrix form. The distances among the adjacent dots are (100. When we have the coordinates (rX. and its image coordinates (u. Eq.119 describes the linear relationship among vectors A. we place the calibration rig at three different locations along the Z axis. it is necessary to have at least six pairs of {(u.
Moreover. Accord . U K * ° »••:' * .47 Example of images used for camera calibration.0.000322 1.5 Determining the Parameters of Cameras Interestingly enough. ( r X.*•••P + > .004174 0. V0. If we apply Eq.v).540 The Fundamentals of Robotics: Linking Perception to Action •L • ^ f . they are unit vectors.337599 0.6. they are mutually orthogonal. as shown in Table 8. rY.0.121) Vo with i/ { Ri = (rn r 1 2 r13) R2 = (r21 r 2 2 r 2 3 ) ^3 = (r3i r 3 2 r 3 3 ) . rZ)}.527718 .367294 . when we have a camera's calibration matrix as input. 8.000212 0. we have twentyseven (27) pairs of {(u.120. •3^K1' • • • fir • o •» ^^^^^ HB^^^'' * ' B B K L * " * JM (a) At Z= 0 (mm) (b) At Z= 200 (mm) (c) At Z= 400 (mm) Fig.1. We express motion transformation matrix cMr as follows: / H\ tx \ (8. Since R\. i?2 and i?3 are the row vectors of the rotation matrix in cMr.013599 110. 8.0 / OOOOOOOOOOOOOOOOOO 8. As a result. a dot in the image plane. Refer to Eq.015303 .068044 112. the estimated calibration matrix of the camera is H= / 0.1.104.000209 0. 8.500411 \ 0. we can compute its intrinsic and extrinsic parameters.
175 44.0 ingly.260 20 146.0 0.0 100.0 200.1 Camera Calibration Data 541 Nb I ~~1 u 116.500 114.0 13 107.128 100.0 15 190.0 1 rZ 0.808 80.210 I v 38.789 77.160 38.679 100.075 100.0 0.0 0.0 9 187.0 400.400 44.Visual Perception System of Robots Table 8.0 11 151.0 17 145.500 0.500 100.175 200.512 0.500 200.0 21 193.0 100.128 81.0 200.382 200.0 22 99.200 41.0 0. substituting Eq.933 200.0 2 155.0 400.121 into Eq.000 100.0 0. 7 ^ 1 = 0 .0 100. 8.500 75. RiRi = 0.0 0.0 25 97. 8.0 6 191.0 200.027 200.0 200.0 19 102.0 12 194.500 100.102 100.000 200.842 14 148.0 400.451 110.140 200.0 100.674 100.0 200.444 84.806 115.0 0.055 0.0 0.690 26 139.0 400.0 16 105.0  400.0 10 109.0 0. Pfc = l. 1 ^ 1 = 1.146 130.0 200.814 38.0 100.081 84.0 0.104 yields (8.0 200.0 0.0 3 195.0 400.500 88.583  200.0 24 188. 13 1 Then.0 0.0 0.0 400.0 0.0 100.0 0. R2Rt3 = 0.0 100. we have \\Ri\\ = l.842 1 rX 00 I rY 200.0 0.837 125.358 40.0 5 152.0 200.0 200.122) .0 200.0 200.0 200.0 200.852 118.641 100.0 0.0 400.0 400.0 0.0 200.0 0.087 41.891 82.0  0.362 23 143.0 27  184.102 119.580 0.950 200.294 78.0 200.0 7 112.928 120.667 44.075 123.0 100.0 4 114.0 200.0 0.0 8 149.0 200.0 100.0 18 186.
124 as follows: R3 = tz • (h31 h32 h33.123) From Eq.124) = (/i3i /132/133) / y ^ + V 0 = /l24 Eq. 8.+ « o ^ = (/in h12h13) fyJ7+^ < ^ = {h2ih22h23) (8.) Solution for UQ First multiply R3 to both sides of the first equality in Eq. it is possible to derive its intrinsic and extrinsic parameters as follows: Solution for tz If we apply the property i?3 = 1 to the third equality in Eq. the solution for R3 can be directly derived from the third equality in Eq.124.117 and Eq. If a camera's calibration matrix H is known. Since Eq. 8. 8.124. 8.125) \A§i + hh + hh .124 contains a sufficient number of constraints. we have (fxf+uo% fyJ7+v°lf §• \ /*fr+«o\ //in hi2 h13 h14\ fvtt+vo = ^ i h22 h23 h24 . we know the following equalities must hold: ' / x § . 8.127) (8.123.542 The Fundamentals of Robotics: Linking Perception to Action where fx = fc/Dx and fy = fc/Dy. (8. we can directly obtain the following solution for tz: tz = Solution for R3 If we know tz. 8.122 are equal up to a scaling factor.126) 1 . 8. (8. apply properties R\R3 = 0 and RsR3 = 1. 1 / W /132 /133 1 / (8. The result will be uo = (h11h12h13)*(tzRt3). Then.
The result will be vo={h2lh22h23)*{tzR\). apply properties R2R3 = 0 and R3R3 = 1. 8. multiply R3 to both sides of the second equality in Eq.124.v0R3\\.130) (8. (8. 8. (8. the computation of the norm on both sides of the above equation yields fx = \\tz O n hi2 h13)u0R3\\. 8.129) If we compute the norm on both sides of the above equation.124. Solution for R\ and R2 From the first and second equalities in Eq. Then. Solution for fx From the first equality in Eq.v0R3.131) [R2 = fy{h21h22h23)fyR3.Visual Perception System of Robots 543 Solution for vo Similarly. 8. we can directly write down the solution for R\ and R2. as follows: (Ri = \ ^(hnh12h13)fR3 ' (8.124.124. .128) Since \\Ri \\ — 1. Solution for fy Similarly. we have fxR\ = tz ( hu hi2 hi3 ) — u0R3. and apply > the property HR2II = 1 the solution for fy will be fy = \\tz ( h 2 1 h22 h23) . we have fyR2 = tz (/i 2 i h22 h23 ) . from the second equality in Eq.
One can see that the estimated parameters are very close to the groundtruth data. v) in a digital image? If we know index coordinates (u. 8.0074 0.48.9848.u0) \y = Dy»(vvo).544 The Fundamentals of Robotics: Linking Perception to Action Solution for tx and ty Finally.9848) R3 = (0. as shown in Fig. 0.0. .3337 .0130) and { Ri = (1.3.1737.9149 \ 1.5822. oooooooooooooooooo Example 8.8926 0.0366 748.0. Assume that the camera in a monocular vision system has been calibrated and its parameters are known.7446 1. 0.5989. y) will be ( x = Dx • (u . 8. 256.0 / If we apply the solutions discussed above.0013 1. Then.9976. its corresponding image coordinates (x.9166 H=[ 0 V 0 0. v) of the target.124 directly produce the solution for tx and ty as follows: { Example 8. v0) = (365. 365.2069). 255.31 For visionguided walking (or locomotion) on a manmade road. 133.27.ty. fy.V0). 0.tz) = (199.4160 . . The groundtruth data in this example can be found in Example 8.132) ty = 7j(/l24 . the results of the camera's intrinsic and extrinsic parameters are as follows: (fx. 0. it is necessary to estimate the robot's walking direction.0) R2 = (0. u0.26 and Example 8.5961.292.1737) {tx. Assume that a camera's calibration matrix is Z2.30 tx = J^(/ii4«o) (8. the interesting question is: What is the walking direction if the robot is programmed to follow a target seen at location q = (u. 179.9932. 0. the fourth and fifth equalities in Eq.0.
the walking direction with respect to reference frame r will be rv= {cMryl*cv.fu~u° \ f \ Jx ' v~v° f i\ J ' Jy This walking direction indicates the direction of the projection line passing through point a.48 Example of visionguided walking. Clearly.u 0 ). •<>0000^<><><>00<>0^^ . If the camera's extrinsic parameters in cMr are known.fc]. 8. When we normalize vector CV with scaling factor fc. Digital Image / » / Camera Frame \ Fig.y. will be cV={x. it is easy to compute the inverse of cMr. Dy*(v vo). then the vector which indicates the walking direction expressed in camera frame c.fc) or CV = [Dx • (u . Accordingly. the projection of vector rV onto the road surface indicates the robot's walking direction. the walking direction becomes cv .Visual Perception System of Robots 545 If the focal length of the camera is fc.
we can establish the following two systems of equations: s i'[ v )= H '\r Z \ (8133) and U + A U \ rY+AY) s2. This problem is known as structure from motion in computer vision. is equivalent to the displacement of a camera with respect to a stationary object (or scene). The coordinates of any point on the object can then be uniquely determined if the displacement vector of the camera is known. this is possible.^v + Avj=H. 8. if an object is stationary.134) Clearly. as shown in Eq. AZ) be its displacement vector. Ay. let (rX. is known.6. (8. one can move the camera to two different locations. For example. but only three constraints. Instead of specifying the model of an object (the relative geometry of three points).109. there is an implicit assumption that the camera must be stationary with respect to a scene. 8. and (AX. As a result.546 The Fundamentals of Robotics: Linking Perception to Action 8. Interestingly enough.109. we have five unknowns and six constraints. . In other words. one may question whether it is possible to determine the coordinates of a point in a threedimensional scene if the displacement vector of this point. with reference to frame r. An even more challenging question is whether it is possible to determine the threedimensional structure of a scene when the displacement vector of a scene or object is unknown.\r Z++AZ . From Eq.2 Binocular Vision From our study of monocular vision. This is because there are four unknowns. the coordinates of point A can be uniquely determined. the displacement of an object (or scene). with respect to a stationary camera. This principle is called motion stereo or dynamic monocular vision. we know that if there is no additional knowledge about the model of an object. the coordinates of a point in a threedimensional space cannot be determined from the forward projectivemapping of a single camera. rY. rZ) be the coordinates of point A in a scene. For dynamic monocular vision to work properly. Mathematically.
then.Visual Perception System of Robots 547 In a dynamically changing environment. and H2 the calibration matrix of the right camera.49. the constraint on the immobility of an object (or a camera) may not be acceptable. rY. 8. we assign a common reference frame (frame r) to the scene. 8. we can simultaneously place two cameras at these two locations. each of which has been assigned with a camera frame.134 illustrate that the coordinates of a point in a threedimensional space can be fully determined if two locations and the images captured at these two locations are known in advance. we have to calibrate these two cameras separately. The projection of point Q onto . there are two cameras. ^WIPllBiiH!. becomes the wellknown binocular vision. 8.". Therefore. let us denote H\ the calibration matrix of the left camera.1 Forward ProjectiveMapping As shown in Fig. 8.49 again.6. Mathematically.u .2. Consider point Q in the scene. rZ). ^ H Reference Frame Fig.  : ] v^ Right Camera t  [r^v*t ^T^* 3 : ' I i y "z \ ' . It is unlikely that the two cameras used for the binocular vision will be identical. Eq. Its coordinates are (rX. In addition. This. Refer to Fig.133 and Eq. Now. w ^ ^ Correspondence of 2D Features Computing of 3D Geometry \ ' I *b —[*' Left ''^Camera pZJ ^. the intrinsic and extrinsic parameters of these two cameras will not be the same. Instead of moving a camera to these two locations. 8. 8. Let us assume that frame C\ is assigned to the left camera and frame c% to the right camera. Thus.49 Illustration of the geometric projection in binocular vision.
50a and Fig. 100. 8.Vi).0366 748. and its projection onto the right camera's image plane is b = («2. The calibration matrix of the left camera is Hx = /2.3. 253). 8.50b respectively.If we apply Eq. T x \.4574 \ 0 1.50c shows a virtual scene and two virtual cameras.109.0074 0.4160 .vi) = (316.0013 1. ^2). 8.136) Eq.3337 238.0 / and the calibration matrix of the right camera is /2. 8. (8.18.8926 0. 253) 1&=(U2^2) = (268. Fig.548 The Fundamentals of Robotics: Linking Perception to Action the left camera's image plane is a = (ui. The locations of image points a and b are shown in Fig.0) (cm).0.0366 748.9166 #2=1 0 V 0 0. 8.4160 .32 Let us construct two virtual cameras in order to form virtual binocular vision. 500.0013 1.136 describe the forward projectivemapping in a binocular vision system.135 and Eq.0 / When we have point A.3.3337 .7446 1.0234\ 1.8926 0. 8.0074 0.9166 I V 0 0. Example 8. we obtain ( and rX \ 7Z (8135) s2 • U 2 I = H2 .7446 1. and its coordinates in reference frame r are A = (200. ••oooooooooooooooo . then the projection of point A onto the image planes of the two virtual cameras is (a= (ui.0.
2 Example of forward projectivemapping in virtual binocular vision. 100 . Moreover. the question here is whether it is possible to determine the coordinates of a point in a threedimensional scene from its images in binocular vision.137) .49. 8. 8. This means that inverse projectivemapping is a major concern.2. let us represent the two calibration matrices in binocular vision as follows: ( hi hi hi i\2 iiz iu\ «22 «23 «24 I hi *3 3 1 / (8. 400 J 500 .v2). 3 • . And. 300 . Assume that the correspondence between a pair of image feature points (a.\ 0 400 \ .50 8. 500 I . 1 0 0 ^^X~200 ^ 100 0 ^^~400 °°"\ 0 ^^^00 400 400 100 200 300 400 500 Fig. the index coordinates of feature points a and b are given as input.Visual Perception System of Robots (a) Left Image Plane 549 1 00 (c) Camera and Scene 200 a 300 300 ^ 400 250 s . * ^ _ ^ • (b) Right Image Plane 100 • ^ . Inverse ProjectiveMapping One primary goal we hope to achieve with a robot's visualperception system is that it is capable of inferring the geometry of objects in a threedimensional space. We would like to know how to determine the coordinates of point Q.vi). 200 . Refer to Fig.50. and • b= (u2.^ ~ — ^ _ _ ^ I 100 I^ 0>l 50 ^ .6. b) has been established. ^^pS'* • L + 1 Y •• Y 200 300 Z I 2 0 0 ^ . Let us denote the coordinates of feature points a and b as • a = (ui. In other words.
i 3 3 u 2 ) • rZ = u2.135.h2V\ «23 ~ «33^1 /o j ^ ^ ) Eq.144) rx\ \rz) .J32^2 i23 .J31^2) • VX u + {j22 .«31^1 «22 . 8.«3i^i) • rX + (i22 .141 can be combined intoi l 3 ~ following matrix form: i l l .i33Vi) • rZ .J32U2) • rY + (ii 3 .h2vi) • rY + (i23 . «1 = r*31 •are the icoordinates«33 •point +Q. let us eliminate rY. (8. 8.i31«2 i l 2 .141) If we define ( hi . 8. the optimal solution which minimizes the squared errors is \V2324/ r y = (A* A)'1 • (A*B) (8.24 . 8.J31W2) • rX + (ii2 .i2i(8.i u 1 («2i . y = h1"1.140 and Eq.j 1 (i21 . Z) scaling factor si from Eq.138) If we apply Eq.«32«i) • ry + (h3 ~ «33«i) • rZ = m.J32V2) • ^ + {J23 .143) Accordingly.140) If we do the same to Eq.«32«i i\3 ~ *33«i \ *21 . Now.J31"2 i22 .550 The Fundamentals of Robotics: Linking Perception to Action and ( s1»v1=i21» rX i n J12 J13 ju\ 321 322 323 324 • J31 J32 J33 1 / (8. we will obtain f ( i n .hiui in .vi . we have + i22» rY + i23» rZ + i2A (8.i 3 3 ^ 2 ) # rZ = V2~ J24 (8.J32«2 the J33W2 J21 .139) rX + 32 • ^ + of ^ 1 where (rX.136. This results in f (in .J33«2 / A. 8.139.hiui) • rX + (i12 .
200) \b=(u2.v2) From Eq.4574 \ 548.144 describes inverse projectivemapping in a binocularvision system. For easy reference.3253 0.7759 I 2.51b.2cm.7446 0. we call it direct 3D reconstruction.0. 8. 200).32.34 Figure 8. 8.0626 . we obtain = (200. the coordinates of the object point with respect to reference frame r assigned to the scene can be computed using Eq. rZ) = (35.145.9166/ Finally. 8. 000000000000000000 Example 8. (8.51 shows an example of the results obtained using binocular vision mounted on a mobile robot. we can see that vector B is: / 318. 195. 8. 7 7 5 9 / From Eq. An object (a box) is placed on the floor. The two images captured by the robot's binocular vision are displayed in Fig. Example 8.33 Let us use the same virtual binocular vision set up in Example 8. Assume that we have a pair of imagefeature points (a. 8.142.7446 0. .8207cm. And.51a and Fig. 193.0234 I ' \548.2 . b) as follows: (a= (ui.0574 \ 0 . rY.0730 0 0.J24 / Eq. 8.0626 .0.9166 I ~ 438.Visual Perception System of Robots 551 with /ui B= Vl~i24 iu\ \.2. ( 2. the results are (rX.144 have values within a continuous range. 8.33cm).4140 .144. We can see that all the parameters and variables in Eq.145) V V2 .«i) = (300.
8. the coordinates of the endpoints of the line segments with respect to frame r assigned to the scene can be computed. j (d) 3D Line Segments (c) Binocular Correspondence Fig. (a) Left Image (b) Right Image J . Fig. if the calibration matrices of the two cameras in binocular vision are known. The final results of binocular correspondence are shown in Fig. Then. . 8. In this way. Here. let us assume that the characteristics of the two cameras in the binocular vision are similar.. Subsequently.51c. and an algorithm on the detection of temporal uniformity (template matching) is used to establish the correspondence of edges. a critical step is establishing correspondence between the line segments in the left image and the line segments in the right image.51d shows one view of the results of 3D line segments. then the correspondence of line segments is decided by a voting technique. Finally. 8.51 Example of 3D reconstruction using binocular vision. the two images are treated as a pair of consecutive images captured by a single camera. simple contours are approximated by line segments.552 The Fundamentals of Robotics: Linking Perception to Action The vision process starts with edge detection and is followed by edge linking. If we know the correspondence of the edges.
• Necessary Condition 2: When we have imagefeature point a in the left image.144). However. for some applications.3 Unresolved Issue: Binocular Correspondence For feature points. is still a major problem of the visual perception system. Since the parameters will be different. When we have imagefeature point a in the left image. (2) Difficulty 2: The binocular vision's two cameras may not be optically and electronically identical. 8. This means that it is unlikely that the intrinsic parameters of these two cameras will be the same.2. their extrinsic parameters will not be the same either.Visual Perception System of Robots 553 000000000000000000 8. we have to search for its corresponding point in the right image which is a twodimensional array of pixels. This indicates that the determination of 3D geometry through binocular vision is only possible if the correspondence of imagefeatures has already been established. If we have to verify all the possible locations in the right image. its corresponding point b in the right image must be known in advance.49. 8. In other words.6. The first necessary condition is easy to meet because it is not difficult to manually calibrate a camera.H2) must be known in advance. However. since these two cameras are not placed in the same location. the search will be computationally prohibitive. this solution depends on two necessary conditions: • Necessary Condition 1: The two cameras in the binocular vision must be calibrated. it is still not desirable to know the calibration matrices. The second necessary condition. . the calibration matrices (Hi. And. known as binocular correspondence. these two cameras will "see" the same object differently in terms of geometry and appearance. Refer to Fig. inverse projectivemapping in binocular vision has a closedform solution. imagespace control). (See Eq. This problem is hard to solve for two reasons: (1) Difficulty 1: Images in binocular vision are digital images. We will study this issue in the next chapter when we discuss the iterative approaches to robotic limbeye coordination (or.
In fact. we will derive the (continuous) epipolarline constraint in a progressive manner. This results in the reduction of search space for binocular correspondence.y) are the right camera's intrinsic parameters. ^ £ V i). 8. As we discussed in Example 8.0. Assume that a pair of imagefeature points (a. let us examine what constraint is imposed onto the index coordinates of these two corresponding feature points (a. b) are projections of the same object point onto the binocular vision's two image planes.y) a r e the left camera's intrinsic parameters. fi. Eq. the direction of the projection line passing through feature point a in the left camera.148 ) V o where (M2. Here. 8.x.6.x. assume that the motion transformation between the left .31.49. is c l K=fe^. b) are projections of the same object point onto the binocular vision's two image planes. Now. fi. J (8.x Jl.0i fi.146 can also be written in a matrix form as follows: (8.147) Vo o i ) \i) Similarly.554 The Fundamentals of Robotics: Linking Perception to Action 8. vectors clVa and c2Vb will intersect.146) h. f2.y Vector clVa is expressed with respect to camera frame cl and (wi.vx) \b= {u2. for feature point b in the right camera. If imagefeature points (a. Refer to Fig. Let us denote the index coordinates of these two feature points as follows: (a = (ui.o.0J ^2.v2). b).4 Continuous EpipolarLine Constraint A common solution which addresses the first difficulty with binocular correspondence is to use the epipolarline constraint. Now. This constraint helps limit the search for correspondence to within an interval along a line. we have the following result: C2V»=\ 0 Jt 7% « U o i /\ i / ( 8 .2. vi.
clVb and c l T c 2 are coplanar.(^ ). any set of two images are related by a 3 x 3 transformation matrix. Then. Eq. it means that the binocular vision's two cameras coincide at the origins of their assigned frames.150.152) Eq.o I /2.150) Mathematically. Unfortunately. b) will be superimposed. Since c2 Vb is a vector expressed with respect to camera frame c2. By applying Eq.148 and Eq. 8.0. vector clTC2 is not a vector of zero length because the binocular vision's two cameras cannot physically overlap in the centers of their optical lenses. (8. c 2 Vb. the same vector expressed with respect to camera frame cl will be clVb = (clRc2) .Visual Perception System of Robots 555 and right cameras is dMc2=(C^2 Cl[c2) (8. when clTC2 = (0.151 describes computer vision's wellknown homography transform. 8. 8. we will have clVa = c l Vb. 0 \ 0 jL0  . It states that under the pure rotation of a camera. The normal vector of \ 0 0 ^ 0  g 1 / .x c 2 ¥&\ 11.149) where clRC2 is the orientation of camera frame c2 with respect to camera frame cl.i 1 . the above equality becomes (ux\ \V! with = M3x3 • /u2\ «2 (8151) /i I Jl.^ \ Jl. As the translational vector clTC2 connects the origins of frames cl and c2 together. 8.1 (±. 1 / (8. When the origins of the frames assigned to these two cameras are at the same location. vectors clVa.147.x M 3X3= o . and clTC2 is the origin of camera frame c2 with respect to camera frame cl. the two projection lines passing through feature points (a. x I ^ .0)*.
Substituting Eq. This means that the following equality holds: [( cl T c2 ) x ( c l K)]* .^2) m the right image plane.^ \ * Jl. 8. (8.ty.vi) in the left image plane are given. \ty tx 0 / Eq. 8.x J2. 0 0 1 / o .x \ (£. 8.156 imposes one constraint in which matrix ^3x3 only depends on the intrinsic parameters and relative posture (clMC2) of the binocular vision's two cameras.tzy (8. Eq.^ \ 0 0 1 / (8157) Fo = (ST)* • (ClRc2) In fact. 8.154 becomes {clVa)t»(ST)i*{clVb)=0. 8.156 imposes a continuous linear constraint on index coordinates (1*2. In practice. Eq./V)4 • ( c l H) = 0. we just need to search the pixel locations along the continuous line described by Eq. 8. This linear constraint imposed to the index coordinates is known as the epipolarline constraint. the search interval is the intersection between . If index coordinates (ui.153) Accordingly.148 into Eq. If we denote clTC2 = (tx.155 yields (8.x J2.fc .^ \.156) (^I 0 . we have (cl.155) fu2\ (Ul with vi l ) « F 3 x 3 » [ v 2 1 = 0 (8. (clVb) = 0. Matrix F3X3 is commonly called the fundamental matrix of binocular vision.vi).556 The Fundamentals of Robotics: Linking Perception to Action the plane passing through these three vectors can be computed as follows: clAT = ( cl T c2 ) x ( c l K ) .0 .154) and its skewsymmetric matrix ST as / 0 ~tZ ty \ ST = S ( cl T c2 ) = \ tz 0 tx . 8.^ \ / Jl.156.x \ ^3X3= \ and 0 fa . In order to identify a possible match for the feature point at (ui.Fo.147 and Eq.
6125 \ 0. Example 8.9848 0.6059.9698 0.1710 0.1534 0 0 0 10 / .52 Example of continuous epipolarline constraint in a binocular vision system. The left camera's parameters are (/i. 256. 365. ^ ^ 200 oJ 500 \ . 100 . v2to) = (365. "2.6059.0 0.9848 0. f2.1710 0. Although the continuous epipolar line is a onedimensional space.52 shows a virtual binocular vision.1710 189.1736 0. • = = m i ^ ^ .9848 0.9698 0.0 0.1710 179.9848 0. 256) and ( 1 °0 0.5967 113. Y z 30 —~~ 0 "~~ 400 ' 400 500 4o°^>\ i ioo^ N l^ioo o o ^^<r~m° 100 200 300 Fig.o) = (365./i.1736 0.0302 121.1736 0. ^^^Z~^~~^~z£~\. UU" 150H> HigHI Imaas Plan.35 Fig.8321 \ 0.0302 153.8218 0 (a) Left Image Plane 200 300 300 5001 . J 200 300 400 500 0 0 (c) Camera and Scene 10 / .y. 256) and ( 0. . The right camera's parameters are (f2. 256.6059.ullo. 8.*.Visual Perception System of Robots 557 the continuous epipolar line and the image plane.x.0.t>i. 8. 365.6059.1736 0. the search interval will depend on the size of the image plane.4448 164.y.
0. we can see that the match candidates are actually on the corresponding epipolar lines. Refer to Fig. The result is ( 0. lowerleft image).0001 0.558 The Fundamentals of Robotics: Linking Perception to Action From the extrinsic parameters of these two cameras.6883 = 0. 8.3420 79.9397 12. 8. 8.0616 0.9397 0. In this way. 8. upperright image). We treat the upperleft image as the master image. We can see that the projection (point b) of object point A onto the right image plane is on the epipolar line.0 / If we apply Eq.0 1.0 0. and the rest the slave images.i) = (252.0 0. 8.55. 253).0045 \ F 3 x 3 = I 0.2396 19.157. In this example.53. we have three pairs of images as follows: • (upperleft image.0 0.53.36 In a fourcamera vision system such as the one shown in Fig.0 0.8481 0.0.t.0258 •u2.156 is 0.0 0. oooooooooooooooooo . lowerright image).1818 0 0 0 1. the continuous epipolar line computed from Eq. its match candidate in a slave image must be on an epipolar line. the motion transformation from camera frame c2 to camera frame cl can be computed. a set of four images is available at a timeinstant. oooooooooooooooooo Example 8.1848 \0.2167 • v2 + 64.0862 \ 0.52c.0001 .0640 / When we have an imagefeature point at the following location in the left image: (ui.3420 0. • (upperleft image. the selected image features are marked by "+". 8. From Fig. we can obtain the fundamental matrix of binocular vision as follows: / 0.0 9. When we select an image feature in the master image. • (upperleft image. This line is shown in Fig.
Mathematically. Before we study a solution. Interestingly enough.ry) increases. let us first examine the following two interesting observations: • Finite Dimensions of Objects of Interest: As we already mentioned. for most applications.5 Discrete EpipolarLine Constraint Continuous epipolarline constraint partially solves the issue of search space because a continuous epipolar line still depends on the size of the image plane. Therefore. the search interval along the continuous epipolar becomes infinite as well. the search interval along the continuous epipolar line increases. it is interesting to investigate whether it is possible to break the dependence between the search interval and the image resolution.53 Example of continuous epipolarline constraint. 8. the relevant dimensions of a scene (or object) are . For example. when image resolution (rx.6. if the image resolution becomes infinite. in a fourcamera vision system. one primary goal we hope to achieve with a robot's visual perception system is the ability to infer the 3D geometry of a scene (or object) from 2D images. 8.2. as well.Visual Perception System of Robots 559 Fig.
n only depends on the range [Zmin. the desired accuracy of the results in a visualperception system should depend on the intended application.159) T Zmin + 1 • AZ 1 / / rX \ . 8. In this way. In theory.1.158) with n • AZ — Zmax. n] (8. we can specify the desired accuracy for the Z coordinates to be AZ. etc. / v rX \ . When inferring the 3D geometry from 2D images. i e [0. when a robot is walking down a road. then Eq. In other words. it is reasonable to assume that the Z coordinates from any object of interest should vary within a fixed range. (8. we should have the freedom to specify the desired accuracy..e.n (8. From the above two observations.n... the accuracy of the results obtained from images is not infinite (i.2. 300cm].. If the height of a robot is measured along the Z axis of the reference frame assigned to a scene. say: [0cm.158. It is independent of the image size.135 and Eq.160) Zmin + i • AZ ' V i / . Clearly. the error is not infinitely small).. we can choose the Z coordinates to be within lmm or within lcm. let us assume that the Z coordinates from the objects of interest in a scene vary within a certain range: [Zmin.. i = 0. 8. Still. If we apply Eq. • Finite Accuracy of a Geometry of Interest: The robot's visualperception system deals with digital images. as this is not a primary concern. 8.. For example. all the possible values of the Z coordinates which are consistent with the specified accuracy will be Zi = Zmin + i • AZ. for AZ. i = 0.560 The Fundamentals of Robotics: Linking Perception to Action not infinite.2. Zmax] and accuracy AZ. Zmax\. For example.1.136 becomes ( and / S2»\v2 \ \=H2m I . the primary concerns are: a) where does the road go? b) are there any obstacles to avoid? A robot's visualperception system should not be programmed to watch for objects falling from the sky.
• Step 5: Compute the difference between a subimage centered at («i.144. This algorithm is fundamentally different from the one which makes use of the inverse projectivemapping described by Eq.^2) are on a continuous epipolar line. Eq. rZ). it will be the match for the feature point at location (ui. This dissimilarity should also be smaller than a userspecified threshold value in order to eliminate false matching. rY. In summary.160. 4. We call this the discrete epipolarline constraint. This difference is called a dissimilarity.i»i) have been given as input. Eq. 8. we have the freedom to specify the range and accuracy for one coordinate. • Step 4: When the predicted coordinates (rXi. rYi.159. matches with the image feature at (u\.i>i) in the left image. Z). 8. 8. compute the predicted location (u2i.Visual Perception System of Robots 561 Interestingly enough. • Step 3: Find the value of index i within interval [0. Eq.n] for index i. depends on the value of n. Most importantly.g. and a subimage centered at (u2i. If such a location exists in the right image. rZi) from Eq. • Step 6: Choose location (u2j. This is explained as follows: When we have a value for index i. rZj) will be the results of the binocular vision's inverse projectivemapping. rZ) because coordinates («i. and is independent of image size.159 and Eq. coordinates (rXj. 8. 8.160 describes a discrete constraint along an epipolar line because all the predicted locations («2. rYi.160. Here.V2j) where the dissimilarity is minimum. This property does not hold true with a continuous epipolarline constraint. n] and compute the predicted coordinates (rXi.vi) in the left image. this algorithm simultaneously solves the issues of both . rY. 8. rY. rZ). an algorithm which implements inverse projectivemapping and takes into account the discrete epipolarline constraint will do the following: • Step 1: Specify the range and accuracy of one coordinate (e. And. Then. 8. 8. determine interval [0.159 allows us to compute coordinates (rX.vi) in the left image. If we know coordinates (rX. regarding a discrete epipolarline constraint.V2i) in the right image. but also the possible binocular correspondence.160 allows us to directly determine the location where the image feature at (1*2^2) m the right image. the dimension of search space. rZ{) are known. • Step 2: Set up Eq. index i helps us predict not only the possible coordinates (rX. and 5 for all the values of index i. Most importantly. • Repeat Steps 3. rYj.v2i) from Eq.
Refer to Fig. Refer to Eq. 8. 8. 8. Now. For easy reference. and permits the use of dissimilar cameras. so that the measurement of similarity or dissimilarity between the image features is meaningful.6 Differential EpipolarLine Constraint In literature. and the rest the slave images.2. Example 8. However. if not identical. the second difficulty with binocular vision has been largely overlooked. we call this algorithm the indirect 3D reconstruction.54. a set of four images is available. and the desired accuracy to 10mm. The selected image features in the master image are marked by "x". if we apply the discrete epipolar line constraint.300mm]. When we undertake 3D reconstruction. such as edges or corners. if we could eliminate this stipulation.54. from the master image. its match candidate in a slave image must be on an epipolar line. if we assume that these intrinsic parameters and . Here. we set the range for the Y coordinate to [0.145. it would be inappropriate to form a binocular vision system with one monochrome camera and one color camera.562 The Fundamentals of Robotics: Linking Perception to Action binocular correspondence and determination of 3D coordinates.37 We use a fourcamera vision system. 8. We treat the upperleft image as the master image.6. In other words. the search space for a match candidate in a slave image is a set of discrete spatial locations. Because of this stipulation. However. we could form a binocular vision system with two cameras which have very different characteristics. the computational time necessary for each feature point is independent of image size. If an image feature in the master image is given as input. as shown in Fig. At a timeinstant. In this example.55. Moreover. it is necessary that the two cameras are optically and electronically similar. we propose a simple solution which overcomes the second difficulty with binocular vision. as shown in Fig. All existing algorithms for binocular correspondence still require that the two cameras in the binocular vision system see the same object in as similar a manner as possible. •••••••••ooooooooo 8. (NOTE: The Y axis is perpendicular to the table's topsurface). A binocular vision system's fundamental matrix ^3x3 depends only on the intrinsic parameters and relative posture (clMc2) of its two cameras. we normally select image features.
161 the differential epipolarline constraint. It can be used in two ways: • Dynamic Binocular Vision: When a binocular vision system moves along with the head of a humanoid robot (or mobile vehicle). differentiating Eq.54 Example of discrete epipolarline constraint where a search space for a match candidate is a set of discrete locations. 8. then the fundamental matrix becomes a constant matrix. We can estimate the displacements of the image features within the left and right image sequences respectively using a technique for detecting temporal uniformity.144 with respect to time allows us to obtain fu2\ (Aui Awi l ) « F 3 x 3 » v2 +(u1 vi l ) « F 3 x 3 » /Au 2 \ Av2 = 0 . And. Clearly.Visual Perception System of Robots 563 Fig. 8. the estimation of . (8. relative posture remain unchanged. the visualsensory system outputs two streams of digital images: a) the left image sequence and b) the right image sequence.161) We call Eq. 8.
Since feature extraction and displacement estimation are carried out independently using the outputs from the two composite cameras. we can not only detect the image features. In this way. the left and right cameras can be very different. but also estimate their displacements from the output of the composite camera. the composite camera's output is a sequence of two images at a time instant.564 The Fundamentals of Robotics: Linking Perception to Action (AMI . The rest are treated as false . we can use two composite cameras.55. the estimation of (Aw. 8. This can be easily explained as follows: When there is an image feature at a = [u\. the differential epipolarline constraint is unique if and only if a scene (or an object) does not undergo a pure rotation about the origin of camera frame cl (or c2). A composite camera is made up of two similar cameras. mL . AV2) is independent of the characteristics of the left camera. Among the possible matches. ' « H 1 \ ^^^^^^^^^^^^^H^^^P^^^P*t ! • • • (b) Composite cameras (a) Composite binocular vision for robots Fig. it is not necessary for the (composite) cameras to be optically and electronically similar. As a result. there is only one correct one. Interestingly enough. Similarly. as shown in Fig. its possible matches in the right image plane are the projections of the object points on the projection line passing through feature point a. v\) in the left image plane.2. • Composite Binocular Vision: Instead of forming a binocular vision system with two simple cameras. As a result.55 Illustration of a composite binocular vision system for a humanoid robot. 8. A«i) in the left image sequence does not depend on the characteristics of the right camera.
the correct match is the one which verifies Eq. Avb) can be computed directly. For a selected image feature. 8. a pair of two consecutive images in the left camera will not be related by a homography transform.Vl) = ( A « ! Avi l ) • F3x3» \ V2 + ( f l W 1) •i r 3x3» l /A« 2 \ Av2 • \i/ V i / (8.va) = 0.144 and Eq. We form two pairs of images as follows: • (upperleft image.144 and minimizes /u2\ £(ui.Visual Perception System of Robots 565 matches.162. the computed values of differential epipolar line constraint. we select image features from the upperleft image. 8. the matches in the pair of upperright and lowerright images can be easily determined by the intersections of epipolar lines. 8. when there is an image feature point at (ui. with regard to the two selected image features. Because of numerical error.56.248). In other words. lowerleft image). displacement vectors (Au a .v\). Then. are ( E(ua. two features are selected from the locations: Uua. If the scene (or object) does not undergo a pure rotation about the origin of camera frame cl.a) and (Auj. none of the false matches will reappear on the epipolar line which is determined by feature point (ui + Aui. By applying Eq.161 provide the necessary and sufficient constraints for the selection of a correct match. Eq.vb) = (256. 8.599551.vb) =0.174416 \E(ub. lowerright image).162) Example 8. we use template matching technique and epipolar line constraint to find its match in the lowerleft image.38 A composite binocular vision system outputs a set of four images at a timeinstant. As a result.va) = (160. 8.121) I {ub. In the pair of upperleft and lowerleft images. In practice. Az. Therefore. Refer to Fig. . differential epipolar line constraint is unlikely to be exactly equal to zero. In this example. • (upperright image.vi + Av\).
This is a very challenging task as digital images do not explicitly contain any geometric information about a scene (or object). In a robot's visualperception system. We learned that the purpose of image transformation is to obtain alternative representations of the original input image. And. 8.56 Example of differential epipolar line constraint. and geometry measurement.566 The Fundamentals of Robotics: Linking Perception to Action mm * JL^J^H ^' —^ * ^M Fig. feature extraction. one has to go through a series of informationprocessing functions in order to derive geometric information from digital images. oooooooooooooooooo 8. These functions include image processing. Therefore. the purpose . feature description.7 Summary One primary goal we hope to achieve through a robot's visualperception system is the ability to infer the threedimensional (3D) geometry of a scene (or object) from twodimensional (2D) digital images. there are two important categories of imageprocessing techniques: a) image transformation and b) image filtering.
We learned that geometric features are implicitly encoded into the discontinuity. depending on the types of analytical curves to be used for feature description. and is useful in the detection of uniform regions in color images. if we have a set of simple contours as input. The output from imagefeature extraction is individual feature points. luminance. Here. the algorithm depends on the adopted definition of corner. continuity and uniformity of an input image's chrominance. These algorithms always start with edge enhancement followed by a process of edge selection. the results are qualitative descriptions of the geometry of a scene (or object). it is necessary to study the important topic of geometry measurement using either monocular or binocular vision. We discussed the probabilistic RCE neural network which deals efficiently with uncertainty and redundancy. We studied three typical examples of feature grouping which make use of three different neighborhood types: a) the 4neighborhood. we can choose linear. most edgedetection algorithms can be explained using the principle of dynamics resonance. We also learned that feature description performed in the index coordinate system assigned to a digital image does not have any unit of measurement. we studied two methods of contour splitting.Visual Perception System of Robots 567 of image filtering is to enhance specific dynamic characteristics in an input image. circular. We learned that geometricfeature description involves three sequential steps: a) feature grouping. imagefeature extraction is mainly the detection of edges. Subsequently. Thus. each of which can be described by analytical curves. As for uniformity detection. it is necessary to group these feature points into a set of clusters. These two principles are clearly explained using the convolution formulae and Laplace transform. we studied a zerogradient algorithm for corner detection which is conceptually simple and works well with real images. For a linear filter. or texture. or elliptic curves to approximate simple contours. most algorithms rely on a complex decisionmaking process because of the challenges caused by uncertainty and redundancy. and c) curve fitting. we studied the principle of dynamics conservation and the principle of dynamics resonance. Interestingly enough. . The purpose of imagefeature extraction is to detect geometric features from images. corners and uniform regions. And. b) contour splitting. and c) the causal neighborhood. b) the 8neighborhood. We know that strategies for contour splitting are different. Theoretically. In order to obtain a quantitative description. As a result. With regard to corner detection.
we studied binocular vision. 8. (10) What are the criteria for designing a good featuredetector? (11) If f(x) represents a onedimensional ridgetype edge.568 The Fundamentals of Robotics: Linking Perception to Action We learned that the most important issue underlying geometry measurement is inverse projectivemapping. what is the exact mathematical description of f{x)l (12) How do you make a feature detector possess the degree of freedom to . Finally. 8. (9) Explain the importance of the localization of a detected feature. We know that if a camera's calibration matrix is known. (2) Is there any difference between the process behind computer graphics and that behind the visualsensory system of a robot? (3) Does a digital image explicitly contain any geometric information? (4) What are the typical applications of a visualperception system? (5) Explain information processing in a visualperception system. (7) Explain the fundamental difference between image transformation and image filtering.8 Exercises (1) Explain the process of visual perception. we studied two possible Solutions: a) the application of a discrete epipolarline constraint. and b) the binocular correspondence of the image features must be established in advance as well. it is necessary to know a camera's parameters. the camera's intrinsic and extrinsic parameters can be fully determined.9. or introduce extra knowledge (resulting in modelbased geometry measurement). and b) the application of a differential epipolarline constraint. With monocular vision. (6) Write a sample Cprogram that calculates the intensity histogram of an input image. In order to measure the geometry. (8) Write a sample Cprogram that automatically determines the optimal threshold value VQ in Eq. However. We know that the inverse projectivemapping in binocular vision has a closed form solution. we must reduce a scene's dimension (resulting in 2D vision). These resulted in two different algorithms for binocular vision. The first condition is easy to meet but the second one largely undermines binocular vision's practicality. For this reason. In order to deal with the second issue. this solution depends on two necessary conditions: a) the two cameras must be calibrated in advance. we also studied camera calibration.
what are the coordinates of the object point with respect to the reference frame of the scene? (25) In Example 8. if we change the focal length to 10.0°). /124)? (22) What is binocular vision? (23) What is the sufficient condition for binocular vision to work properly? (24) In Example 8. (33) Project 5: Implement an edgelinking algorithm.256).45. (18) For curve fitting with circular or elliptic arcs.0cm. what is the projective matrix 7PC? Now.200).33.0°. (26) What is the epipolarline constraint? (27) Explain the differences between the continuous epipolarline constraint and the discrete epipolarline constraint. (19) A virtual camera's focal length is 1. Propose some practical solutions to this problem. if a = (400. what is the meaning of (/114. 100. (30) Project 2: Implement a cornerdetection algorithm. (15) What are the possible definitions of a corner feature? (16) Explain why a boundarytracing algorithm uses the 4neighborhood instead of the 8neighborhood. Its aperture angles are (100. (32) Project 4: Implement a templatematching algorithm. .100). (34) Project 6: Implement modelbased inverse projective mapping in monocular vision. Where are locations B and C if image points b and c are inversely projected onto plane rZ = 0? (21) In a camera's calibration matrix . one problem is how to compute the tangential angle at each edge pixel of a simple contour.35. If the image resolution is (256.0 cm. compute the epipolar line corresponding to the image feature point a = (100. (17) Explain why contour splitting is necessary before performing curve fitting.Visual Perception System of Robots 569 compensate for the shift of feature localization caused by noise? (13) If the coefficients in Canny's function satisfy the following constraints: oi • a2 < 0 and a3»a. 8. (31) Project 3: Implement a colorsegmentation algorithm. (28) Do the two cameras in a binocular vision have to be absolutely similar in terms of optical and electronic characteristics? (29) Project 1: Implement an edgedetection algorithm. what is the new projective matrix 7PC? (20) Refer to Fig.200) and b = (200.4 > 0. what is the physical meaning behind these constraints? (14) Explain why Deriche's function is a special case of Canny's function.
(1993). (1990). (1994). (7) Fostner. (11) Jain. Hildreth (1980). International Journal of Computer Vision. Robot Vision. A Combined Corner and Edge Detector. IEEE Trans. G. R. (1997). D. B.9 Bibliography (1) Banks. R. J. W. Machine Vision. PrenticeHall. (3) Castleman. (8) Haralick. Computer and Robot Vision. (4) Crane. Gulch (1987). (36) Project 8: Implement the solution for continuous epipolarline constraint. (37) Project 9: Implement a binocularvision algorithm with a discrete epipolarline constraint. (9) Harris. D. on Pattern Analysis and Machine Intelligence. MIT Press. A Fast Operator for Detection and Precise Location of Distinct Points. and E. (6) Faugeras. (1986). Kasturi. A. C. Inc. R&D Publications. (2) Canny. R. (12) Marr. Computer Vision. M. (14) Philips. 8. (10) Horn. London. ThreeDimensional Computer Vision. B207. G. Theory of Edge Detection. Image Processing in C. and L. PrenticeHall. K. G. Using Canny's Criteria to Derive Optimal Edge Detector Recursively Implemented. Models and Inspection. Shapiro (1993). and E. (1996). Image Processing and Pattern Recognition. 2. Martin (1992). and B. Stephens (1988).570 The Fundamentals of Robotics: Linking Perception to Action (35) Project 7: Implement solutions for determining a camera's parameters from its calibration matrix. O. R. ISPRS Intercommission Workshop. A Simplified Approach to Image Processing. PrenticeHall. The 4th Alvey Vision Conference. P. D. R. (13) Marshal. (38) Project 10: Implement a binocularvision algorithm with a differential epipolarline constraint. MIT Press. A Computational Approach to Edge Detection. Schunck (1995). 8. AddisonWesley. Digital Image Processing. (5) Deriche. K. (1994). Signal Processing. and M. World Scientific. Proceedings of Royal Society. R.. Corners and Centers of Circular Features. S. (1987). . McGrawHill. and R. R.
MIT Press. Object Enhancement and Extraction. in Picture Processing and Psychopictorics. (17) Roberts. . Mapping or Associative Memory. M. I. Academics Press. L. International Journal of Computer Vision. S.A New Approach to Low Level Image Processing. in Optical and Electrooptical Information Processing. A Neural Model for Category Learning. L.. Journal of ACM. (1965). (18) Smith. M. (19) Specht. M.Visual Perception System of Robots 571 (15) Prewitt. SUSAN . S. Boundary and Object Detection in Real World Images. IEEE International Conference on Neural Networks. D. J. Probabilistic Neural Networks for Classification. Cooper and C. (1988). Brady (1997). and J. L. Machine Perception of Threedimensional Solids. 45. Elbaum (1982). Biological Cybernetics. 23. D. (20) Yakimovsky. 23. (1976). G. (16) Reilly. F. N. Y. (1970).
.
From an engineering point of view. decision making is an indispensable part of any physical or mental actor. a control law is a functionbased. Thus. However. we know that any complex behavior is the result of the interaction among mental and physical actors embedded in a physical body. and forms an important pillar in an autonomous system. this is still an unresolved issue which needs more research. as we studied in Chapter 6. an autonomous robot must also be a sociable and educable robot. Mathematically. we learned that motion control can be fully automated if and only if: a) the desired output is given and b) the chosen control law makes the closedfeedback loop stable. such as the switching functions studied in Chapter 5. we only addressed 573 . a physical actor is a sensingcontrol loop which acts on a physical kinetodynamic chain. however. which can not only be deployed in industry for better productivity.Chapter 9 DecisionMaking System of Robots 9. In Chapter 5. In Chapter 5. Humans have an innate mechanism for autonomously developing mental and physical abilities. and. or system. it must have the ability to develop and learn behaviors compatible with human society. What is this innate mechanism which determines the degree of autonomy in a physical system. like a human being or a robot? So far. such as a humanoid robot. but can also coexist in human society for better service.1 Introduction The ultimate goal of robotics research is to develop autonomous robots. If a robot coexists with humans. a mental actor is a perceptiondecisionaction loop which acts on the physical actors and may also alter the mental world's internal representation(s). Clearly. decisionmaking process which may be complemented by some decision rules.
2 The Basics of Decision Making Generally speaking. etc. control laws may be complemented by some decisionmaking rules such as switching functions. 9. unknown or unmodelled system dynamics. military. there would be no automated actions. Mathematically. decision making is the key to the automation of many industrial tasks. the decisionmaking process exists everywhere. In particular. regardless of uncertainty caused by noise. the control law(s) must achieve certain design specifications in terms of absolute and relative stabilities. 9. we can consider a control law to be a functiondriven decisionmaking process. Accordingly. tasks or even behaviors. a critical issue is the design of suitable control law(s). for an automaticfeedback control system to work properly. From this perspective. The emphasis of our study in this chapter will be on imageguided motion planning and control. This means that an action taken by a system can be fully automated if the desired output of the automaticfeedback control loop can be specified in advance. etc. . we will address the first as we study the basics of decision making and motion planning for autonomous robots. However. Depending on the complexity of a system under control.1 The Key to Automated Actions In Chapter 5. This requirement must be satisfied by all automated systems in manufacturing. disturbance. In this chapter. transportation. a set of control laws may be used to cope with variation caused by the system's configuration.2. we learned that control system is synonymous with automaticfeedback control system.1 Automation is the interaction between a decisionmaking process and an actiontaking system which does not require any human intervention.574 The Fundamentals of Robotics: Linking Perception to Action the second condition. we can attempt to formally define the buzzword automation as follows: Definition 9. Therefore. Without decision making.
we can attempt to define the term autonomy as follows: Definition 9. Since behavior is a sequence of ordered actions for the achievement of a desired outcome.2 The Key to Automated Behaviors As we discussed above. there is no absolute autonomy. the elimination of this dependence will make an automated system an autonomous system. an automated system is not an autonomous system. first it is necessary to define scientifically the term autonomy. which. It would signal tremendous progress if we could develop a robot which could decide without human intervention the desired outcome to be achieved. "automation" means automated actions implemented by feedback control loops. the ability to selfspecify the desired outcome is the key to autonomous behaviors.2. Autonomy is related to the state of independence. As a result. how do we make an automated system an autonomous system? Perhaps. From Chapter 5. Formally. Hopefully. It still depends on the specification of desired output or outcome. we know that without a specification of desired output. This indicates that an automation system has at least one degree of dependence. So.3 An autonomous system is an automated system which has gained at least one level of independence in specifying its desired output without human intervention in programming or reprogramming. it may be useful to attempt to define the term autonomous system as follows: Definition 9. As there is no absolute independence for any physical system on Earth. for many industrial automation systems including robots. an automated system will not produce any useful outcome. which is: the specification of desired output. Because of this human intervention. the thoroughness in eliminating the degree of dependence will result in the degree of autonomy attributed to an autonomous system. is programmed and reprogrammed by humans. An automated action has only gained independence in action taking. autonomy is better described by a degree (or level) of independence. Therefore.DecisionMaking System of Robots 575 9. And.2 Autonomy is a characteristic which describes an automated system's degree (or level) of independence. .
9. Accordingly. Fig.3 DecisionMaking Processes According to the level of sophistication in performing actions. or the outside world. is called an active system.576 The Fundamentals of Robotics: Linking Perception to Action 9. A system in this category does not have the ability to interact with other systems. Thus. decisionmaking itself is a process which has its own input and output. . the term intelligent can only be attributed to a system which encompasses a decisionmaking subsystem. we denned the term intelligence as a measurement which is inversely proportional to the effort spent achieving a predefined outcome or target. An openloop control system is a reactive system. a system can be classified into one of these categories: • Active System: A system which acts on its own. it is called a reactive system. either an automated or autonomous system can be attributed with the qualifier intelligent if it has the ability to achieve a predefined outcome or target in different ways. Most importantly. • Automated System: If a reactive system is equipped with a decisionmaking module. • Intelligent System: In Chapter 1. and must be embedded in a native system (like robot) in order to be useful.2. A reactive system can interact with the environment. regardless of desired outcome or sensory input. effort depends on a subjective decision made in the planning of an action sequence. Au automated system acts according to desired output or outcome which has been specified in advance as input. the enabling technology for the development of autonomous systems is decision making. • Reactive System: If a system is able to respond to outside stimulus. Clearly. it is called an autonomous system. As a result. The term effort means energy spent in the planning and execution of a sequence of mental and physical actions. it becomes an automated system.1 shows a simplistic view of a generic decisionmaking process. like a controller. The purpose here is to highlight the complexity of the inputoutput relationship. • Autonomous System: If an automated system specifies its own desired output or outcome.
• i^ iZ— ^ 1 OUTPUT • Actions • Confirmations Belief System ^ • Sensory data • Constraints 'Goals •Contexts I I f\ \ ) / II j^M^^BuHUI^H^^H I DecisionmakingProcess b I 1 Statements • Others —7 _ / \l^ ) 1/ O U T P U T •Facts • Previous Results • Others I 1/ L  j ^_F " 1f _^ U . representations and hypotheses imposed on the environment and physical systems.2. In contrast to hard input. kinematics imposes kinematic constraints on a robot's mechanism. ^ . the common forms of hard input to a decisionmaking process include: • Sensory Data: These are measurements directly obtained from sensors embedded onto physical systems. 9. the input to a decisionmaking process can be divided into two parts: a) hard input. and b) soft input. • Constraints: These are models. soft input refers to all data and facts which are subject to linguistic interpretation. For example. we mean all data and facts which are the direct outcome of the physical systems.3.1.1 Inputs of DecisionMaking As shown in Fig.— ^ . By hard input. . Hard Input Generally speaking. 9.DecisionMaking System of Robots 577 . ("Knowledge _) ( Experience ~) Learning ^— Fig. 9.1 A simplistic view of a generic decisionmaking process.
In this case. Thus. while an imaging sensor can be very accurate. the instruction "Place the cup on the dining table" is a goal specified by others.g. but is acquired from experiments or trials . For example. the common forms of soft input include: • Knowledge: Knowledge itself has no elasticity. the statement "I will do it in a better way" is typically a selfgenerated goal. However. However. when it comes to computing with words (the semantic level). and the application of knowledge may also introduce some approximations (i.578 The Fundamentals of Robotics: Linking Perception to Action • Goals: These can be instructions received from the outside world or selfgenerated goals.g. As shown in Fig. image transformation or image filtering). • Contexts: These generally refer to conditions imposed by the environment. effects or actions). softness or elasticity). For another example. 9. One can interpret it as "An increase in velocity increases the kinetic energy of a body". • Facts and Previous Results: This is knowledge and information about what a physical system can achieve. color has a specific range of wavelengths. the solutions and results of inverse kinematics tell us what can be done with an open kinetodynamic chain.g. the interpretation of knowledge is subjective. the statement itself has a lot of elasticity. "indoor" and "outdoor" are two different contexts. This pattern of mapping is not preestablished. This makes a decisionmaking process very different from a linear transformation or filtering process (e. • Experience: The skill or mental syntax of a physical system can be described as the specific pattern of mapping (or association) between the input (e. For example.e. decision making is a difficult and complicated job for a robot to perform. there is no concrete answer to the question of how to deal with the softness (also called elasticity) of soft input. let us interpret the equation WK = \rrf v2. causes) and the output (e. the interpretation of color in a digital image does not have a rigid range of numerical values. However. So far. and is a kind of hard input. For example. For example. Soft Input A decisionmaking process must consider not only hard input but also soft input.1.
g. it will help us to focus our research effort in this direction.DecisionMaking System of Robots 579 which result in what is called experience. Here. conjectures. probable. A belief system cannot exist on its own. but must be embedded in a physical system (e.2 Outputs of DecisionMaking As shown in Fig. words with elastic meanings include: possible. desired path or trajectory). c) curiositydriven behaviors and d) statedriven behaviors. of words used.g. conclusions. etc. the output from a belief system may also be taken as input to the belief system itself. . Without a belief system. Statements will introduce uncertainty or fuzziness depending on the lexical elasticity.2. However. Interestingly enough.g. likely. voltage or current level). When output from a belief system remains permanent. As we studied in Chapter 6.1. Accordingly. It is the source from which all goals and desired outcomes are formulated. • Statement: This is the linguistic description of facts.g. motion descriptions (e. For example. or signal descriptions (e. 9. a body). Action can be in the form of action descriptions (e. A belief system takes all the data from a body's sensory systems as input. experience cultivates the skill or mental syntax of a physical system. we can raise these challenging questions: What should the ultimate goal of a humanoid robot be? Could this ultimate goal be programmed once without any further reprogramming? 9. In fact.3. • Belief: Belief is the "motor" which powers an autonomous system. it conditions the tendency for a physical system to plan familiar sequences of ordered actions. b) expectationdriven behaviors. what to do). the possible forms of output from a decisionmaking process include: • Action: This is the most common sort of output from a decisionmaking process. a human being or a physical system would not be able to selfdevelop and evolve. or softness. if we understand the importance of the belief system. How to develop a belief system for a robot is still an unresolved issue. the belief system determines four types of autonomous behaviors: a) valuedriven behaviors. hypotheses. it may be designated as the ultimate goal(s) or ambition(s) of a physical system.
• Confirmation: This refers to statements without any ambiguity. fact or context is in the form of a statement. the goal specification. 9. In fact.4. Depending on the size of the spatial or temporal window. "Place part A on top of part B" contains the elastic term "on top of". and output.2. In general. uncertainty can exist in input. large. 9. we studied the meaning of a sensor's precision.4 Difficulties in DecisionMaking We mentioned that decisionmaking is a very difficult job for a robot.2. many. • Incompleteness: Mathematically. In a physical system. Due to the presence of noise. "No". a set of simple decisionmaking processes can be nested to form a complex decisionmaking system. For example. This incompleteness of sensory data will cause uncertainty. a robot must be equipped with the appropriate mathematical tools to handle not only certainty and redundancy but also uncertainty (including ambiguity). uncertainty is the result of one or a combination of the following three causes: • Imprecision: In Chapter 5. Because of the nature of soft input in a decisionmaking process. "It is". the observation of a sensor's output at a specific timeinstant may not contain complete information about a physical quantity.2. • Ambiguity: When the specification of a goal. 9. If a decisionmaking .580 The Fundamentals of Robotics: Linking Perception to Action very. Thus. the statistical distribution of a sensor's output is spread over a region. a spatialtemporal signal from a sensor has a limited range in both space and time. etc. the term precision refers to the statistical distribution of a sensor's output s(t) when its input y(t) is set at a fixed value. specification of goal. "It is not". ambiguity can arise depending on the elasticity of the words used. This is largely due to uncertainty and redundancy. The simplest confirmations include: "Yes". this goal specification has what is called fuzziness or vagueness. resulting in the imprecision of the sensory data.1 Uncertainty As shown in Fig. small. etc.
[ Ik. For example.lUii"~ I Vui.il Vl'.>  I I / * * ' " ••*. DecisionMilking ^ .lllll\ Ki'ihiiKl. the difficulty of decision making due to redundancy has been overlooked.iiK\ COM .4.iiii. gripper or hand) attached to a robot can be determined from the measurements of joint angles and forward kinematics. the same physical quantity can be derived from the same set of sensory data by using different signalprocessing techniques. the edges of a surface's boundary can be detected by an edgedetection algorithm. In some cases. For example.mik 1 Methodology Fig.2.2 Redundancy In literature.I IM. specification of goal. _"/' I \'i.lllslk. For example. This posture can also be measured by a robot's visual perception system.~ I u / / \ IniciciKC I Sl.ll KlI. 9. redundancy may come from: • Multiple Sources of Input: The same physical quantity can be measured by a set of different sensors. the posture of an endeffector (e.2.'I[.knluml.g.nm <Jl ll'l I . and output.DecisionMaking System of Robots 581 Difficult?  l\l'l I I IK0ll. Alternatively. take the solutions of inverse kinematics for a . • Multiple Mappings: The pathway for associating input to output may not be unique. process is not able to completely eliminate fuzziness.IIII1> Ki'iluiul. 9. 9.llllU . These edges can also be determined by a uniformitydetection algorithm followed by a boundarytracing algorithm. redundancy may also appear in input.il Illlil. In general...ll.2 Difficulties and methodologies in decisionmaking. As shown in Fig. ambiguity will certainly reappear in the output.'VK k'llL'i. there may be an infinite number of mappings from input to output.
redundancy is the source of information which can be used advantageously by a decisionmaking process (e. sensor fusion. the environment and human society. For example. an open kinetodynamic chain).e. A typical example is face recognition. optimization. efficie