This action might not be possible to undo. Are you sure you want to continue?

BooksAudiobooksComicsSheet Music### Categories

### Categories

### Categories

Editors' Picks Books

Hand-picked favorites from

our editors

our editors

Editors' Picks Audiobooks

Hand-picked favorites from

our editors

our editors

Editors' Picks Comics

Hand-picked favorites from

our editors

our editors

Editors' Picks Sheet Music

Hand-picked favorites from

our editors

our editors

Top Books

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Audiobooks

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Comics

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Sheet Music

What's trending, bestsellers,

award-winners & more

award-winners & more

Welcome to Scribd! Start your free trial and access books, documents and more.Find out more

This page intentionally left blank

This page intentionally left blank

To My Parents .

This page intentionally left blank .

neural modeling. Attention is paid to the selection of appropriate model structures in terms of the dynamic properties. Recently. Many systems are not amenable to conventional modeling approaches due to the lack of precise. a great deal of research activity has focused on the development of methods to build or update fuzzy models from numerical data. as well as the internal structure of the fuzzy rules. not only from the scientific community but also from industry. aiming at . basic mathematics. The orientation of the book is towards methodologies that in the authors experience proved to be practically useful. The presentation reflects theoretical and practical issues in a balanced way. education and research. This book is intended to cover widely primarily the topics on neural computing. which provide little insight to help understand the underlying process. which can facilitate the effective development of models. Neurofuzzy models can be regarded as black-box models. and design procedures have been firmly established and applied for many years. Fuzzy modeling along with other related techniques such as neural networks have been recognized as powerful tools. Thus. Most approaches are based on neuro-fuzzy systems. This combination of fuzzy systems and neural networks enables a more effective use of optimization techniques for building fuzzy systems. due to strongly non-linear behaviour. Some of the material in this book contains timely material and thus may heavily change throughout the ages. The approach adopted in this book aims at the development of transparent rule-based fuzzy models which can accurately predict the quantities of interest.Preface This book attempts to provide the reader with basic concepts and engineering applications of Fuzzy Logic and Neural Networks. and neural memory. formal knowledge about the system. especially with regard to their approximation accuracy. or due to the time varying characteristics. and at the same time provide insight into the system that generated the data. The field of neural networks has a history of some five decades but has found solid application only in the past fifteen years. neural learning. Neural networks are useful for industry. it is distinctly different from the fields of control systems or optimization where the terminology. Modeling and control of dynamic systems belong to the fields in which fuzzy set techniques have received considerable attention. the absence of any state-of-the-art (Indian Domain) textbook forced me to write this book. Back in 2000. which exploit the functional similarity between fuzzy reasoning systems and neural networks. due to the high degree of uncertainty. The choice of describing engineering applications coincides with the Fuzzy Logic and Neural Network research interests of the readers. and the field is still developing rapidly.

Chennakesava R. Examples are given throughout the text and six selected real-world applications are presented in detail.viii PREFACE readership from the academic world and also from industrial practice. Alavala .

2 What is Fuzzy Logic? 2.6.6.2 Support 2.3 Historical Background 2.4 Applications Question Bank References LE 1-5 1 2 4 4 4 5 Part I: Chapter 2: Fuzzy Sets and Fuzzy Logic 2.3 Similarities and Dissimilarities Between FL and NN 1.5 Convex Fuzzy Set 2.8 Triangular Fuzzy Number 2.6.11 Equality of Fuzzy Sets 2.7 Quasi Fuzzy Number 2.5 Characteristics of Fuzzy Systems 2.6 Fuzzy Sets 2.6.1 Fuzzy Logic (FL) 1.Contents 2HAB=?A Chapter 1: Introduction 1.4 =-Cut 2.6.6.3 Normal Fuzzy Set 2.9 Trapezoidal Fuzzy Number 2.6.1 Introduction 2.1 Fuzzy Set 2.6 Fuzzy Number 2.10 Subsethood 2.2 Neural Networks (NN) 1.6.6.6.6.4 Characteristics of Fuzzy Logic 2.12 Empty Fuzzy Set Fuzzy Logic 6-18 6 6 8 9 9 9 9 11 11 11 11 12 12 13 14 14 15 15 .6.

6 Transitivity 3.6.1 Intersection 2.5 Anti-Symmetricity 3.3.13 Universal Fuzzy Set 2.3.2 The Linguistic Variable Truth Question Bank References 19-28 19 19 19 20 20 20 20 20 20 20 20 21 21 21 22 23 24 24 26 27 27 29-40 29 30 33 34 35 38 39 .1 Introduction 3.3 Operations on Fuzzy Relations 3.1 Classical N-Array Relation 3.2 Fuzzy Relations 3.1 Introduction 4.2.3.2.2.2.14 Fuzzy Point 2.3.8 Partial Order 3.3.4 Symmetricity 3.9 Total Order 3.3 Anti-Reflexivity 3.3.2 Fuzzy Implications 4.3 Modifiers 4.2 Union 2.1 Intersection 3.2.x CONTENTS 2.4 Cartesian Product of Two Fuzzy Sets 3.7.3.7 Operations on Fuzzy Sets 2.3 Complement Question Bank References 15 15 16 16 16 17 17 18 Chapter 3: Fuzzy Relations 3.2 Reflexivity 3.2.3.7.2.3 Projection 3.2 Union 3.7.10 Binary Fuzzy Relation 3.2.6 Sup-Min Composition of Fuzzy Relations Question Bank References Chapter 4: Fuzzy Implications 4.6.1 Linguistic Variables 4.2.7 Equivalence 3.5 Shadow of Fuzzy Relation 3.2.

3.6.1 Introduction 7. References.3 Rational Properties 5.2 Conjunction Rule 5.4 t-norm-based Intersection 6.9 Larsen System 6.2 Translation Rules 5.3.6 Compositional Rule of Inference 5.1 Entailment Rule 5.2 Total Indeterminance 5.1 Mamdani inference Mechanism 7. xi 41-53 41 43 43 43 43 44 44 44 45 45 46 46 47 50 51 Chapter 6: Fuzzy Rule-Based Systems 6.3.1 Introduction 6.3 Inference Mechanisms In Fuzzy Rule-base Systems 7.2 Triangular Norm 6.4 Larsen inference Mechanism 54-70 54 54 55 57 57 58 58 60 63 66 66 67 68 68 71-80 71 71 72 73 73 75 77 .2.2.3 Triangular Conorm 6.4 Superset Question Bank.2.2 Ordered Weighted Averaging 6.6.2.1 Basic Property 5.CONTENTS Chapter 5: The Theory of Approximate Reasoning 5.2 Tsukamoto inference Mechanism 7.5 Negation Rule 5.2.3.3.7 Measure of Dispersion or Entropy of an Owa Vector 6.3 Disjunction Rule 5.5 t-conorm-Based Union 6.3.1 Introduction 5.3.3 Subset 5.4 Projection Rule 5.8 Mamdani System 6.1 An Averaging Operator is a Function 6.2 Fuzzy Rule-base System 7.6 Averaging Operators 6.2.10 Defuzzification Question Bank References Chapter 7: Fuzzy Reasoning Schemes 7.3 Sugeno Inference Mechanism 7.3.

4.5.5 Conclusions 9.xii CONTENTS 7.4.3.5 Effectivity Of Fuzzy Logic Control Systems Question Bank References Chapter 9: Fuzzy Logic-Applications 9.3 When Not to use Fuzzy Logic? 9.3.5.4.3 Rule Application 9.4.7 Conclusions 9.4.1 Traffic Accidents and Traffic Safety 9.2 Fuzzy Logic Approach 9.2 First-of-Maxima 8.5 Rule Base 9.4.1 Why use Fuzzy Logic? 9.4.5.3 Application 9.6 Fuzzy Logic Model for Grading of Apples 9.3.3 Fuzzy Logic Control Systems 8.2 Fuzzification 9.1 The Mechanics of Fuzzy Logic 9.3.4 Defuzzification 9.2 Mamdani Type of Fuzzy Logic Control 8.2 Basic Feedback Control System 8.4.5 Fuzzy Logic Model to Control Room Temperature 9.4.4.4 Max-Criterion 8.3 Middle-of-Maxima 8.5 Height defuzzification 8.4 Fuzzy Logic Model for Prevention of Road Accidents 9.3 Fuzzy Logic Controller 8.5.5.1 Two-Input-Single-Output (TISO) Fuzzy Systems 8.4 Defuzzification Methods 8.4.4.6 Output 9.6.1 Center-of-Area/Gravity 8.1 Apple Defects Used in the Study 81-93 81 81 82 82 82 84 86 87 87 87 88 88 89 91 91 94-120 94 95 96 96 96 97 97 98 99 99 99 100 100 101 102 103 104 104 105 .2 Applications of Fuzzy Logic 9.5 Simplified Fuzzy Reasoning Question Bank References 77 79 79 Chapter 8: Fuzzy Logic Controllers 8.1 Introduction 8.4 Membership Functions 9.

1 Notation 10.1 Introduction 11.1 Paradigms of Learning 10.6.7.3 Some Observations Question Bank References xiii 105 106 108 109 110 111 112 112 112 116 117 118 118 Part II: Neural Networks Chapter 10: Neural Networks Fundamentals 10.5.6.2 Modifying Patterns of Connectivity 10.3.6.3 A Framework for Distributed Representation 10.2 Convergence Theorem 11.4 Fuzzy Rules 9.6.2 Networks with Threshold Activation Functions 11.4 Network Topologies 10.7.7 Results and Discussion 9.2 Biological Neural Network 10.5 Determination of Membership Functions 9.3.3 Perceptron Learning Rule and Convergence Theorem 11.1 Processing Units 10.3.2 Materials and Methods 9.4 Adaptive Linear Element (Adalime) 121-128 121 121 122 123 123 124 125 125 125 126 126 126 127 128 128 129-138 129 129 131 131 131 133 .6.6 Notation and Terminology 10.2 The Fuzzy Approach 9.CONTENTS 9.2 Connections between Units 10.1 Introduction 10.3.7.8 Conclusion 9.3 Activation and Output Rules 10.6.5.3.6 Defuzzification 9.5 Training of Artificial Neural Networks 10.7 An Introductory Example: Fuzzy v/s Non-fuzzy 9.3 Application of Fuzzy Logic 9.6.6.6.1 The Non-Fuzzy Approach 9.2 Terminology Question Bank References Chapter 11: Perceptron and Adaline 11.1 Perceptron Learning Rule 11.

8.2 Multi .1 The Jordan Network 13.1 Network Paralysis 12.3.2 The Generalised Delta .xiv CONTENTS 11.4 Boltzmann Machines Question Bank References 157-189 157 157 158 159 161 161 162 163 164 164 165 167 167 .3 Back-Propagation in Fully Recurrent Networks 13.6.4.6.4.3 Learning Per Pattern 12.Layer Feed .2.2 Local Minima 12.Rule In Recurrent Networks 13.3 The Hopfield Network 13.3 Neurons with graded response 13.1 Introduction 13.3.6 Deficiencies of Back-propagation 12.5 Other Activation Functions 12.2 Hopfield Network as Associative Memory 13.8.6 Exclusive-or Problem 11.2 The Effect of the Number of Hidden Units 12.8 How Good are Multi-layer Feed-forward Networks? 12.1 Introduction 12.3.2 The Elman Network 13.3 The Generalised Delta Rule 12.1 The Effect of the Number of Learning Samples 12.7 Advanced Algorithms 12.3.4 Working with Back-propagation 12.7 Multi-layer Perceptrons Can do Everything Question Bank References Chapter 12: Back-Propagation 12.2.1 Description 13.3.2 Learning Rate and Momentum 12.1 Understanding Back-Propagation 12.9 Applications Question Bank References 134 135 137 138 138 139-156 139 139 140 142 143 143 144 144 146 146 148 148 148 151 152 153 153 155 155 Chapter 13: Recurrent Networks 13.4.4 Hopfield networks for optimization problems 13.Forward Networks 12.2.5 The Delta Rule 11.1 Weight Adjustments with Sigmoid Activation Function 12.

2 Principal Component Extractor 14.1 Introduction 14.5 Normalization of the Original Model 14.2 The Critic 15.2.2 ART1: The Simplified Neural Network Model 14.6 Contrast enhancement Question Bank References Chapter 15: Reinforcement Learning 15.2 Adaptive Critic 15.3 Dynamics 16.1 Introduction 16.5.2.3 The Controller Network 15.3 More eigenvectors 14.4 ART 1: The Original Model 14.3 Operation 14.2 Inverse Kinematics 16.4 Learning Vector Quantisation 14.2.4.CONTENTS xv 169 169 170 170 174 174 176 177 179 180 181 181 182 182 183 184 185 186 187 188 188 Chapter 14: Self-Organising Networks 14.1 Introduction 15.5.1 Clustering 14.1 Forward Kinematics 16.2.4.2.4.5.4 Principal Component Networks 14.4.2 Robot Control 16.3 Counter propagation 14.2 Vector Quantisation 14.3 The Cart-Pole System 15.5.1 Normalized Hebbian Rule 14.1 Background: Adaptive Resonance Theory 14.4 Bartos Approach: The ASE-ACE Combination 15.2.2.5.2.2 Competitive Learning 14.4.3 Kohonen Network 14.5 Adaptive Resonance Theory 14.5.1 Associative Search 15.5 Reinforcement Learning Versus Optimal Control Question Bank References Chapter 16: Neural Networks Applications 16.4 Trajectory generation 190-198 190 190 191 192 193 194 194 195 197 197 199-215 199 200 200 200 201 201 .4.

2 Results and Discussion Question Bank References 201 202 202 203 206 207 210 211 213 215 215 Part III: Hybrid Fuzzy Neural Networks 217-232 217 217 217 218 218 219 220 221 222 223 223 224 226 228 229 231 232 232 233-252 233 233 234 235 237 Chapter 17: Hybrid Fuzzy Neural Networks 17.3.6b Approach 2: Topology conserving maps 16.2.10 Adaptive Neuro-fuzzy Inference System (ANFIS) 17.1 Sequential Hybrid Systems 17.10.4 Fuzzy Neurons 17.7 Robot Arm Dynamics 16.1 Unsupervised Adaptive Resonance Theory (ART) Neural Networks 16.2 Simulation 17.1 Introduction 18.2.2 Neural Networks 18.1 Strong L-R Representation of Fuzzy Numbers 17.2.2.8 Committee of Networks 17.3 Fuzzy Logic in Learning Algorithms 17.2.6 Neural Networks as Tuners of Fuzzy Logic Systems 17.2.9.2 Tool Breakage Monitoring System for end Milling 18.2.2.9.1 ANFIS Structure Question Bank References Chapter 18: Hybrid Fuzzy Neural Networks Applications 18.9 Fnn Architecture Based On Back Propagation 17.5 Neural Networks as Pre-processors or Post-processors 17.5 End-Effector Positioning 16.2.7 Advantages and Drawbacks of Neurofuzzy Systems 17.3.6 Camera-Robot Coordination in Function Approximation 16.xvi CONTENTS 16.2 Hybrid Systems 17.2.6a Approach-1: Feed-forward Networks 16.5a Involvement of neural networks 16.2.2.2 Auxiliary Hybrid Systems 17.1 Introduction 17.1 Methodology: Force signals in the end milling cutting process 18.3 Embedded Hybrid Systems 17.3 Experimental Design and System Development Experimental Design .3 Detection of Tool Breakage in Milling Operations 16.

5 Findings and Conclusions 18.3.4 Optimization of the PI-Controllers using Genetic Algorithms Question Bank References Index xvii 238 241 243 243 245 246 247 251 251 253 .3.3.3.1 Adaptive neuro-fuzzy inference system 18.3 Control of Combustion 18.2 Learning Method of ANFIS 18.3 Model of Combustion 18.2.2.4 Neural Network-BP System Development 18.CONTENTS 18.

This page intentionally left blank .

This approach to set theory was not applied to control systems until the 70s due to insufficient small-computer capability prior to that time. manufacturers have not been so quick to embrace this technology while the Europeans and Japanese have been aggressively building real products around it. and presented not as a control methodology. U. approximate reasoning and computing with words. Unfortunately. Professor Zadeh reasoned that people do not require precise. imprecise input. Zadeh. have ability to deal with non-linearities. utilize self-learning.S. Notions like rather tall or very fast can be formulated mathematically and processed by computers. and yet they are capable of highly adaptive control. high/low. follow more human-like reasoning paths than classical methods. they would be much more effective and perhaps easier to implement. but as a way of processing data by allowing partial set membership rather than crisp set membership or non-membership. neural networks have rooted in many application areas (expert systems. . Fuzzy logic is mainly associated to imprecision. they have many common features . Although these methodologies seem to be different. pattern recognition. If feedback controllers could be programmed to accept noisy.+ 0 ) 2 6 . etc. Basically. numerical information input. and neural networks to learning and curve fitting (also to classification).).1 FUZZY LOGIC (FL) The concept of Fuzzy Logic was conceived by Lotfi A. etc. 1 1. a professor at the University of California at Berkley. These methods have in common that they are non-linear. in order to apply a more human-like way of thinking in the programming of computers. fuzzy logic. FL is a multivalued logic that allows intermediate values to be defined between conventional evaluations like true/false. yes/no.4 Introduction Nowadays. system control.like the use of basis functions (fuzzy logic has membership functions and neural networks have activation functions) and the aim to estimate functions from sample data or heuristics.

These types of networks were called perceptrons. The precision of mathematics owes its success in large part to the efforts of Aristotle and the philosophers who preceded him. see Fig. drives smoother) when compared with a conventional PID controller.) there were strong and immediate objections: for example. for example washing machine controllers and autofocus cameras. Rockwell. For example. and later mathematics. But it was Lukasiewicz who first proposed a systematic alternative to the bi-valued logic of Aristotle. It was Plato who laid the foundation for what would become fuzzy logic.2 NEURAL NETWORKS (NN) The study of neural networks started by the publication of Mc Culloch and Pitts [1943]. complex aircraft engines and control surfaces. industrial processes and so on. Even when Parminedes proposed the first version of this law (around 400 B. Mitsubishi. The most famous controller is the subway train controller in Sengai. Fuzzy control includes fans. states that every proposition must either be True or False. Commercially most significant have been various household and entertainment electronics. which did not seem to be more difficult could not be solved.1). Fuji. Japan. + – Input Fuzzy Controller Control Plant to be controlled Output Fig. Nissan. missile guidance. The singlelayer networks. 1. One of these. the Law of the Excluded Middle. Samsung. If the conventional techniques of system analysis cannot be successfully incorporated to the modeling or control problem.2 FUZZY LOGIC AND NEURAL NETWORKS Fuzzy systems is an alternative to traditional notions of set membership and logic that has its origins in ancient Greek philosophy. etc. indicating that there was a third region (beyond True and False) where these opposites tumbled about.C. Fuzzy system performs better (uses less fuel. Heraclitus proposed that things could be simultaneously True and not True. then turn the wheel slightly left. wheel slip control. the use of heuristic linguistic rules may be the most reasonable solution to the problem. with threshold activation functions. 1. automatic transmission. It has been made a rough guess that 90% of applications are in control (the main part deals with rather simple applications. McDonnell Douglas. These limitations of one-layer perceptron were mathematically shown by Minsky and Papert in their book Perceptron [1969]. in which the truck must be guided from an arbitrary initial position to a desired final position. In the 1960s it was experimentally shown that perceptrons could solve many problems. Humans and fuzzy systems can perform this nonlinear control task with relative ease by using practical and at the same time imprecise rules as If the trailer turns slightly left.1 Example of a control problem. Companies that have fuzzy research are General Electric. helicopter control. were introduced by Rosenblatt [1959]. The most significant application area of FL has been in control field. Omron. Sharp. Siemens. Honda. In their efforts to devise a concise theory of logic. 1. Canon. The result of this publication was that the neural networks lost their . the so-called Laws of Thought were posited. Hitachi. but many problems. there is no mathematical model for truck and trailer reversing problem.

which changes the weights (or the parameters of activation functions) in such a way that the network will reproduce a correct output with the correct input values. In the mid-1980s.2 Simple illustration of biological and artificial neuron (perceptron). adaptivity and fault tolerance. and Williams [1986]. and the modification of connections. The learning may be based on two mechanisms: the creation of new connections. Each neuron has an activation level which. which are connected to each other with transmission lines called axons and receptive lines called dendrites (see Fig. Neural networks are often used to enhance and optimize fuzzy logic based systems. The simulating is based on the present knowledge of brain function. 1.2). in contrast to Boolean logic. input-output mapping. Neural networks offer nonlinearity. Nonlinearity is a desired property if the generator of input signal is inherently nonlinear.. The operation of brain is believed to be based on simple basic elements called neurons. This learning ability is achieved by presenting a training set of different examples to the network and using learning algorithm. The high connectivity of the network ensures that the influence of errors in a few terms will be minor.INTRODUCTION 3 interestingness for almost two decades. which revived the study of neural networks. In artificial neural networks the inputs of the neuron are combined in a linear way with different weights. The significance of this new algorithm was that multiplayer networks could be trained by using it. and this knowledge is even at its best primitive.2). The result of this combination is then fed into a non-linear activation unit (activation function). 1. back-propagation algorithm was reported by Rumelhart. So. ranges between some minimum and maximum value. 1. . The difficulty is how to guarantee generalization and to determine when the network is sufficiently trained. which ideally gives a high fault tolerance. which can in its simplest form be a threshold unit (See Fig. Synapse Nucleus Axon dendrites 1 X1 X2 W0 W1 W2 S Summing threshold unit z Fig. by giving them a learning ability. Hinton. it is not absolutely wrong to claim that artificial neural networks probably have no close relationship to operation of human brains.g. e. NN makes an attempt to simulate human brain.

A very interesting combination is the neuro-fuzzy architecture. good results have been achieved by combining both the methods. computer networks. 3. business forecasting. What is the ancient philosophy of fuzzy logic? What are the various applications of fuzzy logic? What is the historical evolution of neural networks? What are the similarities and dissimilarities between fuzzy logic and neural networks? What are the various applications of neural networks? . credit rating. 2. speech processing. fault diagnostics and information security. The number of this kind of hybrid systems is growing. database management. Possible new application areas are programming languages. robotics control. automaticized programming. Fuzzy logic may also be employed to improve the performance of optimization methods used with neural networks. user-friendly application interfaces.3 SIMILARITIES AND DISSIMILARITIES BETWEEN FL AND NN There are similarities between fuzzy logic and neural networks: estimate functions from sample data do not require mathematical model are dynamic systems can be expressed as a graph which is made up of nodes and edges convert numerical inputs to numerical outputs process inexact information inexactly have the same state space produce bounded signals a set of n neurons defines n-dimensional fuzzy sets learn some unknown probability function can act as associative memories can model any system provided the number of nodes is sufficient. pattern recognition. The main dissimilarity between fuzzy logic and neural network is that FL uses heuristic knowledge to form rules and tunes these rules using sample data. 1. Most neuro-fuzzy systems are fuzzy rule based systems in which techniques of neural networks are used for rule induction and calibration. adaptive process control. In many cases. etc. QUESTION BANK. 4. 1. 5. in which the good properties of both methods are attempted to bring together. quality assurance and industrial inspection.4 FUZZY LOGIC AND NEURAL NETWORKS 1. natural-language understanding.4 APPLICATIONS Applications can be found in signal processing. whereas NN forms rules based entirely on data.

A logical calculus of the ideas immanent in nervous activity. pp. 1969. NY: 1967. Jan Lukasiewicz. C. Loses Focus on Fuzzy Logic.417. Pitts. Hinton and R. 115-133. Vol.E.. 414. 1991. Papert. IEEE Software.E. 20. 4. Perceptrons: An Introduction to Computational Geometry. Vol. S. Smith.A. pp. Why the Japanese are going in for this fuzzy logic. Learning representations by backpropagating errors. 1965. L. pp.S. pp. Lejewski. 1986. Information and Control. 1990. New York: Spartan Books. Zadeh. Encyclopedia of Philosophy. 11. 48-56. Feb. pp. L. 5. IEEE. Machine Design. L. Encyclopedia of Philosophy. 8. Zadeh. 2. 1984. 104-107. 6.J. McCulloch and W. . Spectrum. 11. 338-353. F. NY: 1967. Rosenblatt. M. 1943. Business Week. 7. Williams.A. Vol. 9. 5. 39. Vol. MacMillan. 4. Rumelhart. 1993. 1959. Korner. Vol. Minsky and S. T. 12. pp. 26-32. Europe Gets into Fuzzy Logic. Principles of Neurodynamics. Making computers think like people. The MTT Press. Bulletin of Mathematical Biophysics. 5. 323. U. MacMillan. Fuzzy Sets. G. Soft computing and fuzzy logic. 3. 1994.INTRODUCTION 5 REFERENCES.S. W. pp. Electronics Engineering Times. (November). 10. Vol. Zadeh. 533-536. Laws of thought. D. 1. Nature.A. June 21. 8. pp. Vol. 8. Nov. 23-26.

2 2. such as thinking and reasoning. Henri Matisse Sometimes the more measurable drives out the most important.2 WHAT IS FUZZY LOGIC? Fuzzy logic is all about the relative importance of precision: How important is it to be exactly right when a rough answer will do? All books on fuzzy logic begin with a few good quotes on this very topic. Rene Dubos Vagueness is no more to be done away with in the world of logic than friction in mechanics. As a consequence. and this is no exception. since such knowledge is by its nature both lexically imprecise and noncategorical. the approaches based on first order logic and classical probability theory do not provide an appropriate conceptual framework for dealing with the representation of commonsense knowledge.A Zadeh in 1965 to represent/manipulate data and information possessing nonstatistical uncertainties. Fuzzy logic provides an inference morphology that enables approximate human reasoning capabilities to be applied to knowledge-based systems. It was specifically designed to mathematically represent uncertainty and vagueness and to provide formalized tools for dealing with the imprecision intrinsic to many problems.+ 0 ) 2 6 . The development of fuzzy logic was motivated in large measure by the need for a conceptual frame work which can address the issue of uncertainty and lexical imprecision. Here is what some clever people have said in the past: Precision is not truth. The conventional approaches to knowledge representation lack the means for representing the meaning of fuzzy concepts. Charles Sanders Peirce . 1 INTRODUCTION Fuzzy sets were introduced by L. The theory of fuzzy logic provides a mathematical strength to capture the uncertainties associated with human cognitive processes.4 Fuzzy Sets and Fuzzy Logic 2.

L. but once you become acquainted with it. and Ill shift the gears for you. You tell me how hot you want the water. Mencken So far as the laws of mathematics refer to reality. You tell me how far away the subject of your photograph is. H. Fuzzy logic is a fascinating area of research because it does a good job of trading off between significance and precision . precise statements lose meaning and meaningful statements lose precision. they are not certain. And so far as they are certain. they do not refer to reality.FUZZY SETS AND FUZZY LOGIC 7 I believe that nothing is unconditionally true. although the modern and methodical science of fuzzy logic is still young. In this sense fuzzy logic is both old and new because. and Ill tell you what the tip should be. LOOK OUT!! Precision Significance Fig. L. This is the starting point for everything else. What do I mean by mapping input space to output space? Here are a few examples: You tell me how good your service was at a restaurant. and hence I am opposed to every statement of positive truth and every man who makes it. Precision and significance in the real world A 1500 kg mass is approaching your head at 45. and Ill adjust the faucet valve to the right setting. You tell me how fast the car is going and how hard the motor is working. 2. Albert Einstein As complexity rises. 2. the concepts of fuzzy logic reach right down to our bones.1 Precision and significance. A Zadeh Some pearls of folk wisdom also echo these thoughts: Dont lose sight of the forest for the trees. and Ill focus the lens for you. and the great emphasis here is on the word convenient. .something that humans have been managing for a very long time (Fig. Dont be penny wise and pound foolish. Fuzzy logic sometimes appears exotic or intimidating to those unfamiliar with it. Fuzzy logic is a convenient way to map an input space to an output space.1). it seems almost surprising that no one attempted it sooner.3 m/sec.

fuzzy mathematical programming. in a lattice. Soft computing came into existence in 1981. alongside the boom in fuzzy logic applications. Other combinations. in general. with the launching of BISC (Berkeley Initiative in Soft Computing) at UC Berkeley. among them the genesis of possibility theory and possibilistic logic. for the most part. An important development in the evolution of fuzzy logic. and (b) the concept of a fuzzy if-then rule. Today. fuzzy arithmetic. or is allowed to be. The principal members of the coalition are: fuzzy logic. evolutionary computing. is a union of FLn. that is. is a logical system which is a generalization of multivalued logic. the term fuzzy logic is used. more generally. A key component of computing with words is the concept of Precisiated Natural Language (PNL). and (b) a wide sense. Basically. more generally. chaotic computing. rough set theory and machine learning. fuzzy logic is used in two different senses: (a) a narrow sense. The basic issues and applications which were addressed were. e. as they are at present. 1973-1999. neurocomputing. fuzzy mathematics. and the impact of soft computing is growing on both theoretical and applied levels. better results can be obtained through the use of constituent methodologies of soft computing in combination rather than in a stand-alone mode. especially in Japan. Today. calculus of fuzzy quantifiers and related concepts and calculi. three principal phases may be discerned. fuzzy topology and. neuro-fuzzy-genetic systems. design and utilization of intelligent systems. There were many other major developments in fuzzy-logic-related basic and applied theories. probabilistic computing. in its wide sense.8 FUZZY LOGIC AND NEURAL NETWORKS 2. Where do we stand today? In viewing the evolution of fuzzy logic. The term fuzzy logic was used for the first time in 1974. decision analysis. fuzzy set theory. The first phase. The distinguishing characteristic of FL is that in FL everything is. two key concepts were introduced in this paper: (a) the concept of a linguistic variable. almost all applications of fuzzy set theory and fuzzy logic involve the use of these concepts. abbreviated as FLn. was concerned in the main with fuzzification. pattern recognition.g. set-theoretic in nature. development of computing with words and perceptions brings together earlier strands of fuzzy logic and suggests that scientific theories should be based on fuzzy logic rather than on Aristotelian. from 1965 to 1973. Today. and logic and reasoning were not at the center of the stage.. 1996 is the genesis of computing with words and the computational theory of perceptions. fuzzy arithmetic. soft computing is a coalition of methodologies which collectively provide a foundation for conception. PNL opens the door to a major enlargement of the role . with generalization of the concept of a set. knowledge representation. Basically. with two-valued characteristic function generalized to a membership function taking values in the unit interval or.3 HISTORICAL BACKGROUND Almost forty years have passed since the publication of first paper on fuzzy sets. for the most part. bivalent logic. a matter of degree. A combination which has attained wide visibility and importance is that of neuro-fuzzy systems. abbreviated as FL. cluster analysis. calculus of fuzzy if-then rules. possibility theory. are appearing. in which fuzzy logic. Fuzzy control applications proliferated but their dominance in the literature became less pronounced. marking the beginning of the third phase. Perhaps the most striking development during the second phase of the evolution was the naissance and rapid growth of fuzzy control. in which fuzzy logic. The second phase. The basic tenet of soft computing is that.

1] . It may well turn out to be the case that..FUZZY SETS AND FUZZY LOGIC 9 of natural languages in scientific theories. Any logical system can be fuzzified. But. In fuzzy logic. in English. for some fuzzy logic is hard to accept because by abandoning bivalence it breaks with centuries-old tradition of basing scientific theories on bivalent logic.6 FUZZY SETS 2. everything is a matter of degree. 2. A fuzzy set A in X is characterized by its membership function (Fig. It may take some time for this to happen.6.1) Let X be a nonempty set. fuzzy constraint on a collection of variables.5 CHARACTERISTICS OF FUZZY SYSTEMS There are two main characteristics of fuzzy systems that give them better performance for specific applications: Fuzzy systems are suitable for uncertain or approximate reasoning.. in coming years. knowledge is interpreted a collection of elastic or. centering on the conception and design of search engines and question-answering systems. exact reasoning is viewed as a limiting case of approximate reasoning. will be the Internet. and µA (x) is interpreted as the degree of membership of element x in fuzzy set A for each x Î X. 2..(2. Fuzzy logic allows decision making with estimated values under incomplete or uncertain information. the word fuzzy is usually used in a pejorative sense. 2.(2. In fuzzy logic. 2. Inference is viewed as a process of propagation of elastic constraints. From its inception.2). and especially PNL. fuzzy logic has been (and to some degree still is) an object of skepticism and controversy. more importantly. especially for the system with a mathematical model that is difficult to derive. one of the most important application-areas of fuzzy logic.2) . equivalently.. skepticism about fuzzy logic is a reflection of the fact that.4 CHARACTERISTICS OF FUZZY LOGIC Some of the essential characteristics of fuzzy logic relate to the following: In fuzzy logic. but eventually abandonment of bivalence will be viewed as a logical development in the evolution of science and human thought. It is clear that A is completely determined by the set of tuples A = {(u. µA(u)) |u Î X} . In part.1 Fuzzy Set µA : X ® [0.

(2. Example 2. 2. 1 –1 1 2 3 4 Fig. 2. and depends on his purse (Fig. The family of all fuzzy sets in X is denoted by F(X).. .2 A discrete membership function for x is close to 1. For instance.. If X = {x1.. 450000 Rs. Frequently we will write A(x) instead of µA(x).. Cheap can be represented as a fuzzy set on a universe of prices. can be A(t) = exp ( b(t 1)2) where b is a positive real number. i =1. 300000 Rs.3) where the term µi / xi..4 Membership function of "cheap". 2... 300000 cars are considered as cheap.4). 2.3 A membership function for x is close to 1.1: defined as The membership function (Fig. 2.2: Assume someone wants to buy a cheap car. .n signifies that µi is the grade of membership of xi in A and the plus sign represents the union. . 600000 Fig. from the Figure cheap is roughly interpreted as follows: Below Rs. + µn/xn . and prices make no real difference to buyers eyes. xn} is a finite set and A is a fuzzy set in X then we often use the notation A = µ1/x1 + . 1 Rs...10 FUZZY LOGIC AND NEURAL NETWORKS 1 –2 –1 0 1 2 3 4 Fig. Example 2.3) of the fuzzy set of real numbers "close to 1".. ..

2. 1. .3: Assume X = { 2.6 0. Beyond Rs. 4} and A = 0.3/3 + 0. In this case [A] a R{-1. 600000 the costs are too high (out of consideration)..3 Normal Fuzzy Set A fuzzy subset A of a classical set X is called normal if there exists an xÎX such that A(x) =1. 1. These are examples of what are called fuzzy numbers. 0. about 5000.5) 2.0/4.FUZZY SETS AND FUZZY LOGIC 11 Between Rs. 3. Between Rs. is the crisp subset of X whose elements all have nonzero membership grades in A. near zero. denoted supp (A). An a-cut of a triangular fuzzy number is shown in Fig.6. a variation in the price induces a weak preference in favor of the cheapest car.(2. Example 2.3 0.. 1]. or essentially bigger than 5000.5. a small variation in the price induces a clear preference in favor of the cheapest car.6/2 + 0..0/1 + 0.3 < a £ 0. 600000. Otherwise A is subnormal. 450000. 450000and Rs. 3} | = S{0. 2. supp(A) = {x ÎX | A(x) > 0}.(2. 0.6. Using the theory of fuzzy subsets we can represent these fuzzy numbers as fuzzy subsets of the set of real numbers. More exactly. the support of A. 2.. 2} |{1} T if if if 0 £ a £ 0. 1. 1.5 Convex Fuzzy Set A fuzzy set A of X is called convex if [A]a is a convex subset of X"a Î [0.4 =-Cut An a-level set of a fuzzy set A of X is a non-fuzzy set denoted by [A]a and is defined by [A]a = R{t e X | A(t ) ³ a} Scl(sup p A) T if if a>0 a=0 .2 Support Let A be a fuzzy subset of X.6. In many situations people are only able to characterize numeric information imprecisely. 2.3/ 1 + 0. . 2.6) where cl (supp A) denotes the closure of the support of A. people use terms such as.6/0 + 1. 300000 and Rs.6 < a £ 1 2.6. For example.0/ 2 + 0.

2.(2..6 Fuzzy Number A fuzzy number (Fig. fuzzy convex and continuous membership function satisfying the limit conditions 1® ¥ lim A(t) = 0 .cut. 1] ® Â is monotone decreasing and upper semicontinuous.. a2(g) max [A]g .12 FUZZY LOGIC AND NEURAL NETWORKS a a – cut Fig. a1 (g) denotes the left-hand side and a2 (g) denotes the right-hand side of the g .11) ..(2.. The family of fuzzy numbers will be denoted by F .1]. Let us introduce the notations a1(g) = min [A]g.(2.6. ..(2. 2.6. It is easy to see that if a £ b then [A]a É [A]b Furthermore. 2.6 Fuzzy number. 2. 2. and the right-hand side function a2 : [0.6) A is a fuzzy set of the real line with a normal.. 1] ® Â is monotone increasing and lower semicontinuous.5 An a-cut of a triangular fuzzy number.10) .8) In other words.. the left-hand side function a1 : [0...(2.7 Quasi Fuzzy Number A quasi-fuzzy number A is a fuzzy set of the real line with a normal.7) Let Abe a fuzzy number.9) . 1 –2 –1 1 2 3 Fig.. (fuzzy) convex and continuous membership function of bounded support. Then [A]g is a closed convex (compact) subset of Â for all g Î [0.

1] . a. 2. a + (1 g)b]. . a2(0)]. The not fuzzy number is shown in Fig. 2. 2.a . b + b).. 1] such that [A]g is not a convex subset of R. 2.14) The support of A is (a a.. 1 –3 –2 –1 1 2 3 Fig.7.8.(2... It can easily be verified that [A]g = [a (1 g)a.(2.(2. If A is not a fuzzy number then there exists an gÎ[0.. a2(g)] The support of A is the open interval [a1 (0). A 1 g a1(g) a2(g) .9) with center a may be seen as a fuzzy quantity x is approximately equal to a. A triangular fuzzy number (Fig.7 The support of A is [a1(0).a . 2. b).FUZZY SETS AND FUZZY LOGIC 13 We shall use the notation [A]g = [a1(g).a £ t £ a if a £ t £ a + b otherwise . 2.12) a1(0) a2(0) Fig.8 Not fuzzy number. left width a > 0 and right width b > 0 if its membership function has the following form A(t) = R1 . a2 (0)] and it is illustrated in Fig.6.13) and we use the notation A = (a.. "g Î[0.t S b |0 | | T if a .8 Triangular Fuzzy Number A fuzzy set A is called triangular fuzzy number with peak (or center) a.t | a |1 .

14 FUZZY LOGIC AND NEURAL NETWORKS 1 a–a a a+b Fig. "t Î X.10) may be seen as a fuzzy quantity x is approximately in the interval [a. 2.(2. b. b]. R1 .6. .. b + (1 g)b]. The subsethood is illustrated in Fig.(2. 2. 1 a–a a b b+b Fig.9 Trapezoidal Fuzzy Number A fuzzy set A is called trapezoidal fuzzy number with tolerance interval [a. 2. b + b).16) The support of is (a a.10 Trapezoidal fuzzy number. left width and right width b if its membership function has the following form.10 Subsethood Let A and B are fuzzy subsets of a classical set X.b |1 b |0 | T and we use the notation if a a £ t £ a if a £ t £ b if a £ t £ b + b otherwise . We say that A is a subset of B if A(t) £ B(t). "g Î[0.t | a |1 | A(t) = S t .. a.6. 2. b].. A trapezoidal fuzzy number (Fig.15) A = (a. 1] . It can easily be shown that [A]g = [a (1 g)a.. 2. 2. b).11.a .9 Triangular fuzzy number.

13 Universal Fuzzy Set The largest fuzzy set in X. 1 X0 X0 Fig. 2. .FUZZY SETS AND FUZZY LOGIC 15 B A Fig. denoted by 1X. If supp (A) = {x0}. 2. "t Î X. A and B are said to be equal.6. 2. called universal fuzzy set (Fig. if A Ì B and B Ì A.11 A is a subset of B.14 Fuzzy Point Let A be a fuzzy number. 2. 2.13 Fuzzy point. is defined by 1X(t) = 1. We note that A = B if and only if A(x) = B(x) for x Î X. 2.13) and we use the notation A = x0.12 The graph of the universal fuzzy subset in X = [0.6. 10]. 2.6. 1 1x 10 x Fig. It is easy to see that Ø Ì A holds for any fuzzy subset A of X.12) in X. denoted A = B.12 Empty Fuzzy Set The empty fuzzy subset of X is defined as the fuzzy subset Ø of X such that Ø(x) = 0 for each x Î X. It is easy to see that A Ì 1X holds for any fuzzy subset A of X. 2. then A is called a fuzzy point (Fig. 2.11 Equality of Fuzzy Sets Let A and B are fuzzy subsets of a classical set X.6.

For this reason..14.15.7.(2.. We note that all those operations which are extensions of crisp concepts reduce to their usual meaning when the fuzzy subsets have membership degrees that are drawn from {0. B(t)} = A(t) Ù B(t) for all t Î X .(2. 2.1 Intersection (A Ç B)(t) = min {A(t).(2. 2. A B . The union of A and B is defined as (A È B) (t) = max {A(t). when extending operations to fuzzy sets we use the same symbol as in set theory.7 OPERATIONS ON FUZZY SETS We extend the classical set theoretic operations from ordinary set theory to fuzzy sets.15 Union of two triangular fuzzy numbers. 1}. Let A and B are fuzzy subsets of a nonempty (crisp) set X. 2. It is easy to see that [A]g = [x0.7. B(t)} = A(t) Ú B(t) for all t Î X The union of two triangular numbers is shown in Fig.. 2. . x0] = {x0}. 1] .14 Intersection of two triangular fuzzy numbers.17) 2.. "g Î [0.18) The intersection of A and B is defined as The intersection of A and B is shown in Fig.16 FUZZY LOGIC AND NEURAL NETWORKS Let A = x0 be a fuzzy point.2 Union A B Fig..19) Fig. 2. 2..

"t Î R. List all a-cuts. however. Given A ={a. then it is easy to see that (ØA Ú A)(t) = mix {ØA(t). 3. Given X = {1.FUZZY SETS AND FUZZY LOGIC 17 2. Let A be a fuzzy set defined by A = 0. 1/2} = 1/2 ¹ 1 Lemma2. Find A È B. 5.. 4. and A Ç B.7.5/x1 + 0. 12. 2. 8.7/x3 + 0. c}.22) It is clear that Ø1X = f and Øf = 1X. 1/2} = 1/2 ¹ 0 However. 2. the laws of excluded middle and noncontradiction are not satisfied in fuzzy logic. What are the characteristics of fuzzy logic? What are the characteristics of fuzzy systems? What are the different fuzzy sets? Define them. 3. b.} and A = {2. 2. 3..2: The law of non-contradiction is not valid.. .3 Complement The complement of a fuzzy set A is defined as (ØA)(t) = 1 A(t) A Ú ØA = X and the law of non-contradiction principle A Ù ØA = f . 6}. A(t)} = max {1 1/2. b. 11. 9. Ø(A Ú B = ØA Ù ØB) . 1. 6. c. (iii) universal fuzzy set. 4. 4. What are the operations on fuzzy sets? Explain with examples. 7. 2} and B = {1.(2. A(t)} = mix {1 1/2. 10. then it is easy to see that (ØA Ú A)(t) = max {ØA(t). (ii) empty fuzzy set. What are the roles of a-cut in fuzzy set theory? What are the different fuzzy numbers? Define them.21) A closely related pair of properties which hold in ordinary set theory are the law of excluded middle QUESTION BANK.. "t Î R.(2. 1.8/x4 + 1/x5. Define the following: (i) equality of fuzzy sets.4/x2 + 0.1: The law of excluded middle is not valid.. Let A(t) =1/2. 5.. fuzzy logic does satisfy De Morgans laws Ø(A Ù B) = ØA Ú ØB. Let A(t) =1/2. 6. What is fuzzy logic? Explain the evolution phases of fuzzy logic. Lemma2.(2.20) . Find ØA.

A. 8. Zadeh. Journal of Mathematics Analysis and Applications.A. . on systems. Zadeh. Zadeh. Prade. Zadeh. L. Gottwald. 1972. SMC-3. Possibility theory and soft data analysis. 40. 1965. Vol. Man and Cybernetics. Vol. 1996. 5. 94-102. No. Vol. Computational Intelligence: Soft Computing and Fuzzy-Neuro Integration with Applications. G. 613-626.A. neutral networks and soft computing. Termini. No. Part 1. 40. 16. Foundations of fuzzy sets. 1991. pp. 8. 14.A.A. Information Sciences. P. Zadeh. Zadeh. Boulder. L. 2. pp. edited by O. Information and Control. A generalized fuzzy set theory. L. No. Vol. J. A note on fuzzy sets. 12. Information and Control. Set theory for fuzzy sets of higher level. 18. 4-34.10-37. IEEE Computer. (November). 1968. 2. L. A fuzzy-set-theoretic interpretation of linguistic hedges. Fuzzy Sets and Systems. 203-230. pp. Vol. pp. Fuzzy Sets and Systems.4. 1988. The concept of a linguistic variable and its application to approximate reasoning. L. Vol. Fuzzy algorithms. man and cybernetics. 18. 20. S. 48-56. IEEE Transactions. Fuzzy logic. Watanable. 12. S. pp. International Journal of Systems Science. Information Sciences.257-296. Westview Press. pp. 1981. Vol. Stout. 1. Zadeh. No. The concept of a linguistic variable and its application to approximate reasoning.18 FUZZY LOGIC AND NEURAL NETWORKS REFERENCES. IEEE Transactions on systems. Vol. 11. Vol. 1. No.A. pp.N. 77-84. L. 10. 6. Zadeh. No.A.A. L.A. 125-151. Information and Control. DeLuca. IEEE Software. Vol. D. 8. 10. No. Albert. design and deployment of information/intelligent systems.A. 8. 2. Rudas. Vol. pp. Zadeh. 7. 1. 1998. The algebra of fuzzy logic. 4. 373-386.A. L. 1974. 1978. 26-32. pp. Berlin. 9. L. 338-353. 756-759. Vol. Roles of soft computing and fuzzy logic in the conception. Vol. No. Vol. Mathematics Frontiers of the Social and Policy Sciences. Algebraic properties of fuzzy sets. Springer-Verlag. No. Brown. 103-111. Vol.8. Zadeh. 3.A. Journal of Cybernetics.A. B. Throll (eds. 3. 32-39. Zadeh.Cobb and R. 3.). L. 4. 1. pp. 2. pp. 13. 2. 8. 301-357. pp. Vol. and I. pp. 1975. pp. Zadeh. 2. pp. Part 2. 1994. Communications of the ACM. 1979. Fuzzy Sets. 2. 37. Kaynak. 28-44.A. 9.M. L. Fuzzy logic. IEEE Spectrum. L. 1978. 1984. J. Hohle and L. pp. 1971. pp. Fuzzy Sets and Systems. Zadeh. 1973. and S. 1994. Dubois. Vol. 21. pp. A. 1978. 1972. 15. pp. No. L. L. and H. L. 199-249. 17. U. 83-93. No. Soft computing and fuzzy logic. Outline of a new approach to the analysis of complex systems and decision process. 19. Making computers think like people. 2. IEEE Transactions on Fuzzy Systems. Vol. Fuzzy logiccomputing with words. Turksen. Zadeh. 69-129. Operations on Fuzzy Numbers.

v)..2) if (u. If X1 = ... v) = Example 3. c] . an example of a ternary tuple is (u... Xn be classical sets. then R is called an n-ary relation in X. v)ÎR Û uÎ[a... (John.(3. for example {(Charles. Eva}. v)ÎR otherwise . James} and Y the domain of women {Diana. The subsets of the Cartesian product X1 x x Xn are called n-ary relations. b] and XR(u. Xn). Rita)} 3.2 R1 S0 T R1 S0 T if (u. .4 Fuzzy Relations 3. where a tuple is an ordered pair.. A binary tuple is denoted by (u. v. v) Î[a . Example 3.+ 0 ) 2 6 .(3. (James. then the relation married to on X ´ Y is. Diana). . c] otherwise . Charles.1 INTRODUCTION 3 A classical relation can be considered as a set of tuples. Then the characteristic function of R is defined as XR(u..1 Classical N-Array Relation Let X1.. b]x[0. v) = vÎ[0. Eva).2 FUZZY RELATIONS 3.1: Let X be the domain of man {John.2.. w) and an example of n-ary tuple is (X1.1) Consider the following relation (u.. Rita. = Xn and R Ì X n. Let R be a binary relation in Â.

.

10 Binary Fuzzy Relation Let X and Y be nonempty sets. 2) = R(2. Example 3.3) The intersection of R and S is defined by Note that R: X ´ Y ® [0. 1) = R(2.3 The membership function of R is given by R1 | R(u.FUZZY RELATIONS 21 Consider the relation mod 3 on natural numbers {(m. R the domain of R is the whole Cartesian product X ´ Y.3 0.v| = 2 LM MM1 MN2 3 3. Then R (u.8 1 0. 3. 2. v). n) | (n m) mod 3 º 0} This is an equivalence relation. 2) = R(3.3 1 1 2 3 0.(3.2. . Let R and S be two binary fuzzy relations on X ´ Y.3 0.3.8P P 1 Q OPERATIONS ON FUZZY RELATIONS Fuzzy relations are very important because they can describe interactions between variables.3 T In matrix notation it can be represented as if u = v if |u . 1) = R(2.8 0. 3) = R(3. 1]. called approximately equal can be defined as R(1. 3) = 1 R(1.8 OP P 0. 3. If X = Y then we say that R is a binary fuzzy relation in X. Let R be a binary fuzzy relation on R.1 Intersection (R Ù S) (u.e.8 |0. v) = S0. v) is interpreted as the degree of membership of the ordered pair (u. i. R Î F (X ´ Y). 3}. A fuzzy relation R is a fuzzy subset of X ´ Y.. v) = min {R(u. v) in R. In other words.v| = 1 if |u . S(u. 1) = 0.5: A simple example of a binary fuzzy relation on U = {1. v)} .8 R(1.. 2) = 0. 3) = R(3.

8 0..7 0. c].4 0.4 0 0 0. y) = M MM x Nx Consider a classical relation R on Â..7 The union of R and S means that x is considerable larger than y or x is very close to y. S(u.7 0.8 01 .b] x [0.8Q OP PP P 0.5 0. 2 3 0 0.(3. 1 2 3 y1 y2 y3 y4 0.3 0 0.9 0.8 0 0. c] otherwise It is clear that the projection (or shadow) of R on the X-axis is the closed interval [a.9 1 0.4 0 01 0. 0 0. LM x (R Ù S) (x.8 0.4 0 0.3 0 0..22 FUZZY LOGIC AND NEURAL NETWORKS 3.7 0.9 0.9 1 0. v ) Î[a .8 0 0.7 .6: Let us define two binary relations R = x is considerable larger than y . v).7 OP P 0P P 0. .8 The intersection of R and S means that x is considerable larger than y and x is very close to y. v) = min{R(u. v)} Example 3.5) S = x is very close to y 1 2 3 y1 y2 y3 y4 0.9 0.3. y) = M MM x Nx LM x (R Ú S) (x. R(u.6 .(3. b] and its projection on the Y-axis is [0.5Q OP PP P 0.. y3 y4 01 0. v) = 1 2 3 y1 y2 y3 y4 0.5 0.5Q OP PP PQ .6 0.4) LM MM x MN x x LM MM x MN x x 1 y1 y2 0.9 0.8 R1 S0 T if (u.2 Union The union of R and S is defined by (R Ú S) (u.

8 01 01 0.7) 3.8.. . y) | y Î Y} and the projection of R on Y is defined as Õy (y) = sup{R (x. y2).e. y4).(3.e. Õx (x2) = 0. y3).FUZZY RELATIONS 23 If R is a classical relation in X ´ Y.. y1). y) Î R} Õy = {y Î Y| $x Î X(x..7: Consider the relation . y3).7 OP PP P 0Q x1 is assigned the highest membership degree from the tuples (x1. y4).3. Õx (x1) = 1. i..e. X Y Fig.3 Projection Let R be a fuzzy binary fuzzy relation on X ´ Y. (x1. i.. y) | x Î X} Example 3. which is the maximum of the first row.6) .(3. (x2.2 Shadows of a fuzzy relation. i. y1). (x1. ..7 . y3). (x2. 3. (x3. The projection of R on X is defined as Õx (x) = sup{R (x.9) . (x2.(3. x2 is assigned the highest membership degree from the tuples (x2. (x3. y2). then Õx = {x Î X| $y Î Y(x.8 0 0 0. which is the maximum of the second row.(3.9 1 0. 0 0. x3 is assigned the highest membership degree from the tuples (x3. Õx (x3) = 1.8) LM x R = x is considerable larger than y = M MM x Nx then the projection on X means that 1 2 3 y1 y2 y3 y4 0. y1). . y4).. y2).. (x3. (x1. which is the maximum of the third row. y) Î R} where Õx denotes projection on X and Õy denotes projection on Y.

y)} . B(v)} ..3 Cartesian product of two fuzzy sets. sup{B(y)}| y} = min {A(x). then Õy (A ´ B) = B and Õx (A ´ B) = A. R(x.3) is a fuzzy relation in X ´ Y.3.24 FUZZY LOGIC AND NEURAL NETWORKS 3.5 Shadow of Fuzzy Relation (Co R) (y) = sup min{C(x)..(3.(3. A B A´B Fig. The composition of a fuzzy set C and a fuzzy relation R can be considered as the shadow of the relation R on the fuzzy set C (Fig. v) = min{A(u)...(3. y)| y} = sup {A(x) Ù B(y)| y} = min {A(x)..4 Cartesian Product of Two Fuzzy Sets The Cartesian product of A Î F (X) and B Î F (Y) is defined as (A ´ B) (u.12) The sup-min composition of a fuzzy set C Î F (X) and a fuzzy relation R Î F (X ´ Y) is defined as for all x Î X and y Î Y. Really. 3.3.4). If A and B are normal. 3.11) 3. It is clear that the Cartesian product of two fuzzy sets (Fig. 1} = A(x) .. Õx(x) = sup {A ´ B (x. 3. .10) for all u Î X and v Î Y.

2/1 + 1/2 + 0.8: Let A and B be fuzzy numbers and let R=A´B a fuzzy relation.2/3) o M MM2 N3 1 2 3 1 0.2/1 + 1/2 + 0.8 1 0.8 1 .3 0. 3} and let R be a binary fuzzy relation in {1.8 1 0.8 0. 2.2/3 and LM 1 R= M MM2 N3 1 1 2 3 0.3 0. y) Y Fig.8 OP P 0. 2.8 0.8 0. 3.FUZZY RELATIONS 25 C(x) R(x. Example 3. Observe the following property of composition A o R = A o (A ´ B) = A B o R = B o (A ´ B) = B Example 3.8P P 1Q OP PP PQ Using the definition of sup-min composition we get LM 1 C o R = (0.3 0.8/1 + ½ + 0.9: Let C be a fuzzy set in the universe of discourse {1.8/3 0. 3}. Assume that C = 0.4 Shadow of fuzzy relation 4 on the fuzzy set +.3 = 0. y¢) (CoR) (y¢) X Y¢ R(x.

6 Sup-Min Composition of Fuzzy Relations Let R Î F(X ´ Y ) and S Î F(Y ´ Z).8 0 0 0. The sup-min composition of R and S.8 0. S(v.9 1 0. 0 0..5 0 0.6 0.13) LM x R = x is considerable larger than y = M MM x Nx LM MM y S = y is very close to z = y MM y MN y Then their composition is 1 2 3 1 2 3 y1 y2 y3 y4 0. Example 3.5 0.10: Let C be a fuzzy set in the universe of discourse [0.3.6 0.8 01 01 0.4 0. denoted by R o S is defined as (R o S) (u. Using the definition of sup-min composition we get C o R(y) = sup min{x.7 OP PP PQ . w) = sup min{R(u. 1] and let R be a binary fuzzy relation in [0. w)} for v Î Y It is clear that R o S is a binary fuzzy relation in X ´ Z. y) = 1 |x y|.9 0.11: Consider two fuzzy relations . 1] and y Î[0.5P Q LM x RoS= M MM x Nx 1 2 3 z1 z2 z3 0. v). .7 .7 0. Assume that C(x) = x and R(x..26 FUZZY LOGIC AND NEURAL NETWORKS Example 3.9 0.3 0 0.(3. 1 |x y|} = for all x Î[0.4 0 0. 1].8 0.9 0.7 0. 1] 1+ y 2 3.8 OP PP PQ z1 z2 z3 0.4 0 0.7 4 OP PP PP 0.

Academic Press. Information and Control. Similarity relations and fuzzy orderings.4 0. x3.R. Structure of fuzzy binary relations.6 0. D. y2} respectively. 1981. REFERENCES.5 0. 123-140. x2.9 OP P 0P P 0. pp.F. 5. d3. Vol.C. Some properties of fuzzy relationships. 5. Dubois. 5. 2.7 2 3 OP LM y PP o MM y 0 P My 0.e. and H. 6. . 3. 169-195. d4} of the various diseases affecting the plants and Y = {y1. Cohen and P. (y2. Find SUP-MIN composition. Fuzzy Sets and Systems.. 8.3). 1971. 1. 6. 0.5 0 0. R.8 0. Fuzzy Sets.9 0.9 1 0.8 0 0. QUESTION BANK. D = {d1. Zadeh. 2. 0.V. y2. 0. Vol. 0.1). 3. 2.4)} be the two fuzzy sets on the universes of discourse X = {x1. L. Zadeh.A.8 x3 0. Vol. LM MM x MN x x 1 y1 y2 y3 y4 0.3 x 0 0. (x2.8Q M MN y 1 2 3 4 z1 z2 z3 0. pp. Frankl.5).6 0. y4} be the common symptoms of the diseases. 3. 4. 338-353. 1965. .4 0.FUZZY RELATIONS 27 formally. 11. (x3. Cybernetics and Systems. but the classical product of the matrices R and S with the difference that instead of addition we use maximum and instead of multiplication we use minimum operator. No. Guild. pp. Given X = {x1. the composition of R and S is nothing else. Ovchinnikov. Information Sciences. pp.7Q i. Problems of Control and Information Theory. Vol. x4} of four varieties of paddy plants. 4.A. Prade.9 0. 1980.4 0 = 1 x2 0. 2. No. how many different projections of the relation can be taken? Given A = {(x1.7 0. Yager. J. 1.7 0. x2. 7. No. Given any N-ary relation. Find the Cartesian product of A and B. y3. Metrical properties of fuzzy relations. 0 0. Kybernetes. 389-396. 1981. x3} and Y = {y1. 177-200. 223-229. NY. 2. and N. d2. pp. G. Vol. B.F. Bouchon. L. 1980. 1982. 9. 2.3)} and B = {(y1.8 01 01 0. Vol. What are the fuzzy relations? Explain them.7 . No.5 OP PP PP PQ LM MM MN z1 z2 z3 0. S. Modelling Controllers Using Fuzzy Relations. 0. Bladwin. Explain the operations on the fuzzy relations. 12. No. Fuzzy Sets and Systems: Theory and Applications.

Prentice Hall. No. Chichester. No. and B. De Baets and E. No. 109-120. 15. 83-93. A. 16. U. . 1992. Information Sciences. Van Nostrand Reinhold. Vol. 12. 307-315. An upper bound on indices of finite fuzzy relations. pp. 3. 49.J.28 FUZZY LOGIC AND NEURAL NETWORKS 8. Gebhardt. pp. On-new types of homomorphisms and congruences for partial algebraic structures and n-ary relations. and M. 1993. 1988. No. 1988. pp. Bandler and L. International Journal of General Systems.C. R. No. 149-157. J.X. Vol.P. 1988. 13. pp. NJ. and F. 213-221. G. 4. J. Foundations of Fuzzy Systems. 1993. 1995. Klir. IEEE Computer. Hohle. J. 27. Li. 21. Kohout. 1991. 331-341. Vol. 11. Fuzzy Sets and Fuzzy Logic: Theory and Applications. 69-79. Vol. Decomposition problem of fuzzy relations: Further results.M. Fillard. Fuzzy relational compositions. 31-44. 134-146. 1991. Vol. J. 18. pp. No. No. Vol.S. W. Fuzzy Sets and Systems. and C. 159-170. pp. 1. 1994. Zadeh. 12. Fuzzy Sets and systems. 317-321. Vol. Gupta. 3. 17. 75. Kruse. pp. Inc. J. 21. W.E. 1994. Operations on fuzzy graphs. No. 10. Fuzzy Logic with Engineering Applications. Kerre. Upper Saddle River. B.. pp. 14. 9. Fodor. A new approach to the similarity in the fuzzy set theory. 54. 20. NY. Vrba. New York. General decomposition problem of fuzzy relations. 60. Fuzzy Sets and Systems. Fuzzy Sets and Systems. 1986. 3. Faurous and J. L. Vol. P. Ross. pp. Quotients with respect to similarity relations. Introduction to Fuzzy Arithmetic: Theory and Applications. Peng.J. 3. 2.A. Vol. Fuzzy logic. Yuan. McGraw-Hill. Information Sciences.1995. Mordeson.J. 50. pp. 1993. Fuzzy Sets and Systems. International Journal of General Systems. Klawon. Vol. 79. T. No. 14. No. Traces of fuzzy binary relations. pp. Kolodziejczyk. Wiley. 1. 1. 19. NY. 4. Kaufman.

p ® q = Øp Ú q p entails q means that it can never happen that p is true and q is not true. This property of material implication can be interpreted as: if X Ì Y then X ® Y .(4. p ® q is true Û t(p) £ t(q) p®q= p 1 0 0 1 q 1 1 0 0 .1 INTRODUCTION ....2) 4 Let p = x is in A and q = y is in B are crisp propositions.4) .e.+ 0 ) 2 6 .1) .. It is easy to see that The full interpretation of the material implication p ® q is that the degree of truth of p ® q quantifies to what extend q is at least as true as p. where A and B are crisp sets for the moment. It is easy to see that p ® q is true. i.(4.4 Fuzzy Implications 4.. The implication p ® q is interpreted as Ø (p Ù Øq).(4. because it can never happen that x is bigger than 10 and x is not bigger than 9.3) R1 S0 T if t(p) £ t(q) otherwise p®q 1 1 1 0 The truth table for the material implication.. Example 4..1: Let p = x is bigger than 10 and let q = x is bigger than 9.(4..

1 1 5 X Fig.1 can be interpreted as 1 is in the fuzzy set big pressure with grade of membership 0 2 is in the fuzzy set big pressure with grade of membership 0.(4. .. 4. 4. 4. The membership function of the fuzzy set A.. illustrated in the Fig..u A(u) = S1 |0 4 T if u ³ 5 if 1 £ u £ 5 otherwise . big pressure. for all x ³ 5 R1 | 5.2) 1 1 5 y Fig.1 Membership function for big pressure. small volume.2 FUZZY IMPLICATIONS Consider the implication statement.6) The membership function of the fuzzy set B.. can be interpreted as (See Fig.2 Membership function for small volume.5) 4.30 FUZZY LOGIC AND NEURAL NETWORKS Other interpretation of the implication operator is X ® Y = sup{Z|X Ç Z Ì Y} . 4.(4.25 4 is in the fuzzy set big pressure with grade of membership 0.75 x is in the fuzzy set big pressure with grade of membership 1. if pressure is high then volume is small.

. i. big pressure and q is a proposition of the form y is B for example.25 2 is in the fuzzy set small volume with grade of membership 0.7) where A is a fuzzy set.(4. for example.. and B(v) is considered as the truth value of the proposition v is small volume.(4. small volume then we define the fuzzy implication A ® B as a fuzzy relation. for all x £1 R1 | v -1 B(v) = S1 |0 4 T If p is a proposition of the form x is A if v ³ 1 if 1 £ v £ 5 otherwise ... v) = I(A(u)..(4.10) 4 is big pressure ® 1 is small volume A(4) ® B(1) = 0.8) R1 S0 T R1 S0 T if t( p) £ t(q ) otherwise . That is (A ® B)(u. v) should be defined pointwise and likewise. that is u is big pressure ® v is small volume º A(u) ® B(v) Remembering the full interpretation of the material implication p®q= .75 x is in the fuzzy set small volume with grade of membership 1.9) One possible extension of material implication to implications with intermediate truth values can be A(u) ® B(v) = if t( p) £ t(q ) otherwise .. B(v)) = A(u) ® B(v) In our interpretation A(u) is considered as the truth value of the proposition u is big pressure.e. It is clear that (A ® B)(u. (A ® B)(u.75 ® 1 = 1 .(4... v) depends only on A(u) and B(v).FUZZY IMPLICATIONS 31 5 is in the fuzzy set small volume with grade of membership 0 4 is in the fuzzy set small volume with grade of membership 0.

7999...(4.e.15) It is easy to see this is not a correct extension of material implications.. ® q = Øp Ú q using the definition of negation and union A(u) ® B(v) = max {1 A(u). it is easy to see that this fuzzy implication operator (called Standard Strict) sometimes is not appropriate for real-life applications.(4.8.14) This operator is called Kleene-Dienes implication...12) . Namely. B(v)} . Then we have A(u) ® B(v) = 0. in knowledge-based systems.. A smoother extension of material implication operator can be derived from the equation X ® Y = sup {Z| X Ç Z Ì Y} That is A(u) ® B(v) = sup {z| min {A(u).32 FUZZY LOGIC AND NEURAL NETWORKS However.8 = 1 Suppose there is a small error of measurement in B(v). and instead of 0..17) . z} £ B(v)} so..13) This operator is called Godel implication.16) . Other possibility is to extend the original definition. A(u) ® B(v) = .(4.. we are usually not interested in rules.18) . Larsen Lukasiewiez Mamdani x ® y = xy x ® y = min{1.(4..(4. i.(4. Then A(u) ® B(v) = 0....20) ..(4.(4. 1 x + y} x ® y = min{x.8 ® 0. B(v)} .11) R1 S B (v ) T if A(u) £ B(v) otherwise ...19) Standard Strict x ® y = Godel R1 S0 T R1 x®y= S Ty if x £ y otherwise if x £ y otherwise .8 we have 0.(4.8 and B(v) = 0.7999 = 0 This example shows that small changes in the input can cause a big deviation in the output..8 ® 0.. y} . because 0 ® 0 yields zero... our system is very sensitive to rounding errors of digital computation and small errors of measurement. where the antecedent part is false. let A(u) = 0. In many practical applications they use Mamdanis implication operator to model causal relationship between fuzzy variables.(4. However. This operator simply takes the minimum of truth values of fuzzy predicates A(u) ® B(v) = min {A(u).

24) The use of fuzzy sets provides a basis for a systematic way for the manipulation of vague and imprecise concepts.(4.FUZZY IMPLICATIONS 33 Gains x®y= R1 S y/x T if x £ y otherwise . In particular.. A linguistic variable can be regarded either as a variable whose value is a fuzzy number or as a variable whose values are defined in linguistic terms.(4.. More or less old Old 30 60 Fig.. we can employ fuzzy sets to represent linguistic variables..23) Kleene-Dienes x ® y = max {1 x.3 MODIFIERS Let A be a fuzzy set in X.(4. y} Kleene-Dienes-Luk x ® y = 1 x + xy 4. .(4.4 More or less old. Old Very old 30 60 Fig. (more or less A)(x) = A( x) ....3 Very old.22) .21) . Then we can define the fuzzy sets very A and more or less A by (very A)(x) = A(x)2. 4. 4..

the set of names of linguistic values of x with each value being a fuzzy number defined on U. In many practical applications we normalize the domain of inputs and use the type of fuzzy partition shown in Fig.. . 1]. For example.6. that is. 4.(4. 1 Slow Medium Fast 40 55 70 Speed Fig.5 Values of linguistic variable speed. NB NM NS ZE PS PM PB –1 1 Fig.1 LINGUISTIC VARIABLES A linguistic variable is characterized by a quintuple (x. and fast as a speed above about 70 mph. moderate.5. moderate as a speed close to 55 mph. G is a syntactic rule for generating the names of values of x. fast.. then its term set T (speed) could be T = {slow.34 FUZZY LOGIC AND NEURAL NETWORKS 4. T(x). M) ..25) in which x is the name of variable. 100]. G. very slow. if speed is interpreted as a linguistic variable. 4. 4. We might interpret slow as a speed below about 40 mph. T(x) is the term set of x. These terms can be characterized as fuzzy sets whose membership functions are shown in Fig.3. 4. and M is a semantic rule for associating with each value its meaning. U. .6 A possible partition of [ 1. more or less fast.} where each term in T (speed) is characterized by a fuzzy set in a universe of discourse U = [0..

(4... Very false..FUZZY IMPLICATIONS 35 Here we used the abbreviations NB Negative Big. Fairly true.7... False (u) = 1 u for each u Î [0.26) R1 S0 T R1 Absolutely true (u) = S T0 1 False Absolutely false if u = 0 otherwise if u = 1 otherwise .. False. 4. Truth True Absolutely true 1 Fig. [PM] Positive Medium. Very true. [PB] Positive Big.3. Absolutely false (u) = .(4. 1]. Absolutely true}. 4. [PS] Positive Small. [ZE] Zero..28) .. NS Negative Small.(4.27) .29) The interpolation if absolutely false and absolutely true are shown in Fig. 4.(4. [NM] Negative Medium. One may define the membership function of linguistic terms of truth as True (u) = u for each u Î [0.7 Interpretation of absolutely false and absolutely true. 1].2 The Linguistic Variable Truth Truth = {Absolutely false. . True.

Truth Fairly false 1.(4... 1].. 1].. Fairly true (u) = for each u Î [0..30) . Very true (u) = u2 for each u Î [0. 4. 1]..9 Interpretation of fairly false and very false.36 FUZZY LOGIC AND NEURAL NETWORKS The word Fairly is interpreted as more or less. Let t be a term of linguistic variable Truth..(4.. Truth Fairly true u .33) Very false 1 Fig. Where (t o A)(u) = t(A(u)) for each u Î [0. 1]. Fairly false (u) = for each u Î [0. 1].u . The word Fairly is interpreted as more or less.8 Interpretation of fairly true and very true..34) . Then the statement x is A is t is interpreted as x is t o A. Very false (u) = (1 u) 2 for each u Î [0.32) . Suppose we have the fuzzy statement x is A.(4.(4.. . 4..31) Very true 1 Fig.(4.

4.FUZZY IMPLICATIONS 37 For example..(4. Then x is A is true is defined by x is t o A = x is A because (t o A)(u) = t(A(u)) = A(u) for each u Î [0. Then the statement x is A is Absolutely false is defined by x is t o A. let t = true.35) 1 A is absolutely true a–a a b b–b Fig. Let t = absolutely false.(4.10 Interpretation of A is true. where (t o A) (x) = R1 S0 T if A( x) = 0 otherwise . where (t o A)(x) = R1 S0 T if A( x) = 1 otherwise . 4. It is why everything we write is considered to be true. 1 A = “A is true” a–a a b b–b Fig. 1].... Then the statement x is A is Absolutely true is defined by x is t o A.11 Interpretation of A is absolutely true.36) . Let t = absolutely true.

2.. Then the statement x is A is Fairly true is defined by x is t o A. where (t o A) (x) = A( x) .. What are the fuzzy modifiers? Explain with an example.14 Interpretation of A is very true. Then the statement x is A is Fairly true is defined by x is t o A. What are the fuzzy implications? Explain with examples. 4. QUESTION BANK.(4. Let t = Very true. 4.37) 1 “A is fairly true” a–a a b b–b Fig. Let t = Fairly true. 4. .38 FUZZY LOGIC AND NEURAL NETWORKS 1 A is absolutely false a–a a b b–b Fig. What are the linguistic variables? Give examples.12 Interpretation of A is absolutely false.. where (t o A)(x) = (A(x))2 ..13 Interpretation of A is fairly true.38) 1 “A is very true” a–a a b b–b Fig. 1.(4. 3.

K. 10. 89-116. 199-249. R. 13-30. 267-279. Valverde. Fuzzy Sets and Systems. pp. No. Ahlquist. 5.FUZZY IMPLICATIONS 39 4. Willmott. E.H. 1987. 22. REFERENCES. Man. 4. pp. A. A theorem on implication functions defined from triangular norms.A. Zadeh. 12. No. 8. Given the set 6 of people in the following age groups: 0 10 10 20 20 30 30 40 40 50 50 60 60 70 70 80 80 and above Represent graphically the membership functions of young. No. pp. Fuzzy Sets and Systems. Bandler. 1980. Kohout. pp. 12. International Journal of Man-Machine Studies. 3.H. Kisska [Eds. Advances in the linguistic synthesis of fuzzy controllers. 115-134. and L. New York.J. 1983. 193-219. 1. negations and implications based on t-norms and t-conorms. 7. 3. Vol.B.J. 12. W. S. 1. Vol. 8. 1980. Vol. 1.]. No. 8. pp. 9. 6. 229-244. M. International Journal of Man-Machine Studies. Trillas and L. Bandler and L. Information Sciences. On mode and implications in approximate reasoning. pp. middle-aged and old.F. No. Vol. Vol. 1182-1191. 1. Baldwin and B. No. Application of fuzzy implication to probe nonsymmetric relations: Part 1. Fuzzy Sets and Systems.W. 1975. No. Vol. Properties of fuzzy implication operators. E. Stochastica. Oh and W. Prade. pp. W. 3. Gupta. Fuzzy power sets and fuzzy implication operators. In: M. 1987. Pilsworth. Fuzzy Sets and Systems. 31-36. 4. Bandler and J. 1985. pp. 1. E. 3. Kandel. The concept of a linguistic variable and its application to approximate reasoning. L. Approximate Reasoning in Expert Systems.E. Kohout. Part 1. 11. 2. Explain the linguistic variable TRUTH with examples. 273-285. pp. 5. Vol. No. Weber. 1980. 6. pp. W. pp. Fuzzy Sets and Systems. 1984. pp. 1976. Two fuzzier implication operators in the theory of fuzzy power sets. and Cybernetics. Mamdani. IEEE Transactions on Systems. 2. Dubois and H. J. 1977. Vol. 8. Axiomatic approach to implication for approximate reasoning with fuzzy logic.W. No. 4. 3. D. 669-678. 1980. Vol. 2. Semantics of implication operators and fuzzy relational products. Vol. 26. 157-166. No. A general concept of fuzzy connectives. Mamdani. Bandler. 11. NorthHolland. Vol. J. . Applications of fuzzy logic to approximate reasoning using linguistic systems. International Journal of Approximate Reasoning.

Vol. W. Fuzzy Sets and Systems. Z. 31. . pp. Magrez. Fuzziness and Knowledge-Based Systems. Implication in fuzzy logic. 14. No. Vol. Kandel. pp. 1988. J. 4. International Journal of Approximate Reasoning. Fuzzy Sets and Systems. 151-186. 1994. 15. pp. Fuzzy implication operators and generalized fuzzy method of cases. 1992.L. 42. pp. No. 18. 333-341. 45. No. 4.M. Delgado and E. pp. 3. pp. G. Castro. M. 1. Magrez. 293-300. 25. 16. Commutative implications on complete lattices. No. Vol. Vol. A. International Journal of Approximate Reasoning. Smets and P. 17. 307-318. De Cooman. B. E. 1991. Trillas. 3. Fuzzy implication in fuzzy systems control. 20. International Journal of Approximate Reasoning. Smets and P. Vol. 235-250. Vol. 1994. 1987. No. 10. 327-347. Applicability of Some Fuzzy Implication Operators. pp. No.40 FUZZY LOGIC AND NEURAL NETWORKS 13. 1989. 1. 4. Wu. 2. No. No. Ruan and E. D. International Journal of Uncertainty. Kerre. 54. J. Vol. Vanmassenhove. Fuzzy Sets and Systems.E Kerre. On fuzzy implication operators. 3. P. 67-72. 1993. Vol. pp. Da. Influence of the fuzzy implication operator on the method-of-cases inference rule. Cappelle and F. 23-37. The measure of the degree of truth and the grade of membership. R. 19. Fuzzy Sets and Systems. 21.E. 1. pp. Inducing implication relations. 1990. 25-35. Piskunov. Cao and A. Fodor.C. 1. No. P. Vol. 2.

More often than not we do not know the complete causal link f between x and y. Namely.1 INTRODUCTION In 1975 Zadeh introduced the theory of approximate reasoning.4 The Theory of Approximate Reasoning 5 5. only we know the values of f (x) for some particular values of x Â1 : also Â2 : If x = x2 then y = y2 If x = x1 then y = y1 . we know that y is a function of x y = f(x) Then we can make inferences easily Premise Fact Consequence y = f(x) x = x1 y = f(x1) This inference rule says that if we have y = f (x). This theory provides a powerful framework for reasoning in the face of imprecise and uncertain information. then y takes the value f (x1). " x Î X and we observe that x = x1. Suppose we have two interactive variables x Î X and y Î Y and the causal relationship between x and y is completely known. Central to this theory is the representation of propositions as statements assigning fuzzy sets as values to variables.+ 0 ) 2 6 .

Suppose that we are given an x1ÎX and want to find an y1ÎY which corresponds to x1 under the rule-base. Â1 : also Â2 : also also Ân : fact: If x = xn then y = yn x = x1 Consequence: y = y1 This problem is frequently quoted as interpolation. 5. x is high and y is small. Let x and y be linguistic variables.g. .1 Simple crisp inference. . e. .42 FUZZY LOGIC AND NEURAL NETWORKS also also Ân : If x = xn then y = yn y y = f (x) y =f (x¢) x = x¢ x Fig. Ân} and the fact A. . The basic problem of approximate reasoning is to find the membership function of the consequence C from the rule-base {Â1. Â1: also Â2 : If x is A2 then y is C2 If x is A1 then y is C1 If x = x2 then y = y2 If x = x1 then y = y1 .

2.2 Conjunction Rule x is A x is B x is A Ç B Temperature is not very high Temperature is not very low Temperature is not very high and not very low 5. 5. which allow us to represent some common linguistic statements in terms of propositions in our language.2.THE THEORY OF APPROXIMATE REASONING 43 also also Ân : fact: If x is An then y is Cn x is A Consequence: y is c Zadeh introduced a number of translation rules.2.3 Disjunction Rule x is A or x is B x is A È B Temperature is not very high or Temperature is not very low Temperature is not very high or not very low .2 5.1 Entailment Rule x is A AÌB x is B TRANSLATION RULES Menaka is very young very young Ì young Menaka is young 5.

The classical Modus Ponens inference rule says: premise fact consequence: if p then q p q This inference rule can be interpreted as: If p is true and p ® q is true then q is true.44 FUZZY LOGIC AND NEURAL NETWORKS 5.5 Negation Rule not (x is A) x is ØA not (x is high) x is not high In fuzzy logic and approximate reasoning.6 Compositional Rule Of Inference premise fact consequence: if x is A then y is B x is A1 y is B1 . The fuzzy implication inference is based on the compositional rule of inference for approximate reasoning suggested by Zadeh. 5. y) have relation R y is Õy(R) (x. 2) x is close to 3 (x. y) have relation R x is Õx (R) (x.2.2.4 Projection Rule (x. 2) y is close to 2 5. the most important fuzzy implication inference rule is the Generalized Modus Ponens (GMP). y) is close to (3. y) is close to (3.2.

The Generalized Modus Tollens. is closely related to the forward data-driven inference which is particularly useful in the Fuzzy Logic Control. especially in the realm of medical diagnosis.3 RATIONAL PROPERTIES Suppose that A.(5.1 Basic Property if x is A then y is B x is A y is B if pressure is big pressure is big volume is small then volume is small . (A ® B) (u.2) The consequence B1 is nothing else but the shadow of A ® B on A1. The Generalized Modus Ponens.1) .. which reduces to calssical modus ponens when A1 = A and B1 = B..3. 5. premise fact consequence: if x is A then y is B y is B1 x is A1 which reduces to Modus Tollens when B = ¬B and A1 = ¬A. is closely related to the backward goaldriven inference which is commonly used in expert systems. The classical Modus Tollens inference rule says: If p ® q is true and q is false then p is false. v)}.. B1 = A1 o (A ® B) that is. B and A1 are fuzzy numbers. B¢(v) = sup min {A¢(u).. v Î V uÎU .(5. The Generalized Modus Ponens should satisfy some rational properties. 5.THE THEORY OF APPROXIMATE REASONING 45 where the consequence B1 is determined as a composition of the fact and the fuzzy implication operator.

5.3.46 FUZZY LOGIC AND NEURAL NETWORKS A¢ = A B¢ = B Fig. 5.2 Total Indeterminance if x is A then y is B x is ¬A y is unknown if pressure is big pressure is not big volume is unknown then volume is small –A – A¢ –B – B¢ Fig. 5.3 Subset if x is A then y is B x is A1 Ì A y is B .2 Basic property.3. 5.3 Total indeterminance.

Suppose that A. Example 5. 5. B and A1 are fuzzy numbers. 5.1: (The GMP with Mamdani implication) if x is A then y is B x is A1 y is B1 .4 Subset property.5 Superset property. We show that the Generalized Modus Ponens with Mamdanis implication operator does not satisfy all the four properties listed above.3.4 Superset if x is A then y is B x is A1 y is B1 É B –A – A¢ –B – B¢ x Fig. 5.THE THEORY OF APPROXIMATE REASONING 47 if pressure is big then pressure is very big volume is small volume is small B¢ = B –A – A¢ Fig.

1 . . B(y)} x = min B( y). Then we have B¢(y) = sup min {1 A(x). Subset: Let A¢ Ì A and let y Î R be arbitrarily fixed. min {A(x). min {A(x). B(y)} x = sup min B( y ). 1/2} = 1/2 B(y) < 1 This means that the total indeterminance property is not satisfied. Then we have B¢(y) = sup min {A¢(x). Then we have B¢(y) = sup min {A(x). sup min { A( x). B(y)}} x = sup min {A(x). B(y)} x = min B( y).A( x). 1} = B(y) So the subset is satisfied. B(y)}} x = sup min {A(x).} x R S T U V W = min {B(y). y Î R Basic property: Let A1 = A and let y Î R be arbitrarily fixed. min {A(x). B(y)}} x = sup min {A(x).48 FUZZY LOGIC AND NEURAL NETWORKS where the membership function of the consequence B1 is defined by B¢(y) = sup {A¢(x) Ù A(x) Ù B(y) |x Î R}. A¢(x). Total indeterminance: Let A1 = ØA = 1 A and let y Î R be arbitrarily fixed. sup A¢( x) x R S T U V W = min {B(y). sup A( x ) x x R S T U V W = min {B(y). 1 A(x). 1} = B(y) So the basic property is satisfied.

THE THEORY OF APPROXIMATE REASONING 49 Superset: Let y Î R be arbitrarily fixed. A(x) B(y)} x = B( y ) <1 1 + B( y ) This means that the total indeterminance property is not satisfied. min {A(x). Then we have B¢(y) = sup min {A¢(x). A¢(x). 5. Then we have B¢(y) = sup min {1 A(x). Total indeterminance: Let A1 = ØA = 1 A and let y Î R be arbitrarily fixed. B(y)}} x = sup min {A(x).6 The GMP with Mamdanis implication operator. . B(y)} £ B(y) x So the superset property of GMP is not satisfied by Mamdanis implication operator. Then we have B¢(y) = sup min {A(x). A –B – B¢ A(x) x Fig.2: (The GMP with Larsens product implication) if x is A then y is B x is A1 y is B1 where the membership function of the consequence B1 is defined by B¢(y) = sup min {A¢(x). A(x) B(y)} = B(y) x So the basic property is satisfied. Example 5. A(x) B(y) |x Î R} y Î R x Basic property: Let A1 = A and let y Î R be arbitrarily fixed.

7 The GMP with Larsens implication operator. 5. A(x) B(y)} x = sup min {A(x). A(x) B(y)} £ B(y) x So. the superset property is not satisfied. Then we have B¢(y) = sup min {A¢(x). Explain generalized modus ponens with Larsens implication. A¢(x) B(y)} x = B(y) So the subset property is satisfied. QUESTION BANK. What are the translation rules? Explain them with examples. 4. Given CÚD ~ H Þ (A Ù ~ B) CÚDÞ~H (A Ù ~ B) Þ (R Ú S) Can (R Ú S) be inferred from the above? . 2. 6. 5. Superset: Let y Î R be arbitrarily fixed. A A¢ –B – B¢ x Fig.50 FUZZY LOGIC AND NEURAL NETWORKS Subset: Let A¢ Ì A and let y Î R be arbitrarily fixed. Explain generalized modus ponens with Mamdanis implication. 3. 1. Explain the theory of approximate reasoning. What are the rational properties? Explain them. Then we have B¢(y) = sup min {A¢(x).

Zadeh. Vol. pp. 7. Multi-dimensional Fuzzy Reasoning.J. 301-357. 1. Fuzzy Sets and Systems. Man and Cybernetics. Information Sciences. L. 11. pp. The concept of a linguistic variable and its application to approximate reasoningPart I. 3. The role of fuzzy logic in the management of uncertainty in expert systems. E. pp. A computational theory of dispositions. Vol. 1. 9. Fuzzy Sets and Systems.A. 16. 8. 17. Baldwin. Vol. 1. 2. pp. pp. 15. No. 1986. Vol. pp. Syllogistic reasoning in fuzzy logic and its application to usually and reasoning with dispositions. Vol. 15. No. 1975.THE THEORY OF APPROXIMATE REASONING 51 REFERENCES. 5. pp. Vol. Takagi. Yager. Fuzzy Sets and Systems. L. International Journal of Intelligent Systems. 1. 16. Vol. 2. 1983. 18. Man and Cybernetics. Prade. No. A new approach to approximate reasoning using a fuzzy logic. 754-765. L. 2. International Journal of Man-Machine Studies. 7. Information Sciences. 20. pp. 4. 67-78. 2. 1985. 12. 1985. Cybernetics and Systems. 1982. 4. Zadeh. Fuzzy Sets and Systems. 23-63. Vol. No. Prade. 1987. Zadeh. 3. 1979. W. 1985. No. 6. No. Fuzzy logic and approximate reasoning. 43-80. Vol. 10.A. 9. Default and inexact reasoning with possibility degress. 1983. . 14. 199-228. 8. pp. Sugeno and T. 270-276. Zimmermann. Man and Cybernetics. Zadeh. 1979. IEEE Transactions on Systems. Medical Information. pp. 1182-1191. 260-283. 1983. 39-63.A.A. pp. IEEE Transactions on Systems. 3.A. 309-325.H. 11. 30. Zadeh. 4. Vol. 407-428. 1. International Journal of Man-Machine Studies. L. Comparison of fuzzy reasoning methods. Semantics for fuzzy reasoning. pp. J. Vol. No. Application of fuzzy logic to approximate reasoning using linguistic systems. No. 1986. 623-668. B. Zadeh. 401-415. pp. 17. pp. 6. Mamdani. L. 1975. A computational approach to approximate and plausible reasoning with applications to expert systems. 313-325. No. IEEE Transactions on Pattern Analysis and Machine Intelligence. No. 13. pp. Foundations of fuzzy reasoning. Fuzzy Sets and Systems. Mizumoto and H. pp. No. Strong truth and rules of inference in fuzzy logic and approximate reasoning. 26.M. Vol. Farreny and H. Wu. 199-251. 16. Vol. No. Fuzzy reasoning and fuzzy relational equations. 9. 4. Syntheses. Vol. 1976. approximate reasoning and dispositions. Gainess. Information Sciences. 2. M. Zadeh.F. R.R. 309-325. Vol. H. 1975. 6. 3. R. 1977. Vol. Linguistic variables. L. Giles. L. 8. Vol. Vol. 173-186. H. pp. M. The concept of a linguistic variable and its application to approximate reasoningPart III. IEEE Transactions on Systems. 12. The concept of a linguistic variable and its application to approximate reasoningPart II.R.A.A. No. 1975. L.A. 8. pp. No. 8. Zadeh.

Fuzzy Sets and Systems. Gorzalczany. pp. D. D. approximate reasoning and prototypical knowledge. Unklesbay and N. R. pp. 143-202. Fuzzy Sets and Systems.S. No. Approximate reasoning for production planning.H. Wang. Cybernetics and Systems. 4. 29. Vol. 5. Vol.Z. M. 1990.52 FUZZY LOGIC AND NEURAL NETWORKS 19. Ruspini. 37. Dutta. M. Part 1: Inference with Possibility distributions. 1991. 47-68. 297-317.B. No. A new model of fuzzy reasoning.L Grize and K. International Journal of Intelligent Systems. 57. 25. T. 77-81. pp. Console. P. 157-178. No. 1988. 21. J. Ellis Horwood. Fuzzy reasoning in a multi-dimensional space of hypothesis. No. 1991. IEEE Transactions on Systems. 1990. International Journal of Approximate Reasoning. pp. 26. X. pp. 1-11. Cao. 1991. 1989. Man and Cybernetics. 1988. Fuzzy Sets and Systems. A system for Reasoning with Imprecise Linguistic Information. 22. Turksen. 1987. 27. Vol. Unklesbay. 3. 23-37. Wang. Theory of t-norms and fuzzy inference methods. D. No. Fuzzy Sets and Systems. No. Vol. S. 1990.M. P. pp. rules and fuzzy reasoning: A factor space approach. 35. approximate spatial reasoning: Integrating qualitative and quantitative constraints. M.G. Fuzzy Sets and Systems. pp. Vol. pp. No. Smets. present. A. Chischster. Prade. 3. No. 34. Vol. 4.M. 1. Peng. Luo and Z. No. 2. K. No. 2. 331-342. 202-244. Li. Prade. pp. 181200. 1. 1. 311-325.S. Kruse and E. Vol. 1991. No. Dehnad. No. 4. International Journal of Intelligent Systems. 40. Dubois and H. 28. 33. Part 2: Logical approaches. 40. future. Vol. 1987. Vol. Magrez and P. 36. pp. Lopez de Mantaras. Vol. pp. Vol. Kandel and L. Quantitative evaluation of university teaching quality An application of fuzzy set and approximate reasoning. Concepts. 1991. 281-293. Keller. Kandel and P. 1. International Journal of General Systems. pp. Qi. Torasso and L. Vol. 307-330. Some notes on multi-dimensional fuzzy reasoning. 1. 40. Y. 31. 1991. pp. Fuzzy Sets and Systems. 5. 431-450. Vol. 15-38. Ying. 1990. Feng. 16. Fuzzy sets in approximate reasoning. 1. Gupta and J. 21. 1. 1989. Qualitative models for reasoning under uncertainty in knowledgebased expert systems. Dubois and H. 24. 1989. An approximate reasoning technique for recognition in color images of beef steaks. Information Sciences.B. A method of inference in approximate reasoning based on interval-valued fuzzy sets. 20. 1991. pp. No. R. Z. Fuzzy Sets and Systems. Lee.P. pp. N. 4. 1-17. C. Schwecke. 1. International Journal of Approximate Reasoning. International Journal of Approximate Reasoning. 21. 1990. . A. C. 1. 26. 36. 32.Z. pp. Approximate Reasoning Models. Schwartz. Vol. Approximate reasoning: past. I. 37. Vol. Fuzzy modus ponens: A new model suitable for applications in knowledge-based systems. 36. Fuzzy Sets and Systems. No. 19. 3. E. Vol. 5. 4. Representation of compositional relations in fuzzy reasoning. Vol. pp. 30. pp. No. 3. 194-205. No. Fuzzy sets in approximate reasoning. No. D. Subhangkasen. Vol. 463-488. 23.

pp. 1. 1994. 177-180. 431-440. 1994. Anderson. 40. No. 2. P. 23. No. 1993. Chen. 51. IEEE Transactions on Fuzzy Systems. Vol. pp. 315-326. pp. Vol. 48. Vol. Z. 39. No. No. 3. pp. D. 3. Some methods of reasoning for fuzzy conditional propositions. Formal frameworks for approximate reasoning. International Journal of Approximate Reasoning. Hirota. 2. Larsen and R. Man and Cybernetics. S.F. IEEE Transactions. No. 3.L. Vol.S. 49. No. An approximate reasoning system: Design and implementation. pp. 5. Fuzzy Sets and Systems. Z. 52. 1992. No. 41. pp. 45. Koczy and K. 51. 1993. 47. Fuzzy Sets and Systems. pp. No. 229-250. K. Vol. 1992. An inference network for bi-directional approximate reasoning based on an equality measure. 7. Cohen and M. on Systems. 43. 1994. 3. 463-481. Vol. 349358. 4.R. pp. Raha and K. 7. 1-25.G. Vol. C. No. J. L. No. Q. M. Ray. pp. 1993. 31-41. 11. 57. 71-79. 1993. No. Rule-based Fuzzy Classification for Software Quality Control. 9. Babu. 1. Metric truth as a basis for fuzzy linguistic reasoning.A.V. 197-225. Vol. 9. 1993. 44. Lano. Cybernetics and Systems. 63. 2. 50. Chun. Kacprzyk. International Journal of Pattern Recognition and Artificial Intelligence. pp. 29-53. Niskanen. Zhang. Vol. V. 131-146. pp. 1. Yager. Fuzzy Sets and Systems. 23. Bien and M. 46.S. An improved algorithm for inexact reasoning based on extended fuzzy production rules.T. 3. . International Journal of Intelligent Systems. 1992. Hudson. S. 1992. Reddy and M. Approximate reasoning with IF-THEN-UNLESS rule in a medical expert system. Elbert. 259-266. Analogy between approximate reasoning and the method of interpolation. Vol. pp.M.L. Fuzzy Sets and Systems. International Journal of Approximate Reasoning. 1992. No. 42. The use of fuzzy relational thesauri for classificatory problem solving in information retrieval and expert systems. Zhao and B. Li. No. International journal of approximate reasoning.THE THEORY OF APPROXIMATE REASONING 53 38. 1. Vol. Vol. pp. Approximate reasoning by linear rule interpolation and general approximation.E. An approximate reasoning system. Vol. On measuring the specificity of IF-THEN rules. H.

T(y. z)) = T(T(x. In fuzzy sets theory triangular norm are extensively used to model the logical connective and. T(x.. y. the standard fuzzy operations occupy specific positions in the whole spectrum of fuzzy operations: the standard fuzzy intersection is the weakest fuzzy intersection. The standard fuzzy union (max operator) produces.3) ...(6.2 TRIANGULAR NORM A mapping T: [0... 1]. Furthermore. respectively. 1] ® [0..(6. while the standard fuzzy union is the strongest fuzzy union.6) £ukasiewicz: TL(a.(6.. 1] .. y Î [0. b) = min {a. b) = max {a + b. 0} .. "x Î [0. 1) = a. y) = T(y. associative. z). x).. on the contrary. y). for all a Î [0.2) . non-decreasing in each argument and T(a.1 INTRODUCTION Triangular norms were introduced by Schweizer and Sklar to model the distances in probabilistic metric spaces. "x.1. The basic t-norms are: minimum : min (a.+ 0 ) 2 6 . Functions that qualify as fuzzy intersections and fuzzy unions are usually referred to in the literature as t-norms and t-conorms. z Î [0.1) . the standard fuzzy intersection (min operator) produces for any given fuzzy sets the largest fuzzy set from among those produced by all possible fuzzy intersections (t-norms).(6.(6. 1) = x. 1] T(x. 6. "x.4) These axioms attempt to capture the basic properties of set intersection.5) .4 Fuzzy Rule-Based Systems 6 6. y) £ T(x¢. y¢) if x £ x¢ and y £ y¢.. That is.. In other words. 1] T(x. 1] ´ [0. b} . any t-norm T satisfies the properties: Symmetricity : Associativity : Monotonicity : One identy : T(x.(6. the smallest fuzzy set among the fuzzy sets produced by all possible fuzzy unions (t-conorms). 1] is a triangular norm (t-norm for short) if it is symmetric.

a Î (0. p [(1 ...(6.(6.. b) = Da(a. associative.1) (l .a ) p + (1 .3 TRIANGULAR CONORM A mapping S : [0.1) ú ï l -1 ê ú ë û î if l = 0 if l = 1 if l = ¥ otherwise .(6.b) p ] .12) All t-norms may be extended.11) Frank : ì min {a. b} = 1 otherwise . a2.. S(y. X an TL (a1. g³0 g + (1 . b) ï P ï Fl(a.FUZZY RULE-BASED SYSTEMS 55 product : weak : TP(a. b) = ab Tw(a. The minimum t-norm is automatically extended and TP (a1....14) R a .... 1) max {a . In other words. 6. through associativity. 1] .n + 1.. 0) = x...... 1]. b) = 1 min 1. any t-conorm S satisfies the properties: Symmetry : S(x. an) = a1 X a2 X ... a2.10) Hamacher : Dubois and Prade : Yager : ab .(6.(6. y¢) if x £ x¢ and y £ y¢ S(x. 0U |å | S V | | T W n i i =1 A t-norm T is called strict if T is strictly increasing in each argument. b) = íTL (a..(6. b} S0 T if max {a ..log l ê1 + (l .... an) = max .15) . is a triangular co-norm (t-conorm) if it is symmetric. b) ï a b é ù ï1 . Triangular co-norms are extensively used to model logical connectives or.(6. z)) = S(S(x.7) Rmin {a. for all a Î [0.. z) S(x.8) .. to n > 2 arguments..g )(a + b .(6.16) .. b} ïT (a...17) .. y) £ S(x¢.. b.. y) = S(y. 1].(6.. b) = Hg(a.13) . a} Yp(a.. b) = . x) S(x.ab) ab . 1] ´ [0.(6.. 0) = a.9) .18) Associativity : Monotonicity : Zero identity : . y). "x Î [0. 1] ® [0.(6. non-decreasing in each argument and S(a.(6. { } p>0 .

g )ab . 0) ³ x S(x. "x. From commutativity of T it follows that a = T(a..(6. b) = 0 otherwise .. The basic t-conorms are: maximum : max (a. y) £ T(x.(6. Then the following statement holds max{a. Proof: If T(a.21) . 1].. y) £ min {x. y) £ T(x.(6. Suppose T(a. b} for any a Î [0.(6. b} £ s (a. y Î[0. "a. b) £ min {a.. y}.2: Proof: Let S be a t-conorm. b) = max {a. Then the following statement holds Tw(x.. 1]. These equations show that T(a. 1] if and only if T is the minimum norm. symmetricity and the extremal condition we get S(x. and a £ b £ 1.(6. b} S1 T if min (a . a) = a for any aÎ[0.g )ab . y) ³ max{x... P > 0 { } Let T be a t-norm. a) £ min {b. b) = min {a + b.1] From monotonicity. b}. b) = min {a. a) £ T(a.. b Î [0.22) £ukasiewicz : probabilistic : strong : .19) Rmax {a. y) £ min {x. b) = min 1.25) YORP (a. g³0 1 ..(6. b) £ STRONG (a. b) = a + b . a) £ T(b..3: T(a.. b) = a + b.(6.23) Hamacher Yager Lemma 6. y) = T (y. y}. x) ³ S(y. b) then T(a. a}. b) = .24) .(1 . 1) £ y This means that T(x. Lemma 6.. b) = 1 T(1 a.. b) = min (a. y) ³ S(x.56 FUZZY LOGIC AND NEURAL NETWORKS If T is a t-norm then the equality S(a. y) ³ S(y. 0) ³ y This means that S(x. b} SL(a. 1} SP(a. We can obtain the following expression using monotonicity of T a = T(a. p a p + b p .(2 . a) = a holds obviously. b). x) £ T (y. . 1] From monotonicity. Lemma 6.20) . 1) £ x T(x. 1 b) defines a t-conorm and we say that S is derived from T.ab STRONG (a. symmetricity and the external condition we get T(x. y}.1: Proof: : : HORg (a. a) = a holds for any aÎ[0..

The T-intersection of A and B is defined as (A Ç B) (t) T (A(t). Let A and B be fuzzy subsets of X = {x1. b Î[0. The S-union of A and B is defined as (A Ç B) (t) = S(A(t).(6. . y) = LOR (x. Example 6.2/x7 Then A È B has the following form A È B = 0. B(t)1} for all t Î X.0/x2 + 0. c) = max {T(a.0/x4 + 1. x5.0/x4 + 1. x2. Then A Ç B has the following form A Ç B = 0.3/x6 + 0.6/x5 + 0. Example 6.3/x2 + 0.0/x1 + 0.0/x4 + 0.6/x3 + 1.2/x7. b}.FUZZY RULE-BASED SYSTEMS 57 Lemma 6.26) Let T be a t-norm.9/x3 + 1. x3..3/x2 + 0. b) £ max {a. If we are given an operator C such that min {a.6/x3 + 1.4 J-NORM-BASED INTERSECTION .(6.0/x5 + 0.0/x1 + 0.6/x6 + 0.6/x5 + 0.28) . B(t)) for all t Î X.2/x7.27) Let S be a t-conorm. T(b.6/x2 + 1.0/x7 B = 0. 6. x6.5 J-CONORM-BASED UNION . "a. Let A and B be fuzzy subsets of X = {x1.3/x2 + 0.. 1}be the £ukasiewicz t-conorm. Then we have 6.3/x6 + 0. x4.0/x3 +1. b}.6/x5 + 0.. 1].4: The distributive law of t-norm T on the max operator holds for any a. 0)} for all t Î X.0/x1 + 0. c)}. Then we have (A È B) (t) = min {A(t).(6.3/x6 + 0. B(t)) for all t Î X.3/x2 + 0. b.2/x7. x7} and be defined by A = 0. x7} and be defined by A = 0.1/x1 + 0. x6. y) = { x + y 1. c Î[0. 2: Let (S(x. b} £ C(a. x5. T(max{a.1/x1 + 0.9/x3 + 1. y) = min {x + y. y)= Ð AND (x. c). x2. 1: Let T(x.0/x5 + 0.0/x4 + 0. x3. The operation union can be defined by the help of triangular conorms. 0} (A Ç B) (t) = max {A(t) + B(t 1.0/x7 B = 0.1] then we say that C is a compensatory operator.0/x5 + 0. x4.3/x6 + 0.0/x4 + 0.0/x4 + 1..0/x6 + 0..5/x3 +1. be the £ukasiewicz t-norm..1/x1 + 0.

We prove that whatever is the particular definition of an averaging operator. y Î[0.(6.. x).. M(1. 1] ® [0.(6. by allowing a positive compensation between ratings. y}. 1) = 1 Monotonicity M(x. 0) = 0.y) = M(y.(6. 6. This occurs in the presence of conflicting goals. M. . Union produces a high output whenever any one of the input values representing degrees of satisfaction of different features or criteria is high.(6. if x £ x¢ and y £ y¢ . y) £ M(x¢. union connectives provide full compensation and intersection connectives provide no compensation. y} £ M(x. Intersection connectives produce a high output only when all of the inputs have high values... y¢). 1] Extremal conditions M(0. "x. These connectives can be categorized into the following three classes union. 1] satisfying the following properties Idempotency M(x. "x Î[0. y Î[0.58 FUZZY LOGIC AND NEURAL NETWORKS 6. Averaging operators realize trade-offs between objectives.34) M is continuous..6 AVERAGING OPERATORS A typical compensatory operator is the arithmetical mean defined as MEAN (a.1 An Averaging Operator is a Function M : [0. 1] ´ [0.. 1] Commutativity M(x. y) £ max {x. b) = a+b 2 .. Averaging operators represent a wide class of aggregation operators. the global evaluation of an action will lie between the worst and the best local ratings: Lemma 6. In a decision process the idea of trade-offs corresponds to viewing the global evaluation of an action as lying between the worst and the best local ratings.29) Fuzzy set theory provides a host of attractive aggregation connectives for integrating membership values representing uncertain information...5: If M is an averaging operator then min {x.31) .6.32) . when a compensation between the corresponding compabilities is allowed. "x.(6.33) .. 1].(6. Compensative connectives have the property that a higher degree of satisfaction of one of the criteria can compensate for a lower degree of satisfaction of another criteria to a certain extent. x) = x..30) . intersection and compensation connectives.. In the sense.

1) An important family of averaging operators is formed by quasi-arithmetic means M(a1.. y}. the quasi-arithmetic mean of a1 and a2 is defined by M(a1. The only associative averaging operators are defined by Ry | M(x. y.(1 . y} Which ends the proof..x )(1 . For example. Table 6.(6.2 xy ) (2 .1 Name Harmonic mean Geometric mean Arithmetic mean Dual of geometric mean Dual of harmonic mean Median Generalized p-mean Mean operators M(x. y) and M{x.an) = f 1 F1 I GH n å f (a )JK n 1 i =1 This family has been characterized by Kolmogorov as being the class of all decomposable continuous averaging operators.. y. a) = Sa |x T if if if x£ y£a x£a£ y a£x£y .y ) ( x + y .x .y ) med (x.(6. y) 2xy ( x + y ) xy (x + y ) 2 1 . y). A strictly increasing averaging operator cannot be associative. y. y}) = max {x.FUZZY RULE-BASED SYSTEMS 59 Proof: From idempotency and monotonicity of M it follows that min {x. aÎ(0. Averaging operators have the following interesting properties: Property 1... y}) £ M(x. max {x. Property 2. 1) e(x p + y p ) 2j 1/p . p³1 .36) The next table shows the most often used mean operators. a2) = f 1 FG f (a ) + f (a ) IJ H 2 K 1 2 . a). min {x. y} = M(min {x. y} £ M(max {x..35) where aÎ(0.. a) = med (a.

a2... an) = max {a1. an) £ max {a1... in particular an aggregate ai is not associated with a particular weight wi but rather a weight is associated with a particular ordered position of aggregate. an) = T T å wjbj j =1 n .60 FUZZY LOGIC AND NEURAL NETWORKS 6... 0. 1/n) and FA(a1.(6. For any OWA operator F holds F*(a1.40) A number of important properties can be associated with the OWA operators. .. From the above it becomes clear that for any F min {a1.0)T and F*(a1. One sees aggregation in neural networks. Furthermore F(a1.37) . fuzzy logic controllers.4. a2..an) = w1b1 + w2b2 +.. an) £ F(a1..... an} FA: In this case W = WA = (1/n ..... a2.wn)T such as wi Î[0.(6... a2. 0. 0...6.. then F(0. 1 £ i £ n and w1 + w2 +. a2. . 0) and F*(a1.2 = 0.. 1..... When we view the OWA weights as a column vector we shall find it convenient to refer to the weights with the low indices as weights at the top and those with the higher indices with weights at the bottom....7 + 0. . . We shall now discuss some of these.+ wnbn = where bj is the j-th largest element of the bag (a1..+ wn = 1. w2.2 ´ 0. A fundamental aspect of this operator is the re-ordering step. 0.... that has an associated weighting vector W = (w1. a2... a2...6 + 0...2 Ordered Weighted Averaging The process of information aggregation appears in many applications related to the development of intelligent systems. .. In 1988 Yager introduced a new aggregation technique based on the ordered weighted averaging (OWA) operators.41) Thus the upper an lower star OWA operator are its boundaries.. an). vision systems. It is noted that different OWA operators are distinguished by their weighting function. a2.(6.2.. an} F*: In this case W = W* = (1. .. a2..75.1)T....6) = 0...39) a1 + .2. 0. a2.. An OWA operator of dimension n is mapping F: Ân ® Â..(6. . an) = min {a1..3.. a2... + an n .. expert systems and multi-criteria decision aids. a2.. . 0. 1]...(6.3: Assume W = (0.(6... . an} . In 1988 Yager pointed out three important special cases of OWA aggregations: F*: In this case W = W* = (1.3 ´ 0....1 ´ 0. Example 6.7. 0. .42) .. an) £ F *(a1.... an) .38) ... an} £ F(a1.4 ´ 1 + 0.. .

. ..6: Let us consider the vectors W * = (1... . If ai = a for all i then for any OWA operator F(a1... Furthermore. .. . Assume ai and ci are a collection of aggregates. .. .(6. dn} be any permutation of the ai. 4: A window type OWA operator takes the average of the m arguments around the center.FUZZY RULE-BASED SYSTEMS 61 The OWA operator can be seen to be commutative... Lemma 6. 0... a measure of orness.. an) = a.. note that the nearer W is to an or... an) ³ (c1. For this class of operators we have R0 |1 w = S m |0 T i if if if i<k k £i £k +m i³ k +m 1/m 1 k k+m–1 n Fig.. dn) .(6.. Then for any OWA operator F(a1... Another characteristic associated with these operators is idempotency.. cn) ... a2. an) = F(d1.. . c2.1 Window type OWA operator. . associated with any vector W is introduce by Yager as follows orness (W) = 1 n-1 å (n i)wi i =1 n It is easy to see that for anyW the orness(W) is always in the unit interval... Let (a1. a2.43) A third characteristic associated with these operators is monotonicity. the closer is to zero. i = 1. Example 6. 0)T.. 0.. an) be a bag of aggregates and let {d1.45) From the above we can see the OWA operators have the basic properties associated with an averaging operator. W* = (1. d2. . 1/n)T. .(6... In order to classify OWA operators in regard to their location between and and or.44) where F is some fixed weight OWA operator.. 6. Then F(a1. 0)T and WA = (1/n. ai ³ ci. a2. n such that for each i. . while the nearer it is to an and...... the closer its measure is to none. .....

i ) w + ( n .6 = 0. wk e.5 and when much of the weights are non-zero near the bottom.6 and 3 andness (W) = 1 orness (W) = 1 0.. Then orness (W) = 1 (2 ´ 0. that is.5.0b3 where bj is the j-th largest element of the bag (a1.. a2. A measure of andness is defined as andness (W) = 1 orness (W)..k ) e i 1 i i 1 orness (W¢ ) = orness (W ) + since k > j.j ) e . that is. wn)T where e > 0. ...0)T. andness (W) ³ 0.. orness (W ¢ ) > orness (W ). . Example 6.46) (Yager..2b2 + 0..8b1 + 0.1 å ( n . orness (W*) = 0 and orness (WA) = 0. Theorem 6. an OWA opeartor with much of non-zero weights near the top will be an orlike operator. Generally.8.. a3) is an or like aggregation.2) = 0. wn)T orness (W ¢) = 1 n-1 å ( n . orness (W) ³ 0.2.5: Let W = (0.. the OWA operator will be andlike. 0. j < k.(6.. a2.1: and W ¢ = (w1. The following theorem shows that as we move weight up the vector we increase the orness. .5.i ) w¢ = n . a3) = 0.( n . 0. Then orness (W¢ ) > orness (W) Proof: From the definition of the measure of orness we get .8 + 0. defined by F(a1.. 1993) Assume W and W ¢ are two n-dimensional OWA vectors such that W = (w1.62 FUZZY LOGIC AND NEURAL NETWORKS Then it can easily be shown that orness (W*) = 1.. 1 e(k j) n-1 . while moving weight down causes us to decrease orness(W).4 This means that the OWA operator. wj + e.

.. C1(w)} for all w.49) . w)} = sup min { x0 (u). 6. if we use Mamdanis implication operator in the GMP then Rule 1: Fact: consequence: if x is A1 then z is C1 x is x0 z is C where the membership function of the consequence C is computed as C(w) = sup min { x0 (u). the supremum turns into a simple minimum C(w) = min { x0 (x0) Ù A1 (x0) Ù C1(w)} = min {1 Ù A1(x0) Ù C1(w)} = min {A1(x0).48) for all w...2 Fuzzy singleton. Suppose now that the fact of the GMP is given by a fuzzy singleton. "u ¹ x0. Then the process of computation of the membership function of the consequence becomes very simple. (A1 ® C1) (u.(6.. C1(w)}} u u . For example... Observing that x0 (u) = 0.7 MEASURE OF DISPERSION OR ENTROPY OF AN OWA VECTOR In 1988 Yager defined the measure of dispersion (or entropy) of an OWA vector by disp (W) = å w ln w i i i .47) We can see when using the OWA operator as an averaging operator Disp (W) measures the degree to which we use all the aggregates equally. min {A1(u).(6.(6. 1 x0 X0 Fig.FUZZY RULE-BASED SYSTEMS 63 6. We can see when using the OWA operator as an averaging operator Disp(W) measures the degree to which we use all the aggregates equally.

. 3: Inference with Mamdanis implication operator...(6.. So. (A1 ® C1) (u.4 u Inference with Godel implication operator.64 FUZZY LOGIC AND NEURAL NETWORKS A1 C1 C A1(x0) X0 U W Fig.(6. w)} = A1(x0) ® C1(w) u .. 6.. If we use Godel implication operator in the GMP. then C(w) = sup min { x0 (u). 6.52) where the membership function of the consequence C is computed as C(w) = sup min { x0 (u)..51) C A1 C1 X0 Fig. (A1 ® C1) (u.50) for all w. w)} = A1(x0) ® C1(w) u for all w. C(w) = R1 SC (w) T 1 if A1 ( x0 ) £ C1 ( w) otherwise . W Rule 1: Fact Consequence: if x is A1 then z is C1 x is x0 z is C .(6.

.. Then combine the C¢1 component wise into C¢ by some aggregation operator: C = U C¢ = x0 o R1 È .58) C(w) = A1(x0) ® C1(w) Ú .(6..(6. Find C from the input x0 and from the rule base R = {R1.56) .(6.. C¢1 is called the output of the i-th rule C ¢ (w) = Ai(x0) ® Ci(w) 1 for each w..55) ..FUZZY RULE-BASED SYSTEMS 65 Consider a block of fuzzy IF-THEN rules R1 : also R2: also also Rn : fact: Consequence: The i-th fuzzy rule from this rule-base Ri : if x is Ai then z is Ci is implemented by a fuzzy implication Ri and is defined as Ri(u. Ú An(x0) ® Cn(w) .(6...... w) = (Ai ® Ci)(u. n.(6........54) .(6.. w) = Ai(u) ® Ci(w) for i = 1..57) . .. Rn} Interpretation of sentence connective also implication operator then compositional operator o We first compose x0 with each Ri producing intermediate result C¢1 = x0 o Ri for i = 1...... È x0 o Rn 1 i =1 n if x is A1 then z is C1 if x is A2 then z is C2 if x is An then z is Cn x is x0 z is C . n..53) .

. ÈC¢ 1 n Overall system output = union of the individual rule outputs.(6.9 LARSEN SYSTEM For the Larsen (Fig..59) 6...(6...61) (a ® b = a Ù b) input to the system is x0 fuzzified input is x0 firing strength of the i-th rule is Ai(x0) the i-th individual rule output is C¢ (w) = Ai(x0) Ù Ci(w) 1 overall system output (action) is C(w) = V Ai(x0) Ù Ci(w) i =1 n ..(6..8 For the Mamdani (Fig..66 FUZZY LOGIC AND NEURAL NETWORKS So.5) system MAMDANI SYSTEM .(6.... 6.6) system (a ® b = ab) input to the system is x0 fuzzified input is x0 firing strength of the i-th rule is Ai(x0) the i-th individual rule output is C ¢ (w) = Ai(x0) Ci(w) 1 overall system output (action) is C(w) = V Ai(x0) Ci(w) i =1 n .66) . 6.... the inference process is the following input to the system is x0 fuzzified input is x0 firing strength of the i-th rule is Ai(x0) the i-th individual rule output is C¢1(w): = A1(x0) ® C1(w) overall system output (action) is C = C¢ È ..(6. .62) ..64) ..(6.65) .(6.(6.63) 6.60) .

specifying a possibility distribution of the (control) action. namely: .5 Illustration of Mamdani system. A1 A1(X0) C1 C ¢1 A2 A2(X0) X0 C2 C ¢2 C = C ¢2 Fig.FUZZY RULE-BASED SYSTEMS 67 A1 C1 C ¢1 Degree of match X0 Individual rule output A2 C2 = C ¢2 Degree of match X0 Individual rule output Overall system output Fig. 6. 6.10 DEFUZZIFICATION The output of the inference process so far is a fuzzy set. one must defuzzify the fuzzy control action (output) inferred from the fuzzy reasoning algorithm. 6.6 Illustration of Larsen system. a non-fuzzy (crisp) control action is usually required. Consequently. In the on-line control.

pp. What is defuzzification? REFERENCES.68 FUZZY LOGIC AND NEURAL NETWORKS z0 = defuzzifier (C) . Explain Larsen rule-based system. Schwartz and A. 1963. Associative functions and abstract semigroups. y) £ T(x. Debrecen. 1960. 15. 20. Defuzzification is a process to select a representative element from the fuzzy output C inferred from the fuzzy control algorithm. B. What are the averaging operators? What are the important properties of averaging operators? Explain order weighted averaging with an example. Schwartz and A. 10. B. 1] 8. Sklar. 6. 1961. 4. Debrecen. 11. 10. 1]. "x. Publication Mathematics. b Î [0.. 313-334. 9. 14. 169-186. Vol. b) £ STRONG (a. What is entropy of an ordered weighted averaging (OWA) vector? Explain the inference with Mamdanis implication operator. y}. 12. 1. Explain the inference with Godels implication operator. Vol. Prove the following statement TW (x. 19. 7. Prove the following statement: max {a. Pacific Journal of Mathematics. 17. 18.. b).67) where z0 is the crisp action and defuzzifier is the defuzzification operator. " a. y Î[0. Schwartz and A. Statistical metric spaces. Associative functions and statistical triangle inequalities. Sklar. Explain Mamdani rule-based system. Publication Mathematics. B. y) £ min{x. 13. 1. 16. What is t-norm? What are the properties to be satisfied by a t-norm? What are the various basic t-norms? What is t-conorm? What are the properties to be satisfied by a t-conorm? What are the various basic t-conorms? Let T be a t-norm. b} £ (S(a. 69-81. 10. QUESTION BANK. 3. Explain the Measure of dispersion. 3. 2. What is t-norm based intersection? Explain with an example. Sklar. Let S be a t-conorm. 2. What is t-conorm based union? Explain with an example. pp. pp. Vol. 8.(6. 5. .

2.F. 275-29358. Fuzzy rule generation methods for high-level computer vision. 113-127. 5. Vol. 51. Takefuji. 7. R. 3. F. No. Yagar. Yagar. 1982. No. Dubois and H. Lim and T. 16. No. 16. Fuzzy Sets and Systems. 195199. 59. 1.R. pp. A fuzzy logic rule-based automatic target recognition. Fodor. No. Keller.H. 23. Fuzzy Sets and Systems. pp. 1992.R. Krishanpuram. No. and J. pp. No. Fuzzy Sets and Systems. 6.M. Cybernetics and Systems. Fuzzy Sets and Systems. 1. Mendel. 1992. 17. J. 103-122. R. J. 267-275.L. Applied Intellignece. No. pp. Vol. 2. 5. M. Czogala and W. pp. 12. No. 9. 687-687. Anderson. Wang. 295-312. 1991. Strong truth and rules of inference in fuzzy logic and approximate reasoning. Vol. 40. 3-13. 60. Qi. 335-351.H. 1. 245-258. Generating fuzzy rules by learning through examples. pp. International Journal of Man-machine Studies. Theory of t-norms and fuzzy inference methods. 7. Vol. International Journal of Intelligent Systems. 207-229. Information Sciences. Generating rules for fuzzy logic controllers by functions. Gradual inference rules in approximate reasoning. 1991. 6. 4. 15-23. Vol. pp. Vol. Fuzzy rule generation for fuzzy control. 3.T. Stochastica. Vol. 431-450.J. 15. Filev and R. E. 65-86. Fuzzy Sets and Systems.R. D. IEEE Transactions on Systems. No. 5.M. pp. 1. Fuzzy control rules and their natural control laws. Vol. No. Vol. Use of rule-based system for process control. 3. 1. Cao. pp. Implementing fuzzy rule-based systems on silicon chips. 1. Novak and W. 21. 1992. M. Man and Cybernetics. Pedrucz. L. 13. Vol. 48. pp. 1992. Pedrycz. No. Vol. pp. 36. A remark on constructing t-norms. International Journal of Intelligent Systems. IEEE Control Systems Magazine. 3. 1992. Coben and M. 6. 83-89. Vol. 1982. pp. Vol. 1. R. 10. 29. pp. Gupta and J. 23-63.R. On computation of the compositional rule of inference under triangular norms. Yagar. 1990. Nafarich and J.A. Vol. 1. Vol.M. International Journal of Intelligent Systems. J. A generalized defuzzification method via bad distributions. Vol. 1991. Input-output mathematical model with t-fuzzy sets. No. No. Bouslama and A Ichikawa. Peng. pp. 19. pp. Vol. 1988. 41. pp. 1992. Approximate reasoning with IF-THEN-UNLESS rule in a medical expert system. No. 71-79. 1991.P.FUZZY RULE-BASED SYSTEMS 69 4. No. 1985. 8. 6. 3. 7. pp. No. 8.J. No. 22. Cybernetics and Systems. Vol. Fuzzy Sets and Systems. D. B. 14. 48. Yagar. M. A general approach to rule aggregation in fuzzy logic control. V. 1993. 31-45. Fuller and H. No. Prade. Zimmerman. No. X. Bukley. 13. 18. 11. A. Vol.X. 1.E. Vol. pp. Fuzzy Sets and Systems. 61.C. F. Hudson. 1990. 3. Fuzzy sets and t-norms in the light of fuzzy logic. A general theory of uncertainty based on t-conorms. D. 1414-1427. 1988. Measures of fuzziness based on t-norms. 1992. Rhee and R. . IEEE Expert.C. pp. No. R. Fuzzy Sets and Systems. Bernard. 6. 20. 289-296. 1993. pp. 22.

Sudkamp. Vol. Fuzzy Sets and Systems. Rule-based fuzzy classification for software quality control. pp. 29-53. Vol. pp. Turksen. No. interpolation and fuzzy rule construction. 11. 32. Uchino. 1994. International Journal of Uncertainity. Bonissone. 1. No. 1993. pp. pp. V. Yamakawa. 3. Patterns of fuzzy-rule based interference. 11. 59. 3. 163-204. Y. On measuring the specificity of IF-THEN rules. Hellendoom. International Journal of Approximate Reasoning. Fuzzy Sets and Systems. Combination of rules or their consequences in fuzzy expert systems. 2. P. 259-270. Fuzzy IF-THEN-UNLESS rules and their implementation. T. Dutta and P. 58. 31. 73-86. 8. 3-40. 25. pp. 28. Fuzziness and Knowledge-based Systems. 1. 27. pp. Vol.B. 3. E. . 33. A rule-based method to calculate exactly the widest solutions sets of a max-min fuzzy relations inequality. 167-182. pp. Vol. Vol. Tano. 58. S. 39-58. 1993. 64. pp. Vol. W. 1993. pp. pp. No. T. 26. 1994. 3. Cross and T. Why triangular membership functions? Fuzzy Sets and Systems. Sudkamp. International Journal of Approximate Reasoning. Fuzzy Sets and Systems. 1993. 1993. International Journal of Approximate Reasoning. Fuzzy Sets and Systems. No.P. Pedrycz. Kacprzyk. 1. No. Miki and S. 1. Vol. Driankov and H.Tian and I. Doherry.70 FUZZY LOGIC AND NEURAL NETWORKS 24. C. Vol. 30. 29. T. 64. No. 349-358. Integrating case and rule-based reasoning. 63. 235-255. No. No. Vol. Similarity. 1. J. Arnould and S. 1994. 1994. 1. Ebert. P. No. No. 1994. Fuzzy rule-based simple interpolation algorithm for discrete signal. Nakamura. T. Vol. 21-30. Fuzzy Sets and Systems.

it is exemplified by the generalized modus tollens form of logical inference. 7.... In this case.4 Fuzzy Reasoning Schemes 7 7.. since only the rules leading to the objective need to be evaluated.. which then uses them to evaluate relevant production rules and draw all possible conclusions.. it is commonly called backward chaining. There exist two approaches to evaluating relevant production rules. The first is data-driven and is exemplified by the generalized modus ponens.. it is commonly called forward chaining.. the expert system searches for data specified in the IF clauses of production rules that will lead to the objective. since the goal-driven method proceeds backward from THEN clauses to the IF clauses. Since the data-driven method proceeds from IF clauses to THEN clauses in the chain through the production rules. in its search for the required data.1 INTRODUCTION This chapter focuses different inference mechanisms in fuzzy rule-based systems with examples. Similarly.+ 0 ) 2 6 . The inference engine of a fuzzy expert system operates on a series of production rules and makes fuzzy inferences. Rn : if x is An and y is Bn then z is Cn z is C x is x0 and y is y0 if x is A1 and y is B1 then z is C1 if x is A2 and y is B2 then z is C2 . available data are supplied to the expert system. Backward chaining has the advantage of speed. Here. An alternative method of evaluation is goal-driven.. or by querying the user. in the THEN clauses of other production rules..2 FUZZY RULE-BASE SYSTEM R1 : R2 : .. these data are found either in the knowledge base.

(7.(7..4) C(w) = Ai(x0) ´ Bi(y0) ® C1(w) Ú . Ú An(x0) ´ Bn(y0) ® Cn(w) input to the system is (x0. Here C¢ is called the output of the i-th rule 1 C¢(w) = [Ai(x0) Ù Bi(y0)] ® Ci(w) 1 for each w. y0 ) firing strength of the i-th rule is Ai(x0) Ù Bi(y0) the i-th individual rule output is C¢ (w): = A1(x0) Ù B1(x0) ® C1(w) 1 overall system output is C = C¢ È .1) .. Then combine the C¢ component wise into C¢ by some aggregation operator: 1 C = U C¢ = x0 ´ y0 o R1È . È x0 ´ y0 o Rn 1 i =1 n . ..72 FUZZY LOGIC AND NEURAL NETWORKS The i-th fuzzy rule from this rule-base Ri : if x is Ai and y is Bi then z is Ci is implemented by a fuzzy relation Ri and is defined as Ri(u... v.. w) = [Ai(u) Ù Bi(v)] ® Ci(w) for i = 1. . For simplicity we assume that we have two fuzzy IF-THEN rules of the form ....(7....(7.. Find C from the input x0 and from the rule base R = {R1......(7.... n.2) .5) 7. . w) = (Ai ´ Bi ® Ci)(u. y0) fuzzified input is ( x0 . Rn} Interpretation of logical connective and sentence connective also implication operator then compositional operator o We first compose x0 ´ y0 with each Ri producing intermediate result C¢ = x0 ´ y0 o Ri 1 for i = 1. ... ÈC¢ . i n overall system output = union of the individual rule outputs. n.3) ...3 INFERENCE MECHANISMS IN FUZZY RULE-BASE SYSTEMS We present five well-known inference mechanisms in fuzzy rule-based systems.

are computed by a1 = A1(x0) Ù B1(y0). 2. a2 = A2 (x0) Ù B2(y0) .(7.FUZZY REASONING SCHEMES 73 R1 : also R2 : fact: Consequence: if x is A1 and y is B1 then z is C1 if x is A2 and y is B2 then z is C2 x is x0 and y is y0 z is C 7. i = 1.1 Mamdani Inference Mechanism The fuzzy implication is modelled by Mamdanis minimum operator and the sentence connective also is interpreted as oring the propositions and defined by max operator.(7. a2 = A2 (x0) Ù B2(y0) The individual rule outputs are obtained by C 1 (w) = (a1 Ù C1(w)). 7. 2...2 Tsukamoto Inference Mechanism All linguistic terms are supposed to have monotonic membership functions.6) A1 B1 C1 u B2 v C2 w A2 x0 u y0 v Min w Fig. 7. denoted by ai. denoted by ai.(7. we employ any defuzzification strategy. are computed by a1 = A1(x0) Ù B1(y0)...9) . The firing levels of the rules..1 Inference with Mamdanis implication operator. .(7. i = 1.3.8) .7) ..3. The firing levels of the rules... to obtain a deterministic control action. C 2 (w) = (a2 Ù C2(w)) ¢ ¢ Then the overall system output is computed by oring the individual rule outputs C(w) = C¢ (w) Ú C 2 (w) = (a1 Ù C1(w)) Ú (a2 Ù C2(w)) ¢ 1 Finally.

(7. C2(z2) = 0...6) = 6 ...6)/(0.3 + 0.e.6 and the crisp control action is z0 = (8 ´ 0. n Example 7.8 It follows that the firing level of the second rule is a2 = min{A2(x0).1: We illustrate Tsukamotos reasoning method by the following simple example R1 : also R2 : fact: Consequence: if x is A2 and y is B2 then z is C2 x is x0 and y is y0 z is C if x is A1 and y is B1 then z is C1 Then according to the figure we see that A1(x0) = 0..6 The individual rule outputs z1 = 8 and z2 = 4 are derived from the equations C1(z1) = 0.3. B2(y0) = 0.3} = 0. 0. z0 is computed by the discrete Center of-Gravity method.74 FUZZY LOGIC AND NEURAL NETWORKS In this mode of reasoning the individual crisp control actions z1 and z2 are computed from the equations a1 = C1(z1).12) i where ai is the firing level and zi is the (crisp) output of the i-th rule.6.6. B2(y0)} = min{0.. i = 1.11) i. B1(y0) = 0.10) a1z1 + a 2 z2 a1 + a 2 . 0.3 + 4 ´ 0..7. a2 = C2(z2) and the overall crisp control action is expressed as z0 = .3 and from A2(x0) = 0.3 Therefore. the firing level of the first rule is a1 = min{A1(x0).. If we have n rules in our rule-base then the crisp control action is computed as n z0 = åaz i =1 n i i åa i =1 .(7.7.8} = 0...(7. B1(y0)} = min{0.

(7. n.(7.. 7..3..6 Z2 = 4 w Fig.15) If we have n rules in our rule-base then the crisp control action is computed as z0 = åa z i =1 n n * 1 i åa i =1 ...14) if x is A1 and y is B1 then z1 = a1x + b1y Sugeno and Takagi use the following architecture The firing levels of the rules are computed by a1 = A1(x0) Ù B1(y0).(7. .8 Y0 v Min 0.2 Tsukamotos inference mechanism.3 v Z1 = 8 C1 w A2 B2 C2 0.(7. i = 1..3 u B1 0. a2 = A2(x0) Ù B2(y0) then the individual rule outputs are derived from the relationships z* = a1x0 + b1y0..6 X0 u 0. . 7. z * = a2x0 + b2y0 1 2 and the crisp control action is expressed as z0 = * * a1z1 + a 2 z2 a1 + a 2 ..16) i where ai denotes the firing level of the i-th rule.13) ..3 Sugeno Inference Mechanism R1 : also R2 : fact: Consequence: if x is A2 and y is B2 then z2 = a2x + b2y x is x0 and y is y0 z0 ..FUZZY REASONING SCHEMES 75 A1 0..7 0.

6) = 4. z* = 2x0 y0 = 2 ´ 3 2 = 4 1 2 So the crisp control action is z0 = (5 ´ 0. the firing level of the first rule is a1 = min {mBIG (x0).9} = 0. 7.76 FUZZY LOGIC AND NEURAL NETWORKS A1 A2 u v a1 a1x + b1y B1 B2 a2 x u y v Min a2x + b2y Fig. mBIG (y0)} = min {0.8 mSMALL (y0) = mSMALL (2) = 0.6.6. 0. mSMALL (y0)} = min {0.2 + 4 ´ 0.25 .3 Sugenos inference mechanism.2 Therefore.6 The individual rule outputs are computed as z * = x0 + y0 = 3 + 2 = 5.6)/(0.9 It follows that the firing level of the second rule is a2 = min {mMEDIUM (x0).2 and from mMEDIUM (x0) = mMEDIUM (3) = 0.8.2 + 0. 0.2: We illustrate Sugenos reasoning method by the following simple example R1 : also R2 : fact : Consequence: if x is MEDIUM and y is BIG then z2 = 2x y x is 3 and y is 2 z0 if x is BIG and y is SMALL then z1 = x + y Then according to the figure we see that mBIG (x0) = mBIG (3) = 0. Example 7. mBIG (y0) = mBIG (2) = 0.2} = 0.

6 2x – y = 4 Fig. 7.18) .3. a2 = A2(x0) Ù B2(y0) Then membership function of the inferred consequence C is pointwise given by C(w) = (a1C1(w)) Ú (a2C2(w)) To obtain a deterministic control action.6 0..19) where ai denotes the firing level of the i-th rule.4 Example of Sugenos inference mechanism..8 0..(7. 7. 2 a1 = A1 (x0) Ù B1(y0). If we have n rules in our rule-base then the consequence C is computed as C(w) = V Ú (ai Ci(w)) i =1 n .17) .... we employ any defuzzification strategy. n 7. i = 1.FUZZY REASONING SCHEMES 77 1 0. Let us denote ai the firing level of the i-th rule.2 u v a1 = 0.2 x+y=5 1 0.3.4 Larsen Inference Mechanism The fuzzy implication is modeled by Larsens product operator and the sentence connective also is interpreted as oring the propositions and defined by max operator.5 Simplified Fuzzy Reasoning R1 : also R2 : fact: Consequence: if x is A2 and y is B2 then z2 = C2 x is x0 and y is y0 z0 if x is A1 and y is B1 then z1 = C1 . i = 1. ...9 3 u 2 v Min a2 = 0.(7..(7.

78 FUZZY LOGIC AND NEURAL NETWORKS A1 B1 C1 u A2 B2 v C2 w X0 Fig.. i = 1. 7. 7..(7.. The firing levels of the rules are computed by a1 = A1(x0) Ù B1(y0). L1 H2 L3 a1 C1 M1 M2 M3 a2 C2 H1 H2 H3 a3 Min Z3 Fig..(7..5 u Y0 v Min w Inference with Larsens product operation rule.. .(7..22) i where ai denotes the firing level of the i-th rule..6 Simplified fuzzy reasoning. ..20) a1c1 + a 2 c2 a1 + a 2 n . n. a2 = A2(x0) Ù B2(y0) then the individual rule outputs are c1 and c2.21) If we have n rules in our rule-base then the crisp control action is computed as z0 = åa C i =1 n i i åa i =1 . and the crisp control action is expressed as z0 = .

No. 23-37. 12. 163-175. What are the different approaches to evaluating relevant production rules? Explain them. Fuzzy Sets and Systems. Pedrycz. Fuzzy logic and reasoning. Synthese. 1976. 3. pp.FUZZY REASONING SCHEMES 79 QUESTION BANK. 9. 7. 270-276. 465-480. Sugeno and T. 21.A. 11. E. Vol. Explain Sugeno inference mechanism.H. Approximate Reasoning in Intelligent Systems. The concept of a linguistic variable and its application to approximate reasoning III. pp. Vol. Turksen. Sanchez and L. 5. Takagi. Baldwin. 10. Zadeh.A. Mamdani and B. Prade. Explain simplified reasoning scheme. Turksen. A method of inference in approximate reasoning based on interval-valued fuzzy sets. Information Sciences. Explain Larsen inference mechanism. Man and Cybernetics. 6. pp. Vol.H. 26. 1987. Four methods of approximate reasoning with interval-valued fuzzy sets. Farreny and H. pp. Oxford. Default and inexact reasoning with possibility degrees. 1983. L. London. 199-251. 16. Approximate reasoning for production planning. Foundations of fuzzy reasoning. 1986. 1975. Vol. Zadeh. pp. Decision and Control. 6. 8. Fuzzy Sets and Systems. IEEE Transactions on Systems. Pergamon Press. Explain Mamdani inference mechanism. E. Fuzzy Sets and Systems. Mamdani. 1985. pp. Fuzzy Reasoning and Its Applications. I. 9. pp. W. Vol. Vol. Applications of fuzzy relational equations for methods of reasoning in presence of fuzzy data. Gaines. I. No. 26. Vol. 1975. 2. pp. pp. Man and Cybernetics. No. Fuzzy Sets and Systems. 11. L. 1-17. Vol. 8.B. 12. 2. 1988. No. REFERENCES. B. The concept of a linguistic variable and its application to approximate reasoning I. Zadeh. 1977. J. IEEE Transactions on Systems. 13.R. No. Zadeh. 4.B. U. Vol. 1. International Journal of Approximate Reasoning. No. M. 8. No. 1975. Zadeh.R. 2. 1. pp. 121-142. 313-325. Academic Press. Vol. 2. Fuzzy logic and approximate reasoning. Explain Tsukamoto inference mechanism. No. 2. M.B. 1987. pp.F. H. 3. 3. 43-80. L. 1. 1981. The concept of a linguistic variable and its application to approximate reasoning II. 15. L.A. 1979. 14.A. 8. No. 1. Gorzalczany. Vol. No. 4. Vol. 30. Information sciences. pp. 6. 301-357. 623-668. 1182-1191. International Journal of Man-machine Studies. 5. 407-428. 1. 4. International Journal of Man-machine Studies.A. Information Sciences. 9. Applications of fuzzy logic to approximate reasoning using linguistic systems. Multidi-mensional fuzzy reasoning. 3. 1989. E. Vol. Gaines.K. 16. pp. 1975. .

Z. 1992. 1991. Z. 17. Fuzzy sets in approximate reasoning. No. Ruspini. 22. Chun. 1992.E. R. 47-68. 1. Dutta.M. Approximate reasoning: past. 2. . Wang.80 FUZZY LOGIC AND NEURAL NETWORKS 16. 4. Basu and A. Vol. pp. A. Dutta. A. 143-202. 24. 77-81. IEEE Transactions on Fuzzy Systems. H. Turksen and M. Pawlak. pp. Approximate spatial reasoning: Integrating qualitative and quantitative constraints. No. No. No. Luo and Z. pp. 1993. Reasoning with imprecise knowledge to enhance intelligent decision support. Fuzzy reasoning in a multidimensional space of hypotheses. Vol.B. 756-770. 25. pp. Nakanishi. pp. E. An inference network for bidirectional approximate reasoning based on an equality measure. International Journal of Intelligent Systems. 1. 1990. 1990. Fuzzy Sets and Systems.L. Vol. Kluwer. Vol. 2. 1991. C. Vol. Vol. 27. Bostan. 1991. Kruse and E. Bien and M. 23. 1. 409-420.P. No. pp. 177-180. S. 307-330. pp. D. 19. 311-325. A review and comparison of six reasoning methods. No. pp. 1989. Z.G. Cybernetics and Systems. pp.H. 1. IEEE Transactions on Systems. 3. 18. Kandel and L. 1991. 21.Z. Vol. No. Part I: Inference with possibility distributions. pp. 20. Representation of compositional relations in fuzzy reasoning. 57. A new improved algorithm for inexact reasoning based on extended fuzzy production rules. No. No. 3. 1994. Rough sets: Theoretical aspects of reasoning about data. Schwecke. 297-317. International Journal of Approximate Reasoning. Information Sciences. I. Vol. Li. present. 5. pp. man and cybernetics. 1990. Vol.F. 36. 3. No. Coben and M. 57. 40. 257-294. Cao. future. Approximate reasoning with IF-THEN-UNLESS rule in a medical expert system. S. Vol. D. 7. Fuzzy Sets and Systems. 36. Prade. Anderson. 19. 4. International Journal of Approximate Reasoning. 71-79. M. Dubois and H. Chen. 5. Sugeno. Vol. 23. Fuzzy Sets and Systems. Fuzzy Sets and Systems. 26. A new model for fuzzy reasoning. Hudson.

8.(8.1 A basic feedback control system..1) The general form of the discrete-time control law is u(k) = f(e(k)...+ 0 ) 2 6 . parameter t defines the order of the controller. . . e represents the error between the desired set point y* and the output of the system y.. to be controlled. u(k 1).. despite the presence disturbances of the system parameters.1 INTRODUCTION Conventional controllers are derived from control theory techniques based on mathematical models of the open-loop process.. called system. y* e Controller u System y Fig. The output of the controller (which is the input of the system) is the control action u. The purpose of the feedback controller is to guarantee a desired response of the output y. and noise measurements.. e(k t)) providing a control action that describes the relationship between the input and the output of the controller. . 8.. is called regulation. e(k 1). The process of keeping the output y close to the set point (reference input) y*. f is in general a non-linear function. e(k t).2 BASIC FEEDBACK CONTROL SYSTEM .4 Fuzzy Logic Controllers 8 8.

The knowledge-based nature of FLC dictates a limited usage of the past values of the error e and control u because it is rather unreasonable to expect meaningful linguistic statements for e(k 3)... e(k t). The expert knowledge is usually of the form IF (a set of conditions are satisfied) THEN (a set of consequences can be inferred). several linguistic variables might be involved in the antecedents and the conclusions of these rules. a fuzzy control rule is a fuzzy conditional statement in which the antecedent is a condition in its application domain and the consequent is a control action for the system under control. Zadeh (1973) was introduced the idea of formulating the control algorithm by logical rules. Ai. . it does not mean that the FLC is a kind of transfer function or difference equation.2 Mamdani Type of Fuzzy Logic Control We can represent the FLC in a form similar to the conventional control law u(k) = f(e(k).3. 8. e(k 4).A. Since the antecedents and the consequents of these IF-THEN rules are associated with fuzzy concepts (linguistic terms). In a fuzzy logic controller (FLC).. y and z in the universes of discourse U.. .. . e(k 1). a rule-base.82 FUZZY LOGIC AND NEURAL NETWORKS 8. where x and y are the process state variables.. 8. equivalently. they are often called fuzzy conditional statements.. and W.3 FUZZY LOGIC CONTROLLER L. z is the control variable.. the dynamic behaviour of a fuzzy system is characterized by a set of linguistic description rules based on expert knowledge. and an implicit sentence connective also links the rules into a rule set or..1 Two-Input-Single-Output (TISO) Fuzzy Systems R1 : also R2 : also . Furthermore. fuzzy control rules provide a convenient way for expressing control policy and domain knowledge. respectively.2) where the function F is described by a fuzzy rule base. In our terminology.. u(k 1)..(8..3. . e(k t) . e(k t).. the system will be referred to as a multi-input-multioutput (MIMO) fuzzy system. When this is the case. also Rn : if x is An and y is Bn then z is Cn if x is A2 and y is B2 then z is C2 if x is A1 and y is B1 then z is C1 For example. fuzzy control rules have the form. V. Bi. and Ci are linguistic values of the linguistic variables x. However. in the case of two-input-single-output fuzzy systems. Basically.

3) N Error ZE P Fig.5) and is a manifestation of the general FLC expression with t = 1.FUZZY LOGIC CONTROLLERS 83 A typical FLC describes the relationship between the changes of the control Du(k) = u(k) u(k 1) On the one hand.(8... So.. The actual output of the controller u(k) is obtained from the previous value of control u(k 1) that is updated by Du(k) u(k) = u(k 1) + Du(k). if x is A2 and y is B2 then z is C2 if x is A1 and y is B1 then z is C1 ... ...(8....6) This type of controller was suggested originally by Mamdani and Assilian in 1975 and is called the Mamdani type FLC.(8. 8.4) . our task is the find a crisp control action z0 from the fuzzy rule-base and from the actual crisp inputs x0 and y0: R1 : also R2 : also .2 Membership functions for the error. such control law can be formalized as Du(k) = F(e(k).(8. A prototypical rule-base of a simple FLC realizing the control law above is listed in the following R1 : R2 : R3 : R4 : R5 : if e is "positive" and De is "near zero" then Du is "positive" if e is "negative" and De is "near zero" then Du is "negative" if e is "near zero" and De is "near zero" then Du is "near zero" if e is "near zero" and De is "positive" then Du is "positive" if e is "near zero" and De is "negative" then Du is "negative" . De(k)) . and the error e(k) and its change De(k) = e(k) e(k 1) On the other hand..

3 Fuzzy logic controller. 1 X0 . Crisp x in U Fuzzifier Fuzzy set in U Fuzzy rule base Fuzzy inference engine Fuzzy set in V Crisp y in V Defuzzifier Fig. and therefore.(8. Furthermore. 8. the inputs of fuzzy rule-based systems should be given by fuzzy sets. A fuzzification operator has the effect of transforming crisp data into fuzzy sets. we have to fuzzify the crisp inputs. In most of the cases we use fuzzy singletons as fuzzifiers fuzzifier (x0): = x0 where x0 is a crisp input value from a process.3) usually consist of four major parts: Fuzzification interface.3.84 FUZZY LOGIC AND NEURAL NETWORKS also Rn : input output if x is An and y is Bn then z is Cn x is x0 and y is y0 z0 Of course. .4 Fuzzy singleton as fuzzifier. and therefore to get crisp value we have to defuzzify it... Fuzzy inference machine and Defuzzification interface. 8. Fuzzy rulebase.3 Fuzzy Logic Control Systems Fuzzy logic control systems (Figure 8. the output of a fuzzy system is always a fuzzy set.7) X0 Fig. 8.

FUZZY LOGIC CONTROLLERS 85 Suppose now that we have two input variables x and y. a fuzzy system can be characterized by a single fuzzy relation which is the combination in question involves the sentence connective also..e.(8. To infer the output z from the given process states x. also Rn : if x is A1 and y is B1 then z is C1 if x is A2 and y is B2 then z is C2 . Since each fuzzy control rule is represented by a fuzzy relation. also Rn : if x is An and y is Bn then z is Cn input x is x0 and y is y0 z is C Consequence : . Symbolically. w) = [Ai(u) and Bi(v)] ® Ci(w) where the logical connective and is implemented by the minimum operator. if we have the collection of rules R1 : also R2 : also . In other words. Aggregate the individual rule outputs to obtain the overall system output.9) .. the overall behavior of a fuzzy system is characterized by these fuzzy relations... A fuzzy control rule Ri : if (x is Ai and y is Bi then (z is Ci) is implemented by a fuzzy implication Ri and is defined as R(u. Fuzzy control rules are combined by using the sentence connective also. y and fuzzy relations Ri.(8.v. i.. we can use any t-norm to model the logical connective and... Bi(v)] ® Ci(w)} Of course. [Ai(u) and Bi(v)] ® Ci(w) = [Ai (u) ´ Bi(v)] ® Ci(w) = min {[Ai(u)..8) if x is An and y is Bn then z is Cn The procedure for obtaining the fuzzy output of such a knowledge base consists from the following three steps: Find the firing level of each of the rules. Find the output of each of the rules. we apply the compositional rule of inference: R1 : if x is A1 and y is B1 then z is C1 also R2 : if x is A2 and y is B2 then z is C2 also .

In the on-line control.(8.(8..18) .17) Example 8. An(x0) ´ Bn(y0) ® Cn(w)} for all w Î W.(8. C. .. ...12) ...15) 8.. C¢ } for all w Î W. x0 ´ y0 o Rn) taking into consideration that ..(8.. is obtained from the individual rule outputs C i by C(w) = Agg {C¢ .. namely: z0 = defuzzifier (C) where z0 is the nonfuzzy control output and defuzzifier is the defuzzification operator...16) ... . ..11) .....È x0 ´ y0 o Rn) That is C(w) = A1(x0) ´ B1(y0) ® C1(w) V..(8.. 1 n .(8.. The procedure for obtaining the fuzzy output of such a knowledge base can be formulated as The firing level of the I-th rule is determined by Ai(x0) ´ Bi(y0) The output of the I-th rule is calculated by C¢ (w) = Ai(x0) ´ Bi(y0) ® Ci(w) for all w Î W 1 The overall system output. C = Agg ( x0 ´ y0 o R1 . .. Consequently. fact o Rn) That is.. ..... v ¹ y0 The computation of the membership function of C is very simple: .(8..(8. u ¹ x0 and y0 (v) = 0.14) C(w) = Agg {A1(x0) ´ B1(y0) ® C1(w).(8.. specifying a possibility distribution of control action..4 DEFUZZIFICATION METHODS The output of the inference process so far is a fuzzy set.. a nonfuzzy (crisp) control action is usually required..86 FUZZY LOGIC AND NEURAL NETWORKS Where the consequence is computed by consequence = Agg (fact o R1.1: If the sentence connective also is interpreted as oring the rules by using minimum-norm then the membership function of the consequence is computed as C = ( x0 ´ y0 o R1 È.. one must defuzzify the fuzzy control action (output) inferred from the fuzzy control algorithm.10) x0 (u) = 0.13) .V An(x0) ´ Bn (y0) ® Cn(w) for all w Î W.

The most often used defuzzification operators are: 8.e.3 Middle-of-Maxima The defuzzified value of a discrete fuzzy set C is defined as a mean of all values of the universe of discourse. 8..5 First-of-maxima defuzzification method..19) w The calculation of the Center-of-Area defuzzified value is simplified if we consider finite universe of discourse W and thus discrete membership function C (w) z0 = å z C(z )dz å c(z ) j j j .4.2 First-of-Maxima The defuzzified value of a fuzzy set C is its smallest maximizing element.1 Center-of-Area/Gravity The defuzzified value of a fuzzy set C is defined as its fuzzy centroid: z0 = z z w zC( z ) dz c( z) dz ..(8. having maximal membership grades z0 = 1 N åz j =1 n j .22) ..(8. 8.FUZZY LOGIC CONTROLLERS 87 Defuzzification is a process to select a representative element from the fuzzy output C inferred from the fuzzy control algorithm.4. i.21) Z0 Fig..4. z0 = min z C ( z ) = max C ( w) u R S T U V W ..(8.(8.20) 8...

the plausible control action depicted in Figure could be interpreted as turn right or left Both Center-of-Area and Middle-of-Maxima defuzzification methods result in a control action driveahead straightforward which causes an accident..25) a where [C]a denotes the a-level set of C as usually..(8... z0 Î z C( z ) = max C( w) w R S T U V W .4. If an obstacle occurs right ahead. . 8..24) 8.(8.4.. If C is not discrete then defuzzified value of a fuzzy set C is defined as z0 = z z G zdz dz ..5 Height Defuzzification The elements of the universe of discourse W that have membership grades lower than a certain level a are completely discounted and the defuzzified value z0 is calculated by the application of the Center-ofArea method on those elements of W that have membership grades not less than a: z0 = [C ] [ C ]a z z zC( z)dz c( z )dz . Z0 Fig. . 8.88 FUZZY LOGIC AND NEURAL NETWORKS where {z1. i.4 Max-Criterion This method chooses an arbitrary value.. Example 8.(8.2: Consider a fuzzy controller steering a car in a way to avoid obstacles..6 Middle-of-maxima defuzzification method.e. zN} is the set of elements of the universe W which attain the maximum value of C.23) G where G denotes the set of maximizing element of C. from the set of maximizing elements of C.

8.26) i2 2 i i3 i3 Singleton fuzzifier fuzzifier (x): = x .G MN 2 H b JK PQ L 1 F w .a I OP MN 2 GH b JK PQ L 1 F v .a I OP B (u) = exp M. A suitable defuzzification method would have to choose between different control actions (choose one of two triangles in the Figure) and then transform the fuzzy set into a crisp value.28) ... 8..G MN 2 H b JK PQ 2 Ai(u) = exp - i1 i1 2 i i2 . Wang (1992) showed that fuzzy logic control systems of the form Ri: if x is Ai and y is Bi then z is Ci.5 EFFECTIVITY OF FUZZY LOGIC CONTROL SYSTEMS Using the Stone-Weierstrass theorem. fuzzifier (y): = y Product fuzzy conjunction [Ai(u) and Bi(v)] = Ai(u) Bi(v) Product fuzzy implication (Larsen implication) [Ai(u) and Bi(v)] ® Ci(w) = Ai(u) Bi(v) Ci(w) (8.a I OP C (w) = exp M.(8.27) .(8. i = 1. ..FUZZY LOGIC CONTROLLERS 89 C Z0 Fig. n with Gaussian membership functions LM 1 F u .(8.29) .7 Undesired result by Center-of-Area and Middle-of-Maxima defuzzification methods...

33) .|c . he proved the following theorem Theorem 8.. n with Symmetric triangular membership functions R1 .w| g C (w) = exp S T0 Ai(u) = exp i i if |ai ..(8..... i = 1. Namely. R2.v| b B (v) = exp S T0 R1 . i.90 FUZZY LOGIC AND NEURAL NETWORKS Centroid defuzzification method z= åa i =1 n i =1 n i 3 Ai ( x ) Bi ( y ) å A ( x) B ( y ) i i .(8. R2.31) Castro in 1995 showed that Mamdanis fuzzy logic controllers Ri : if x is Ai and y is Bi then z is Ci..(8. are universal approximators.35) . (8. .w| £ g i otherwise .... . Rn} ..(8. Bi(v).u| a S0 T R1 .|b .v| £ bi otherwise if |ci ...34) ..32) i i i i i i Singleton fuzzier fuzzifier (x0): = x0 Minimum norm fuzzy conjunction [Ai(u) and Bi(v)] = min {Ai(u)Bi(v)} Minimum norm fuzzy implication [Ai(u) and Bi(v)] ® Ci(W) = min {Ai(u).36) .|a .30) where ai3 is the center of Ci.u| £ a i otherwise if |bi . they can approximate any continuous function on a compact set to arbitrary accuracy. . Ci(W)} Maximum t-conorm rule aggregation Agg (R1. there exists a fuzzy logic control system with output function f such that sup ||g(x) f(x)|| £ e x ÎU ..1 For a given real-valued continuous function g on the compact set U and arbitrary e > 0..e... (8... Rn ) = max {R1.(8.

H. 94. 3-4. pp. Explain Mamdani type of fuzzy logic controller. QUESTION BANK. Fuzzy Sets and Systems.M. 3. Selection of parameters for a fuzzy logic controller. No. E. pp. 3. 3. Machine Studies. 13. Vol. 1.J. 3. No. Control problems in fuzzy systems. 1971. M. E. No.H. 3. 185-199. 6. 6. Measurement and Control. . No. No. Lee. Assilian. Mamdani. 4. 8. Mamdani. 1977. What is the effectivity of fuzzy logic control systems? REFERENCES.J. Fuzzy Sets and Systems. 1-13. What is fuzzy logic controller? Explain two-input-single-output fuzzy system. 9. No. pp. 2. Zadeh. 4.H. An experiment in linguistic synthesis with a fuzzy logic controller. Brase and D. 235-242.FUZZY LOGIC CONTROLLERS 91 Centroid defuzzification method z= å c min { A ( x) B ( y)} i i i i =1 n n å min { A ( x) B ( y)} i i i =1 . 5. 7. C. 13. King and E.(8. W. What are the various parts of fuzzy logic control system? Explain them. Selection of parameters for a fuzzy logic controller. Vol. Fuzzy sets and systems. pp. International Journal of Machine Studies.A. Vol. Analysis of a fuzzy logic controller. Mamdani and S. Automatica. No. Pedrycz. No. 3. pp. Kickert and E. Advances in the linguistic synthesis of fuzzy controllers. E. 2. L. 1. 2. 669-678. Rutherford. 7. 1976. 8.A. P. Vol. 1. No. Vol. 1979. 29-44. Pedrycz. E. Vol.H. 275-293. Cybernetics and Systems. 1. Fuzzy Sets and Systems. 1982. pp. 1978. a rationale for fuzzy control.C. What are the various defuzification methods? Explain them. 257-274. 3. pp. Fuzzy rule generation for fuzzy control. International Journal of Man. Czogala and W. Mamdani. 7. Vol. Czogala and W. Vol..37) where ci is the center of Ci are also universal approximators. pp. 2. 1. 1979. 185-199. Vol. 5. 1. 1975. The application of fuzzy control systems to industrial process.. pp. 1982. Journal of dynamical systems. 6.

Yager. pp. Control of dynamic systems using fuzzy learning algorithm. X. 2. No.M. Chen. 8. Boullama and A. 2.C. 4. pp. B. Man and Cybernetics. 11. Sugeno. No.S. 127130. Stability analysis and design of fuzzy control systems. 159-168. Bladwin and N. Vol. 1984. No. Use of rule-based system for process control. 1. 26. 1. No. Vol. 1985. pp. 1825-1848. J. Man and Cybernetics. Kiszks and G. 223-229. Infromation Sciences. 1992. Multivariable structure of fuzzy control systems. Vol. 57. 99-111. pp. 3-13. IEEE Transactions on Systems. Chen and L.A. Buckley. 48. 1993. 19.Y. Wong. A. No.Q. IEEE Control Systems Magazine. Theory of the fuzzy controller: an introduction. 21. J. 2. J. Application of circle criteria for stability analysis of linear SISO and MIMO systems associated with fuzzy logic controller. No. Vol. pp. Fuzzy identification and control of a liquid level rig. pp. 1-14. Vol. 3. 59. No. 5983. 1993.M. Vol. Fuzzy Sets and Systems. pp. Fuzzy Sets and Systems. 2. pp. No. 51. Vol. K. 116-132. C. 27. 22. Trojan. IEEE Transactions on Systems. No. No. 1992. Lamotte. pp. Vol. No. 1986. pp. An introductory survey of fuzzy control. 25. Buckley. Control and Cybernetics. Tanaka and M.Q. 1992. Sugeno. 5. Oh. 36. G. 36. No. Kybernets. 18. Vol. Fuzzy Sets and Systems. 65-86. Man and Cybernetics. 23. 2. Fuzzy Sets and Systems.F. International Journal of Systems Sciences. 14.F. No.92 FUZZY LOGIC AND NEURAL NETWORKS 10.B. Design of a fuzzy controller using input and output mapping factors. T.J. 13. Ragot and M. 28. Vol.H. 17. 16. 1991. M. Chung and J. Sugeno. C. 1988. 5. R. F. Gupta. Cao. No. Chou and D. 1993. pp. No. IEEE Transactions on Systems. Fuzzy identification of systems and its applications to modeling and control. Vol. 5. A general approach to rule aggregation in fuzzy logic control. 24. Li and Z. 135-156. K. Huang and J. J.J. 638-656. Fuzzy logic control.M. 3. No. Fuzzy v/s non-fuzzy controllers. pp. Newell. IEEE Transactions of on Systems. Dutta Majumdar. 1991. No. Generating rules for fuzzy logic controllers by functions. 15. 20. 3. Vol. No. 1. Abdelnour. 12. Study on stability of fuzzy closed-loop control systems. Man and Cybernetics. 45. Cheung. . 10.1. 249-258. C. B. 16. 335-351. 255-273. 1988. 1992. No . 21. Vol. 2. 149-158. Applied Intelligence. Vol. Fuzzy Sets and Systems. M. 57. 1992. Fuzzy Sets and Systems. 1993. Fuzzy Sets and Systems. J. L. R. 1990. 1985. pp. Vol. 1.P.H. 14.B. pp. 83-89. Vol. Kandel . 15. pp. F. Studies on the output of fuzzy controller with multiple inputs.T. 26. 24. Ray and D. Fuzzy control rules and their natural control laws. 1. J. Fuzzy inference and its applicability to control systems. Fuzzy Sets and Systems. 18. Modeling controllers using fuzzy relations.H. Ichikawa. Fuzzy Sets and Systems. Bernard. Takagi and M. pp. pp. Graham and R. Vol. pp. Vol. Mon. Chang. 48. pp. Vol. Peng. 345-349.R. 9. 925-960. 1989.J. Guild.

Altrock. J. 1. . 1994. 30.2. John Wiley. Vol. pp.P. Fuzzy control architectures. 29-36. Journal of Systems Science. pp. B. No. Vol. C. Vol.125-146. Three models of fuzzy logic controllers. 91-114. Vol. Rommler. Yager. S. Journal of Intelligent and Fuzzy Systems. Vol. Fuzzy controllers: Principles and architectures. Mc Murray. Yagar. 61. pp.Y. Pedrycz. 1993. No. pp. 1993.B. No. No.1905-1914.R. Vol. 31. 34. 1994.M. Asia-Pacific Engineering Journal. N. No. D. 1993. H.2. Fuzzy Sets and Systems. 23. Han and V. Adaptive fuzzy control applied to home heating system. Two-layer multiple-variable Fuzzy logic controller.O. 1-32. 33. 2. Krause. 1994. Kiupel and P. Bugarin. R. Cybernetics and Systems. 24. 3. Filev and R. 277-285.J. pp.P. Filev. 1. Barro and R. W.FUZZY LOGIC CONTROLLERS 93 29. Essentials of Fuzzy Modeling and Control. New York. Steffess and E. Frank.R. 1993. C.V. A. Man and Cybernetics. Fuzzy control of steam turbines. 1. 10. and D. pp. No. 32. IEEE Transactions of Systems. 24. Ruiz. Arend. 35.

What makes fuzzy nice is the “naturalness” of its approach and not its far-reaching complexity. You can create a fuzzy system to match any set of input-output data. Fuzzy logic can model nonlinear functions of arbitrary complexity. Fuzzy logic is flexible. most things are imprecise even on careful inspection. 7. . Everything is imprecise if you look closely enough. Fuzzy logic is conceptually easy to understand. which are available in the Fuzzy Logic Toolbox. fuzzy logic lets you rely on the experience of people who already understand your system. which take training data and generate opaque. 4. This process is made particularly easy by adaptive techniques like ANFIS (Adaptive Neuro-Fuzzy Inference Systems). but more than that. Fuzzy logic can be built on top of the experience of experts. impenetrable models. 3. it’s easy to massage it or layer more functionality on top of it without starting again from scratch. In many cases fuzzy systems augment themand simplify their implementation. In direct contrast to neural networks. 6.C H A P T E R Fuzzy Logic Applications 9 9. The basis for fuzzy logic is the basis for human communication. This observation underpins many of the other statements about fuzzy logic. 2. The mathematical concepts behind fuzzy reasoning are very simple.1 WHY USE FUZZY LOGIC? Here is a list of general observations about fuzzy logic: 1. Fuzzy logic can be blended with conventional control techniques. Fuzzy reasoning builds this understanding into the process rather than tacking it onto the end. 5. Fuzzy logic is based on natural language. Fuzzy systems don’t necessarily replace conventional control methods. With any given system. Fuzzy logic is tolerant of imprecise data.

A cross section of applications that have successfully used fuzzy control includes: 1. Exposure and Anti-shake • Hi-Fi Systems 4. Control engineers also use it in applications where the on-board computing is very limited and adequate control is enough. Commercially. Domestic Goods • Washing Machines/Dryers • Vacuum Cleaners • Toasters • Microwave Ovens • Refrigerators 3. Sentences written in ordinary language represent a triumph of efficient communication. but for control problems where simplicity and speed of implementation is important then fuzzy logic is a strong candidate. has been shaped by thousands of years of human history to be convenient and efficient. Automotive Systems • Vehicle Climate Control • Automatic Gearboxes • Four-wheel Steering • Seat/Mirror Control Systems . Natural language. Fuzzy logic is not the answer to all technical problems. Environmental • Air Conditioners • Humidifiers 2. 9. In the right application fuzzy logic systems are simple to design. We are generally unaware of this because ordinary language is. In most cases someone with a intermediate technical background can design a fuzzy logic controller. fuzzy logic has been used with great success to control machines and consumer products.2 APPLICATIONS OF FUZZY LOGIC Fuzzy logic deals with uncertainty in engineering by attaching degrees of certainty to the answer to a logical question. and can be understood and implemented by nonspecialists in control theory. Why should this be useful? The answer is commercial and practical. of course. The control system will not be optimal but it can be acceptable. Consumer Electronics • Television • Photocopiers • Still and Video Cameras – Auto-focus.FUZZY LOGIC APPLICATIONS 95 The last statement is perhaps the most important one and deserves more discussion. something we use every day. that which is used by ordinary people on a daily basis. Since fuzzy logic is built.

4 FUZZY LOGIC MODEL FOR PREVENTION OF ROAD ACCIDENTS Traffic accidents are rare and random. for example. but also through physical obstacles on the roads. might also be reflected in a higher acceptance of other measures. and whether acceptance of safety measures is also reflected in their perception of road traffic. road humps. do a fine job without using fuzzy logic. weather conditions etc. a model was developed which would obtain to prevent the vehicle pursuit distance automatically. cyclists and other traffic groups. Speed reduction can be accomplished by police surveillance. which has increasing usage area in Intelligent Transportation Systems (ITS). for example single vehicle accidents or some accidents without injury or fatality. and might reduce dangerous behaviour in traffic. Another aspect is whether the individual’s acceptance. When statistics are investigated India is the most dangerous country in terms of number of traffic accidents among Asian countries. and to increase safety for drivers. However. Using velocity of vehicle and pursuit distance that can be measured with a sensor on vehicle a model has been established to brake pedal (slowing down) by fuzzy logic. This goal forms the background for the present traffic safety program. literacy. One way to reduce the number of accidents is to reduce average speeds. These obstacles can cause damages to cars. you will see it can be a very powerful tool for dealing quickly and efficiently with imprecision and nonlinearity. they can cause difficulties for emergency vehicles. using fuzzy logic method. In this study. Obstacles such as flower pots. pedestrians. Cost of traffic accident is roughly 3% of gross national product. An alternative to these physical measures is different applications of Intelligent Transportation Systems (ITS). However. .4. The major objectives with ITS are to achieve traffic efficiency. One important aspect when planning and implementing traffic safety programs is therefore drivers’ acceptance of different safety measures aimed at speed reduction. physical measures are not always appreciated by drivers. Many reasons can contribute these results. if you take the time to become familiar with fuzzy logic. 9. However. Fuzzy logic is the codification of common sense-use common sense when you implement it and you will probably make the right decision. which are mainly driver fault. try something else.1 Traffic Accidents And Traffic Safety The general goal of traffic safety policy is to eliminate the number of deaths and casualties in traffic. If you find it is not convenient.96 FUZZY LOGIC AND NEURAL NETWORKS 9. However. 9. use it. If a simpler solution already exists. The program is partly based on the assumption that high speed contributes to accidents. Many researchers support the idea of a positive correlation between speed and traffic accidents. When should you not use fuzzy logic? Fuzzy logic is a convenient way to map an input space to an output space. agree that this rate is higher in India since many traffic accidents are not recorded. many people died or injured because of traffic accidents all over the world. by for instance redirecting traffic. lack of infrastructure. and in winter these obstacles can reduce access for snow clearing vehicles. small circulation points and elevated pedestrian crossings are frequently found in many residential areas around India.3 WHEN NOT TO USE FUZZY LOGIC? Fuzzy logic is not a cure-all. Many controllers. when there is a certain degree of freedom of choice. environment.

4. minimum inference or product inference is used. 9. The inference engine of the fuzzy logic maps fuzzy sets onto fuzzy sets. fuzzifier. Based on this fact we can infer another fact that is called a conclusion or consequent (the fact following “Then”). Models based on fuzzy logic consist of “If-Then” rules. In the next step this fuzzy rule base can (but need not) be supplemented with the rules collected from human experts. numerical data has been used as well as through a combination of numerical data-human experts. 9. A typical “If-Then” rule would be: If the ratio between the flow intensity and capacity of an arterial road is SMALL Then vehicle speed in the flow is BIG The fact following “If” is called a premise or hypothesis or antecedent. A set of a large number of rules of the type: If premise Then conclusion is called a fuzzy rule base. one value is chosen for the output variable. In most papers and practical engineering applications. 9. rules.3 Application In the study. .2 Fuzzy Logic Approach The basic elements of each fuzzy logic system are. The literature also contains a large number of different defuzzification procedures. In fuzzy rule-based systems.FUZZY LOGIC APPLICATIONS 97 9. recently. a model was established which estimates brake rate using fuzzy logic. inference engine. The task of the fuzzifier is to map crisp numbers into fuzzy sets (cases are also encountered where inputs are fuzzy variables described by fuzzy membership functions). An interesting case appears when a combination of numerical information obtained from measurements and linguistic information obtained from human experts is used to form the fuzzy rule base.1.1 Basic elements of a fuzzy logic. The general structure of the model is shown in Fig.2.4. In this case. The final value chosen is most often either the value corresponding to the highest grade of membership or the coordinate of the center of gravity. A large number of different inferential procedures are found in the literature. Input data are most often crisp values. the rule base is formed with the assistance of human experts. rules are extracted from numerical data in the first step. Input Fuzzifier Defuzzifier Crips output Rules Inference Fig. and defuzzifier. as shown in Figure 9. During defuzzification.

4.4. Because of the fact that current distance sensors perceive approximately 100-150 m distance.98 FUZZY LOGIC AND NEURAL NETWORKS Speed Rule base Distance Brake rate Fig. 9. and 9. Membership functions are given in Figures 9. .4 Membership Functions In the established model. Low Medium High 1 0.5 0 0 20 40 60 80 100 120 Fig. distance and brake rate. 9.5 0 0 50 100 150 Fig. Brake rate membership function is used 0-100 scale for expressing percent type.3. speed scale selected as 0-120 km/h on its membership function. 1 Low Medium High 0.3 Membership function of speed. distance membership function is used 0-150 m scale. different membership functions were formed for speed. 9.4 Membership function of distance. 9.5.2 General structure of fuzzy logic model. For maximum allowable car speed (in motorways) in India. 9.

Fig.7 Conclusions Many people die or injure because of traffic accidents in India. a model was established for estimation of brake rate using fuzzy logic approach.FUZZY LOGIC APPLICATIONS 99 Low Medium High 1 0. For this model.6 Output Fuzzy logic is also an estimation algorithm. So.1.4.4. various alternatives are able to crossexamine using the developed model.1: Speed LOW LOW LOW MEDIUM MEDIUM MEDIUM HIGH HIGH HIGH Fuzzy allocation map of the model Distance LOW MEDIUM HIGH LOW MEDIUM HIGH LOW MEDIUM HIGH Brake rate LOW LOW MEDIUM MEDIUM LOW LOW HIGH MEDIUM LOW 9. Figure 6 shows that the relationship between inputs. In this study. Fuzzy Allocation Map (rules) of the model was constituted for membership functions whose figures are given on Table-9.4.5 0 0 10 20 30 40 50 60 70 80 90 100 Fig.5 Membership function of brake rate. It is important that the rules were not completely written for all probability. weather conditions etc. it can be said that this fuzzy logic approach can be effectively used for reduce to traffic accident rate. 9. Car brake rate is estimated using the developed model from speed and distance data. 9. 9. Many reasons can contribute these results for example mainly driver fault. . 9. speed and distance. environment.6 is an example for such the case. Table 9. This model can be adapted to vehicles. lack of infrastructure. and brake rate.5 Rule Base We need a rule base to run the fuzzy model.

quantitative observations are often required to make quantitative control decisions. In conventional set theory. 9. 9. The power of a fuzzy model is the overlap between the fuzzy values.5 FUZZY LOGIC MODEL TO CONTROL ROOM TEMPERATURE Although the behaviour of complex or nonlinear systems is difficult or impossible to describe using numerical models. Although the controllers are simple to construct.7.1 The Mechanics of Fuzzy Logic The mechanics of fuzzy mathematics involve the manipulation of fuzzy variables through a set of linguistic equations. This implies a crisp . A single temperature value at an instant in time can be a member of both of the overlapping sets. In these cases. which obscures the ease of the formulation of a fuzzy controller.6 Relationship between inputs and brake rate. but the complexity or non-linearity of a process may make a numerical model unfeasible. Here the process is described in common language. The room temperature is the variable shown in Fig. Although Zadeh was attempting to model human activities. The form of the control model also determines the appropriate level of precision in the result obtained. an object (in this case a temperature value) is either a member of a set or it is not a member. Numerical models provide high precision. Three fuzzy sets: ‘hot’. Mamdani showed that fuzzy logic could be used to develop operational automatic control systems. which can take the form of if–then rules. which describe the control model. 9. The linguistic model is built from a set of if-then rules.5. linguistic models provide an alternative. which can take one or more fuzzy values. A fuzzy variable is one of the parameters of a fuzzy model. These decisions could be the determination of a flow rate for a chemical process or a drug dosage in medical practice. The outline of fuzzy operations will be shown here through the design of a familiar room thermostat. Much of the fuzzy literature uses set theory notation. each represented by a fuzzy set and a word descriptor. ‘cold’ and ‘comfortable’ have been defined by membership distributions over a range of actual temperatures. 9. the proof of stability and other validations remain important topics.100 FUZZY LOGIC AND NEURAL NETWORKS 80 Brake rate 60 40 0 20 0 50 50 Distance 100 150 100 Speed Fig.

7. 9. 9. 0. as its computation is simple.4 0.2 Fuzzification For a single measured value. In fuzzy logic. The Gaussian form from statistics has been used. The membership functions are used to calculate the memberships in all of the fuzzy sets. usually in the form of if-then rules. The second step is the application of the linguistic model. Guidance for these choices is provided by Zadeh’s Principle of Incompatibility: As the complexity of a system increases. The membership functions defining the three fuzzy sets shown in Fig. our ability to make precise and yet significant statements about its behaviour diminishes until a threshold is reached beyond which precision and significance (or relevance) become almost mutually exclusive characteristics.2 0. the boundaries between sets are blurred. the linguistic imprecision is taken into account. the fuzzification process is simple. a temperature of 15°C becomes three fuzzy values.FUZZY LOGIC APPLICATIONS 101 1. In the overlap region. The first is fuzzification. an object can be a partial member of each of the overlapping sets. The blurred set boundaries give fuzzy logic its name. 9. but the triangular form is commonly chosen.67 0.7 Room temperature.00 ‘hot’.8 0. The operation of a fuzzy controller proceeds in three steps. but the computation cost increases.5. boundary between the sets. Finally the resulting fuzzy output is converted back into physical values through a defuzzfication process. where measurements are converted into memberships in the fuzzy sets. There are no constraints on the specification of the form of the membership distribution. By admitting multiple possibilities in the model. 9.6 Cold 0.7 are triangular.33 Comfortable Hot 0. The number of values and the range of actual values covered by each one are also arbitrary.0 Membership value 0.0 0 5 10 15 20 25 30 35 Temperature (Degrees C) 40 45 50 Fig.2 1. .66 ‘cold’. Thus.33 ‘comfortable’ and 0. as shown in Fig. Finer resolution is possible with additional sets. 0.

40 and 0.8 0.8. The membership of the histogram in ‘cold’ is given by: max {min [mcold(T).8.6 Cold 0.then rules. For complex systems the number of rules required may be very large. the rule consequents.4 Comfortable Hot 0.73. but more . The rules needed to describe a process are often obtained through consultation with workers who have expert knowledge of the process operation. the membership of the histogram in ‘comfortable’ and ‘hot’ are 0. Although each rule is simple. By similar operations.3 Rule Application The linguistic model of a process is commonly made of a series of if .102 FUZZY LOGIC AND NEURAL NETWORKS A series of measurements are collected in the form of a histogram and use this as the fuzzy input as shown in Fig. the rule antecedents.8 the measurement data histogram is normalized so that its peak is a membership value of 1.0 Membership value 0. It is interesting to note that there is no requirement that the sum of all memberships be 1. 9. 1.2 0.2 1. In Fig. The fuzzy inference is extended to include the uncertainty due to measurement error as well as the vagueness in the linguistic descriptions.8 Fuzzification with measurement noise. is 0. mhistogram(T)]} where the maximum and minimum operations are taken using the membership values at each point T over the temperature range of the two distributions. 9.00. 9.00. The membership of the histogram in ‘cold’.0 0 5 10 15 20 25 30 35 Temperature (Degrees C) 40 45 50 Fig. Thus. These use the measured state of the process. 9. the simplicity of the rules trades off against the number of rules.0 and it can be used as a fuzzy set. These experts include the process designers. 9. to estimate the extent of control action. indicated by the arrow in Fig. there must be a rule to cover every possible combination of fuzzy input values. The minimum operation yields the overlap region of the two sets and the maximum operation gives the highest membership in the overlap.5.

rules of the form: IF (Temperature is Cold) AND (Humidity is High) THEN (Heater is ON) might be used. the membership in ‘on’ will be the minimum of the two antecedent membership values. (Temperature is Cold) is the membership value of the actual temperature in the ‘cold’ set. there must be a consistent method to resolve conflict and define an appropriate compromise.g. A simple temperature control model can be constructed from the example of Fig.66 membership in ‘on’ and 0. It is possible that several outputs are recommended and some may be contradictory (e. In decision support systems.33 membership in ‘off’.5. mB) and OR = Max (mA. one physical value of a controller output must be chosen from multiple recommendations. Mamdani used the maximum of the membership values. These can be used directly where the membership values are viewed as the strength of the recommendations provided by the rules. Similar values from rules 2 and 3 are 0. For example.FUZZY LOGIC APPLICATIONS 103 importantly. To extend these to more complex control models. Two methods are commonly used. All of the output membership functions are combined using the OR operator and the position of the highest membership value in the range of the output variable is used as the controller output. Exception handling is a particular strength of fuzzy control systems. The first is the maximum membership method. Zadeh defined the logical operators as AND = Min (mA. 9.00 in the ‘off’ setting for the heater. In the above rule. Zadeh also defined the NOT operator by assuming that complete membership in the set A is given by mA = 1. In automatic control. the experts may not be able to identify their thought processes in sufficient detail for rule creation. but gives a distribution corresponding to the overlap between A and its adjacent sets. The result for the three rules is then 0. When several rules give membership values for the same output set.66 membership in the heater setting ‘on’. Here the method becomes indecisive and does not produce a satisfactory result. The rules can include both the normal operation of the process as well as the experience obtained through upsets and other abnormal conditions.7: Rule 1 : Rule 2 : IF (Temperature is Cold) THEN (Heater is On) IF (Temperature is Comfortable) THEN (Heater is Off) Rule 3 : IF (Temperature is Hot) THEN (Heater is Off) In Rule 1. compound rules may be formulated. Defuzzification is the process for converting fuzzy output values to a single value or final decision. This gives the interesting result that A AND NOT (A) does not vanish. Rules may also be generated from operating data by searching for clusters in the input data space. if humidity was to be included in the room temperature control example. where mA and mB are membership values in sets A and B respectively. heater on and heater off). For very complex systems.4 Defuzzification The results of rule application are membership values in each of the consequent or output sets. The membership in NOT (A) is then given by m NOT (A) = 1 – mA. mB). The rules presented in the above example are simple yet effective. the process operators. . Rule 1 transfers the 0.66 membership in ‘cold’ to become 0. This method fails when there are two or more equal maximum membership values for different recommendations.33 and 0. 9.

the heater power decreases smoothly between fully on and fully off as the temperature increases between 10°C and 25°C. Therefore.5. ‘off’ at 0% power and ‘on’ at 100% power.33. a heater power of 65%. By taking the center of gravity.104 FUZZY LOGIC AND NEURAL NETWORKS The second method uses the center of gravity of the combined output distribution to resolve this potential conflict and to consider all recommendations based on the strengths of their membership values. such as template matching and fixed object modeling are unlikely to produce satisfactory results in the classification or control of input from agricultural products. Techniques used in industrial applications. With singletons. display high variation due to their inherent morphological diversity. tediousness. 9.5 Conclusions Linguistic descriptions in the form of membership functions and rules make up the model. applying the same rules gave mON = 0. soil. etc. The center of gravity is given by XF = z z x ( x ) dx ( x ) dx where x is a point in the output range and XF is the final control value. The output values used in the thermostat example are singletons. The sum of the membership functions was normalized by the denominator of the center of gravity calculation.40. The rules are generated a priori from expert knowledge or from data through system identification methods. but can be revised for controller tuning. in this case. in terms of weather. Singletons are fuzzy values with a membership of 1. Although only two singleton output functions were used. self-learning techniques such as neural networks (NN) and fuzzy logic (FL) seem to represent a good approach. These integrals are taken over the entire range of the output.6 FUZZY LOGIC MODEL FOR GRADING OF APPLES Agricultural produce is subject to quality inspection for optimum evaluation in the consumption cycle. 9. biological materials. the agricultural environment is highly variable. Output membership functions can be initially set. such as plants and commodities. However. Finally. The rules are applied using formalized operations to yield memberships in output sets. In the example there were two. applying automation in agriculture is not as simple as automating the industrial operations. with center of gravity defuzzification. There are two main differences. First. labor requirements. .73 and mOFF = 0. Defuzzifying these gives a control output of 67% power. Efforts to develop automated fruit classification systems have been increasing recently due to the drawbacks of manual grading such as subjectivity. cost and inconsistency. Second. the center of gravity equation integrals become a simple weighted average. conflicting rules essentially cancel and a fair weighting is obtained. availability. Once these are defined. the operating procedures for the calculations are well set out. Input membership functions are based on estimates of the vagueness of the descriptors used.00 at a single value rather than a membership function between 0 and 1 defined over an interval of values.67 and mOFF = 0. these are combined through defuzzification to give a final control output. Center of gravity defuzzification gave. Applying the rules gave mON = 0. In the histogram input case. Measurement data are converted to memberships through fuzzification procedures.

to manage a food supply and to predict peanut maturity. to decide the transfer of dairy cows between feeding groups. Maximum circumference measurement was performed using a Cranton circumference measuring device. a and b. where L is the lightness factor and a and b are the chromaticity coordinates. 2. punctures and bruises were among the defects encountered on the surfaces of Golden Delicious apples. To establish a multi-sensor measuring system for quality features in the long term. to manage crop production. It provides a means of translating qualitative and imprecise information into quantitative (linguistic) terms. to predict the yield for precision farming. Programming for fuzzy membership functions. external defects. weight and size. bitter pit.6. Weight was measured using an electronic scale. To compare the classification results from the FL approach and from sensory evaluation by a human expert. fuzzification and defuzzification was done in Matlab. to steer a sprayer automatically. Scars.2 Materials and Methods Five quality features. shape. In addition to these defects. Sizes of surface defects (natural and bruises) on apples were determined using a special figure template. The following objectives were included in this study: 1. 3.1 Apple Defects Used in the Study No defect formation practices by applying forces on apples were performed. . maintaining flexibility in making decisions even on complex biological systems. which consisted of a number of holes of different diameters. To design a FL technique to classify apples according to their external features developing effective fuzzy membership functions and fuzzy rules for input and output variables based on quality standards and expert expectations. a size defect (lopsidedness) was also measured by taking the ratio of maximum height of the apple to the minimum height. 9. to predict corn breakage. The main purpose of this study was to investigate the applicability of fuzzy logic to constructing and tuning fuzzy membership functions and to compare the accuracies of predictions of apple quality by a human expert and the proposed fuzzy logic model. defect. Fuzzy logic is a nonparametric classification procedure. color. were measured. Only defects occurring naturally or forcedly on apple surfaces during the growing season and handling operations were accounted for in terms of number and size. to reduce grain losses from a combine. which can infer with nonlinear relations between input and output categories.FUZZY LOGIC APPLICATIONS 105 Fuzzy logic can handle uncertainty. leaf roller. Color was measured using a CR-200 Minolta colorimeter in the domain of L. Grading of apples was performed in terms of characteristics such as color. Readings of these properties were obtained from different measurement apparatuses. ignoring their age. russeting. ambiguity and vagueness. Fuzzy logic was successfully used to determine field trafficability. to control the start-up and shutdown of food extrusion processes. assuming that the same measurements can be done using a sensor fusion system in which measurements of features are collected and controlled automatically.6. Size defects were determined measuring the maximum and minimum heights of apples using a Mitutoya electronic caliper. 9. shape. weight and size.

medium and good). In addition.(9..(9. A total of 181 golden delicious apples were graded first by a human expert and then by the proposed fuzzy logic approach. based on the expert’s expectations and USDA standards (USDA. an exponential function with the base of the irrational number e was used to simulate the inclination of the human expert in grading apples in terms of size (Fig.9 and 9. such as scars and leaf roller. medium and good. 1976)..3 ¥ SD . “defect” after normalizing each defect component such as bruises. bad..3 Application of Fuzzy Logic Three main operations were applied in the fuzzy logic decision making process: selection of fuzzy inputs and outputs. size and color. Coefficients used in the above equations were subjectively selected. The USDA standards for apple quality explicitly define the quality criteria so that it is quite straightforward for an expert to follow up and apply them. 21 of the apples were harvested before the others and kept for 15 days at room temperature for the same purpose of creating a variation in the appearance of the apples to be tested. Along with the measurements of features. Extremely large or small apples were already excluded by the handling personnel. and fuzzy inference. 9. was shown to be the best representation of human recognition of color. After the combinations of features given in the above equations. To simplify the problem. natural defects.6.2) where C is the circumference of the apple (normalized). The grading performance of fuzzy logic proposed was determined by comparing the classification results from FL and the expert.10). defects were collected under a single numerical value. The expert was trained on the external quality criteria for good. ND is the amount of natural defects. R is the total area of russeting defect (normalized) and SD is the normalized size defect. 9. russetting and size defects (lopsidedness). the apples were graded by the human expert into three quality groups.. 9. as total area (normalized).11). expectations and USDA standards (USDA. Eighty of the apples were kept at room temperature for 4 days while another 80 were kept in a cooler (at about 3°C) for the same period to create color variation on the surfaces of apples. 1976). input variables were reduced to 3 defect. A trial and error approach was used to develop membership functions. as it was difficult for the human expert to quantify it nondestructively. Defect = 10 ¥ B + 5 ¥ ND + 3 ¥ R + 0. Although triangular and trapezoidal functions were used in establishing membership functions for defects and color (Fig. blush (reddish spots on the cheek of an apple) percentage and weight were combined under “Size” using the same procedure as with “Defect” Size = 5 ¥ C + 3 ¥ W + 5 ¥ BL . 1976). circumference. Although it was measured at the beginning. The Hue angle (tan-1(b/a)). . which was used to represent the color of apples. Fuzzy logic techniques were applied to classify apples after measuring the quality features. formation of fuzzy rules. W is weight (normalized) and BL is the normalized blush percentage.106 FUZZY LOGIC AND NEURAL NETWORKS The number of apples used was determined based on the availability of apples with quality features of the 3 quality groups (bad. medium and bad apple groups defined by USDA standards (USDA.1) where B is the amount of bruising. firmness was excluded from the evaluation. depending on the expert’s experience. Similarly.

11 Membership functions for the size feature.3) .10 Membership functions for the color feature.5 7.5 106 Hue values 114 116 117 Fig.9 Membership functions for the defect feature.05 6. 9.0 2.15 11.10 7. .05 Size 11.1 1. 9.71828 and x is the value of size feature.2 1...(9.4 Defects 4.13 7. 1 Small Medium Big 6. Size = ex where e is approximately 2.FUZZY LOGIC APPLICATIONS 107 1 Low Medium High 0. 9.6 Fig.27 Fig. 1 Yellow Greenish-yellow Green 90 95 100 104.80 8.7 2.

then quality is very bad (rule Q3. The rules used in the evaluations of apple quality are given in Table 9. and it is a badly formed (small) apple. mD} complement = = 1 – mD ... then quality is very good (rule Q1.6.6 Q3.(9.5) . there are a lot of defects.14 C 3 + S3 Q3.2 C1 + S3 Q2.4) The minimum method given by equation (9.4 C2 + S1 Q1.3 Q3. there is no defect.6 Q3. Degree of membership for any set ranges from 0 to1.(9.4 Fuzzy Rules At this stage.1 Q3.3 Q2.11 Q3. while D2 and D3 represent moderate (medium) and high (bad) amounts of defects. and the Complement.0 represents a 100% membership while a value of 0 means 0% membership. on the other hand.6) .7 Q3.3 Q3. C1 is the greenish color quality (desired). which ranges from 1 to 17 for the bad quality group.16 Q3.2: Fuzzy rule tabulation C1 + S1 D1 D2 D3 Q1.2 Q3.17 in Table 9.1 in Table 9.7 C2 + S3 Q3. S3 is badly formed size (bad). .. S1..2.5) was used to combine the membership degrees from each rule established.7) .2 is given as follows. An example of the fuzzy AND (the minimum method) used in if-then rules to form the Q11 quality group in Table 9. Three primary set operations in fuzzy logic are AND.. S2 is moderately formed size (medium). 1] where X represents the universal set.12 C 2 + S2 Q2.8 Q3. human linguistic expressions were involved in fuzzy rules.2). The second subscript of Q shows the number of rules for the particular quality group.15 Q3. C2 is greenish-yellow color quality medium).1 C 1 + S2 Q1.13 Q3. which are given as follows AND: OR: mC Ÿ mD = min {mC .10 C3 + S1 Q2. the first subscript 1 stands for the best quality group. Two of the rules used to evaluate the quality of Golden Delicious apples are given below: If the color is greenish. D1 represents a low amount of defects (desired). A fuzzy set is defined by the expression below: C D = {X.4 Q3.. then three memberships are required to express the size values in a fuzzy rule. A value of 1. D is a fuzzy subset in X and μD(x) is the membership function of fuzzy set D.(9. respectively.(9.5 Q3.. OR.9 Q3. If the color is pure yellow (overripe). The minimum method chooses the most certain output among all the membership degrees. For quality groups represented with “Q” in Table 1.2). is well formed size (desired). and C3 is yellow color quality (bad).1 Q2. m0(x))| x Œ X} m0(x): Æ [0.2 Q2.5 C2 + S2 Q2. mD} mC » mD = (mC ⁄ mD) = max {mC. Finally. respectively. and it is a well formed large apple. while 2 and 3 stand for the moderate and bad quality groups. Table 9.108 FUZZY LOGIC AND NEURAL NETWORKS 9. If there are three subgroups of size..17 Where.

14 .2 . Q3. the membership function is m(D2) = 0. Q3.(9.8) On the other hand...2 . Q2. when x (1) < 1. Q1.14) m(D3) = 1. for example. k1 = (Q1.3 ) k2 = (Q2. when x(1) ≥ 4. max (k1) = (Q1.6 m(D2) = . Q2. for instance. the challenge is to assign input data into one or more of the overlapping membership functions.52 or 2. Q3. when 0.. Q3. equation 11 produces the membership degree for the best class (Lee. Q3. If. 3) = max {Q1.(9. determination of the quality group that an apple would belong to.75 £ x(1) £ 4. Q1. was formed as given below: If the input vector x is given as x = [defects.13) ...1 .12 .11 . Q2.(9. Q3.. color].15) .11) where k is the quality output group that contains different class membership degrees and the output vector y given in equation 10 below determines the probabilities of belonging to a quality group for an input sample before defuzzification: y = [max (k1) max (k2) max (k3)] where..2. 2 ⁄ Q1.(9.1 . Q3.6. Q3.. when 1. D1} .17 ) .3 Q2 .5 Determination of Membership Functions ( using intuition and qualitative assessment of the Membership functions are in general developed by x (1) – 0.(9. Q3.16 . Q1.1.. Q3.10 . for instance.77 . 1 ⁄ Q1.6 .4 .. or both. Q3.75) ..8 .24) relations between the input variable(s) and output 1. 1990).7 .3 Q3. then the membership function for the class of a high amount of defects (D3) is m(D3) = 0.6 ) k3 = (Q3.10) .(9.12) 9.13. 3} then. the fuzzy OR (the maximum method) rule was used in evaluating the results of the fuzzy rules given in Table 9.13 .75 m(D3) = ( x (1) – 1. Q3. Q3.FUZZY LOGIC APPLICATIONS 109 Q11 = (C1 Ÿ S1 Ÿ D1) = min {C1..(9. Q2..9 through 9. size.5 . 2.4.. 4 . 1. These functions can be defined either by linguistic terms or numerical ranges.24 or x (1) > 7. Q1.5 . was done by calculating the most likely membership degree using equations 9. . when defect innput x(1) < 0.76 classes.9 .2 . In the existence of more than one membership function that is actually in the nature of the fuzzy logic approach.52 For a medium amount of defects (D2).. Q3.9) . S1.(9. The membership function used in this study for defect quality in general is given in equation 9.15 . Q3. The membership function for high amounts of defects.24 £ x (1) £ 2 ..

.

.

First the problem is solved using the conventional (non-fuzzy) method. . in addition to the features mentioned earlier. 9.13 Constant tipping.7. tip = 0.S. Then. 9. providing good flexibility in reflecting the expert’s expectations and grading standards into the results. Consider the tipping problem: what is the “right” amount to tip your waitperson? Given a number between 0 and 10 that represents the quality of service at a restaurant (where 10 is excellent). could increase the efficiency of decisions made regarding apple quality.6. the same system is solved using fuzzy logic. 9.05 0 0 2 4 6 Service 8 10 Fig.112 FUZZY LOGIC AND NEURAL NETWORKS 9.13). 9.25 0. Suppose that the tip always equals 15% of the total bill.8 Conclusion Fuzzy logic was successfully applied to serve as a decision support technique in grading apples. It was also seen that color. is 15%.7 AN INTRODUCTORY EXAMPLE: FUZZY V/S NON-FUZZY To illustrate the value of fuzzy logic. variables such as firmness. internal defects and some other sensory evaluations.1 0. However. defects and size are three important criteria in apple classification. what should the tip be? This problem is based on tipping as it is typically practiced in the United States.15 0.15 Tip 0. Grading results obtained from fuzzy logic showed a good general agreement with the results from the human expert. though the actual amount may vary depending on the quality of the service provided. writing MATLAB commands that spell out linear and piecewise-linear relations. fuzzy and non-fuzzy approaches are applied to the same problem.2 0.1 The Non-Fuzzy Approach Let’s start with the simplest possible relationship (Fig. An average tip for a meal in the U.

FUZZY LOGIC APPLICATIONS 113 This does not really take into account the quality of the service.05 0.20/20 ¥ (service + food) + 0.2 Tip 0. so we need to add a new term to the equation. The formula does what we want it to do. we might have the tip go linearly from 5% if the service is bad to 25% if the service is excellent (Fig. and it is pretty straight forward. 14 Linear tipping. 9. Now our relation looks like this: tip = 0.1 0.1 0.15 0.25 0. .05 10 5 Food 0 0 5 Service 10 Fig. we may want the tip to reflect the quality of the food as well. 9.05 0 2 4 6 Service 8 10 Fig. what should the tip be? Let’s see how the formula will be affected now that we’ve added another variable (Fig. However. 9.2 Tip 0.20/10 * service + 0.15 0.25 0.14). Since service is rated on a scale of 0 to 10. 9.15). Suppose we try: tip = 0.15 Tipping depend on service and quality of food. This extension of the problem is defined as follows: Given two sets of numbers between 0 and 10 (where 10 is excellent) that respectively represent the quality of the service and the quality of the food at a restaurant.05 0.

8. tip = (0. 9.e. but when you look at them closely. We can still salvage things by using a piecewise linear construction (Fig. else if service < 7 .15 0. the results look pretty. Let’s return to the one-dimensional problem of just considering the service. Suppose you want the service to be a more important factor than the food quality. The response is still somehow too uniformly linear. means that those nice linear mappings no longer apply.25 0. You can string together a simple conditional statement using breakpoints like this: if service < 3.16).10/3) ¥ (service –7) + 0. you want to give a 15% tip in general.05.15.. tip = 0.16 Tipping based on the service to be a more important factor than the food quality.05 10 5 Food 0 0 5 Service 10 Fig. Suppose you want more of a flat response in the middle.05).20/10 ¥ food + 0. 9. Try: servRatio = 0. end .2 Tip 0. i. tip= servRatio ¥ (0.10/3) ¥ service + 0.20/10 ¥ service + 0. 0.17). 9.114 FUZZY LOGIC AND NEURAL NETWORKS In this case. else if service < =10. they do not seem quite right. and will depart from this plateau only if the service is exceptionally good or bad (Fig.05) + (1– servRatio) ¥ (0.1 0.15. This. tip = (0. in turn. Let’s say that the service will account for 80% of the overall tipping “grade” and the food will make up the other 20%.

1 0.20/10 ¥ food + 0.05).20/10 ¥ food + 0.05).FUZZY LOGIC APPLICATIONS 115 0.15) ¥ servRatio + (1 – servRatio) ¥ (0. where we take food into account again. else if service < 7. something like this result: servRatio = 0. 9. tip = ((0.15 0.2 Tip 0.15) ¥ servRatio + (1 – servRatio) ¥ (0.8.18 Tipping with two-dimensional variation.10/3) ¥ service + 0. tip = (0.15 0. .05).25 0. if service < 3.20/10 ¥ food + 0. 9.18).05) ¥ servRatio + (1 – servRatio) ¥ (0.1 0.10/3) ¥ (service – 7) + 0. else.25 0.05 10 5 Food 0 0 5 Service 10 Fig. If we extend this to two dimensions (Fig.05 0 2 4 6 Service 8 10 Fig. 9. tip = ((0. 17 Tipping using a piecewise linear construction.2 Tip 0. end 0.

9. 9. Now if we give mathematical meaning to the linguistic variables (what is an “average” tip. and it is definitely not easy to modify this code in the future. It does not matter which rules come first. . If food is delicious. leaving aside all the factors that could be arbitrary.7.2 The Fuzzy Approach It would be nice if we could just capture the essentials of this problem. If we wanted to include the food’s effect on the tip. If service is poor or the food is rancid. simple. we can combine the two different lists of rules into one tight list of three rules like so: 1. Of course. It was a little tricky to code this correctly.2 Tip 0. there’s a lot left to the methodology of fuzzy logic that we’re not mentioning right now. then tip is generous These three rules are the core of our solution. then tip is cheap 5. If service is poor. What matters is what we have shown in this preliminary exposition: fuzzy is adaptable. If we make a list of what really matters in this problem. 0. then tip is cheap 2. and easily applied. then tip is average 3. we might add the following two rules: 4. Moreover. And coincidentally.19 Tipping using fuzzy logic. for example?) we would have a complete fuzzy inference system. If food is rancid. If service is good. then tip is cheap 2. then tip is average 3. If service is excellent or food is delicious. If service is good. we have just defined the rules for a fuzzy logic system. it is even less apparent how the algorithm works to someone who did not witness the original design process. then tip is generous In fact.25 0.15 0. but the function is surprisingly complicated.116 FUZZY LOGIC AND NEURAL NETWORKS The plot looks good. If service is excellent.05 10 5 Food 0 0 5 Service 10 Fig. things like: • How are the rules all combined? • How do I define mathematically what an “average” tip is? The details of the method do not really change much from problem to problem .the mechanics of fuzzy logic are not terribly complex.1 0. we might end up with the following rule descriptions: 1. then tip is generous The order in which the rules are presented here is arbitrary.

% If service is good. badService=0. We found a piecewise linear relation that solved the problem. but the underlying logic the same: if the service is good. the subsequent modification was pretty easy. Moreover.. the maintenance of the structure of the algorithm decouples along fairly clean lines. but it was something of a nuisance to derive. badFood=0. (1–servRatio)*(tipRange/foodRange*food+lowTip). okayService=3.. here is the piecewise linear tipping problem slightly rewritten to make it more generic. You can do this sort of thing with lists of piecewise linear functions.19). tip is cheap if service<okayService. it was not very easy to interpret.FUZZY LOGIC APPLICATIONS 117 Here is the picture associated with the fuzzy system that solves this problem (Fig. foodRange=greatFood–badFood. The notion of an average tip might change from day to day.15. In other words.05. averTip=0. tipRange=highTip–lowTip. by using fuzzy logic rules.25.. tip is average elseif service<goodService. the tip should be average. greatService=10. % Establish constants lowTip=0. country to country. It worked. we were able to add two more rules to the bottom of the list that influenced the shape of the overall output without needing to undo what had already been done. The picture above was generated by the three rules above.. highTip=0. greatFood=10. goodService=7. tip=averTip*servRatio + (1–servRatio)* . 9. . 9. city to city. % If service is poor or food is rancid. You can recalibrate the method quickly by simply shifting the fuzzy set that defines average without rewriting the fuzzy rules. the fuzzy system is based on some “common sense” statements. tip=(((averTip–lowTip)/(okayService–badService)) . but there is a greater likelihood that recalibration will not be so quick and simple. On the other hand. serviceRange=greatService–badService.7.. For example. only now the constants can be easily changed.. It performs the same function as before. and once we wrote it down as code. Also. *service+lowTip)*servRatio + . (tipRange/foodRange*food+lowTip).3 Some Observations Here are some observations about the example so far.

Automatica. tip=(((highTip–averTip)/ .. 10. but the medium is not on our side. 8. pp.118 FUZZY LOGIC AND NEURAL NETWORKS % If service is excellent or food is delicious. as with all code. QUESTION BANK. Kickert and H. The truly fascinating thing to notice is that if we remove everything except for three comments. A fuzzy logic controller for a traffic junction. 3. (service–goodService)+averTip)*servRatio + . tip is generous else. the comment is identical with the code. 1976. Van Nauta Lemke. What we are doing here is not that complicated. . 4. W. which is why it is a very successful technique for bridging the gap between people and machines. 2. Application of a fuzzy controller in a warm water plat. think how much more likely your code is to have comments! Fuzzy logic lets the language that’s clearest to you. (1–servRatio)*(tipRange/foodRange*food+lowTip). 7.. threatening eventually to obscure it completely. 1977. 2. 338-353. Zadeh. No. 1965.. Vol. what remain are exactly the fuzzy rules we wrote down before: % If service is poor or food is rancid. we can fight this tendency to be obscure by adding still more comments. Information and Control. Washington. (greatService–goodService))* .P. 3. tip is cheap % If service is good. Pappis and E.. 4. 1976. Mamdani. as with a fuzzy system.C. pp. D.M. 1. high level comments. No.J. pp. tip is generous If. USDA Agricultural Marketing Service. REFERENCES. United States Standards for Grades of Apples. 301-308. Man and Cybernetics. 12...A. C. Vol. L. 4. 1.R.H. Why use fuzzy logic? What are the applications of fuzzy logic? When not use fuzzy logic? Compare non-fuzzy logic and fuzzy logic approaches. 707-717. for creeping generality to render the algorithm more and more opaque. end Notice the tendency here. or perhaps by trying to rewrite it in slightly more self-evident ways.. Fuzzy sets. IEEE Transactions on Systems. True. Vol. also have meaning to the machine. tip is average % If service is excellent or food is delicious.

45. Kikuchi. St.S. Vol. 54. 2. 3. Transactions of the ASAE. Vol. 1339. No. A fuzzy multi-criteria decision making method for technology transfer strategy selection in biotechnology. Edan. Agriculture. 1. Peleg and P. Grinspan.J. 255-273.H. pp. Information Sciences: Applications. Vol. T. pp. pp. pp. Transportation Research Part C. Control of a redundant manipulator using fuzzy rules. No. pp. A practical application of fuzzy control for an air-conditioning system. 37. E.C. 63. S. Yang. 2. Czogala and T. No. 943028. No. Fuzzy logic in control systems: Fuzzy logic controller. 2. pp. Fuzzy control of steam turbines.L. Newell. 11. Transactions of the ASAE. 19.Part I and Part II. 279-298. 1-12.E. Q. 1999-2009. Liu and J. Evaluation of cabbage seedling quality by fuzzy logic. 13-22. Fuzzy Sets and Systems. 5. Chen. 961-968. 267-276. 20. Takahasgi. J. B. 27-35. International Journal of Approximate Reasoning. U. pp. 1905-1914. 1988.B. Fuzzy control of model car. Sayyarodsari and J. 10. Fuzzy identification and control of a liquid level rig. pp. Fuzzy Sets and Systems. 22. Vol. P. Westin. pp. Fuzzy controller robot arm trajectory. No. 331-348. Frank. Vol. 21. 1991. No. Intelligent Transportation System and Traffic Safety Drivers Perception and Acceptance of Electronic Speed Checkers. Thangavadivelu and T. 8. Chakroborty and H. Classification of fruits by a Boltzman perceptron neural network. No. K. 2. 5. Vol. Tobi and T. 1994. A model for rider-motorcycle system using fuzzy control. Kiupel and P. 20: 404-435. 18. Maltz. 23. pp. pp. Colvin. A fuzzy logic expert system for dairy cow transfer between feeding groups. 15. Fuzzy Sets and Systems. Graham and R. Karlen. 1994. Kahn and E. 1993. Computer. Transportation research record. Fuzzy Sets and Systems. 12. A. 24. 1992. 69-83. 23. 13.C.L. 1.2278. T. Joseph. Ben-Hannan. 103-113. Rawlik. Vol. 1993. Vol. Wu. 3. Y. Park. 1993. Chang and Y.O. 131147.P. Vol. No. A. 1992. 9. ASAE Paper No. IEEE Transactions on Systems. 1994. S. 1999. 1991. Transactions of the ASAE. 1985. 28.S. Trafficability determination using fuzzy set theory. Modeling of driver anxiety during signal change intervals. No. 7 pp. Vol. Nishida. International Journal of Systems Science. T. Man and Cybernetics. Song and S. 121-133. 7. pp. P. 6. E. Vol.C.G. 34. Vol. pp.FUZZY LOGIC APPLICATIONS 119 5. Vol. 37. Vol. 5. Perincherry. 2. A Fuzzy Dynamic Learning Controller for Chemical Process Control. Marell and K. Man and Cybernetics. 1993. 2272. pp. P. 1993. 3. Colvin and D. S. IEEE Transactions on Systems. R. Modeling of a fuzzy controller with application to the control of biological processes. 17.R. Classification of apple surface features using machine vision and neural networks. J. M. Vol. 14. Sugeno and M.S. Roger. Automatica. Electron. 1990. Vol. Palm. V. Ambuel. 1994. B. No. Hanafusa. Gutman. 10. 9. 26. USA. 1647-1654. . MI. A fuzzy logic yield simulator for prescription farming. 1994. Lee.M. 16. No. 16. Hofaifar. N. pp. 6. Fuzzy Sets and Systems. 31. Chen and E. Hogans. 1989. pp. 131-139. C. and No. No. No.

Verma. USA. How much do road accidents cost the national economy. Vol. Fuzzy logic model for predicting peanut maturity. pp: 849-851. 2. R. 483-490. M. 2000. No. Accident Analysis and Prevention. 337-364. Transportation Research Part A. 25. Shahin. Fuzzy Logic Systems for Transportation Engineering: The State Of The Art. Volume: 32. and E. 33.W. . 1999. 43. 26.A. B.P. 2000. pp. D. Vol. Tollner. Elvik. Teodorovic.120 FUZZY LOGIC AND NEURAL NETWORKS 24. pp. Transactions of the ASAE.

are all variations on the parallel distributed processing (PDP) idea. the message is transferred from one neuron to the other. 10.+ 0 ) 2 6 . Learning strategies as a basis for an adaptive system will be presented in the last section. which perform the processing. received by neuron. In this chapter we first discuss these processing units and discuss different network topologies. with which the axon is connected to (Fig. In a typical neuron.1). Extending out from the nucleus is the axon. The weighted value is passed to the neuron. The modified value is directly presented to the next neuron. . The units combine usually by a simple summation. that is. 10. which transmits. The architecture of each network is based on very similar building blocks. the neurons or the processing units may have several input paths corresponding to the dendrites. The human brain consists of nearly 1011 neurons (nerve cells) of different types.2 BIOLOGICAL NEURAL NETWORK The term neural network comes from the intended analogy with the functioning of the human brain adopting simplified models of biological neural network.4 Neural Networks Fundamentals 10 10. In this way. the weighted values of these paths (Fig.2). by means of complex chemical process. it triggers sending an electric signal of constant level and duration through axon. which we describe in this course. When signals. become equal or surpass their threshold values.1 INTRODUCTION The artificial neural networks. one can find nucleus with which the connections with other neurons are made through a network of fibres called dendrites. where it is modified by threshold function such as sigmoid function. 10. In the neural network. electric potentials to the neurons.

cells). an activation function Fk.3 A FRAMEWORK FOR DISTRIBUTED REPRESENTATION An artificial network consists of a pool of simple processing units. which determines the effective input sk of a unit from its external inputs.2 Wnj Schematic representation of mathematical neuron network.. a state of activation yk for every unit. 10. Generally each connection is defined by a weight wjk which determines the effect which the signal of unit j has on unit k. which communicate by sending signals to each other over a large number of weighted connections. A set of major aspects of a parallel distributed model can be distinguished as: a set of processing units (neurons. . connections between the units.122 FUZZY LOGIC AND NEURAL NETWORKS Dendrite Cell body Nucleus Myelin sheath Axon Nerve ending Synapse Fig. which determines the new level of activation based on the effective input sk(t) and the current activation yk(t) (i. 10. X1 X2 Wij W2j 1 2 Wij Xi Xn Fig.1 Schematic representation of biological neuron network. which is equivalent to the output of the unit. 10. a propagation rule. the update).e.

a method for information gathering (the learning rule). each unit has a (usually fixed) probability of updating its activation at a time t. and hidden units (indicated by an index h) whose input and output signals remain within the neural network. 10. output units (indicated by an index o) which send data out of the neural network. which is propagated to other units. w w wjk w j yj qk Sk = k å j w jk y j + q k Fk yk Fig.. some of which will be discussed in the next sections. a second task is the adjustment of the weights. with asynchronous updating. and usually only one unit will be able to do this at a time.2 Connections between Units In most cases we assume that each unit provides an additive contribution to the input of the unit with which it is connected.NEURAL NETWORKS FUNDAMENTALS 123 an external input (aka bias. offset) Gk for each unit. units can be updated either synchronously or asynchronously.3 illustrates these basics. Apart from this processing.1) . providing input signals and if necessary error signals.3 The basic components of an artificial neural network. Within neural systems it is useful to distinguish three types of units: input units (indicated by an index i) which receive data from outside the neural network.3. The propagation rule used here is the standard weighted summation. an environment within which the system must operate. With synchronous updating.(10. During operation.1 Processing Units Each unit performs a relatively simple job: receive input from neighbors or external sources and use this to compute an output signal. 10. In some cases the latter model has some advantages. 10.3.. The system is inherently parallel in the sense that many units can carry out their computations at the same time. Figure 10. The total input to unit k is simply the weighted sum of the separate outputs from each of the connected units plus a bias or offset term qk: sk(t) = åw j jk (t) yj(t) + qk(t) . all units update their activation simultaneously.

(10.3.4).. as well as implementation of lookup tables.5) is used. 10. but the neuron input determines the probability p that a neuron get a high activation value: . yk = F(sk) = 1 1 + e sk . Generally. We call units with a propagation rule (10. 10. some sort of threshold function is used: a hard limiting threshold function (a sgn function).3) F w GH å j jk (t ) y j (t ) + q k (t ) I JK . In some cases more complex rules for combining inputs are used... We need a function Fk which takes the total input sk(t) and the current activation yk(t) and produces a new value of the activation of the unit k: yk(t + 1) = Fk ( yk(t).. Although these units are not frequently used. In some applications a hyperbolic tangent is used..4) although activation functions are not restricted to non-decreasing functions. Often.2) Often.(10. the yjm are weighted before multiplication. they have their value for gating of input. the output of a unit can be a stochastic function of the total input of the unit.(10.1) sigma units. is known as the propagation rule for the sigma-pi unit is given by sk(t) = åw j jk (t) Õy m jm (t) + qk(t) . introduced by Feldman and Ballard.(10.124 FUZZY LOGIC AND NEURAL NETWORKS The contribution for positive wjk is considered as an excitation and for negative wjk as inhibition. For this smoothly limiting function often a sigmoid (S-shaped) function like Sgn i Semi-linear i Sigmoid i Fig.. sk (t)). yielding output values in the range [1.4 Various activation functions for a unit.. or a linear or semi-linear function. In that case the activation is not deterministically determined by the neuron input. 10. or a smoothly limiting threshold (see Fig. In some cases.3 Activation and Output Rules We also need a rule. the activation function is a non-decreasing function of the total input of the unit: yk(t + 1) = Fk ( sk(t)) = Fk . +1]. in which a distinction is made between excitatory and inhibitory inputs.. A different propagation rule. which gives the effect of the total input on the activation of the unit.

In some cases..5. As for this pattern of connections. connections extending from outputs of units to inputs of units in the same layer or previous layers. the activation values of the units undergo a relaxation process such that the network will evolve to a stable state in which these activations do not change anymore. These are: · Supervised learning or Associative learning in which the network is trained by providing it with input and matching output patterns. the main distinction we can make is between: Feed-forward networks. that is. such that the dynamical behavior constitutes the output of the network.. Kohonen. This section focuses on the pattern of connections between the units and the propagation of data. Another way is to train the neural network by feeding it teaching patterns and letting it change its weights according to some learning rule. which will be discussed in the next chapter. Contrary to feed-forward networks.NEURAL NETWORKS FUNDAMENTALS 125 p(yk ¬ 1) = 1 1 + e sk / T . One way is to set the weights explicitly.5 TRAINING OF ARTIFICIAL NEURAL NETWORKS A neural network has to be configured such that the application of a set of inputs produces (either direct or via a relaxation process) the desired set of outputs.1 Paradigms of Learning We can categorize the learning situations in two distinct sorts. 10. 10. using a priori knowledge.4 NETWORK TOPOLOGIES In the previous section we discussed the properties of the basic processing unit in an artificial neural network. In all networks we consider that the output of a neuron is to be identical to its activation level. where the data flow from input to output units is strictly feed-forward. Examples of recurrent networks have been presented by Anderson. but no feedback connections are present. 10. Various methods to set the strengths of the connections exist. In other applications. which contains the network (self-supervised). In this paradigm the system is supposed to discover .6) in which T (temperature) is a parameter which determines the slope of the probability function.(10. Classical examples of feed-forward networks are the Perceptron and Adaline. or by the system. These input-output pairs can be provided by an external teacher. the change of the activation values of the output neurons are significant. · Unsupervised learning or Self-organization in which an (output) unit is trained to respond to clusters of pattern within the input. the dynamical properties of the network are important. Recurrent networks that do contain feedback connections. and Hopfield and will be discussed in subsequent chapters. The data processing can extend over multiple (layers of) units.

vectors can. presented to the network). j s p the input to a set of neurons when input pattern vector p is clamped (i. In the next chapters some of these update rules will be discussed. . x p the jth element of the pth input pattern vector.5. . Vectors are indicated with a bold non-slanted font: j. This is often called the Widrow-Hoff rule or the delta rule. Unlike the supervised learning paradigm... k.. there is no a priori set of categories into which the patterns are to be classified rather the system must develop its own representation of the input stimuli. and will be discussed in the next chapter.e.126 FUZZY LOGIC AND NEURAL NETWORKS statistically salient features of the input population.(10. have indices) where necessary. according to some modification rule.6.(10.. often: the input of the network by clamping input pattern vector p. If j receives input from k.8) in which dk is the desired activation provided by a teacher. the unit j. Many variants (often very exotic ones) have been published the last few years.g. p is often not necessary) or added (e. and that in some cases subscripts or superscripts may be left out (e. 10. their interconnection must be strengthened. Another common rule uses not the actual activation of unit k but the difference between the actual and desired activation for adjusting the weights: Dwjk = g yj (dk yk) .g. d p the desired output of the network when input pattern vector p was input to the network..1 Notation NOTATION AND TERMINOLOGY We use the following notation in our formulae.6 10. d p the jth element of the desired output of the network when input pattern vector p was input to the j network.2 Modifying Patterns of Connectivity Both learning paradigms discussed above result in an adjustment of the weights of the connections between units. k. h a hidden unit. contrariwise to the notation below. x p the pth input pattern vector.. . o an output unit.7) where g is a positive constant of proportionality representing the learning rate.. Virtually all learning rules for models of this type can be considered as a variant of the Hebbian learning rule. 10. i an input unit. Note that not all symbols are meaningful for all networks. the simplest version of Hebbian learning prescribes to modify the weight wjk with Dwjk = g yj yk . The basic idea is that if two units j and k are active simultaneously.

the inputs perform no computation and their layer is therefore not counted. Fj the activation function associated with unit j. Uj the threshold of unit j in Fj . y p the activation values of element j of the network when input pattern vector p was input to the j network. the second one is the learning algorithm. The representational power of a neural network refers to the ability of a neural network to represent a desired function.e. is there a procedure to (iteratively) find this set of weights? . although the latter two terms are often envisaged as a property of the activation function. The first one is the representational power of the network. learning: When using a neural network one has to distinguish two issues which influence the performance of the system.2 Terminology Output vs. we consider the output and the activation value of a unit to be one and the same thing. and even for an optimal set of weights the approximation error is not zero. in most cases the network will only approximate the desired function. That is. one hidden layer. q the biases to the units.. this external input is usually implemented (and can be written) as a weight from a unit with activation value 1.NEURAL NETWORKS FUNDAMENTALS 127 y p the activation values of the network when input pattern vector p was input to the network. and one output layer is referred to as a network with two layers. Bias. activation of a unit. wj the weights of the connections which feed into unit j. Representation vs. Thus a network with one input layer. offset.6. They may be used interchangeably. qj the bias input to unit j. Given that there exist a set of optimal weights in the network. A the energy of the network. 10. Ep the error in the output of the network when input pattern vector p is input. Because a neural network is built from a set of standard functions. Furthermore. independent of the network input but adapted by the learning rule) term which is input to a unit. Number of layers: In a feed-forward network. Since there is no need to do otherwise. gjk the learning rate associated with weight wjk . the output of each neuron equals its activation value. The second issue is the learning algorithm. This convention is widely though not yet universally used. threshold: These terms all refer to a constant (i. wjk the weight of the connection from unit j to unit k. W the matrix of connection weights.

B. 263-269. M. Cognitive Science. Widrow. Vol. 1. Neural networks and physical systems with emergent collective computational abilities. 1977. D.H. 3. T. 27-90). Hillsdale. No.J. New York: Wiley. Kohonen. 2. Goldstein. 5. 8. NJ: Erlbaum. 6. 435-461. pp.C. 6. Samuels (Eds. 7. pp. Rumelhart and J. pp. Connectionist models and their properties. PP.). 1962. Learning state space trajectories in recurrent neural networks. Pearlmutter. J. Anderson. 1982. 79. 2. B. 3. 4. Associative Memory: A System-Theoretical Approach. 1982. 2.128 FUZZY LOGIC AND NEURAL NETWORKS QUESTION BANK. . J.J.O. D. 1986. CA: Academic Press.L. Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Jacobi. What are the major aspects of parallel distributed model? Explain the biological neural network. San Diego. 205-254. In D. and D. McClelland. Mel. Feldman. 1989. Connectionist Robot Motion Planning. B. Springer-Verlag.. Ballard.C: Spartan Books. Washington. 2554-2558. Vol. Generalization and Information Storage in Networks of Adaline Neurons. in Self Organizing Systems 1962. Vol. 1990. The MIT Press. What are the paradigms of neural network learning? REFERENCES. 1. Proceedings of the National Academy of Sciences.T. Hopfield. LaBerge and S. Neural Computation. Hebb. 6. Basic Processes in Reading Perception and Comprehension Models (pp. The Organization of Behaviour. 5.A. G. 1977. What are the basic components of artificial neural network? What are the network topologies? What are the various activation function? Explain them schematically. 1949. 1.E. ed.A. 9. D.A. J.W. G. Neural models with cognitive implications. 4. Jovitz.

In the first part of this chapter we discuss the representational power of the single layer networks and their learning algorithms and will give some examples of using the networks.1 (we leave the output index o out). including some of the classical approaches to the neural computing and learning problem.1 INTRODUCTION This chapter describes single layer neural networks. . 11. In the second part we will discuss the representational limitations of single layer networks.1 Single layer network with one output and two inputs.2 NETWORKS WITH THRESHOLD ACTIVATION FUNCTIONS A single layer feed-forward network consists of one or more output neurons o. proposed by Rosenblatt and the Adaline. each of which is connected with a weighting factor wio to all of the inputs i. In the simplest case the network has only two inputs and a single output. as sketched in Fig.+ 0 ) 2 6 . Two classical models will be described in the first part of the chapter: the Perceptron.4 Perceptron and Adaline 11 11. The input of the neuron is the weighted sum of the inputs plus the bias term. The output of the network is formed by the activation of the output neuron. 11. presented by Widrow and Hoff. 11. which is some function of the input: X1 W1 y X2 W2 q +1 Fig.

if the total input is negative.130 FUZZY LOGIC AND NEURAL NETWORKS y=F F I GH å w x + G JK 2 i i i =1 . A geometrical representation of the linear threshold neural network is given in Fig.e.3) can be written as x2 = w1 q x1 w2 w2 .2. 11. The network can now be used for a classification task: it can decide whether an input pattern belongs to one of two classes.. .2 Geometric representation of the discriminant function and the weights. the pattern will be assigned to class +1.. The separation between the two classes in this case is a straight line..(11.. In this section we consider the threshold (sgn) function: F(s) = R+1 S1 T if s > 0 otherwise . 11. i.. the sample will be assigned to class +1. If the total input is positive.1) The activation function F can be linear so that we have a linear network. given by the equation: w1x1 + w2 x2 + q = 0 .2) The output of the network thus is either +1 or 1. or non-linear. depending on the input.. Note that also the weights can be plotted in the input space: the weight x2 + + w1 + w2 –q ||W|| x1 + + + + Fig.(11. Equation (11.(11.3) The single layer network represents a linear discriminant function..(11.. how far the line is from the origin.4) and we see that the weights determine the slope of the line and the bias determines the offset.

which states the following: If there exists a set of connection weights w* which is able to perform the transformation y = d(x). For a classification task the d(x) is usually +1 or 1.(11. 2. modify all connections wi according to: Dwi = d(x)xi. Start with random weights for the connections. where o denotes dot or inner product. . 4.2 Convergence Theorem For the learning rule there exists a convergence theorem.. Because w*is a correct solution. Select an input vector x from the set of training samples. will be greater than 0 or: there exists a d > 0 such that |w* o x| > d for all inputs x. Proof: Given the fact that the length of the vector w* does not play a role (because of the sgn operation)..3. when the network responds correctly. Both methods are iterative procedures that adjust the weights.. we come to the second issue: how do we learn the weights and biases in the network? We will describe two learning methods for these types of networks: the perceptron learning rule and the `delta or `LMS rule.3. Given the perceptron learning rule as stated above. Besides modifying the weights. no connection weights are modified.. the value |w* o x|.(11. we must also modify the threshold q.. Note that the procedure is very similar to the Hebb rule.7) 11.1 Perceptron Learning Rule Suppose we have a set of learning samples consisting of an input vector x and a desired output d(x). we take ||w*|| = 1. the perceptron learning rule will converge to some solution (which may or may not be the same as w*) in a finite number of steps for any initial choice of the weights.6) The learning problem can now be formulated as: how do we compute Dwi(t) and Dq(t) in order to classify the larning patterns correctly? 11. The perceptron learning rule is very simple and can be stated as follows: 1.(11. For each weight the new value is computed by adding a correction to the old value. This q is considered as a connection w0 between the output neuron and a dummy predicate unit which is always on: x0 = 1.3 PERCEPTRON LEARNING RULE AND CONVERGENCE THEOREM 11. The threshold is updated in a same way: wi(t + 1) = wi(t) + Dwi(t) q(t + 1) = q(t) + Dq(t) .5) .. Now that we have shown the representational power of the single layer network with linear threshold units. this threshold is modified according to: Dq = R0 Sd ( x) T if the perceptron responds correctly otherwise .PERCEPTRON AND ADALINE 131 vector is always perpendicular to the discriminant function. If y ¹ d(x) (the perceptron gives an incorrect response). A learning sample is presented to the network. the only difference is that. 3. Go back to 2.

. we know that Dw = d(x)x. If we start with connections w = 0. q = 2. the system modifies its connections only a limited number of times. connection weights are modified at a given input x. after maximally tmax modifications of the weights the perceptron is correctly performing the mapping..8) Example 11. The conclusion is that there must be an upper limit tmax for t.1: A perceptron is initialized with the following weights: w1 = 1. From this it follows that: w¢ o w* = w o w* + d(x) o w* o x = w o w* + sgn(w* o x) w* o x > w o w* + d ||w¢||2 = ||w + d(x)x||2 = w2 + 2d (x) w o x + x2 < w2 + x2 =w +M After t modifications we have: w(t) o w* > w o w* + td ||w(t)||2 < w2 + tM such that cos a(t) = 2 (because d (x) = sgn [w o x]) w* o w(t ) || w(t )|| w* o w + td w2 + tM > From this follows that limt®¥ cos a(t) = limt®¥ d M t = ¥ while cos a £ 1. and the weight after modification is w¢ = w + Dw. The perceptron learning rule is used to learn a correct discriminant function for a number of samples. tmax will be reached when cos a = 1. tmax = M d2 . . sketched in Fig. In other words.132 FUZZY LOGIC AND NEURAL NETWORKS Now define cos a = w o w* .(11. || w|| When according to the perceptron learning rule. w2 = 2. 11.3.

with values x = (0:5. Usually the central block. Dw2 = 0:5. the weight changes are: Dw1 = 0:5.1) it can be calculated that the network output is +1.4) this device consists of a set of controllable resistors connected to a circuit. the summer. 11. so no weights are adjusted. respectively. According to the perceptron learning rule. The new weights are now: w1 = 1:5. 0:5) the network output will be 1. q = 1. The delta-rule uses the net output without further mapping into output values 1 or +1. In a simple physical implementation (Fig. When presenting point C with values x = (0:5. it may be clear that a system with many parallel outputs is directly implementable by multiple units of the above kind. From equation (11. and sample C is classified correctly. The learning rule was applied to the adaptive linear element. Although the adaptive process is here exemplified in a case when there is only one output.. while the target value d(x) = +1. so no change. is also followed by a quantiser. 0:5) and target value d(x) = -1. 1. 1:5) and target value d(x) = +1 is presented to the network. If the input conductances are denoted by wi. which can sum up currents caused by the input voltage signals. also named Adaline. with values x = (0:5.3 Discriminant function before and after weight update. The same is the case for point B. The perceptron learning rule uses the output of the threshold function (either 1 or +1) for learning. The main functional di_erence with the perceptron training rule is the way the output of the system is used in the learning rule. depending on the polarity of the sum..PERCEPTRON AND ADALINE 133 x2 2 + A 1 B +C 1 2 x1 Original discriminant function After weight update + Fig. developed by Widrow and Hoff. In Fig. The first sample A. q = 1. 11. and the input and output signals by xi and y.3 the discriminant function before and after this weight update is shown. n.4 ADAPTIVE LINEAR ELEMENT (Adaline) An important generalisation of the perceptron training algorithm was presented by Widrow and Hoff as the least mean square (LMS) learning procedure.. 11.. 11. the network output is negative. which outputs either +1 or 1. w2 = 2:5. i = 0. also known as the delta rule. then the output of the central block is defined to be .

The problem is to determine the coeficients wi. 2. n. for instance. .. i = 0. but here we focus on the linear relationship and use the network for a function approximation task. 11.134 FUZZY LOGIC AND NEURAL NETWORKS +1 –1 +1 w1 w2 w3 Gains Input pattern switches S – Summer Error S Quantizer + – 1 +1 Reference switch Fig. is applied at the inputs. The delta-rule now uses a cost-or error-function based on these differences to adjust the weights. where y p is the actual output for this pattern.9) where q = w0.(11. For the Adaline.. Widrow introduced the delta rule to adjust the weights. the output of the network differs from the target value d p by (d p y p). to attain the correct values. n. usually iteratively.. . By thresholding the output value..10) Such a simple network is able to represent a linear relationship between the value of the output unit and the value of the input units. . n Level w0 Output y= åw x i =1 i i +q . in the sense of least squares. 11. Suppose we want to train the network such that a hyperplane is fitted as well as possible to a set of training samples consisting of input values x p and desired (or target) output values d p.4 The adaline. For every given input sample. in such a way that the input-output response is correct for a large number of arbitrarily chosen signal sets.(11. In high dimensional input spaces the network represents a (hyper) plane and it will be clear that also multiple output units may be defined. An adaptive operation means that there exists a mechanism by which the wi can be adjusted. If an exact mapping is not possible. the average error must be minimised. The purpose of this device is to yield a given value y = d p at its output when the set of values xip i = 1. a classifier can be constructed (such as Adaline).5 THE DELTA RULE For a single layer network with an output unit with a linear activation function the output is simply given by y= åw x j j j +q . . 1.

(11..(11..(11.16) where d = d y is the difference between the target output and the actual output for pattern p. 11..10). ¶w j ¶y p ¶w j Because of the linear units....(11. The delta rule modifies weight appropriately for target and actual outputs of either polarity and for both continuous and binary input and output units.PERCEPTRON AND ADALINE 135 The error function. That is.12) where g is a constant of proportionality...13) ¶y p = xj ¶w j and . These characteristics have opened up a wealth of new applications. the total error E is defined to be E= åE p p = 1 2 å (d p p y p )2 . eq. .15) . as indicated by the name least mean square. (11.. is the summed squared error.(11. but we have not discussed the limitations on the representation of these networks..14) ¶E p = (d p y p) ¶y p such that Dp wj = g d p xj p p p . The LMS procedure finds the values of all the weights that minimize the error function by a method called gradient descent.. .6 EXCLUSIVE-OR PROBLEM In the previous sections we have discussed two learning algorithms for single layer networks.(11.11) where the index p ranges over the set of input patterns and E p represents the error on pattern p. The derivative is ¶E p ¶E p ¶y p = . The idea is to make a change in the weight proportional to the negative of the derivative of the error as measured on the current pattern with respect to each weight: Dp wj = g ¶E p ¶w j ..

(11.1).5 Geometric representation of input space For the specific XOR problem we geometrically show that by introducing hidden units.5 a geometrical representation of the input domain is given. This simple example demonstrates that adding hidden units increases the class of . as desired. the net input is equal to: s = w1x1 + w2x2 + q .136 FUZZY LOGIC AND NEURAL NETWORKS Table 11. thereby extending the network to a multi-layer perceptron. 1) and (1.1 shows the desired relationships between inputs and output units for this function.1 Exclusive-or truth table. 11. Fig. Table 3. N 1 1 1 1 N 1 1 1 1 @ 1 1 1 1 One of Minsky and Paperts most discouraging results shows that a single layer perceptron cannot represent a simple exclusive-or function. The proof is given in the next section.6a demonstrates that the four input points are now embedded in a three-dimensional space defined by the two inputs plus the single hidden unit. as depicted in Fig. 11. 1) and (1. To see that such a solution cannot be found.18) and equal to zero on the other side of this line. (11. x1 (– 1. The obvious question to ask is: How can this problem be overcome? Minsky and Papert prove that for binary inputs. take a loot at Fig. the output of the perceptron is equal to one on one side of the dividing line which is defined by: w1x1 + w2x2 = q .. 1). 1) cannot be separated by a straight line from the two open circles at (1. 11. 11. These four points are now easily separated by a linear manifold (plane) into two groups. – 1) XOR Fig. any transformation can be carried out by adding a layer of predicates which are connected to all inputs.17) According to eq.(11. In a simple network with two inputs and one output. For a constant q. – 1) OR (1..1. and the two solid circles at (1..5. the output of the perceptron is zero when s is negative and equal to one when s is positive. the problem can be solved.. 1) x1 (1. 11. In Fig. 1) x1 x2 x2 ? ? x2 And (– 1. The input space consists of four points.

For binary units.(11. (a) The perceptron of Fig. With the indicated values of the weights wij (next to the connecting lines) and the thresholds qi (in the circles) this perceptron solves the XOR problem.. 11. one can prove that this architecture is able to perform any transformation given the correct connections and weights.N such that p y h = sgn F wx GH å i p ih i N+ 1 2 I JK . clearly. the weights to the output neuron can be chosen such that the output is one as soon as one of the M predicate neurons is one: p y o = sgn F y GH å M h =1 h +M 1 2 I JK . For every x p Î X+ a hidden unit h can be reserved of which the activation yh is 1 if and only if the specific pattern p is present at the input: we can choose its weights wih equal to the specific pattern xp and the bias qh equal to 1 .(11. Similarly.(11. – 0.7 MULTI-LAYER PERCEPTRONS CAN DO EVERYTHING In the previous section we showed that by adding an extra hidden unit.20) is equal to 1 for xp = wh only. Fig.. the XOR problem can be solved.5 a. 11. (1.7 Solution of the XOR problem. we can divide the set of all possible input vectors into two classes: X + = {x|d(x) = 1} and X = {x|d(x) 1} .6 onto the four points indicated here. 1) 1 1 1 1 (– 1. 1.. 11.1 with an extra hidden unit. perceptron. the total number of possible input vectors x is 2N. (b) This is accomplished by mapping the four points of Fig. by this generalization of the basic architecture we have also incurred a serious loss: we no longer have a learning rule to determine the optimal weights. 11.19) Since there are N input units. – 1) b.. separation (by a linear manifold) into the required groups is now possible. – 1. For a given transformation y = d(x).5 – 1 – 0.like networks. The most primitive is the next one.PERCEPTRON AND ADALINE 137 problems that are soluble by feed-forward.. However.21) ..

1959. Perceptrons: An Introduction to Computational Geometry. 5. In 1960 Ire Wescon Convention Record. 1969. The MIT Press. Describe the perceptron learning rule. Rosenblatt. Explain single layer neural network with one output and two inputs. What are the advantages of multiplayer perceptron over single layer perceptron? REFERENCES. 1949. Derive the convergence theorem for perceptron learning rule. A more elegant proof is given by Minsky and Papert. QUESTION BANK. Principles of Neurodynamics. Widrow. The Organization of Behaviour. 1960. 2. Adaptive Switching Circuits. and we will always take the minimal number of mask units. 1. New York: Wiley. Hoff.O. Hebb. which is equal to the number of patterns in X +. B. . Explain the delta rule used to adjust the weights of Adaline network. 7. 2. which is maximally 2N-1. Papert. 6. 3.138 FUZZY LOGIC AND NEURAL NETWORKS This perceptron will give y0 = 1 only if x Î X+: it performs the desired mapping. which is maximally 2N. but the point is that for complex transformations the number of required units in the hidden layer is exponential in N. 3. and S. New York: Spartan Books.E. 4. Justify this statement. 4. D. Dunno. Single layer perceptron cannot represent exclusive-OR. Minsky. and M. M. Explain Adaline neural network. F. Of course we can do the same trick for X . 1. The problem is the large number of predicate units.

Cun. Hinton and Williams in 1986. just as for networks with binary units (section 11. (10. Keeler. 1989. 1989. The input units are merely fan-out units. Minsky and Papert showed in 1969 that a two layer feed-forward network can overcome many restrictions. provided the activation functions of the hidden units are non-linear (the universal approximation theorem). 12. . 12. The output of the hidden units is distributed over the next layer of Nh. 1990) that only one layer of hidden units suffices to approximate any function with finitely many discontinuities to arbitrary precision. a single-layer network has severe restrictions: the class of tasks that can be accomplished is very limited. Hartman.1). Although back-propagation can be applied to networks with any number of layers. In this chapter we will focus on feed forward networks with layers of processing units. Funahashi. The central idea behind this solution is that the errors for the units of the hidden layer are determined by back-propagating the errors of the units of the output layer. 1 hidden units. until the last layer of hidden units. 1989.2 MULTI . Back-propagation can also be considered as a generalization of the delta rule for non-linear activation functions and multilayer networks.4). & Kowalski.C H A P T E R Back-Propagation 12 12. & White. 1 INTRODUCTION As we have seen in the previous chapter. and similar solutions appeared to have been published earlier (Parker. no processing takes place in these units. 1985. Hornik. There are no connections within a layer. 2 hidden units. For this reason the method is often called the back-propagation learning rule. Stinchcombe. Each layer consists of units. of which the outputs are fed into a layer of No output units (see Fig.FORWARD NETWORKS A feed-forward network has a layered structure. as given in eq. The activation of a hidden unit is a function Fi of the weighted inputs plus a bias. 1985).LAYER FEED . The Ni inputs are fed into the first layer of Nh. An answer to this question was presented by Rumelhart. but did not present a solution to the problem of how to adjust the weights from input to hidden units.7) it has been shown (Cybenko. which receive their input from units from a layer directly below and send their output to units in a layer directly above the unit.

3) The error E p is defined as the total quadratic error for pattern p at the output units: Ep = 1 o p p do yo 2 o =1 åd N i 2 .2) To get the correct generalization of the delta rule as presented in the previous chapter..(12.4) .1 A multi-layer network withlayers of units.140 FUZZY LOGIC AND NEURAL NETWORKS h Ni o N0 Nh. 12.(12. we have to generalise the delta rule.1) in which p sk = åw j p jk yk + qk . given by p p yk = F(Sk) .(12.. The activation is a differentiable function of the total input.. we must set Dpwjk = g ¶E p ¶w jk . 12. (12.3 THE GENERALISED DELTA RULE Since we are now using units with nonlinear activation functions... which was presented in chapter 11 for linear functions to the set of non-linear activation functions. In most applications a feed-forward network with a single layer of hidden units is used with a sigmoid activation function for the units..1 Nh1–1 Nh1–2 Fig...

11) which is the same result as we obtained with the standard delta rule..1) we see that ¶ykp p = F(Sk) p ¶S k ... We can write ¶E p ¶E p ¶Skp = ¶w jk ¶Skp ¶w jk By equation (12..7) we will get an update rule which is equivalent to the delta rule as described in the previous chapter. it follows from the definition of E p that ¶E p p p p = (do yo) ¶yo .9).8) p The trick is to figure out what dk should be for each unit k in the network. Thus.(12..9) Let us compute the second factor.(12. First.2) we see that the second factor is .10) p which is simply the derivative of the squashing function F for the kth unit.. Substituting this and equation (12.(12.BACK-PROPAGATION 141 p where do is the desired output for unit 0 when pattern p is clamped.. we consider two cases.. which we now derive. one factor reflecting the change in error as a function of the output of the unit and one reflecting the change in the output as a function of changes in the input. p To compute dk we apply the chain rule to write this partial derivative as the product of two factors.6) ¶E p ¶Skp . resulting in a gradient descent on the error surface if we make the weight changes according to: p Dpwjk = gdk yjp . we get .. evaluated at the net input Sk to that unit. is that there is a simple recursive computation of these ds which can be implemented by propagating error signals backward through the network.(12. To compute the first factor of equation (12.(12. assume that unit k is an output unit k = o of the network. By equation (12.10) in equation (12.. (12..9).. We further set E = åE p p as the summed squared error..5) ¶Skp = yjp ¶w jk When we define p dk = . we have p dk = ¶E p ¶E p ¶ykp = ¶Skp ¶ykp ¶S kp . In this case..(12. The interesting result.

1 Understanding Back-Propagation The equations derived in the previous section may be mathematically correct.. We have to bring eo to zero. in order to reduce an error.12) and (12. Differently put.13) Substituting this in equation (12.. weighted by this connection. In fact. Lets call this error eo for a particular output unit o. The simplest method to do this is the greedy method: we strive to change the connections in the neural network in such a way that.. However.. (12. Secondly.(12..) and we use the chain rule to write ¶E p p = ¶yh ¶E p ¶Sop = p ¶Shp o = 1 ¶So å No ¶E p ¶ p p ¶yh o = 1 ¶So å No å No wko y jp j =1 ¶E p = w = p ho j = 1 ¶So å No åd j =1 No p o who .15) That is step one. In order to adapt the weights from input to hidden units. We know from the delta rule that. and the actual network output is compared with the desired output values... the weights from input to hidden units are never changed. the error eo will be zero for this particular pattern. next time around.. a hidden unit h receives a delta from each output unit o equal to the delta of that output unit weighted with (= multiplied by) the weight of the .142 FUZZY LOGIC AND NEURAL NETWORKS p p p do = (do yo ) Fo' (S p ) o . the error measure can be written p p as a function of the net inputs from hidden to output layer Ep = Ep (s1. we again want to apply the delta rule. sjp. and we do not have the full representational power of the feed-forward network as promised by the universal approximation theorem. of course. But it alone is not enough: when we only apply this rule. however.. What happens in the above equations is the following... which are then used to compute the weight changes according to equation (12.9) yields p d h = F(Shp ) å@ j =1 No p o who . we do not readily know the contribution of the unit to the output error of the network.14) Equations (12. In this case. 12. yes. we have to adapt its incoming weights according to Dwho = (d° y°) yh .3. the whole back-propagation process is intuitively very clear. but what do they actually mean? Is there a way of understanding back-propagation other than reciting the necessary equations? The answer is.(12. This is solved by the chain rule which does the following: distribute the error of an output unit o to all the hidden units that is it connected to.8).. we usually end up with an error in each of the output units. the activation values are propagated to the output units.. When a learning pattern is clamped.14) give a recursive procedure for computing the ds for all units in the network.. This procedure constitutes the generalized delta rule for a feed-forward network of non-linear units. if k is not an output unit but a hidden unit k = h. s2 . (12. we do not have a value for d for the hidden units..12) for any output unit o.

.. resulting in an error signal do for each output unit.(12. the error signal is given by p p p p do = (do y o ) Fo' (S o ) .4 WORKING WITH BACK-PROPAGATION The application of the generalised delta rule thus involves two phases: During the first phase the input p x is presented and propagated forward through the network to compute the output values y o for each p output unit.1 Weight Adjustments with Sigmoid Activation Function The results from the previous section can be summarised in three equations: The weight of a connection is adjusted by an amount proportional to the product of an error signal d. F ¢ has to be applied to the delta. before the back-propagation process can continue.16) . In symbols: dh = åd w 0 0 ho Well.(12..20) The error signal for a hidden unit is determined recursively in terms of error signals of the units to which it directly connects and the weights of those connections.(12. on the unit k receiving the input and the output of the unit j sending this signal along the connection: Dpwkj = gdkp yjp If the unit is an output unit. not exactly: we forgot the activation function of the hidden unit.17) Take as the activation function F the sigmoid function as defined in chapter 2: y p = F(S p ) = 1 1 + e s p . This output is compared with its desired value do...(12. 12..4. 12..18) In this case the derivative is equal to ¶ 1 1 F ¢(S ) = p = e ¶S p 1 + e s 1 + e s p e j 2 e e j = sp 1 e e j p sp e1 + e j e s 2 1+ e sp j = y p(1 y p ) ..19) such that the error signal for an output unit can be written as: p p p p p do = (do yo ) y o o (1 yo ) . For the sigmoid activation function: . The second phase involves a backward pass through the network during which the error signal is passed to each unit in the network and appropriate weight changes are calculated..BACK-PROPAGATION 143 connection between those units.(12..

This problem can be overcome by using a permuted training method.(12. .. whereas for high learning rates the minimum is never reached because of the oscillations. the back-propagation algorithm performs gradient descent on the total error only if the weights are adjusted after the full set of learning patterns has been presented.21) 12. is to make the change in weight dependent of the past weight change by adding a momentum term: p Dwjk (t + 1) = gdk yjp+ aDwjk (t) . 12. more often than not the learning rule is applied to each pattern separately. it takes a long time before the minimum has been reached with a low learning rate.2 Learning Rate And Momentum The learning procedure requires that the change in weight is proportional to ¶E p .2. E p is calculated. For practical purposes we choose a learning rate that is as large as possible without leading to oscillation. a pattern p is applied. with the order in which the patterns are taught. 12.144 FUZZY LOGIC AND NEURAL NETWORKS p p d h = F ¢(S h ) å No p p dop who = y h(1 y h ) j =1 åd j =1 No p o who . .22) where t indexes the presentation number and a is a constant which determines the effect of the previous weight change. The role of the momentum term is shown in Fig. When adding the momentum term.(12.2 The descent in weight space. 12.4. True gradient descent ¶w requires that infinitesimal steps are taken. The constant of proportionality is the learning rate g. (b) for large learning rate: note the oscillations. There exists empirical indication that this results in faster convergence. One way to avoid oscillation at large.e. the minimum will be reached faster. When no momentum term is used.. and the weights are adapted (p = 1. (a) for small learning rate. and (c) with large learning rate and momentum term added. theoretically.3 Learning Per Pattern Although.. however. P). For example.. when using the same sequence over and over again the network may become focused on the first few patterns. i.4. 2.. Care has to be taken. b a c Fig.

Bottom right: The error in the approximation. Suppose we have a system (for example a chemical process or a financial market) of which we want to know the characteristics. while the function which generated the learning samples is given in Fig. 10 hidden units with sigmoid activation function and an output unit with a linear activation function. d p} as depicted in Fig. Bottom left: The function which generated the learning samples.20) should be adapted for the linear instead of sigmoid activation function. 12. 12. We want to estimate the relationship d = f(x) from 80 examples {x p. The input of the system is given by the two-dimensional vector x and the output is given by the one-dimensional vector d. The approximation error is depicted in Fig. Check for yourself how equation (4.3 (bottom right). The relationship between x and d as represented by the network is shown in Fig.3 (top left). described in the previous section.1: A feed-forward network can be used to approximate a function from examples.3 (top right). We see that the error is higher at the edges of the region within which the learning samples were generated. A feed-forward network was programmed with two inputs.3 Example of function approximation with a feed forward network.3 (bottom left). . The network weights are initialized to small values and the network is trained for 5. The network is considerably better at interpolation than extrapolation.000 learning iterations with the back-propagation training rule. 12. Top right: The approximation with the network.BACK-PROPAGATION 145 Example 12. 12. 12. 1 1 0 0 –1 1 1 0 – 1 –1 0 –1 1 0 0 –1 –1 1 1 1 0 0 –1 1 1 0 0 –1 –1 –1 1 1 0 –1 –1 0 Fig. Top left: The original learning samples.

4. The same function (albeit with other learning points) is learned with a network with eight sigmoid hidden units (see Figure 12. 12. from Fourier analysis it is known that any periodic function can be written as a infinite sum of sine and cosine terms (Fourier series): f(x) = n=0 å(an cos nx+ bn sin nx) åcn sin (nx + qn) n =1 ¥ ¥ . This can be a result of a non-optimum learning rate and momentum. The basic difference between the Fourier approach and the back-propagation approach is that the in the Fourier approach the weights between the input and the hidden units (these are the factors n) are fixed integer numbers which are analytically determined. Most troublesome is the long training process. . A lot of advanced algorithms based on back-propagation learning have some optimized method to adapt this learning rate. which make the algorithm not guaranteed to be universally useful. In some cases this leads to a formula. as will be discussed in the next section. The factor a0 corresponds with the bias of the output unit. For example. From the figures it is clear that it pays off to use as much knowledge of the problem at hand as possible.5). other functions can be used as well.5 OTHER ACTIVATION FUNCTIONS Although sigmoid functions are quite often used as activation functions. a single output unit for f (x) and hidden units with an activation function F = sin (s). whereas in the back-propagation approach these weights can take any value and are typically learning using a learning heuristic. the factors cn correspond with the weighs from hidden to output unit.. This can be seen as a feed-forward network with a single input unit for x. The result is depicted in Fig.146 FUZZY LOGIC AND NEURAL NETWORKS 12.. Outright training failures generally arise from two sources: network paralysis and local minima..(12.24) 2 2 with cn = an + bn and qn = arctan (b/a). which is known from traditional function approximation theories.6 DEFICIENCIES OF BACK-PROPAGATION Despite the apparent success of the back-propagation learning algorithm. four hidden units. the phase factor qn corresponds with the bias term of the hidden units and the factor n corresponds with the weights between the input and hidden layer.23) We can rewrite this as a summation of sine terms f(x) = a0 + . there are some aspects. To illustrate the use of other activation functions we have trained a feed-forward network with one output unit.(12.. and one input with ten patterns drawn from the function f (x) = sin(2x) sin(x). 12.

+1 –4 2 4 6 –1 Fig. 12.5 Fig. . 12.4 The periodic function B(N) = sin (2N) sin (N) approximated with sine activation functions.5 The periodic function B(N) = sin (2N) sin (N) approximated with sigmoid activation functions.BACK-PROPAGATION 147 +1 –4 –2 2 4 6 8 0.

Because of the gradient descent. a set of n directions is constructed which are all conjugate to each other such that minimization along one of these directions uj does not spoil the minimization along one of the earlier directions ui. May be the most obvious improvement is to replace the rather primitive steepest descent method with a direction set minimization method. again results in the system being trapped in local minima. Probabilistic methods can help to avoid this trap. Note that minimization along a direction u brings the function f at a place where its gradient is perpendicular to u (otherwise minimization along u is not complete). when exceeded. j i ¶2 f j p 1 xi xj + . conjugate gradient minimization. As is p clear from equations (12.. the network can get trapped in a local minimum when there is a much deeper minimum nearby. The total input of a hidden unit or output unit can therefore reach very high (either positive or negative) values.e. Flannery.. 12. Another suggested possibility is to increase the number of hidden units. Thus one minimization in the direction of ui suffices. the weight adjustments which are proportional to ykp (1 yk ) will be close to zero. It is too early for a full evaluation: some of these techniques may prove to be fundamental.1 Network Paralysis As the network trains.25) . Instead of following the gradient at every step.148 FUZZY LOGIC AND NEURAL NETWORKS 12. 12.20) and (12.7 ADVANCED ALGORITHMS Many researchers have devised improvements of and extensions to the basic back-propagation algorithm described above. Suppose the function to be minimized is approximated by its Taylor series f (x) = f (p) + where T denotes transpose.6. which directly minimizes in the direction of the steepest descent (Press.g. A few methods are discussed in this section.6. the directions are non-interfering. Although this will work because of the higher dimensionality of the error space..2 Local Minima The error surface of a complex network is full of hills and valleys. but they tend to be slow. & Vetterling. and the training process can come to a virtual standstill.» xT Ax bT x + c 2 ..21). it appears that there is some upper limit of the number of hidden units which. and the chance to get trapped is smaller.(12.. and because of the sigmoid activation function the unit will have an activation very close to zero or very close to one. others may simply fade away.. the weights can be adjusted to very large values. i. Teukolsky. e. such that n minimizations in a system with n degrees of freedom bring this system to a minimum (provided the system is quadratic). This is different from gradient descent. and c º f (p) å ¶x i ¶f i p xi + 1 2 å ¶x ¶x i. 1986).

BACK-PROPAGATION 149

b º Ñ f

¶2 f ¶xi ¶x j

p

[A]ij =

...(12.26)

p

A is a symmetric positive definite n ´ n matrix, the Hessian of f at p. The gradient of f is Ñf = Ax b such that a change of x results in a change of the gradient as d(Ñf ) = A(dx) ...(12.28) Now suppose f was minimized along a direction ui to a point where the gradient gi+ 1of f is perpendicular to ui, i.e., u iTgi + 1 = 0 ...(12.29) and a new direction ui+1is sought. In order to make sure that moving along ui+1 does not spoil minimization along ui we require that the gradient of f remain perpendicular to ui, i.e., u iTgi + 2 = 0 otherwise we would once more have to minimise in a direction which has a component of ui. Combining (12.29) and (12.30), we get 0 = u iT(gi+1 gi+2) = u iTd(Ñf) = u iTAui+1 When eq. (12.31) holds for two vectors ui and ui + 1 they are said to be conjugate. Now, starting at some point p0, the first minimization direction u0 is taken equal to g0 = Ñf (p0), resulting in a new point p1. For i ³ 0, calculate the directions ui+1 = gi +1 + giui where gi is chosen to make u iT Aui 1 and the successive gradients perpendicular, i.e.,

giT+ 1 gi + 1 giT gi

...(12.30)

...(12.27)

...(12.31)

...(12.32)

gi =

with gk = Ñf |pk for all k ³ 0

...(12.33)

Next, calculate pi+2 = pi+1 + li+1 ui+1where li+1 is chosen so as to minimize f(Pi + 2 )3. It can be shown that the us thus constructed are all mutually conjugate (e.g., see (Stoer & Bulirsch, 1980)). The process described above is known as the Fletcher-Reeves method, but there are many variants, which work more or less the same (Hestenes & Stiefel, 1952; Polak, 1971; Powell, 1977). Although only n iterations are needed for a quadratic system with n degrees of freedom, due to the fact that we are not minimizing quadratic systems, as well as a result of round-off errors, the n directions have to be followed several times (see Fig. 12.6). Powell introduced some improvements to correct for behaviour in non-quadratic systems. The resulting cost is O(n) which is significantly better than the linear convergence 4 of steepest descent.

150 FUZZY LOGIC AND NEURAL NETWORKS

Gradient ut +l

ut

**A very slow approximation
**

Fig. 12.6 Slow decrease with conjugate gradient in non-quadratic systems. [The hills on the left are very steep, resulting in a large search vector KE. When the quadratic portion is entered the new search direction is constructed from the previous direction and the gradient, resulting in a spiraling minimization. This problem can be overcome by detecting such spiraling minimizations and restarting the algorithm with K0 = ÑB ].

Some improvements on back-propagation have been presented based on an independent adaptive arning rate parameter for each weight. Van den Boomgaard and Smeulders (Boomgaard & Smeulders, 1989) show that for a feed-forward network without hidden units an incremental procedure to find the optimal weight matrix W needs an adjustment of the weights with Dw(t + 1) = g(t + 1) [d(t + 1) w(t) ´ (t + 1)] ´ (t + 1) ...(12.34) in which g is not a constant but an variable (Ni + 1) ´ (Ni + 1) matrix which depends on the input vector. By using a priori knowledge about the input signal, the storage requirements for can be reduced. Silva and Almeida (Silva & Almeida, 1990) also show the advantages of an independent step size for each weight in the network. In their algorithm the learning rate is adapted after every learning pattern:

RuC | | g (t + 1) = S |dC | T

jk

jk ( t ) jk ( t )

¶E (t + 1) ¶E (t ) and have the same signs ¶w jk ¶w jk ¶E (t + 1) ¶E (t ) if and have the opposite signs ¶w jk ¶w jk if

...(12.35)

BACK-PROPAGATION 151

where u and d are positive constants with values slightly above and below unity, respectively. The idea is to decrease the learning rate in case of oscillations.

**12.8 HOW GOOD ARE MULTI-LAYER FEED-FORWARD NETWORKS?
**

From the example shown in Fig. 12.3 is clear that the approximation of the network is not perfect. The resulting approximation error is influenced by: 1. The learning algorithm and number of iterations. This determines how good the error on the training set is minimized. 2. The number of learning samples. This determines how good the training samples represent the actual function. 3. The number of hidden units. This determines the expressive power of the network. For smooth functions only a few number of hidden units are needed, for wildly fluctuating functions more hidden units will be needed. In the previous sections we discussed the learning rules such as back-propagation and the other gradient based learning algorithms, and the problem of finding the minimum error. In this section we particularly address the effect of the number of learning samples and the effect of the number of hidden units. We first have to define an adequate error measure. All neural network training algorithms try to minimize the error of the set of learning samples which are available for training the network. The average error per learning sample is defined as the learning error rate error rate: Elearning =

1 Plearning

Plearning p=1

åE

p

...(12.36)

in which Ep is the difference between the desired output value and the actual network output for the learning samples: Ep =

1 2

å (d

0=1

No

p o

p yo )

...(12.37)

This is the error, which is measurable during the training process. It is obvious that the actual error of the network will differ from the error at the locations of the training samples. The difference between the desired output value and the actual network output should be integrated over the entire input domain to give a more realistic error measure. This integral can be estimated if we have a large set of samples. We now define the test error rate as the average error of the test set:

1 Etest = Ptest

åE

p=1

Ptest

p

...(12.38)

In the following subsections we will see how these error measures depend on learning set size and number of hidden units.

152 FUZZY LOGIC AND NEURAL NETWORKS

**12.8.1 The Effect Of the Number of Learning Samples
**

A simple problem is used as example: a function y = f(x) has to be approximated with a feed-forward neural network. A neural network is created with an input, 5 hidden units with sigmoid activation function and a linear output unit. Suppose we have only a small number of learning samples (e.g., 4) and the networks is trained with these samples. Training is stopped when the error does not decrease anymore. The original (desired) function is shown in Fig. 4.7A as a dashed line. The learning samples and the approximation of the network are shown in the same figure. We see that in this case Elearning is small (the network output goes perfectly through the learning samples) but Etest is large: the test error of the network is large. The approximation obtained from 20 learning samples is shown in Fig. 12.7B. The Elearning is larger than in the case of 5 learning samples, but the Etest is smaller.

1 A 1 B

0.8

0.8

0.6

y y

0.6

0.4

0.4

0.2

0.2

0

0

0.5 X

1

0

0

0.5 X

1

Fig. 12.7

Effect of the learning set size on the generalization. The dashed line gives the desired function, the learning samples are depicted as circles and the approximation by the network is shown by the drawn line. 5 hidden units are used. a) 4 learning samples. b) 20 learning samples.

This experiment was carried out with other learning set sizes, where for each learning set size the experiment was repeated 10 times. The average learning and test error rates as a function of the learning set size are given in Fig. 12.8. Note that the learning error increases with an increasing learning set size, and the test error decreases with increasing learning set size. A low learning error on the (small) learning set is no guarantee for a good network performance! With increasing number of learning samples the two error rates converge to the same value. This value depends on the representational power of the network: given the optimal weights, how good is the approximation. This error depends on the number of hidden units and the activation function. If the learning error rate does not converge to the test error rate the learning procedure has not found a global minimum.

**BACK-PROPAGATION 153
**

Error rate

Test set

Learning set

**Number of learning samples
**

Fig. 12.8 Effect of the learning set size on the error rate. The average error rate and the average test error rate are as a function of the number of learning samples.

12.8.2

The Effect of the Number of Hidden Units

The same function as in the previous subsection is used, but now the number of hidden units is varied. The original (desired) function, learning samples and network approximation is shown in Fig. 4.9A for 5 hidden units and in Fig. 4.9B for 20 hidden units. The effect visible in Fig. 4.9B is called over training. The network fits exactly with the learning samples, but because of the large number of hidden units the function which is actually represented by the network is far more wild than the original one. Particularly in case of learning samples which contain a certain amount of noise (which all real-world data have), the network will fit the noise of the learning samples instead of making a smooth approximation. This example shows that a large number of hidden units leads to a small error on the training set but not necessarily leads to a small error on the test set. Adding hidden units will always lead to a reduction of the Elearning. However, adding hidden units will first lead to a reduction of the Etest, but then lead to an increase of Etest. This effect is called the peaking effect. The average learning and test error rates as a function of the learning set size are given in Fig. 12.10.

12.9 APPLICATIONS

Back-propagation has been applied to a wide variety of research applications. Sejnowski and Rosenberg (1986) produced a spectacular success with NETtalk, a system that converts printed English text into highly intelligible speech. · A feed-forward network with one layer of hidden units has been described by Gorman and Sejnowski (1988) as a classification machine for sonar signals.

An example is the work of Josin (1988).10 The average learning error rate and the average test error rate as a function of the number of hidden units.4 0. who used a two-layer feed-forward network with back-propagation learning to perform the inverse kinematic transform which is needed by a robot arm controller.9 Effect of the number of hidden units on the network performance.154 FUZZY LOGIC AND NEURAL NETWORKS 1 A 1 B 0.6 0.5 X 1 0 0 0.8 0.5 X 1 Fig.2 0 0 0.6 y y 0. · A multi-layer feed-forward network with a back-propagation training algorithm is used to learn an unknown function between input and output signals from the presentation of examples.4 0. It is hoped that the network is able to generalize correctly. the circles denote the learning samples and the drawn line gives the approximation by the network. 12 learning samples are used. b) 20 hidden units. Error rate Test set Learning set Number of hidden units Fig. a) 5 hidden units. . 12.2 0. The dashed line gives the desired function.8 0. 12. so that input values which are not presented as learning patterns will result in correct output values.

8. 7. 11. 303-314. 4. Press. Explain learning rate and momentum with back-propagation with an example. 3. Explain the effect of the number of hidden links in multi-layer feed forward networks. Vol. 1989. 5. 9. Cambridge.E. pp. MA: Massachusetts Institute of Technology. Cybenko. 1990. Papert. 3. and Systems. Cun. Perceptrons: An Introduction to Computational Geometry. pp. Learning representations by back-propagating errors. 533-536.T. 210-215. Minsky. 6. 1986. Vol. J. pp. 3. Nos. 2. and J. .M. 2. 7. Explain the effect of the number of learning samples in multi-layer feed forward networks. Williams. Mathematics of Control. Hinton. K. Neural Computation. Layered neural networks with Gaussian hidden units as universal approximations.A. 10. Cambridge: Cambridge University Press. The MIT Press. 193-192. Vol. S. and R. D. Nature. What is back-propagation algorithm? Explain. 599-604. and W. 1. 9. 85. Hartman. What are the deficiencies of back-propagation algorithm? Explain various methods employed to overcome the deficiencies of back-propagation algorithm. Proceedings of Cognitiva. No.J. Signals. 4. Stinchcombe. M. Parker. Vol. 2. 359-366. Rumelhart. 4. B.J. Approximation by superpositions of a sigmoidal function. No. 5. Bulirsch. Vol. 323. Explain the multi-layer feed forward networks. 12. New York-Heidelberg. Neural Networks. K. Multilayer feed forward networks are universal approximates. Teukolsky. Introduction to Numerical Analysis. 6. No. What are the applications of back-propagation algorithm? REFERENCES. Keeler. Flannery. How the weights are adjusted with sigmoid activation function? Explain with an example. 10. and S. Vol. 5. Y. Funahashi. 1985. 1. Hornik. 1985. On the approximate realization of continuous mappings by neural networks. White. J. Numerical Recipes: The Art of Scientific Computing. 2. 1986. and R. How good are multi-layer feed forward networks? Explain. 2. and H. 8. G. M. Vol.P. W. Describe the generalized delta rule. Learning-Logic (Tech. Vetterling.L. pp.E. TR (47). 2. Kowalski.D. G. 2. L. Une procedure dapprentissage pour reseau a seuil assymetrique. Explain the sine activation function with an example. Stoer. E. D.B.I. pp. Neural Networks. 1989. Rep.BACK-PROPAGATION 155 QUESTION BANK. No. 1980. 1969.Berlin: Springer-Verlag. Center for Computational Research in Economics and Management Science.H. Y. 1989.

Neural-space generalization of a topological transformation. pp. M. Mathematical Programming. 1. Sejnowski. and C.P. Neural Networks.J. 49. 16. Gorman. pp. NETtalk: A Parallel Network that Learns to Read Aloud (Tech. Biological Cybernetics. Rosenberg. Rep. pp. JHU/EECS-86/01). New York: Academic Press. Stiefel. Vol. 59. 1988.156 FUZZY LOGIC AND NEURAL NETWORKS 11. M. Restart procedures for the conjugate gradient method.R. 1. 1986. Vol. . 12. Vol. and E. 75-89. E. and T. 1971.R. 12. Nos. Sejnowski. Analysis of hidden units in a layered network trained to classify sonar targets. T. 15. No.D. Vol. R. 1952. 13. Methods of conjugate gradients for solving linear systems. Journal of National Bureau of Standards.J. Polak. 283-290. 14. 1977.J. Powell. G. Josin. 409-436. Hestenes. Computational Methods in Optimization. pp. 241-254. 1988. The John Hopkins University Electrical Engineering and Computer Science Department.

but there are also recurrent networks where the learning rule is used after each propagation (where an activation value is transversed over each weight only once). etc. however.4 Recurrent Networks 13 13. to solve the same problem. the activation values in the network are repeatedly updated until a stable point is reached after which the weights are adapted. 13. recurrent extensions to the feed-forward network will be discussed. subsequently we touch upon Boltzmann machines. which can be used for the representation of binary patterns. 1 INTRODUCTION The learning algorithms discussed in the previous chapter were applied to feed-forward networks: all data flows in a network in which no cycles are present. therewith introducing stochasticity in neural computation. while external inputs are included in each propagation. we can connect a hidden unit with itself over a weighted connection. there exist recurrent network. the approximation capabilities of such networks do not increase. it is possible to continue propagating activation values until a stable point (attractor) is reached.. As we will see in the sequel. connect hidden units to input units. which are attractor based. Before we will consider this general case. In such networks. can be easily used for training patterns in recurrent networks. when one is considering a recurrent network. Yet the basics of these networks will be discussed. But what happens when we introduce a cycle? For instance. Also some special recurrent networks will be discussed: the Hopfield network.+ 0 ) 2 6 . Although. as we know from the previous chapter. we will first describe networks . network size. In this chapter. we may obtain decreased complexity. or even connect all units with each other.2 THE GENERALISED DELTA . An important question we have to consider is the following: what do we want to learn in a recurrent network? After all.RULE IN RECURRENT NETWORKS The back-propagation learning rule. i. the recurrent connections can be regarded as extra inputs to the network (the values of which are computed by the network itself).e. The theory of the dynamics of recurrent networks extends beyond the scope of a one-semester course on neural networks. introduced in chapter 12.

x¢¢. . An example of this network is shown in Fig. Due to the recurrent connections. 13. or where output values are fed back into hidden units (the Jordan network). which is slow and difficult to train.1 The Jordan Network One of the earliest recurrent neural networks was the Jordan network. The Jordan and Elman networks provide a solution to this problem. that the input dimensionality of the feed-forward network is multiplied with n. x(t 1). .1 The Jordan network. Suppose we have to construct a network that must generate a control command depending on an external input. we also input its first. Naturally. etc. instead.2. 2. . x(t 2). Create inputs x1. computation of these derivatives is not a trivial task for higher-order derivatives. x¢. . Besides only inputting x(t). the activation values of the output units are fed back into the Input units h o State units Fig. With a feed-forward network there are two possible approaches: 1. 13. In the Jordan network. x2. The disadvantage is. Thus a time window of the input vector is input to the network. of course. xn which constitute the last n values of the input vector. to a set of extra neurons called the state units. derivatives. 13. Create inputs x.1. leading to a very large network. Output activation values are fed back to the input layer.158 FUZZY LOGIC AND NEURAL NETWORKS where some of the hidden unit activation values are fed back to an extra set of input units (the Elman network). which is a time series x(t). second. a window of inputs need not be input anymore. the network is supposed to learn the influence of the previous time steps itself. A typical application of such a network is the following.

Thus the network is very similar to the Jordan network.1: As we mentioned above. the Jordan and Elman networks can be used to train a network on reproducing time sequences. Again the hidden units are connected to the context units with a fixed weight of value +1.RECURRENT NETWORKS 159 input layer through a set of extra input units called the state units. There are as many state units as there are output units in the network. since the object suffers from friction and perhaps other external forces. 4. 13.2. we trained an Elman network on controlling an object moving in 1 D. 13. The schematic structure of this network is shown in Fig. The back-propagation learning rule is applied. 13.2 The Elman Network In the Elman network a set of context units are introduced. t = 1 2. 3. which are extra input units whose activation values are fed back from the hidden units. to a set of extra neurons called the context units. Output layer Hidden layer Input layer Fig. the forward calculations are performed once. This object has to follow a pre-specified trajectory xd. Example 13.2 Context layer The Elman network. except that (1) the hidden units instead of the output units are fed back. To control the object. learning takes place only in the connections between input and hidden units as well as hidden and output units. and (2) the extra input units have no self-connections. With this network. The idea of the recurrent connections is that the network is able to remember the previous states of the input values. . Learning is done as follows: 1. The connections between the output and state units have a fixed weight of +1. The context units at step t thus always have the activation value of the hidden units at step t 1. Pattern xt is clamped.2. Thus all the learning rules derived for the multi-layer perceptron can be used to train this network. As an example. go to 2. t ¬ t + 1. The context units are set to 0. forces F must be applied. the hidden unit activation values are fed back to the input layer.

4 Training a feed-forward network to control an object. 13. The results of training are shown in Fig. The hidden units are connected to three context units. The solid line depicts the desired trajectory N@. 4 2 0 100 200 300 400 500 –2 –4 Fig. . the dashed line the realized trajectory. we use an Elman net with inputs x and xd. We tested this with a network with five inputs.3. and three hidden units. 13. five units feed into the hidden layer.3 Training an Elman network to control an object. The same test can be done with an ordinary feedforward network with sliding window input. 13. the dashed line the realized trajectory. one output F. In total.160 FUZZY LOGIC AND NEURAL NETWORKS To tackle this problem. The third line is the error. The solid line depicts the desired trajectory N@. four of 4 2 0 100 200 300 400 500 –2 –4 Fig. The third line is the error.

independently of each other Pineda (1987) and Almeida (1987) discovered that error back-propagation is in fact a special case of a more general gradient learning method. All connections are weighted. 13. All neurons are both input and output neurons. the network iterates to a stable state.5). However. i ¹ j (see Fig. The disappointing observation is that the results are actually better with the ordinary feed-forward network. the discussion of which extents beyond the scope of our course.5 The auto-associator network. 13.. can be used to train a multi-layer perceptron to follow trajectories in its activation values.e. Hopfiled (1982) brings together several earlier ideas concerning these networks and presents a complete mathematical analysis. and one the desired next position of the object. For instance. x2. x1 and x0. Fig.RECURRENT NETWORKS 161 which constituted the sliding window x3. 1989. which has the same complexity as the Elman network.4. and the output of the network consists of the new activation values of the neurons.3 Back-Propagation in Fully Recurrent Networks More complex schemes than the above are possible.3 THE HOPFIELD NETWORK One of the earliest recurrent neural networks reported in literature was the auto-associator independently described by Anderson (1977) and Kohonen (1977). 13. This learning method. a learning method can be used: back-propagation through time (Pearlmutter. Results are shown in Fig. which can be used for training attractor networks. 15. 13. 1990). i. .2. It consists of a pool of neurons with connections between each unit i and j. also when a network does not reach a fixed point. a pattern is clamped.

3) A state a is called stable if.(13. yk(t) = sgn (Sk(t 1)) . the behavior of the system can be described with an energy function e= 1 2 j¹k åå y j yk w jk - åq k k yk . (13.. We will therefore adhere to the latter convention. when the network is in state a.e. .2).4) is bounded from below.1) and (13. A neuron k in the Hopfield network is called stable at time t if.. 10.5) is always negative when yk changes according to eqs.. A pattern xp is called stable if..2) i. yk (t + 1) = sgn (Sk (t + 1)) For simplicity we henceforth choose Uk = 0. all neurons are stable..(13.2). but using values +1 and 1 presents some advantages discussed below.2) is applied to the net input to obtain the new activation value yi(t + 1) at time t + 1: R+ 1 | y (t + 1) = S.(13.1 | y (t ) T k k if Sk (t + 1) > U k if Sk (t + 1) < U k otherwise . e is monotonically decreasing when state changes occur. but this is of course not essential. Proof: First.. Originally. when xp is clamped. The activation values are binary..162 FUZZY LOGIC AND NEURAL NETWORKS 13. all neurons are stable.1 Description The Hopfield network consists of a set of N interconnected neurons (Fig. because De = Dyk F yw GH å j j¹k jk + qk I JK .. (13. 13. note that the energy expressed in eq.5).. since the yk are bounded from below and the wjk and qk are constant.(13. which update their activation values asynchronously and independently of other neurons.1) A simple threshold function (Fig. The net input Sk(t + 1) of a neuron k at cycle t + 1 is a weighted sum Sk(t + 1) = å yj (t)wjk + qk j¹k .1) and (13. Hopfield chose activation values of 1 and 0. The state of the system is given by the activation values Y = y(k).3. in accordance with equations (13. Secondly..4) Theorem 13.2) has stable limit points.1: A recurrent network with connections wjk = wkj in which the neurons are updated using rule (13.(13. All neurons are both input and output neurons.. When the extra restriction wjk = wkj is made.

1983).6) p i. wjk is increased. (Bruce. There are two problems associated with storing too many patterns: 1. whereas in the 1/0 model this is not always true (as an example.. weights only increase). where the algorithm remains oscillatory (try to find one)! The second problem stated above can be alleviated by applying the Hebb rule in reverse to the spurious stable state. Forrest. and that about 0:15N memories can be stored before recall errors become severe.(13. 13. In this case. The Hebb rule can be used to store P patterns: wjk R xx | = Så |0 T p p =1 p p j k if j ¹ k otherwise . which is in some sense near to the cued pattern. Spurious stable states appear (i.(13. Algorithm 13. however.2 Hopfield Network as Associative Memory A primary application of the Hopfield network is an associative memory. both a pattern and its inverse have the same energy in the +1/1 model. the weights of the connections between the neurons have to be thus set that the states of the system corresponding with the patterns which are to be stored in the network are stable. stable states which do not correspond with stored patterns). Thus these patterns are weakly unstored and will become unstable again. for each pattern x p to be stored and each element x kp in x p define a correction ek such that Ak = R0 S1 T if yk is stable and x p is clamped otherwise .. Gardner. Repeat this procedure until all patterns are stable. wjk = wkj) results in a system that is not guaranteed to settle to a stable state.. its inverse is stable. Similarly.RECURRENT NETWORKS 163 The advantage of a + 1/1 model over a 1/0 model then is symmetry of the states of the network. otherwise decreased by one (note that. For. Feinstein. & Wallace. if xjp and xk are equal. It appears that. Removing the restriction of bidirectional connections (i.7) Now modify wjk by Dwjk = yj yk(ej +ek) if j ¹ k. but 11 11 need not be). this algorithm usually converges. that the network gets saturated very quickly. The first of these two problems can be solved by an algorithm proposed by Bruce et al.1: Given a starting weight matrix W = [wjk]. but with a low learning factor (Hopfield.e. too.. It appears. When the network is cued with a noisy or incomplete test pattern. .. The stored patterns become unstable. These states can be seen as dips in energy space.. it will render the incorrect or missing data by iterating to a stable state. Canning.e..3. 1986). 2.e. when some pattern x is stable. in the original Hebb rule. & Palmer. however. in practice. There exist cases. the pattern 00 00 is always stable.

each neuron has an external bias input Cn.9) is added to the energy. The weights are set as follows: wXJ. few of which lead . Hopfield and Tank (1985) use a network with n ´ n neurons. To ensure a correct solution. a path of minimal distance must be found between n cities. j . For example. As before. the subscripts are defined modulo n. such that the begin. respectively. indicating a specific city occupying a specific position in the tour.4 Hopfield Networks for Optimization Problems An interesting application of the Hopfield network with graded response arises in a heuristic solution to the NP-complete traveling salesman problem (Garey & Johnson. The last term is zero if and only if there are exactly n active neurons. When the network is settled. Whereas Hopfield and Tank state that the network converges to a valid solution in 16 out of 20 trials while 50% of the solutions are optimal. (Wilson and Pawley. each row and each column should have one and only one active neuron.3.. whereas each column represents the position in the tour. 13.10) . 1984). To minimise the distance of the tour. The activation value yxj = 1 indicates that city X occupies the jth place in the tour.3.nJ K 2 . In this problem. an extra term e= D 2 å ååd X Y¹X j XY y Xj ( yY .(13. An energy function describing this problem can be set up as follows..164 FUZZY LOGIC AND NEURAL NETWORKS 13.1 can be generalized by allowing continuous activation values.and end-points are the same.3.. j +1 + yY .8) are zero if and only if there is a maximum of one active neuron in each row and column. For convenience.3 Neurons with Graded Response The network described in section 13. the threshold activation function is replaced by a sigmoid.. j+1 + dk. 1979).8) where A. 1988) find that in only 15% of the runs a valid result is obtained. Here. Each row in the matrix represents a city. The first and second terms in equation (13.1 ) . Yk = AdXY (1 djk) inhibitory connections within each row = Bdjk(1 dXY) inhibitory connections within each column = C global inhibition = DdXY(dk..(13. this system can be proved to be stable when a symmetric weight matrix is used (Hopfield. other reports show less encouraging results. Finally. and C are constants. j1) data term where djk = 1if j = k and 0 otherwise. where dXY is the distance between cities X and Y and D is a constant.. B.(13. the applicability is limited. Although this application is interesting from a theoretical point of view.2) with a sigmoid activation function between 0 and 1. The neurons are updated using rule (13. the following energy must be minimized: A e= 2 ååå y X j k¹ j Xj y Xk B + 2 åå å j X C y Xj yYj + 2 X ¹Y F y GH å å X j Xj I .

RECURRENT NETWORKS 165 to an optimal or near-optimal solution. At high temperatures.12) where Pa is the probability of being in the ath global state. and Sejnowski in 1985 is a neural network that can be seen as an extension to Hopfield networks to include hidden units. the network will ignore small energy differences and will rapidly approach equilibrium. very slowly to a freezing point. it will perform a search of the coarse overall structure of the space of global states. in which a neuron becomes active with a probability p. The weights are still symmetric. .11) where T is a parameter comparable with the (synthetic) temperature of the system. As the temperature is lowered. as first described by Ackley. each of which may be traversed in two directions as well as started in N points. A good way to beat this trade-off is to start at a high temperature and gradually reduce it. without any impurities. Differently put. This is a process whereby a material is heated and then cooled very. such that all but one of the final 2N configurations are redundant. Note that at thermal equilibrium the units still change state.(13. but the probability of finding the network in any global state remains constant. and with a stochastic instead of deterministic update rule.( e . but the time required to reach equilibrium may be long.e )/ T = e a b Pb . Hinton. Since. the network will eventually reach thermal equilibrium and the relative probability of two global states a and b will follow the Boltzmann distribution Pa ..4 BOLTZMANN MACHINES The Boltzmann machine.. and ea is the energy of that state. In accordance with a physical system obeying a Boltzmann distribution. In the Boltzmann machine this system is mimicked by changing the deterministic update of equation (13. As a result. 13. At higher temperatures the bias is not so favorable but equilibrium is reached faster. This stochastic activation function is not to be confused with neurons having a sigmoid deterministic activation function. the crystal lattice will be highly ordered.(13. it will begin to respond to smaller energy differences and will find one of the better minima within the coarse-scale minimum it discovered at high temperature.De k / T 1 . p(yk ¬ + 1) = 1 + e . The degenerate solutions occur evenly within the hypercube. The main problem is the lack of global information. The operation of the network is based on the physics principle of annealing. there are N! possible tours. for an N-city problem.. the number of different tours is N!/2N. such that the system is in a state of very low energy. the N-dimensional hypercube in which the solutions are situated is 2N degenerate. In doing so. At low temperatures there is a strong bias in favor of states with low energy.2) in a stochastic update. The competition between the degenerate tours often leads to solutions which are piecewise optimal but globally inefficient. and will find a good minimum at that coarse level..

**166 FUZZY LOGIC AND NEURAL NETWORKS
**

As multi-layer perceptions, the Boltzmann machine consists of a non-empty set of visible and a possibly empty set of hidden units. Here, however, the units are binary-valued and are updated stochastically and asynchronously. The simplicity of the Boltzmann distribution leads to a simple learning procedure, which adjusts the weights so as to use the hidden units in an optimal way (Ackley et al., 1985). This algorithm works as follows: First, the input and output vectors are clamped. The network is then annealed until it approaches thermal equilibrium at a temperature of 0. It then runs for a fixed time at equilibrium and each connection measures the fraction of the time during which both the units it connects are active. This is repeated for all input-output pairs so that each connection can measure (yj yk)clamped, the expected probability, averaged over all cases, that units j and k are simultaneously active at thermal equilibrium when the input and output vectors are clamped. Similarly, (yj yk)free is measured when the output units are not clamped but determined by the network. In order to determine optimal weights in the network, an error function must be determined. Now, the probability Pfree(Y p) that the visible units are in state Y pwhen the system is running freely can be measured. Also, the desired probability Pclamped(Y p)that the visible units are in state (Y p) is determined by clamping the visible units and letting the network run. Now, if the weights in the network are correctly set, both probabilities are equal to each other, and the error E in the network must be 0. Otherwise, the error must have a positive value measuring the discrepancy between the networks internal mode and the environment. For this effect, the asymmetric divergence or Kullback information is used: E=

åP

p

clamped

(Y p ) log

P clamped (Y p ) P free (Y P )

...(13.13)

Now, in order to minimize E using gradient descent, we must change the weights according to Dwjk = g It is not difficult to show that

¶E ¶w jk

...(13.14)

**¶E 1 = ( y j yk ) clamped - ( y j yk ) free ¶w jk T
**

Therefore, each weight is updated by Dwjk = g ( y j yk ) clamped - ( y j yk ) free

...(13.15)

...(13.16)

RECURRENT NETWORKS 167

QUESTION BANK.

1. 2. 3. 4. 5. 6. 7. 8. 9. What happens when a cyclic data is introduced to feed forward networks? Explain the generalized delta-rule in recurrent networks. Describe the Jordan network with an example. Describe Elman network with an example. Describe the Hopfield network. Describe the Hopfield network as associative memory. Describe Hopfield network for optimization problems. Describe the Boltzman machine. What are the problems resulted while storing too many patterns using associative memory? How these problems can be solved?

REFERENCES.

1. M.I. Jordan, Attractor dynamics and parallelism in a connectionist sequential machine, In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, Hillsdale, NJ: Erlbaum, pp. 531-546, 1986. 2. M.I. Jordan, Serial Order: A Parallel Distributed Processing Approach (Tech. Rep. No. 8604). San Diego, La Jolla, CA: Institute for Cognitive Science, University of California, 1986. 3. J.L. Elman, Finding structure in time. Cognitive Science, Vol. 14, pp. 179-211, 1990. 4. F. Pineda, Generalization of back-propagation to recurrent neural networks, Physical Review Letters, Vol. 19, and pp. 2229-2232, 1987. 5. L.B. Almeida, A learning rule for asynchronous perceptrons with feedback in a combinatorial environment, In Proceedings of the First International Conference on Neural Networks, Vol. 2, pp. 609-618,1987. 6. B.A. Pearlmutter, Learning state space trajectories in recurrent neural networks, Neural Computation, Vol. 1, No. 2, pp. 263-269, 1989. 7. B.A. Pearlmutter, Dynamic Recurrent Neural Networks (Tech. Rep. Nos. CMU-CS-90-196), Pittsburgh, PA 15213: School of Computer Science, Carnegie Mellon University, 1990. 8. J.A. Anderson, Neural Models with Cognitive Implications. In D. LaBerge and S.J. Samuels (Eds.), Basic Processes in Reading Perception and Comprehension Models, Hillsdale, NJ: Erlbaum, pp. 27-90, 1977. 9. T. Kohonen, Associative Memory: A System-Theoretical Approach, Springer-Verlag, 1977. 10. J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences, Vol. 79, pp. 2554-2558, 1982. 11. A.D. Bruce, A. Canning, B. Forrest, E. Gardner, and D.J. Wallace, Learning and memory properties in fully connected networks, In J.S. Denker (Ed.), AIP Conference Proceedings 151, Neural Networks for Computing, pp. 65-70, DUNNO, 1986.

**168 FUZZY LOGIC AND NEURAL NETWORKS
**

12. J.J. Hopfield, D.I. Feinstein, and R.G. Palmer, unlearning has a stabilizing effect in collective memories, Nature, Vol. 304, pp. 159-159, 1983. 13. J.J. Hopfield, Neurons with graded response have collective computational properties like those of two-state neurons, Proceedings of the National Academy of Sciences, Vol. 81, pp. 3088-3092, 1984. 14. M.R. Garey, and D.S. Johnson, Computers and Intractability. New York: W.H. Freeman, 1979. 15. J.J. Hopfield, and D.W. Tank, neural computation of decisions in optimization problems, Biological Cybernetics, Vol. 52, pp. 141-152, 1985. 16. G.V. Wilson, and G.S. Pawley, On the stability of the traveling salesman problem algorithm of Hopfield and tank, Biological Cybernetics, Vol. 58, pp. 63-70, 1988. 17. D.H. Ackley, G.E. Hinton, and T.J. Sejnowski, (1985). A learning algorithm for Boltzmann machines, Cognitive Science, Vol. 9, No. 1, pp. 147-169, 1985.

+ 0 ) 2 6 - 4

Self-Organizing Networks

14

14. 1

INTRODUCTION

In the previous chapters we discussed a number of networks, which were trained to perform a mapping F: Â¢¢ ® Âm by presenting the network examples (x p, d p) with d p = F(x p) of this mapping. However, problems exist where such training data, consisting of input and desired output pairs are not available, but where the only information is provided by a set of input patterns x p. In these cases the relevant information has to be found within the (redundant) training samples x p. Some examples of such problems are: Clustering: the input data may be grouped in clusters and the data processing system has to find these inherent clusters in the input data. The output of the system should give the cluster label of the input pattern (discrete output); Vector quantisation: this problem occurs when a continuous space has to be discretized. The input of the system is the n-dimensional vector x, the output is a discrete representation of the input space. The system has to find optimal discretization of the input space; Dimensionality reduction: the input data are grouped in a subspace, which has lower dimensionality than the dimensionality of the data. The system has to learn an optimal mapping, such that most of the variance in the input data is preserved in the output data; Feature extraction: the system has to extract features from the input signal. This often means a dimensionality reduction as described above. In this chapter we discuss a number of neuro-computational approaches for these kinds of problems. Training is done without the presence of an external teacher. The unsupervised weight adapting algorithms are usually based on some form of global competition between the neurons. There are very many types of self-organizing networks, applicable to a wide area of problems. One of the most basic schemes is competitive learning as proposed by Rumelhart and Zipser (1985). A very similar network but with different emergent properties is the topology-conserving map devised by Kohonen. Other self-organizing networks are ART, proposed by Carpenter and Grossberg (1987), and Fukushima (1975).

170 FUZZY LOGIC AND NEURAL NETWORKS

14.2 14.2.1 Clustering

COMPETITIVE LEARNING

Competitive learning is a learning procedure that divides a set of input patterns in clusters that are inherent to the input data. A competitive learning network is provided only with input vectors x and thus implements an unsupervised learning procedure. We will show its equivalence to a class of traditional clustering algorithms shortly. Another important use of these networks is vector quantisation. An example of a competitive learning network is shown in Fig. 14.1. All output units o are connected to all input units i with weights wio. When an input pattern x is presented, only a single output unit of the network (the winner) will be activated. In a correctly trained network, all x in one cluster will have the same winner. For the determination of the winner and the corresponding learning rule, two methods exist.

O

wio i

Fig. 14.1 A simple competitive learning network. Each of the four outputs o is connected to all inputs i.

**Winner Selection: Dot Product
**

For the time being, we assume that both input vectors x and weight vectors wo are normalized to unit length. Each output unit o calculates its activation value yo according to the dot product of input and weight vector: yo =

å

i

wio xi = w T x o

...(14.1)

In a next pass, output neuron k is selected with maximum activation "o ¹ k : yo £ yk ...(14.2) Activations are reset such that yk = 1 and yo ¹ k = 0. This is the competitive aspect of the network, and we refer to the output layer as the winner-take-all layer. The winner-take-all layer is usually implemented in software by simply selecting the output neuron with highest activation value. This function can also be performed by a neural network known as MAXNET (Lippmann, 1989). In MAXNET, all neurons o are connected to other units o0 with inhibitory links and to itself with an excitatory link: wo, o =

R- e S+ 1 T

if o ¹ o¢ otherwise

...(14.3)

Each time an input x is presented.4) effectively rotates the weight vector wo towards the input vector x. 14. 14.SELF-ORGANIZING NETWORKS 171 It can be shown that this network converges to a situation where only the neuron with highest initial activation survives.1) and (14.5) reduces to (14. whereas the activations of all other neurons converge to zero. Once the winner k has been selected. Note that only the weights of winner k are updated. the weights are updated according to: wk(t + 1) = wk (t ) + g ( x(t ) . using the Euclidean distance measure: k: ||wk x||£||wo x|| "o . the weight update must be changed to implement a shift towards the input: wk(t + 1) = wk(t) + g (x(t) wk(t)) . This procedure is visualized in Fig. weight vectors are rotated towards those areas where many inputs appear: the clusters in the input. Instead of rotating the weight vector towards the input as performed by equation (14.(14. the weight vector closest to this input is selected and is subsequently rotated towards the input.4) where the divisor ensures that all weight vectors w are normalized. The weight update given in equation (14.2. Naturally one would like to accommodate the algorithm for normalized input data.4). The three weight vectors are rotated towards the centers of gravity of the three different input clusters. 14.5) It is easily checked that equation (14.(14. Winner selection: Euclidean distance Previously it was assumed that both inputs x and weight vectors w were normalized.... In Fig.2 Example of clustering in 3D with normalized vectors. we will simply assume a winner k is selected without being concerned which algorithm is used. Consequently.wk (t ))|| .1) gives a biological plausible solution.. From now on. Using the activation function given in equation (14..(14.wk (t )) || wk (t ) + g ( x(t ) . which all lie on the unity sphere..6) .3 it is shown how the algorithm would fail if normalized vectors were to be used. To this end. Weight vector Pattern vector w1 w3 w2 Fig. the winning neuron k is selected with its weight vector wk closest to the input pattern x.2) if all vectors are normalized.2). The Euclidean distance norm is therefore a more general case of equations (14.1) and (14.

a. This is implemented by expanding the weight update given in equation (14. . the dot product xTw1 is still larger than xTw2. Conversely.. vectors x and w1 are nearest to each other. Another more thorough approach that avoids these and other problems in competitive learning is called leaky learning.. b.. Especially if the input vectors are drawn from a large or high-dimensional input space.. The three vectors having the same directions as in a.e. In a. Chen.. 14..3 Determining the winner in a competitive learning network. The weights w are interpreted as cluster centres. However. each neuron records the number of times it is selected winner. & Melton.172 FUZZY LOGIC AND NEURAL NETWORKS W1 W1 X W2 X W2 a b Fig. however. it is customary to initialize weight vectors to a set of input patterns {x} drawn from the input set at random. Krishnamurthy. 1990). Again only the weights of the winner are updated. and their dot product xTw1 = |x||w1| cos a is larger than the dot product of x and w2. i. In b. but with different lengths. it is not beyond imagination that a randomly initialized weight vector wo will never be chosen as the winner and will thus never be moved and never be used. It is not difficult to show that competitive learning indeed seeks to find a minimum for this square error by the negative gradient of the error-function. the less sensitive it becomes to competition. Therefore. Cost function: Earlier it was claimed. neurons that consistently fail to win increase their chances of being selected winner. the pattern and weight vectors are not normalized. A somewhat similar method is known as frequency sensitive competitive learning (Ahalt. as discussed before.. Three normalized vectors.(14.. Similarity is measured by a distance function on the input vectors.6) with wl(t + 1) = wl(t) + g ¢(x(t) wl(t)) "l ¹ k . that a competitive network performs a clustering process on the input data. given by E= å ||wk x p||2 p . A common criterion to measure the quality of a given clustering is the square error criterion.(14.7) with g ¢ < g the leaky learning rate.8) where k is the winning neuron when input xp is presented. The more often it wins. In this algorithm. input patterns are divided in disjoint clusters such that similarities between input patterns in the same cluster are much bigger than similarities between inputs in different clusters. and in this case w2 should be considered the winner when x is applied. A point of attention in these recursive clustering techniques is the initialization.

(14.12) Example 14.5 Fig.10) where g is a constant of proportionality. we calculate the effect of a weight change on the error function. Proof: that As in eq.5 1 Competitive learning for clustering data. (3.1: The error function for pattern x p Ep = å ||wk x p ||2 p .SELF-ORGANIZING NETWORKS 173 Theorem 14.3 0. (14.9) where k is the winning unit.4 0.6) written down for one element of wo. 1 0.6 0. So we have DpWio = C ¶E p ¶wio .xip ¶E p = ¶wio 0 such that R S T if unit o wins otherwise .6). . The data are given by +. The network was trained with g = 0:1 and a g ¢ = 0:001 and the positions of the weights after 500 iterations are shown.8) is minimized by repeated weight updates using eq.5 0. 14.1: In Fig.8 0.6).12).2 0.(14..11) p Dpwio = g (wio x ip) = g (x o wio) which is eq.9 0.. (14..(14. Now.7 + 0. 8 clusters of each 6 data points are depicted. eq. is minimised by the weight update rule in eq.4 ++ + ++ + + ++ + + + + + + + 0 0.. .4...1 0 – 0.(14. (14. (14. The positions of the weight vectors after 500 iterations is given by o.. Therefore. we have to determine the partial derivative of E p: wio . 14. A competitive learning network using Euclidean distance to select the winner was initialized with all weight vectors wo = 0..

the upper part of the input space is divided into two large separate regions. competitive learning has also be used in combination with supervised learning methods. five neurons discretized the input space into five smaller subspaces. 14. A vector quantisation scheme divides the input space in a number of disjoint subspaces and represents each input vector x by the label of the subspace it falls into (i. In this way. We will describe two examples: the counter propagation method and the learning vector quantisation.2. In the areas where inputs are scarce.2. the upper part of the figure.e. index k of the winning neuron). x2 x1 Input pattern Weight vector Fig.3 Counter Propagation In a large number of applications. however. whereas a more coarse quantisation is obtained in those areas where inputs are scarce. The quantisation performed by the competitive learning network is said to track the input probability density function: the density of neurons and thus subspaces is highest in those areas where inputs are most likely to appear. An example of tracking the input density is sketched in Figure 14.. where many more inputs have occurred. The input patterns are drawn from Â2. The lower part. Vector quantisation through competitive learning results in a more fine-grained discretization in those areas of the input space where most input have occurred in the past.5. but more in quantising the entire input space. An example of such a network is given in .174 FUZZY LOGIC AND NEURAL NETWORKS 14. However.2 Vector Quantisation Another important use of competitive learning networks is found in vector quantisation. only few (in this case two) neurons are used to discretized the input space. 14. the weight vectors also lie in Â2.5 This figure visualizes the tracking of the input density. networks that perform vector quantisation are combined with another type of network in order to perform function approximation. The difference with clustering is that we are not so much interested in finding clusters of similar data. and be applied to function approximation problems or classification problems. competitive learning can be used in applications where data has to be compressed such as telecommunication or storage. Thus.

. one can choose to perform the vector quantisation before learning the function approximation. wmo]T which is somehow representative for the function values f(x) of inputs x represented by o. w2o. and the function value [w1k. h)dx . This network can be used to approximate functions from Â2 to Â2. This way of approximating a function effectively implements a look-up table: an input x is assigned to a table entry k with "o ¹ k: ||x wk||£||x wo||. 2. Update the weights wih with equation (14.6). 14. Vector quantisation h i o y Wih Fig.SELF-ORGANIZING NETWORKS 175 Fig..14) It can be shown that this learning procedure converges to who = z yog (x. k) as: g(x. As an example of the latter.. wmk]T in this table entry is taken as an approximation of f(x).6 Feedforward Who A network combining a vector quantisation layer with a 1-layer feed-forward neural network. 14. A well-known example of such a network is the Counter propagation network (Hecht-Nielsen. k) = h .13) å yhwho = wko when k is the winning neuron and the desired R1 S0 T Ân if k is winner otherwise .(14. Present the network with both input x and function value d = f (x).. the network presented in Fig. or one can choose to learn the quantisation and the approximation layer simultaneously. .6 can be supervisedly trained in the following way: 1. the input space Â2 is discretized in 5 disjoint subspaces.15) . 1988). 3..6. Perform the unsupervised quantisation step.(14. Perform the supervised approximation step: wko(t + 1) = wko(t) + g (do wko(t)) This is simply the d rule with yo = output is given by d = f(x). . This network can approximate a function f : Ân ® Âm by associating with each neuron o a function value [w1o.. If we define a function g(x. Depending on the application. For each weight vector.(14. w2k. 14. calculate the distance from its weight vector to the input pattern and find winner k.

g. a class label (or decision of some other kind) yo is associated.. given a large set of exemplary decisions (the training set). A learning sample consists of input vector xp together with its correct class label y o. However. 1984. Not all functions are represented accurately by this combination of quantisation and approximation layers. using the following strategy: ||xp wk1||<||x p wk2||<||x p wi|| "o ¹ k1. 1991) are based on this very idea. to obtain a better or locally more fine-grained quantisation). They are all based on the following basic algorithm: 1.g. The labels y k1.4 Learning Vector Quantisation It is an unpleasant habit in neural network literature. Friedman. 14. Using distance measures between weight vectors wo and input vector x .. and Groen. y k2 are compared with d p. An example of the last step is given by the LVQ2 algorithm by Kohonen (1977).e. e. Olshen. if we expect our input to be (a subspace of) a high dimensional input space <n and we expect our function f to be discontinuous at numerous points. such as Kohonen networks or octree methods (Jansen.176 FUZZY LOGIC AND NEURAL NETWORKS i. 1994).6) is used selectively based on this comparison. Smagt. With each output neuron o. Granted that these methods also perform a clustering or quantisation task and use similar learning rules. Friedman. p 3. they are trained supervisedly and perform discriminant analysis rather than unsupervised clustering. various modern statistical function approximation methods (Breiman. These networks attempt to define decision boundaries in the input space. As we have seen before. the combination of quantisation and approximation is not uncommon and probably very efficient. a simple identity or combinations of sines and cosines are much better approximated by multilayer back-propagation networks if the activation functions are chosen appropriately. the quantisation scheme tracks the input probability density function. The weight update rule given in equation (6. e. which results in a better approximation of the function in those areas where input is most likely to appear. p 2. A rather large number of slightly different LVQ methods is appearing in recent literature.2. In fact. The latter could be replaced by a reinforcement learning procedure (see chapter 15). each table entry converges to the mean function value over all inputs in the subspace represented by that table entry. to also cover Learning Vector Quantisation (LVQ) methods in chapters on unsupervised clustering. k2 and ||x p wk2|| ||x p wk1|| < e then wk2(t + 1) = wk2(t) + g (x wk2(t)) and wk1(t + 1) = wk1(t) + g (x wk1(t)) p p if y k1 ¹ d p and d p = y k2 .g. be a correct class label.. extended with the possibility to have the approximation layer influence the quantisation layer (e. each decision could. Of course this combination extends itself much further than the presented combination of the presented single layer competitive learning network and the single layer feed-forward network.. and Stone. The quantisation layer can be replaced by various other quantisation schemes. but also the second best k2: p p 4. not only the winner k1 is determined.

the learning patterns are random samples from ÂN. k) is a decreasing function of the grid-distance between units o and k. Due to this collective learning scheme. 14.SELF-ORGANIZING NETWORKS 177 i.. in the visual system.8. the weights to this winning unit as well as its neighbours are adapted using the learning rule wo(t + 1) = wo(t) + gg(o. the dimensionality of S must be at least N. a sample x(t) is generated and presented to the network.. The topology-conserving quality of this network has many counterparts in biological brains. which can for example be used for visualization of the data. how many next-best winners are to be determined. the same or neighboring units.16) Here. 1984) can be seen as an extension to the competitive learning network.1. However. although this is chronologically incorrect. 14.. such that (in one dimension!) g(o. k) = exp ( (o k)2) (see Fig.e. Also.3 KOHONEN NETWORK The Kohonen network (1982. the neurons in S. for g( ) a Gaussian function can be used.e. how to define class labels yo. i. the winning unit k is determined. which are near to each other. the output units in S are ordered in some fashion. Thus. although this is application-dependent.(14. which are also near to each other. For example. a Kohonen network can be used of lower dimensionality. the weights to the output units are thus adapted such that the order present in the input space Â2 is preserved in the output. the neurons in the network are folded in the input space. The brain is organized in many places so that aspects of the sensory environment are represented in the form of two-dimensional maps. There are organized mappings of the body surface . i. often in a twodimensional grid or array. which represents a discretization of the input space. The mapping. k) = 1. The ordering. such as depicted in Fig. If the intrinsic dimensionality of S is less than N. input signals.9. if the inputs are restricted to a subspace of ÂN. Usually.g. will be mapped on neighbouring neurons. k)(x(t) wo(t)) . Now.e. e. determines which output neurons are neighbours. while wk1 with the incorrect label is moved away from it. The new LVQ algorithms that are emerging all use different implementations of these different steps. In the Kohonen network. This means that learning patterns which are near to each other in the input space (where near is determined by the distance measure used in finding the winning unit) must be mapped on output units. Using the same formulas as in section 6. At time t. the Kohonen network has a different set of applications... when learning patterns are presented to the network. there are several topographic mappings of visual space onto the surface of the visual cortex. which is chosen by the user. wk2 with the correct label is moved towards the input vector.7). 14. such that g(k. Next.. For example. Thus the topology inherently present in the input signals will be preserved in the mapping. such as depicted in Fig. 14. For example: data on a two. if inputs are uniformly distributed in ÂN and the order must be preserved.dimensional manifold in a high dimensional input space can be mapped onto a two-dimensional Kohonen network. g(o. how to adapt the number of output neurons i and how to selectively use the weight update rule. is said to be topology preserving.

the rightmost when the map is almost completely formed.178 FUZZY LOGIC AND NEURAL NETWORKS 1 0.7 Gaussian neuron distance function g( ).75 0. .5 0. Iteration 0 Iteration 200 Iteration 600 Iteration 1900 Fig.9 The mapping of a two-dimensional input space on a one-dimensional Kohonen network. o2 ) and wi .25 0 –2 –1 0 1 1 2 –2 –1 2 1 0 Fig. Fig. The leftmost figure shows the initial weights. In this case. A line in each figure connects weight wi .8 A topology-conserving map converging. The weight vectors of a network with two inputs and 8 x 8 output neurons arranged in a planar grid are shown.(o1 o2 ) with weights wi . 14. g( ) is shown for a two-dimensional grid because it looks nice. 14.( i1 i 2 + 1) . 14.(o1 + 1.

those connection strengths form a Mexican hat (see Figure 14. It can be easily seen that x1 and x2 are related. An example is shown in Figure 14. the network has been used to merge sensory data from different kinds of sensors. x2) are plotted in the figure.11. The use of topographic representations. looking at the same scene (Gielen. Lateral interaction around the winning neuron as a function of distance: excitation to nearby neurons.10 Mexican hat.10). d x2 ) and (de1 . such as auditory and visual. To explain the plausibility of a similar structure in biological networks. 1984). 1991). Krommenhoek. After the rotation. In one dimension. inhibition to farther off neurons. which map an input vector to a number of binary output elements or neurons. Here the conditional prediction has no use because the points have uncorrelated coordinates. Kohonen himself has successfully used the network for phonemerecognition (Kohonen. Also. and tonotopic mappings of frequency in the auditory cortex. de2 ) in the figures. The weights are adjusted in such a way that they could be considered as prototype vectors (vectorial means) for the input patterns for which the competing neuron wins. such that if we know x1 we can make a reasonable prediction of x2 and vice versa since the points are centered around the line x1 = x2. It does not come as a surprise. and Saramaki. The two dimensional samples (x1.4 PRINCIPAL COMPONENT NETWORKS The networks presented in the previous sections can be seen as (nonlinear) vector transformations. This can be intuitively verified by comparing the spreads (d x1. If we rotate the axes over p/4 we get the (e1. . Kohonen remarks that the lateral inhibition between the neurons could be obtained via efferent connections between those neurons. e2) axis as plotted in the figure. and Gisbergen. 14. 14. the variance of the samples is large along the e1 axis and small along the e2 axis. Another property of this rotation is that the variance or energy of the transformed patterns is maximized on a lower dimension. The self-organizing transform described in this section rotates the input space in such a way that the values of the output neurons are as uncorrelated as possible and the energy or variances of the patterns is mainly concentrated in a few output neurons. that already many applications have been devised of the Kohonen topology-conserving maps. Excitation Lateral distance Fig.SELF-ORGANIZING NETWORKS 179 onto the cortex in both motor and somatosensory areas. Makisara. where some important aspect of a sensory modality is related to the physical locations of the cells on a surface. is so common that it obviously serves an important information processing function. therefore.

Now the operator which computes the vector length. typically 1... This transform is very closely related to the eigenvector transformation known from image processing where the image has to be coded or transformed to a lower dimension and reconstructed again by another transform as well as possible. The output yo(t) of this neuron is given by the usual inner product of its weight w and the input vector x: yo(t) = w(t)T x(t) .17) As seen in the previous sections. and g is a small learning parameter.180 FUZZY LOGIC AND NEURAL NETWORKS x2 dx2 e1 de2 de1 e2 x1 dx1 Fig.. 14. but which scales the vector length to unity.4. which leads to the following learning rule w(t + 1) = w(t ) + gy(t ) x(t ) L( w(t ) + gy(t ) x(t )) . This can be overcome by normalising the weight vector to a fixed length. can be approximated by a Taylor expansion around g = 0: L(w(t)) + gy(t) x(t) = 1 + g ¶L + O(g2) ¶g g = 0 . There the delta rule was normalized. 14.1 Normalized Hebbian Rule The model considered here consists of one linear neuron with input weights w. all models are based on a kind of Hebbian learning. here the standard Hebb rule is. Compare this learning rule with the normalized learning rule of competitive learning.(14. extending the theory in the last section to multidimensional outputs.18) where L() indicates an operator which returns the vector length.(14..(14. The next section describes a learning rule which acts as a Hebbian learning rule.19) . the basic Hebbian rule would make the weights grow uninhibitedly if there were correlation in the input patterns. the norm of the vector... However.11 Distribution of input samples. In the subsequent section we will see that a linear neuron with a normalised Hebbian learning rule acts as such a transform.

(14.. Proof: 1 Since the eigenvectors of R span the N-dimensional space. In the following we assume the signal mean to be zero.2: Let the eigenvectors ei of R be ordered with descending associated eigenvalues li such that l1 > l2 > .. it resolves for small g(t )..(14. 1982)...3 More Eigenvectors In the previous section it was shown that a single neurons weight converges to the eigenvector of the correlation matrix with maximum eigenvalue.21) which is called the Oja learning rule (Oja.22) d w(t) = Rw(t) (w(t)T Rw(t)) w(t) dt .g F GGH ¶L + o( g 2 ) ¶g g = 0 I JJK .20) Since dL = y(t)2 discarding the higher order terms of g leads to dg g =0 w(t + 1) = w(t) + gy(t) x(t)(x(t) y(t)w(t)) .23) Theorem 14.. the first product terms is the Hebb rule yo(t) x(t). 2 w(t + 1) = (w(t) + gy(t) x(t)) 1 . Correlation matrix R = E((x(t) m) (x(t) m)T). From equation (6.4.18).SELF-ORGANIZING NETWORKS 181 When we substitute this expression for the vector length in equation (6. > lN . With equation (6.24) Substituting this in the differential equation and concluding the theorem is left as an exercise.(14. This learning rule thus modifies the weight in the usual Hebbian sense. What exactly does this learning rule do with the weight vector? 14. so m = 0.. the weight of the neuron is directed in the direction ..(14. but normalizes its weight vector directly by the second product term yo(t) x(t) w(t).e..(14...2 Principal Component Extractor Remember probability theory? Consider an N-dimensional signal x(t) with Mean m = E(x(t)).4..23) the weights w(t) will converge to ± e1. the weight vector can be decomposed as w(t) = å i N bi (t)ei .. i.21) we see that the expectation of the weights for the Oja learning rule equals E(w(t + 1)|w(t)) = w(t) + g (Rw(t) (w(t)TRw(t))w(t)) which has a continuous counterpart . 14.

.27) ~ =xy w x o The term subtracted from the input vector can be interpreted as a kind of a back-projection or expectation. 14.. then its weights will lie in the direction of the remaining eigenvector with the highest eigenvalue.(14. the data is not only fed forward but also back from output to input units. Consider the signal x which can be decomposed into the basis of eigenvectors ei of its correlation matrix R.. We call ~ the deflation of x.. from the signal x ~ =xa e x 1 1 . x If now a second neuron is taught on this signal ~.25) If we now subtract the component in the direction of e1. the weight will converge to the remaining eigenvector with maximum eigenvalue..28) . Here we tackle the question of how to find the remaining eigenvectors of the correlation matrix given the first found eigenvector.(14.(14. Compare this to ART described in the next section.(14.. the direction in which the signal has the most energy. Since the deflation removed the component in the direction of the first eigenvector.182 FUZZY LOGIC AND NEURAL NETWORKS of highest energy or variance of the input patterns..5. as with networks in the next chapter. 14.5 ADAPTIVE RESONANCE THEORY The last unsupervised learning network we discuss differs from the previous networks in that it is recurrent.29) å i N aiei = ai . The model has three crucial properties: .1 Background: Adaptive Resonance Theory In 1976.. In the previous section we ordered the eigenvalues in magnitude. We can write the deflation in neural network terms if we see that yo = wT x = e T 1 since w = e1 ~ equals So that the deflated vector x .26) we are sure that when we again decompose ~x into the eigenvector basis. Grossberg introduced a model for explaining biological phenomena. the coefficient a1 = 0. x= å aiei i N ... We can continue this strategy and find all the N eigenvectors belonging to the signal x. simply x because we just subtracted it. so according to this definition in the limit we will find e2.(14.

F1 and F2. The mechanism used here is contrast enhancement. 14.5. and vice versa via the binary-valued backward LTM W b. the classification). If there is a match.e. The expectations. the expectations are strengthened otherwise the classification is rejected. whereas classification takes place in F2. Before the input pattern can be decoded. 14. A neuron outputs a 1 if and only if at least three of these inputs are high: the two-thirds rule. translate the input pattern to a categorization in the category representation field. As mentioned before.2 ART1: The Simplified Neural Network Model The ART1 simplified model consists of two layers of binary neurons (with values 1 and 0). Contrast enhancement of input patterns. The system consists of two layers. which resides in the LTM weights from F2 to F1. residing in the LTM connections. called F1 (the comparison layer) and F2 (the recognition layer) (see Fig. 14. and a reset module. .12). First a characterization takes place by means of extracting features. whereas the STM is used to cause gradual changes in the LTM. Distinguishing a hiding panther from a resting one makes all the difference in the world. The classification is compared to the expectation of the network. Each neuron in F1 is connected to all neurons in F2 via the continuous-valued forward long term memory (LTM) W f. A normalization of the total network activity. The input pattern is received at F1.13). a component of the feedback pattern. For example. The awareness of subtle differences in input patterns can mean a lot in terms of survival. The long-term memory (LTM) implements an arousal mechanism (i. and a gain G1. Short-term memory (STM) storage of the contrast-enhanced pattern. the human eye can adapt itself to large variations in light intensities. giving rise to activation in the feature representation field. Fig. the input is not directly classified.12 The ART architecture. 3. 14. 2. which are connected to each other via the LTM (see Category representation field STM activity pattern LTM STM activity pattern Feature representation field F1 LTM F2 Input Fig. Biological systems are usually very adaptive to large changes in their environment. Each neuron in the comparison layer receives three inputs: a component of the input pattern.SELF-ORGANIZING NETWORKS 183 1. The other modules are gain 1 and 2 (G1 and G2). it must be stored in the short-term memory..

Gain 2 is the logical or of all the elements in the input pattern x. The winning neuron then inhibits all the other neurons via lateral inhibition. and in F2 one neuron becomes active. which reproduces a binary pattern at F1.184 FUZZY LOGIC AND NEURAL NETWORKS F2 M neurons + G2 + j + W b +f W F1 N neurons – + G1 + i – + Reset Input Fig.13 The ART 1 neural network.3 Operation The pattern is sent to F2.5. Finally. If there is a substantial mismatch between the two patterns. This signal is then sent back over the backward LTM. Gain 1 equals gain 2. 14. we use the notation employed by Lippmann (1987): 1. Instead of following Carpenter and Grossbergs description of the system using differential equations. Gain 1 is inhibited. the reset signal will inhibit the neuron in F2 and the process is repeated. 14. the reset signal is sent to the active neuron in F2 if the input vector x and the output of F1 differ by more than some vigilance level. and only the neurons in F1 which receive a one from both x and F2 remain active. except when the feedback pattern from F2 contains any 1. The neurons in the recognition layer each compute the inner product of their incoming (continuousvalued) weights and the pattern sent over these connections. then it is forced to zero. Initialization: wb (0) = 1 ji f w ji = 1 1+ N .

M the number of neurons in F2. This algorithm tries to fit each new input pattern in an existing class. Apply the new input pattern x. 0 £ r £ 1. 1975).(14. Go to step 3. The network incorporates a follow-the-leader clustering algorithm (Hartigan. the distance between the new pattern and all existing classes exceeds some threshold.propagation network. Note that wb o x essentially is the k inner product x* o x.e. Re-enable all neurons in F2 and go to step 2.. Neuron k is disabled from further activity. If no matching class can be found. choose the vigilance threshold r..4 ART 1: The Original Model In later work. and 0 £ j < M. Vigilance test: if b wk (t ) o x >r xox . Compute the activation values of the neurons in F2: y¢ = i åw j =1 N f ij (t)x1 . 7. 6.SELF-ORGANIZING NETWORKS 185 where N is the number of neurons in F1. i. while the previous memory is not corrupted. the teaching of a new pattern might corrupt the weights for all previously learned patterns.. In most neural networks. Fig. which will be large if x* and x are near to each other. Select the winning neuron k(0 £ k < M). The novelty in this approach is that the network is able to adapt to new incoming patterns. Also. 0 £ i < N. We will only discuss the first model. ART 1.14 shows exemplar behaviour of the network. 3. all patterns must be taught sequentially. 5. Set for all l. else go to step 6. 14. go to step 7. a new class is created containing the new pattern.30) 4. 14... By changing the structure of the network rather than the weights. such as the back. Carpenter and Grossberg (1987) present several neural network models to incorporate parts of the complete theory.(14.5. o £ l < N: wb (t + 1) = wb (t)xl kl kl f w lk (t + 1) = b wkl (t ) xl N 1 + 2 åw i =1 b ki ( t ) xi 8.31) where o denotes inner product. ART1 overcomes this problem. . 2.

14.5 Normalization of the Original Model We will refer to a cell in F1 or F2 with k. In order to introduce normalization in the model.5.186 FUZZY LOGIC AND NEURAL NETWORKS Backward LTM from Input pattern Output 1 Output 2 Output 3 Output 4 Not active Not active Not active Not active Not active Not active Not active Fig. the surroundings of each cell have a negative influence on the cell . On the right the stored patterns (i. the weights of W b for the first four output units) are shown..yk ås k and let the relative input ås . . 14. i. we set I = intensity Qk = sk I 1. l l¹k has an excitatory response as far as the input at the cell is concerned +Bsk. The binary input patterns on the left were applied sequentially.e. So we have a model in which the change of the response yk of an input at a certain cell k depends inhibitorily on all other inputs and the sensitivity of the cell. Each cell k in F1 or F2 receives an input sk and respond with an activation level yk.14 An example of the behaviour of the Carpenter Grossberg network for letter patterns..e.

(14. and with I = dt yk(A + 1) = Bsk ås we have that .36) At equilibrium.SELF-ORGANIZING NETWORKS 187 has an inhibitory response for normalization yk sk. A and B are constants. In order to enhance the contrasts.37) Now.32) does not suffice anymore. when we set B = (n 1) C where n is the number of neurons.. (14..6 Contrast Enhancement In order to make F2 react better on differences in neuron values in F1 (or vice versa). we have yk = 1 nCI Qk A+ I n FG H IJ K .. we chop off all the equal fractions (uniform parts) in F1 or F2.(14.35) åy k never exceeds B: it is normalized. then all the yk are zero: the effect of C is enhancing differences..32) with 0 £ yk (0) £ B because the inhibitory effect of an input can never exceed the excitatory input.. we will revert to the simplified model as presented by Lippmann. 14. This can be done by adding an extra inhibitory input proportional to the inputs from the other cells with a factor C: dy k sl = Ayk + (B yk)sk (yk + C) dt l¹k å ...(14. At equilibrium. We can show that eq. when dyk = 0. then more of the input shall be chopped off. since BI £B A+ I The total activity ytotal = .. at equilibrium yk is proportional to Qk. when an input in which all the sk are equal is given.5.(14.. contrast enhancement is applied: the contrasts between the neuronal values in a layer are amplified. . Here. The differential equation for the neurons in F1 and F2 now is dyk sl = Ayk + (B yk)sk . and. If we set B £ (n 1)C or C/(B + C) ³ 1/n..(14.. Instead of following Carpenter and Grossbergs description.33) Because of the definition of Qk = sk I 1 we get yk = Qk BI A+ I .34) Therefore. The description of ART1 continues by defining the differential equations for the LTM. has a decay Ayk .yk dt l¹k å k .(14..

Rumelhart. Vol.H. Vol. 11. 119-130.). 187-202. . pp. Neural Networks. pp.C. P. Wadsworth and Broks/Cole. Ahalt. 7. 1. 3. Fukushima. 1990. S. Chen. Describe the normalization of ART 1. 2.E. Competitive learning algorithms for vector quantisation. P. and F. R. Zipser. pp. Smagt. Vol. L. Explain ART 1 neural network. Biological Cybernetics. 1-38. In A. Describe the learning vector quantisation method.K. REFERENCES. Computer Vision. and Image Processing. No.188 FUZZY LOGIC AND NEURAL NETWORKS QUESTION BANK. 5. 121-134. 6. Cognitive Science. 1984. and D. 1976. Grossberg. K. 3. Describe the vector quantisation scheme. Stone. Vol. 9.A. Neural Network Applications. J. 9. R. Nielsen. 54115. What are the advantages of self-organizing networks? What is competitive learning network? Explain various methods of determining the winner and the corresponding learning rule. What is Kohonen network? Explain. 6. Adaptive pattern classification and universal recoding I & II. pp. 3. Explain normalized Hebbian rule. Counterpropagation networks.A.H. 5. 10. and S. Neural Networks. Groen.P. 23. Vol. 1985. 4. 19.C. Classification and Regression Trees. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Melton. 75-112. Feature discovery by competitive learning. Fukushima. 10. 277-290. 1987. 1. 1. 1988. 1988. S.J. 9. Kluwer Academic Publishers. 37. Annals of Statistics. Neural Networks. Explain the counter propagation network. R. 8. Krishnamurthy. D. Jansen. Biological Cybernetics. Neural Computation. 2. Vol. A.F.P. pp. 131-139. 1-141.A. Vol. pp. Review of neural networks for speech recognition. 1. Describe the square error criterion to measure the quality of a given clustering. J. Friedman. Murray (Ed. 4. Nested networks for robot control. Multivariate adaptive regression splines. Breiman. Graphics. D. 1994. 20. Van der. 11. 1975. Carpenter. A massively parallel architecture for a self-organizing neural pattern recognition machine. Describe adaptive resonance theory. pp. A. Olshen. and C. Vol. 7.Vol. 12. Cognitron: A self-organizing multilayered neural network. pp. 1. Friedman. Lippmann. Grossberg. G. 121-136. 8. K. P. 1989.H. 1991.

Groen. 417-423. 54115. 1982. 187-202. pp. T.A. 21.A. ART 2: Self-organization of stable category recognition codes for analog input patterns. 1984. 4-22. 23. Phonotopic maps|insightful representation of phonological features for speech recognition. 2 No. and Signal Processing. and S. . J. Gisbergen.A. pp. 37. 22.O. 16. 121-134.SELF-ORGANIZING NETWORKS 189 12. Graphics. Biological Cybernetics. 267-273. and Image Processing. and L. 15. Hertzberger (Eds. C. Lippmann. In T. M. Makisara. Self-organized formation of topologically correct feature maps. G. T.P. 1984. Grossberg. G. Kohonen. S. An introduction to computing with neural nets. Applied Optics. Clustering Algorithms. Carpenter. 1991. R. Computer Vision. 14. Grossberg. 43. 59-69. Adaptive pattern classification and universal recoding I & II. Kohonen. 26(23). Vol. A procedure for self-organized sensor-fusion in topologically ordered maps. and T. Oja. Kanade. Carpenter. 1976. In Proceedings of the 7th IEEE International Conference on Pattern Recognition. T. 15. E. 1987. Grossberg. Biological Cybernetics. Proceedings of the Second International Conference on Autonomous Systems. 19. Self-Organization and Associative Memory. Elsevier Science Publishers. Vol. Kohonen. 1982. T. DUNNO. A massively parallel architecture for a self-organizing neural pattern recognition machine. Kohonen. Springer-Verlag. Vol. and S. 18. Gielen. 4919-4930.). A simplified neuron model as a principal component analyzer. K. 13. Speech. 1987. 1975.4. Hartigan. pp. 20. and J. IEEE Transactions on Acoustics.A. pp. 1977. Krommenhoek. F. Saramaki. Vol. Journal of Mathematical Biology. New York: John Wiley & Sons. Associative Memory: A System-Theoretical Approach. 17. 1987. Berlin: Springer-Verlag.C.

learning controller Fig.2 THE CRITIC The first problem is how to construct a critic. performance feedback is straightforward and a critic is not required. Reinforcement learning involves two subproblems. On the other hand.1 Reinforcement learning scheme. 15. If the objective of the network is to minimize a direct measurable quantity r. This temporal credit assignment problem is solved by learning a critic network which represents a cost function J predicting future reinforcement. Often the only information is a scalar evaluation r. Fig. which is able to evaluate system performance.+ 0 ) 2 6 . The two problems are discussed in the next paragraphs.1 INTRODUCTION In the previous chapters a number of supervised training methods have been described in which the weight adjustments are calculated using a set of learning samples. respectively. Critic $ J Reinforcement signal u System x Reinf. The First is that the reinforcement signal r is often delayed since it is a result of network outputs in the past. 15. how is current behaviour to be evaluated . 15. However. The second problem is to Find a learning procedure which adapts the weights of the neural network such that a mapping is established which minimizes J.4 Reinforcement Learning 15 15. not always such a set of learning examples is available.1 shows a reinforcement-learning network interacting with a system. existing of input and desired output values. which indicates how well the neural network is performing.

REINFORCEMENT LEARNING 191

if the objective concerns future system performance? The performance may for instance be measured by the cumulative or future error. Most reinforcement learning methods (Barto, Sutton and Anderson (1983) use the temporal difference (TD) algorithm (Sutton, 1988) to train the critic. Suppose the immediate cost of the system at time step k are measured by r(xk, uk, k), as a function of system states xk and control actions (network outputs) uk. The immediate measure r is often called the external reinforcement signal in contrast to the internal reinforcement signal in Fig. 7.1. Define the performance measure J(xk, uk, k)of the system as a discounted cumulative of future cost. The task of the critic is to predict the performance measure: J(xk, uk, k) =

i=k

åy

¥

ik

r (xk, uk, k)

...(15.1)

in which g Î [0, 1] is a discount factor (usually » 0.95). The relation between two successive prediction can easily be derived: J(xk, uk, k) = r(xk, uk, k) + rJ(xk+1, uk+1, k + 1) be: ...(15.2)

**$ If the network is correctly trained, the relation between two successive network outputs J should $ $ J (xk, uk, k) = r(xk, uk, k) + r J (xk+1, uk+1, k + 1)
**

...(15.3)

If the network is not correctly trained, the temporal difference d(k) between two successive predictions is used to adapt the critic network:

$ $ d(k) = [r(xk, uk, k) + r J (xk+1, uk+1, k + 1)] J (xk, uk, k)

2

...(15.4)

A learning rule for the weights of the critic network wc(k), based on minimizing d (k) can be derived: Dwc(k) = ae(k) in which a is the learning rate.

$ ¶J ( xk , uk , k ) ¶wc ( k )

...(15.5)

15.3

THE CONTROLLER NETWORK

If the critic is capable of providing an immediate evaluation of performance, the controller network can be adapted such that the optimal relation between system states and control actions is found. Three approaches are distinguished: 1. In case of a finite set of actions U, all actions may virtually be executed. The action which decreases the performance criterion most is selected:

$ uk = min J (xk, uk, k)

u ÎU

...(15.6)

**192 FUZZY LOGIC AND NEURAL NETWORKS
**

The RL-method with this controller is called Q-learning (Watkins & Dayan, 1992). The method approximates dynamic programming which will be discussed in the next section. 2. If the performance measure J(xk, uk, k) is accurately predicted, then the gradient with respect to the controller command uk can be calculated, assuming that the critic network is differentiable. If the measure is to be minimized, the weights of the controller wr are adjusted in the direction of the negative gradient: Dwr(k) = b

$ ¶J ( xk , uk k ) ¶u(k ) ¶u( k ) ¶wr (k )

...(15.7)

with b being the learning rate. Werbos (1992) has discussed some of these gradient based algorithms in detail. Sofge and White (1992) applied one of the gradient based methods to optimize a manufacturing process. 3. A direct approach to adapt the controller is to use the difference between the predicted and the true performance measure as expressed in equation 15.3. Suppose that the performance measure is to be minimized. Control actions that result in negative differences, i.e. the true performance is better than was expected, then the controller has to be rewarded. On the other hand, in case of a positive difference, then the control action has to be penalized. The idea is to explore the set of possible actions during learning and incorporate the beneficial ones into the controller. Learning in this way is related to trial-and-error learning studied by psychologists in which behavior is selected according to its consequences. Generally, the algorithms select probabilistically actions from a set of possible actions and update action probabilities on basis of the evaluation feedback. Most of the algorithms are based on a look-up table representation of the mapping from system states to actions (Barto et al., 1983). Each table entry has to learn which control action is best when that entry is accessed. It may be also possible to use a parametric mapping from systems states to action probabilities. Gullapalli (1990) adapted the weights of a single layer network.

15.4

BARTOS APPROACH: THE ASE-ACE COMBINATION

Barto, Sutton and Anderson (1983) have formulated reinforcement learning as a learning strategy, which does not need a set of examples provided by a teacher. The system described by Barto explores the space of alternative input-output mappings and uses an evaluative feedback (reinforcement signal) on the consequences of the control signal (network output) on the environment. It has been shown that such reinforcement learning algorithms are implementing an on-line, incremental approximation to the dynamic programming method for optimal control, and are also called heuristic dynamic programming (Werbos, 1990). The basic building blocks in the Barto network are an Associative Search Element (ASE) which uses a stochastic method to determine the correct relation between input and output and an Adaptive Critic Element (ACE) which learns to give a correct prediction of future reward or punishment (Fig. 15.2). The external reinforcement signal r can be generated by a special sensor (for example a collision sensor of a mobile robot) or be derived from the state vector. For example, in control applications, where the state s of a system should remain in a certain part A of the control space, reinforcement is given by:

**REINFORCEMENT LEARNING 193
**

Reinforcement T ACE WC1 WC2 WCn

$ T Internal

Reinforcement detector

reinforcement

Decoder

WS1 WS2 ASE WSn

yo

System

State vector

Fig. 15.2:

Architecture of a reinforcement learning scheme with critic element.

r=

R0 S1 T

if s Î A otherwise

...(15.8)

15.4.1

Associative Search

In its most elementary form the ASE gives a binary output value yo(t) Î{0, 1}; as a stochastic function of an input vector. The total input of the ASE is, similar to the neuron presented in chapter 2, the weighted sum of the inputs, with the exception that the bias input in this case is a stochastic variable N with mean zero normal distribution: s(t) =

åw

j =1

N

sj x j

(t) + Nj

...(15.9)

The activation function F is a threshold such that yo(t) = y(t)

R1 S0 T

if s(t ) > 0 otherwise

...(15.10)

For updating the weights, a Hebbian type of learning rule is used. However, the update is weighted with the reinforcement signal r(t) and an eligibility ej is defined instead of the product y0(t) xj(t) of input and output: wsj(t + 1) = wsj(t) + ar(t) ej(t) where a is a learning factor. The eligibility ej is given by ... (15.11)

**194 FUZZY LOGIC AND NEURAL NETWORKS
**

ej(t + 1) = dej(t) + (1 d) y0(t) xj(t) ...(15.12) with d the decay rate of the eligibility. The eligibility is a sort of memory; ej is high if the signals from the input state unit j and the output unit are correlated over some time. Using r(t) in expression (15.11) has the disadvantage that learning only finds place when there is an $ external reinforcement signal. Instead of r(t), usually a continuous internal reinforcement signal r (t) given by the ACE, is used. Barto and Anandan (1985) proved convergence for the case of a single binary output unit and a set of linearly independent patterns x p. In control applications, the input vector is the (n-dimensional) state vector s of the system. In order to obtain a linear independent set of patterns x p, often a decoder is used, which divides the range of each of the input variables si in a number of intervals. The aim is to divide the input (state) space in a number of disjunct subspaces or boxes. The input vector can therefore only be in one subspace at a time. The decoder converts the input vector into a binary valued vector x, with only one element equal to one, indicating which subspace is currently visited. It has been shown (Krose and Dam, 1992) that instead of a-priori quantisation of the input space, a self-organizing quantisation, based on methods described in this chapter, results in a better performance.

**15.4.2 Adaptive Critic
**

The Adaptive Critic Element (ACE, or evaluation network) is basically the same as described in section 7.1. An error signal is derived from the temporal difference of two successive predictions (in this case denoted by p!) and is used for training the ACE:

$ r (t) = r(t) + gp(t) p(t 1)

...(15.13) ...(15.14)

p(t) is implemented as a series of weights wCk to the ACE such that p(t) = wCk if the system is in state k at time t, denoted by xk = 1. The function is learned by adjusting the wCk s $ according to a delta-rule with an error signal d given by r (t):

$ DwCk(t) = b r (t)hj(t)

...(15.15) ...(15.16)

b is the learning parameter and hj (t) indicates the trace of neuron xj: hj(t) = lhj(t 1) + (1 l) xj(t 1) This trace is a low-pass filter or momentum, through which the credit assigned to state j increases $ while state j is active and decays exponentially after the activity of j has expired. If r (t) is positive, the $ action u of the system has resulted in a higher evaluation value, whereas a negative r (t) indicates a $ deterioration of the system. r (t) can be considered as an internal reinforcement signal.

15.4.3

The Cart-Pole System

An example of such a system is the cart-pole balancing system (see Fig. 15.3). Here, a dynamics controller must control the cart in such a way that the pole always stands up straight. The controller applies a left or right force F of fixed magnitude to the cart, which may change direction at discrete time intervals. The model has four state variables:

**REINFORCEMENT LEARNING 195
**

q

F

x

Fig. 15.3: The cart-pole system.

x the position of the cart on the track, q the angle of the pole with the vertical,

& x & q

the cart velocity, and the angle velocity of the pole.

Furthermore, a set of parameters specify the pole length and mass, cart mass, coefficients of friction between the cart and the track and at the hinge between the pole and the cart, the control force magnitude, and the force due to gravity. The state space is partitioned on the basis of the following quantisation thresholds: 1. x : ±0.8, ±2.4 m 2. q : 0°, ±1°, ±6°, ±12°

& 3. x : ±0.5, ±¥ m/s

& 4. q : ±50, ±¥ °/s

This yields 3 ´ 6 ´ 3 ´ 3 = 162 regions corresponding to all of the combinations of the intervals. The decoder output is a 162-dimensional vector. A negative reinforcement signal is provided when the state vector gets out of the admissible range: when x > 2.4, x < 2.4, q > 12° or q < 12°. The system has proved to solve the problem in about 75 learning steps.

15.5

REINFORCEMENT LEARNING VERSUS OPTIMAL CONTROL

The objective of optimal control is to generate control actions in order to optimize a predefined performance measure. One technique to find such a sequence of control actions which define an optimal control policy is Dynamic Programming (DP). The method is based on the principle of optimality, formulated by Bellman (1957): Whatever the initial system state, if the first control action is contained

& Watkins. u . the performance measure could be defined as a discounted sum of future costs as expressed by equation 15. 1992). which is assumed to be an exact representation of the system and the environment. Q.6. Barto. uk.. Solving the equations backwards in time is called dynamic programming. k) (the name Q-learning comes from Watkins notation).. The basic idea in Qlearning is to estimate a function.17) . uk. & Wilson. k + 1) + r(xk . a solution can be derived only for a small N and simple systems.. k)] J (xk .. is i i i=k N to be minimized. 1992). k)] u ÎU . the notation with J is continued here: $ J (xk. In order to deal with large or infinity N.. This can be achieved backwards. control actions and disturbances. k + 1) + r(xk. Sutton. uk . uk. uk+ 1.(Sutton. uk .. uk . The requirements are a bounded N. 1992). k) . 1992): (1) the critic is implemented as a look-up table. (3) all actions continue to be tried from all states.19) $ The optimal control rule can be expressed in terms of J by noting that an optimal control action for $ state x is any action u that minimizes J according to equation 7.17) and (15. of states and actions.5. The equations for the discrete case are (White & Jordan. k) with r being the immediate costs. The temporal difference e(k)between the true and expected performance is again used: $ $ e(k) = [g min J (xk+ 1.18) Jmin (xN) = r(xN) The strategy for finding the optimal control actions is solving equation (15. k) u ÎU . .20) Watkins has shown that the function converges under some pre-specified conditions to the true optimal Bellmann equation (Watkins & Dayan. uk+ 1. (2) the learning parameter a must converge to zero. (Werbos.196 FUZZY LOGIC AND NEURAL NETWORKS in an optimal control policy. starting at state xN.(15. k k $ The estimate of minimum cost J is updated at time step k + 1 according equation 7.. where Q is the minimum discounted sum of future costs Jmin(xk. The most directly related RL-technique to DP is Q-learning (Watkins & Dayan. k + 1) + r(xk .. and a model. RL is therefore often called an heuristic dynamic programming technique (Barto. k) = å J( x . The Bellman equations follow directly from the principle of optimality.2.(15. Assume that a performance measure J(xk.(15. 1990). 1992): Jmin (xk. In practice. uk+ 1. k) = g Jmin(xk+ 1. k) = min [Jmin xk+ 1. For convenience. uk. The model has to provide the relation between successive system states resulting from system dynamics. The minimum costs Jmin of cost J can be derived by the Bellman equations of DP. Reinforcement learning provides a solution for the problem stated above without the use of a model of the system and environment. then the remaining control actions must constitute an optimal control policy for the problem with as initial system state the state remaining from the first control action.18) from which uk can be derived.(15. uk.

IEEE Transactions on Systems. Optimal control: a foundation for intelligent control. In D.W. Barto. Neural. 1985. and Adaptive Approaches. Describe dynamic programming to find a sequence of acontrol actions. Learning to avoid collisions: A reinforcement learning paradigm for mobile robot manipulation.H. Werbos (Eds. 1992. Sofge & D. Sutton. III. R.W.J. Sutton. 3. and P.J. Watkins. 6. Werbos. P. Neuron like adaptive elements that can solve difficult learning problems. pp.). and Adaptive Approaches. 671-692. 5. A stochastic reinforcement learning algorithm for learning real-valued functions. Bellman. Describe the Barto network of reinforcement learning. IEEE Transactions on Systems. C. 11. A. Barto. Anderson. White (Eds. Vol. Vol. 295-300 1992. Van Nostrand Reinhold.G. C. C. Vol. In D. Machine Learning. Man and Cybernetics. Vol. and Adaptive Approaches.S. R. 1990. 8. Jordan. 834-846. REFERENCES. 279-292. Delft: IFAC.G.S. 360-375. Approximate dynamic programming for real-time control and neural modeling. Explain the cast-pole balancing scheme. Barto. Sofge. pp. MIT Press/Bradford. Pattern-recognizing stochastic learning automata. Q-learning. 13. Van Nostrand Reinhold. New York. Luxemburg. 2. and C. Neural Networks for Control. 7. pp. D. 1992.W. Man and Cybernetics.REINFORCEMENT LEARNING 197 QUESTION BANK. V. Dayan. 1992. 1. 1957. 8. Princeton University Press. Vol. . What are the building blocks of Barto network? Explain them. and D.S. 15. White. Machine Learning. Fuzzy. 1990. B. Werbos. New York. & P. IEEE Transactions on Systems. White. New York.M. pp. 10. 6. 1983.). 3. pp. 2.A. and P. Fuzzy. R. Sutton. 4. P. 9-44. Anderson. Vol. 1983. R. Handbook of Intelligent Control. White (Eds.). Dam. 9.T. Neuronlike adaptive elements that can solve difficult learning problems. Fuzzy. 3. pp. Neural. Gullapalli. and J. Learning to predict by the methods of temporal differences. Handbook of Intelligent Control.W.).G. pp. D. White (Eds. and M. A. What are the various approaches of control networks used to find optimal relation between system states and control actions? 3. 834-846. Applied learning: optimal control for manufacturing. 4. 13. Neural. 1992. 5.J. Dynamic Programming. In W. A. Krose. Anandan. A menu for designs of reinforcement learning over time. Explain reinforcement learning scheme. 12.M. Handbook of Intelligent Control. Sofge & D. Neural Networks. In Proceedings of IFAC/IFIP/IMACS International Symposium on Artificial Intelligence in Real-Time Control. 1. R. Sofge & D. Man and Cybernetics. In D. Sutton. Van Nostrand Reinhold.S. 1988.

DUNNO. 14. C. Neural. 279-292. and Adaptive Approaches. Approximate dynamic programming for real-time control and neural modeling.). Touretsky (Ed. Watkins. P. Dayan. Machine Learning. 1992. Barto. IEEE Control Systems. New York. Sequential decision problems and neural networks. Wilson. In D. 8.G. . 19-22.H. and C.J. Vol. Fuzzy. 15. Advances in Neural Information Processing II. Vol. Sutton. Werbos. White (Eds. 6. Sutton. pp. Reinforcement learning is direct adaptive optimal control.S. and P. Handbook of Intelligent Control. Barto. In D.198 FUZZY LOGIC AND NEURAL NETWORKS 13. Sofge & D. Van Nostrand Reinhold. R. 1992.S. A.). Q-learning. 1990. and R. R. pp. C. A. 1992. Watkins. 16.

market forecasting. feature extraction and noise suppression. signal/image identification. facial recognition. beer testing. 2. Credit Card Activity Checking: Neural networks are used to spot unusual credit card activity that might possibly be associated with loss of a credit card. 10. product design and analysis. 9. chemical product design analysis. nonlinear modeling. planning and management. process control. credit application evaluation. product optimization. mortgage screening. real-time particle identification. autopilot enhancements. sonar. 3. corporate financial analysis. flight path simulation. 5. machine vision. 4.4 Neural Networks Applications 16. 6. portfolio trading program. Banking: Check and other document reading. Industrial: Neural networks are being trained to predict the output gasses of furnaces and other industrial processes. Entertainment: Animation. warranty activity analysis. . analysis of grinding operations. voice synthesis. 7. new kinds of sensors. project bidding.1 INTRODUCTION 16 A list of some applications mentioned in the literature follows: 1. integrated circuit chip layout. Financial: Real estate appraisal. visual quality inspection systems. Aerospace: High performance aircraft autopilot. Insurance: Policy application evaluation. aircraft component simulation. corporate bond rating. object discrimination. chip failure analysis. 11. target tracking. special effects. Defense: Weapon steering. aircraft component fault detection. welding quality analysis. They then replace complex and costly equipment used for this purpose in the past. radar and image signal processing including data compression. process and machine diagnosis. Automotive: Automobile automatic guidance system. Electronics: Code sequence prediction. computer-chip quality analysis. Manufacturing: Manufacturing process control.+ 0 ) 2 6 . 8. paper quality prediction. loan advisor. aircraft control systems. dynamic modeling of chemical process system. creditline use analysis. machine maintenance analysis. currency price prediction.

2. Solving this problem is a least requirement for most robot control systems. calculate all possible sets of joint angles which could be used to attain this given position and orientation. Telecommunications: Image and data compression. The inverse kinematic problem is not as simple as the forward one. 16. This is a fundamental problem in the practical use of manipulators. given a set of joint angles. Also. optimization of transplant times. There are four related problems to be distinguished (Craig. and of multiple solutions. vision systems. these networks are designed to direct a manipulator. vowel classification. and all higher order derivatives of the position variables. the questions of existence of a solution. 18. customer payment processing systems. Oil and Gas: Exploration. based on sensor data. 16. Securities: Market analysis. Medical: Breast cancer cell analysis. to grasp objects. Because the kinematic equations are nonlinear. 15. manipulator controllers. velocity. Within this science one studies the position. the major task involves making movements dependent on sensor data. stock trading advisory systems. EEG and ECG analysis. Robotics: Trajectory control. This is the static geometrical problem of computing the position and orientation of the endeffector (hand) of the manipulator. which treats motion without regard to the forces. automated information services. A very basic problem in the study of mechanical manipulation is that of forward kinematics. hospital quality improvement. 17. Usually. Another applications include the steering and path-planning of autonomous robot vehicles.2.2 Inverse Kinematics This problem is posed as follows: given the position and orientation of the end-effector of the manipulator. real-time translation of spoken language. hospital expense reduction. forklift robot. the forward kinematic problem is to compute the position and orientation of the tool frame relative to the base frame (see Fig. Speech: Speech recognition. acceleration. prosthesis design. In robotics. routing systems.2 ROBOT CONTROL An important area of application of neural networks is in the field of robotics. Specifically. 16. which cause it. Transportation: Truck brake diagnosis systems. arise. . 1989): Forward kinematics Inverse kinematics Dynamics Trajectory generation 16.1 Forward Kinematics Kinematics is the science of motion. 14. automatic bond rating. vehicle scheduling. 13.200 FUZZY LOGIC AND NEURAL NETWORKS 12. emergency-room test advisement.1). their solution is not always easy or even possible in a closed form. text-to-speech synthesis. which is the most important form of the industrial robot. 16. speech compression.

Typically. pick up an object. Take for instance the weight (inertia) of the robot arm.NEURAL NETWORKS APPLICATIONS 4 3 tool frame 201 2 1 base frame Fig. The robot arm has a memory. previous positions. from the image frame determine the position of the object in that frame.1 An exemplar robot manipulator.g. Its responds to a control signal depends also on its history (e. a complex set of torque functions must be applied by the joint actuators. representing the inverse kinematics in combination with sensory transformation).g.2. glide at a constant end-effector velocity. when this position is not always the same. this is done with a number of fixed cameras or other sensors which observe the work scene.3 Dynamics Dynamics is a field of study devoted to studying the forces required to cause motion. involving the following steps: Determine the target coordinates relative to the base of the robot. and finally decelerate to a stop. . acceleration). With the accurate robot arm that are manufactured. which determines the force required to change the motion of the arm. controlled fashion each joint must be moved via a smooth function of time. 16. The dynamics introduces two extra problems to the kinematic problems. 16. In order to accelerate a manipulator from rest.. this task is often relatively simple. 16. making the relative weight change very small). but also the physical properties of the robot are taken into account. speed. In dynamics not only the geometrical properties (kinematics) are used. and perform a pre-determined coordinate transformation.2.4 Trajectory Generation To move a manipulator from here to there in a smooth. e.5 End-Effector Positioning The final goal in robot manipulator control is often the positioning of the hand or end-effector in order to be able to. If a robot grabs an object then the dynamics change but the kinematics dont. 16.2. This is because the weight of the object has to be added to the weight of the arm (thats why robot arms are so heavy. Exactly how to compute these motion functions is the problem of trajectory generation. In the first section of this chapter we will discuss the problems associated with the positioning of the end-effector (in effect.

the development of more complex (adaptive!) control methods allows the design and use of more flexible (i.202 FUZZY LOGIC AND NEURAL NETWORKS With a precise model of the robot (supplied by the manufacturer). but has the problem that the input space is of a high dimensionality. Also. Move the arm (dynamics control) and close the gripper. 16. Constructing the mapping N() from the available learning samples. need frequent recalibration or parameter determination. Finally... the inverse kinematics). a solution will be found for both the learning sample generation and the function representation. and the samples are randomly distributed. systems. This controller then generates a joint position q for the robot: q = N(xtarget.(16.2.5a Involvement of Neural Networks So if these parts are relatively simple to solve with a high accuracy. yet still with accurate models as starting point) are required and the system must be calibrated. since in useful applications R() is an unknown function. xhand) . . In each of these approaches. Generating learning samples which are in accordance with eq.2).2.. (8. The target position xtarget together with the visual position of the hand xhand are input to the neural controller N().. When the (usually randomly drawn) learning samples are available. why involve neural networks? The reason is the applicability of robots. We will discuss three fundamentally different approaches to neural networks for robot end-effector positioning. 16. a form of self-supervised or unsupervised learning is required. 2.. less rigid) robot systems. This is a relatively simple problem.e. There are two problems associated with teaching N(): 1.6 Camera-Robot Coordination in Function Approximation The system we focus on in this section is a work floor observed by fixed cameras.(16. Some examples to solve this problem are given below. This is not trivial. and a robot arm. but we will not focus on that. When traditional methods are used to control a robot arm.e. Instead. This is evidently a form of interpolation. accurate models of the sensors and manipulators (in some cases with unknown parameters which have to be estimated from the systems behavior. which suffer from wear-and-tear. The visual system must identify the target as well as determine the visual position of the end-effector.2) The task of learning is to make the N generate an output close enough to q0. a neural network uses these samples to represent the whole input space over which the robot is active. xhand) . both on the sensory and motor side. calculate the joint angles to reach the target (i.1) We can compare the neurally generated qq with the optimal qq0 generated by a fictitious perfect controller R(): q0 = R(xtarget. Gripper control is not a trivial matter at all..

Sideris and Yamamura (1988).. The success of this method depends on the interpolation capabilities of the network. The network is then trained on the error e1 = q q¢ (see Fig.. resulting in q¢.. . xn) M ym = fm(x1.. A Jacobian matrix of a multidimensional function F is a matrix of partial derivatives of F.2). which is constrained to two-dimensional positioning of the robot arm.. if we have Y = F(x).. which generates an angle vector q. then for feeding back the error. 2. This x¢ again is input to the network. x2.e. and the cameras determine the new position x¢ of the end-effector in world coordinates.. by a two cameras looking at an object. For example. a self-supervised learning system must be used. One such a system has been reported by Psaltis.g. x2. xn) y2 = f2(x1. i. e.2. This method requires knowledge of the Jacobian matrix of the plant. 3. The manipulator moves to position q. x2.2 Indirect learning system for robotics. learns by experimentation. minimization of e1does not guarantee minimization of the overall error e = x x¢. the mapping II). a Cartesian target point x in world coordinates is generated. the multidimensional form of the derivative.. i. 16. Correct choice of q may pose a problem.. 16. but here the plant input q must be provided by the user.. the network often settles at a solution that maps all xs to a single q (i.. the network is used in two different places: first in the forward step..NEURAL NETWORKS APPLICATIONS 203 16. For example. However. Here. . the network. General learning: The method is basically very much like supervised learning. Specialized learning: Keep in mind that the goal of the training of the network is to minimize the error at the output of the plant: Î = x x¢. x Neural network q x¢ Plant e1 q¢ Neural network Fig. We can also train the network by backpropagating this error trough the plant (compare this with the backpropagation of the error in Chapter 12). y1 = f1(x1. .e. In each cycle..e. Indirect learning: In indirect learning. This target point is fed into the network. Thus the network can directly minimize |q q¢|. Three methods are proposed: 1.6a Approach-1: Feed-forward Networks When using a feed-forward system for controlling the manipulator. xn) .

(16. The total error Î = x x¢is propagated back through the plant by calculating the dj: @j = F(sj) åd i i ¶ P ( q) i ¶q j .(16. the Jacobian matrix can be used to calculate the change in the function when its parameters change.. 1990) and (Smagt and Krose.... Now.4) where J is the Jacobian matrix of F. be approximated by ¶Pi (q) can ¶q j ¶Pi (q) Pi (q + hq j e j ) Pi (q) » ¶q j ¶q j ...7) This approximate derivative can be measured by slightly changing the input the plant and measuring the changes in the output.204 FUZZY LOGIC AND NEURAL NETWORKS then dy1 = dy2 = ¶f1 ¶f ¶f dx1 + 1 dx2 + . Korst.(16. Again a two-layer feed-forward network is trained with back-propagation.3) LM ¶P OP N ¶q Q i i . The learning rule applied here regards the plant as an additional and unmodifiable layer in the neural network. (16. instead of . and Groen..(16...6) di = x x¢ where I is used to change the scalar qj into a vector.. A somewhat similar approach is taken in (Krose.... However. 1991). (16. + 1 dxn ¶x1 ¶ x2 ¶x n ¶f2 ¶f ¶f dx1 + 2 dx2 + . + 2 dxn ¶x1 ¶ x2 ¶x n M dym = or Eq. in this case we have Jij = ¶fm ¶f ¶f dx1 m dx2 + . When the plant is an unknown function. so.5) where Pi(q) the ith element of the plant output for input q. + m dxn ¶x1 ¶ x2 ¶x n dY = ¶F dX ¶X ...3) is also written as dY = J(X) dX ..

Dq) to the network. A feed-forward network with one layer of sigmoid units is capable of representing practically any function. 4. x Ril + 1 x¢. as input for the neural network. . such that the dimensions of the object need not to be known anymore to the system). One step towards the target consists of the following operations: 1. and back-propagation is applied to this new input vector and the existing output vector. 1994). 5. calculating a desired output vector the input vector which should have invoked the current output vector is reconstructed. & Krose. The configuration used consists of a monocular manipulator. where Ril + 1 is the rotation matrix of the second camera image with respect to the first camera image. Measure the distance from the current position to the target position in camera domain. Smagt. Krose. Teach the learning pair (x Ril + 1 x¢. Groen. smooth function consisting of a summation of sigmoid functions. and to be very adaptive to changes in the sensor or manipulator (Smagt & Krose. 2. Use this distance. 3. an accurate representation of the function that governs the learning samples is often not feasible or extremely difficult (Jansen et al. x¢. together with the current state q of the robot. Calculate the move made by the manipulator in visual domain. 1993). the available learning samples are approximated by a single. 1991. Due to the fact that the camera is situated in the hand of the robot. Send Dq to the manipulator. The reason for this is the global character of the approximation obtained with a feed-forward network with sigmoid units: every weight in the network has a global effect on the final approximation that is obtained. and Groen. By using a feed-forward network. which has to grasp objects. a biologically inspired system is proposed (Smagt. But how are the optimal weights determined in finite time to obtain this optimal representation? Experiments have shown that.NEURAL NETWORKS APPLICATIONS 205 X Neural Network q Plant X¢ e Fig. 1992) in which the visual flow-field is used to account for the monocularity of the system. q.3 The system used for specialized learning. Again measure the distance from the current position to the target position in camera domain. the task is to move the hand such that the object is in the centre of the image and has some predetermined size (in a later article. x. 6.. 16. although a reasonable representation can be obtained in a short period of time. This system has shown to learn correct behavior in only tens of iterations. The network then generates a joint displacement vector Dq.

4). since it is the most interesting and straightforward.(16. final . i. the neuronal lattice is a discrete representation of the workspace. As with the Kohonen network. The system described by Ritter et al... Martinetz. y) coordinates of the object and the end effector (see Fig. The system is observed by two fixed cameras which output their (x. The final retinal coordinates of the end-effector after . consists of a robot manipulator with three degrees of freedom (orientation of the end-effector is not included). Thus accuracy is obtained locally (keep it small and simple). which are arranged in a 3-dimensional lattice. 16. We will only describe the kinematics part. Each run consists of two movements. which has to grab objects in 3D-space. During gross move qk is fed to the robot which makes its move. 16.e. the observed location of the object x (a four-component vector) is input to the network. To correct for the discretization of the working space.6b Approach 2: Topology Conserving Maps Ritter. and Schulten (1989) describe the use of a Kohonen-like network for robot control. Fig. 16. an additional move is made which is dependent of the distance between the neuron and the object in space wk x..2. this small displacement in Cartesian space is translated to an angle change using the Jacobian Ak: qfinal = qk + Ak(x wk) which is a first-order Taylor expansion of q this fine move are in xf .8) . the neuron k with highest activation value is selected as winner.206 FUZZY LOGIC AND NEURAL NETWORKS Building local representations is the obvious way out: every part of the network is responsible for a small subspace of the total input space. because its weight vector wk is nearest to x. With each neuron a vector q and Jacobian matrix A are associated. correspond in a 11 fashion with subregions of the 3 D workspace of the robot. resulting in retinal coordinates xg of the end-effector. The neurons. This is typically obtained with a Kohonen network.4 A Kohonen network merging the output of two cameras. In the gross move.

A)jold + g¢(t) g¢ (t) ((q.1 without wrist joint. 16.10) In eq.. In fact. Thus eq. the network can be restricted to one learning layer such that finding the optimal is a trivial task.. and the robot is not too susceptible to wear-and-tear. as with the Kohonen learning rule. which is generated by another subsystem.NEURAL NETWORKS APPLICATIONS 207 Learning proceeds as follows: when an improved estimate (q. It appears that after 6.000 iterations the system approaches correct behavior. Again. A)jnew = (q.7 Robot Arm Dynamics While end-effector positioning via sensor{robot coordination is an important problem to solve. Here. the change in retinal coordinates of the end-effector due to the fine movement. this is similar to perceptron learning.. A)jold) jk j If gjk(t) = g¢ (t) = djk.e.10) can be recognized as an error-correction rule of the Widrow-Hoff type for Jacobians A. This requirement has led to the current-day robots that are used in many factories. the following adaptations are made for all neurons j: wjnew = wjold + g(t) gjk(t) (x wjold) (q.(16. Dx = xf fg.e. the related joint angles during fine movement. jk a distance function is used such that gjk (t) and g¢ (t) are Gaussians depending on the distance between jk neurons j and k with a maximum at j = k. A)* has been found.5). The error between qd(t) and q(t) is fed into the neural model.(16. Dynamics model. the system is a feed-forward network.. But the application of neural networks in this field changes these requirements. 16. A)* (q. The desired trajectory qd (t). The manipulator used consists of three joints as the manipulator in Fig. the robot itself will not move without dynamic control of its limbs. and that after 30. One of the first neural networks which succeeded in doing dynamic control of a robot arm was presented by Kawato. Their system does not include the trajectory generation or the transformation of visual coordinates to body coordinates. 16. and Dq = Ak(x wk).. the basis functions are thus chosen that the function that is approximated is a linear combination of those basis functions. i.000 learning steps no noteworthy deviation is present. i. and Suzuki (1987). accurate control with nonadaptive controllers is possible only when accurate models of the robot are available. Furukawa.. They describe a neural network which generates motor commands from a desired trajectory in joint angles. (16.10). the final error x xf in Cartesian space is translated to an error in joint space via multiplication by Ak. but by carefully choosing the basis functions. An improved estimate (q.2. The network is extremely simple. (16. In eq.9) = Ak + (Dq Ak D x) ( x f xg )T || x f x g ||2 Dx T || Dx||2 . is fed into the inverse-dynamics model (Fig. A)* is obtained as follows: q* = qk + Ak(x xf) A* = Ak + Ak(x wk xf + xg) . (16. In this case. This error is then added to qk to constitute the improved estimate q* (steepest descent minimization of error). .9).

4)T and (16.2. training with a repetitive pattern sin (wkt).2. 2.208 FUZZY LOGIC AND NEURAL NETWORKS Inverse dynamics model q d( t ) Ti(t) + Tf (t) + K T(t) Manipulator q(t) + – Fig.5 The neural model proposed by Kawato et al. such that Tik(t) = with xl1 = f1(qd1(t). 16.6.11) dq(t ) .2. 191. 3) . 746. qd 2(t).. Although the applied patterns are very dedicated. 16. the weights adapt using the delta rule g dwik = xik T1 = xik(Tjk Tik). each one feeding in one joint of the manipulator.7. 2 : 3 is . After 20 minutes of learning the feedback torques are nearly zero such that the system has successfully learned the transformation. (k1. qd3) is feed into 13 nonlinear subsystems.3) . Next. which is shown in Fig. The feedback torque Tf (t) in Fig.3) dt Kvk = 0 unless |qk(t) qdk(objective point) | < e The feedback gains Kp and Kv were computed as (517. qd2. 16.0. 8.. The desired trajectory qd = (qd1.1.. dt (k = 1.2. The resulting signals are weighted and summed.5 consists of Tfk(t) = Kpk(qdk(t) = qk(t)) + Kvk åw i =1 13 lk xlk (k = 1. 16.(16.12) A desired move pattern is shown in Fig. qd 3(t)) and fl and gl as in Table 16.4)T . qd 2(t). The neural model. qd 3(t)) xl 2 = xl3 = gl (qd 1(t).2.(16. consists of three perceptrons. 37.. with w1: w2: w3 = 1: also successful.

6 The neural network used by Kawato et al. 3(t) qd 1(t) qd 2(t) qd 3(t) T1(t) T2(t) Ti1(t) g1 S Ti2(t) Ti3(t) S g13 T3(t) Fig. 3(t) x13.NEURAL NETWORKS APPLICATIONS x1. joint 1 in Figure 16. q2. 2(t) x13. 1(t) 209 f1 S f13 x13. q3) && q1 && sin2 q q1 2 && cos q q1 2 2 g1(q1. The upper neuron is connected to the rotary base joint (cf. the other two neurons to joints 2 and 3. q3) && q2 && q3 && cos q q2 3 && cos q q3 3 && 2 sin q cos q q1 2 2 & q 2 sin (q2 + q3) cos (q2 + q3) 1 & q 2 sin q2 cos (q2 + q3) 1 & q 2 cos q2 sin (q2 + q3) 1 & q 2 sin q3 2 & q 2 sin q3 3 & q q 2 &&3 sin q3 & q2 & q3 && sin (q + q ) q1 2 3 2 && cos2 (q + q ) q1 2 3 && sin q sin (q + q ) q1 2 2 3 && q sin q cos q q1 & 2 2 2 && q sin (q + q ) cos (q + q ) q1 & 2 2 3 2 3 && q sin q cos (q + q ) q1 & 2 2 2 3 && q cos q sin (q + q ) q1 & 2 2 2 3 && q sin (q + q ) cos (q + q ) q1 & 3 2 3 2 3 && q sin q cos (q + q ) q1 & 3 2 2 3 && q1 . q2. 16.1: l 1 2 3 4 5 6 7 8 9 10 11 12 13 Nonlinear transformations used in the Kawato model. f1(q1. There are three neurons. Table 16. Each neuron feeds from thirteen nonlinear subsystems. 1(t) x1.1). one per joint in the robot arm. 2(t) x1.

ART type unsupervised neural network paradigm was used for detection of tool breakage. With the normal and broken tool cutting force variation signals is possible to train neural networks.3 DETECTION OF TOOL BREAKAGE IN MILLING OPERATIONS The recent trend in manufacturing is to achieve integrated and self-adjusting machining systems. ensuring safe and efficient metal removal rate and taking corrective actions in the event of failures and disturbances (Yusuf. Neural networks with parallel processing capability and robust performances provide a new approach to adaptive pattern recognition. (b) The back-propagation technique generalizes the given information in order to store it inside the initially selected hidden layers. Neural networks are also classified as supervised and unsupervised according to their learning characteristics. Unless recognized in time. Adaptive Resonance Theory (ART 2) architectures are neural networks that carry out stable self-organization of recognition codes for arbitrary sequence of input patterns. Also simulation-based training is proposed to reduce the cost of preparing the systems that monitor the real cutting signals. The absence of human supervision requires on-line monitoring of machining operation. Joints 2 and 3 have similar time patterns. 1995). In this section. The cutting force variation characteristics of normal and broken tools are different.210 FUZZY LOGIC AND NEURAL NETWORKS q1 p 0 -p 10 20 30 t/s Fig. which are capable of machining varying parts without the supervision of operators. The usefulness of neural algorithms is demonstrated by the fact that novel robot architectures. (a) The training of paradigm is much faster than the back-propagation technique. 1993). are now constructed. In unsupervised learning. tool breakage can lead to irreparable damage to the workpiece and possibly to the machine tool itself. the neural network classifies the signals by itself. The use of adaptive resonance theory (ART) type neural network was evaluated for detections of tool breakage. Artificial neural networks refer to a group of architectures of the brain (Cheng and Sheng. The milling operations can be monitored with the neural network. which no longer need a very rigid structure to simplify the controller. The back propagation technique cannot give reliable decisions on the sufficiency of previous training. after training. 16. 16. and . One of the most important monitoring requirements is a system capable of detecting tool breakages on-line. ART paradigm was used for the following reasons.7 The desired joint pattern for joint 1. 1998). in this study (Ibrahim and Mclaughlin.

It is extremely expensive and time consuming to collect cutting force data at different cutting conditions with normal and broken tools.(16. where it is normalized. (1987). Another important issue is the training of the neural network.. If the new pattern cannot be classified in a known category. To overcome this problem.1 Unsupervised Adaptive Resonance Theory (ART) Neural Networks The theory of adaptive resonance networks was first introduces by Carpenter and Grossberg.(16. The F1 field in ART 2 includes a combination of normalization and noise suppression. which are temporary and flexible.. F1 and F2. and the long term memory (LTM) nodes.... gain and learning procedures. it is coded and classified as a new category. F1 uses the following equations to calculate the nodes: ui = vi e + ||v|| . or already known patterns.13) . learned.. When as ART network receives an input pattern. Adaptive resonance occurs when the input to a network and the feed back expectancies match.3.(16.(16.16) . To accomplish this. which are permanent and stable..NEURAL NETWORKS APPLICATIONS 211 (c) ART has very important advantage since it can be trained in the field and continuously updates previous experience. in addition to the comparison of the bottom-up and top-down signals needed for the reset mechanism.. The unsupervised ART neural networks can monitor the signal based on previous experience and can update itself automatically while it is monitoring the signals (Carpenter and Grossberg.18) wi = Si + aui pi = ui + S q( yi) z ji qi = pi e + || p|| wi e + || w|| vi = f(xi) + bf(qi ) xi = .(16. matched..(16. 1991). The STM is divided into two sets of nodes. The theoretical background of ART 2 type neural network. Simulated data was used to select the best vigilance of the ART 2 type neural network and to evaluate the performance of paradigm... this bottom-up pattern is compared to the top-down. If the input pattern is matched with known pattern in memory. 16. The input pattern ( i ) is received by the STM.. The ART 2 neural networks developed by Carpenter and Grossberg (1991) self-organize recognition codes in real time. the weights of the model are changed to update the category. the proposed data monitoring system and their performance is presented in the paper. control. simulation-based training of neural networks was introduced. the short term memory (STM) nodes. The STM F1 nodes are used for normalization.17) . and stored in the LTM (zji). The basic ART 2 architectures consist of two types of nodes.15) .14) .

...24) dzij dt = g(yj) [pi zij] ... N} . then equations (16. b.....(16.22) to: .22) Equation (3) takes the following form: pi = Ru Su + dz T if F2 is inactive if jth node is F2 is active ij .24) and (16.23) The bottom-up and top-down LTM equations are bottom-up (F1 ® F2) : dzij dt topbottom ( F2 ® F1) : = g(yj) [pi zij] .20) is satisfied. ||v|| and ||w|| denote the norms of the vectors p.21) Competition on F2 results in contrast enhancement where a single winning node is chosen each time.(16. v and w.. The node that gives the largest sum with the F1.(16. all other nodes in the LTM are inhibited... f(x) = Rx S0 T if x ³ q if 0 £ x ³ q . and si is the input. 2 . and only one is allowed to interact with the STM. N otherwise .. 2. and e are selected based on the particular application.(16. The constants a. 2004] Tj = åpz i i ji . F2 equations select or activate nodes in the LTM..25) When F2 is active. The function f filters the noise from the signal... q can be set to zero for the case where filtering is not desired. The output function of F2 is given by g(yi) = Rd S0 T i i if Ti : j = 1. Bottom-up inputs are calculated as in ART 2 [Fauselt.25) are modified from equation (16.19) where q is an appropriate constant. When F2 chooses a node.(16.20) The jth node is selected if equation (16. The STM F2 nodes are used for the matching procedure. The non-linear signal function in equation (5) is used for noise suppression .212 FUZZY LOGIC AND NEURAL NETWORKS Here ||p||.(16. F2 input pattern (bottom-up) is the key property that is used for node selection...(16. the activation function (f ) is given by the equation. Tj = max { Tj: 1. .

r: ri = ui + cpi e + ||u|| + ||cp|| . the neural network classified the good and broken tools in different categories.016 Feed rate mm/min 50. In each one.26) where d is a constant ( 0 < d < 1).NEURAL NETWORKS APPLICATIONS 213 dzij dt = d[pi zij] .98 in all the tests. The ART neural network monitored the profile of the resultant force in different tests.(16.016 1.5. 1.5) tests for the broken tools.016 1. An orienting ART 2 sub system is used to decide. If no match has been found after all nodes have been activated. On the other hand. Vigilance of the neural network was 0.2-16.8 50. The LTM node weights are recalculated and the pattern is learned by the system.. As seen in Tables 16. 2. if a new pattern can be matched to a known pattern by comparing with a given vigilance parameter. Depth of cut (mm) 1.016 1. 1 and 3 different categories were selected.3) .4) and 4th (Table 16.6 101.27) If ||r|| < r e. For the broken tool 2..8 101. The neural network did not have any prior information at the beginning of each test.016 1. The ART assigned 2. Table-16..2-16.5. a match has been found and the new pattern is learned by the system. 1.(16. 3rd (Table16.96 or 0.07 mm diameter at various cutting conditions. The vigilance of the ART 2 selected either 0.3.2 Results and Discussion The experimental data was collected with a fo1ur flute end mill of 12.016 1. the neural network inspected the resultant force profile and placed it into a category or initiated a new category if it was found to be different. a new node is created.2 254 254 Tool condition G B G B G B G B Category 1 2 1 3 1 3 4 5 Spindle speed (rpm) 500 500 500 500 500 500 500 500 . and depth of cut of these different conditions are out lined in Tables 16. the neural network assigned more nodes to the signal of a good tool with offset.2 203. In the three tests. 16. the neural network generated only one category in the 2nd (Table 16. 1 and 3 different categories for the good tool. feed rate. It indicates that the broken tool signals are more similar to each other at different cutting conditions compared to the force patterns of normal tools. then F2 resets another node. 3. The ART used four categories to classify all of the data.016 1. The spindle speed. If ||r|| ³ r e .6 203.96. and the new pattern is stored.2 Classification of experimental data with the ART. In all the tests.016 1. experiments were done at different feed rates with the good and broken tool..

6 203.2 203.016 1. .8 101.524 1.2 203.98 is used.524 Feed rate mm/min 50.524 1.524 1. which requires a minimum number of nodes and has acceptable error rate.524 1.6 101.8 50.2 203. The ART 2 used four categories to classify all of the data. The ART used three categories to classify all of the data.8 101. After simulation training. the neural network inspected the incoming signals and continued to assign new categories when different types of signals were encountered.6 101.6 203.524 1.6 101.016 1. The studies focused on selection of the best vigilance.524 1.016 1.8 50.2 254 254 Tool condition G B G B G B G B Category 1 2 3 2 4 2 4 2 Spindle speed (rpm) 700 700 700 700 700 700 700 700 Table 16.524 1.524 1.98.2 254 254 Tool condition G B G B G B G B Category 1 2 3 2 1 2 3 2 Spindle speed (rpm) 500 500 500 500 500 500 500 500 Table 16.8 101.016 1. Vigilance of the neural network was 0. Depth of cut (mm) 1. The ART 2 used two categories to classify all of the data.524 1.016 Feed rate mm/min 50. the neural network started to monitor the experimental data collected at different conditions.96.214 FUZZY LOGIC AND NEURAL NETWORKS Table16.6 203.016 1.524 1.016 1.2 254 254 Tool condition G B G B G B G B Category 1 2 1 2 1 2 1 2 Spindle speed (rpm) 700 700 700 700 700 700 700 700 The ART gained first experience on the simulation data and later. the network classified the perfect too input data into seven different categories and classified the broken tool input data into four different categories. Vigilance of the neural network was 0. Depth of cut (mm) 1.524 1.3: Classification of experimental data with the ART.524 Feed rate mm/min 50.524 1.524 1.8 50.5: Classification of experimental data with the ART 2.96. When the vigilance of 0. Depth of cut (mm) 1.524 1. Vigilance of the neural network was 0.016 1.4: Classification of experimental data with the ART 2.

Groen.Tools Manufact. 13. Smagt. 2. 1995. In O. Krose. K. 169-185. 1992. Kangas (Eds. Mach.J. Addison-Wesley Publishing Company. 2. Yusuf In-processs detection of tool breakages using time series monitoring of cutting forces. 1993.J. The neural network approach.Tools Manufact. Learning strategies for a vision based neural controller for a robot arm. Department of Computer Systems.A. 4. Groen. No. Korst. and F. A. University of Amsterdam.. 2. A. 5. Furukawa. In A. 9.J. Krose.). and B. pp.A. Neural Networks. Vol.J. A real-time learning neural robot controller. 351-356.F. & Suzuki. M. A multilayer neural network controller.T. J. In Proceedings of the 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems. North Holland/Elsevier Science Publishers.A. Makisara. Simula. Sideris.A. 1988. Vol. Proceedings of the 1991 International Conference on Artificial Neural Networks. 1993. T. & J. Explain the application of ART for machining applications.A. A. Kaynak (Ed.J. Rep. pp. Krose. of Neural Networks. 3. pp. and K. P.C.A. and A. 159-168. Adaptive hamming net. Explain the application of Kohonon network for robot control. Using time-to-contact to guide a robot manipulator. and B. IEEE Control Systems Magazine. 1989. 33. CS-93-10. No. Neural Network Applications. Biological Cybernetics. M. 605-618.M. Krose. In T. 1. pp. Craig. and F. Enumerate various applications of neural networks. 8. No. Schulten. 12. Ritter. 1994. Groen. N. pp. Vol. Sheng. Kawato. Int. 2. D. B. . Mclaughlin. Psaltis.Cheng and F. akisara. 1989. Kluwer Academic Publishers. Nos. pp.P. P. and B.J. Int. Vol.J. Ibrahim and C. IEEE. 1. 28. In T. 1990. No. REFERENCES. B.C. Nested networks for robot control. Mach. Groen.J. Robot Hand-Eye Coordination Using Neural Networks. 5.NEURAL NETWORKS APPLICATIONS 215 QUESTION BANK. 1991. P. Vol.). B. K.). 351-356.C. Int.P. Kohonen.J.J. and F.A.J. 57.A. A. 4. 1998. J. Murray (Ed. 10. Explain the application of neural network for robot arm dynamics. 17-21. Krose.P. & J. Topology-conserving maps for learning visuomotor-coordination. Simula.1991. F. M.M. 4. pp.H. 6. 545-558. 3. (1987). 8.). Korst. R. Kohonen. Kangas (Eds. Explain the application self-supervised learning to control manipulator. A real-time learning neural robot controller. IEEE International Workshop on Intelligent Motor Control.. Yamamura. Introduction to Robotics. 7. Smagt. Tech.A.A. Proceedings of the 1991 International Conference on Artificial Neural Networks. O. 4. Krose. J. Detection of tool breakage in milling II. pp. Smagt. 8. J. 199-203. A hierarchical neural-network model for control and learning of voluntary movement. 2. Jansen. H.A. O. A Fast learning ART 1 model without searching. 11.C. Martinetz. K. 157-172. North-Holland/Elsevier Science Publishers.

G. 54115. and S. 16. Graphics. 493-504. ART 2-A: Adaptive resonance Algorithm for rapid category learning and recognition. Applied Optics. and Image Processing. L. 1987. . G. and S. 37.A. Carpenter. Carpenter. Fauselt.A. Pearson Education. 2nd Edition. ART 2: Self-organization of stable category recognition codes for analog input patterns. Fundamentals of Neural Networks. Computer Vision. Architectures. 4. and Applications. Vol. of Neural Network. pp. Grossberg. 1987. 2004. Carpenter and Grossberg. 4919-4930. A massively parallel architecture for a self-organizing neural pattern recognition machine. 1991. Algorithms. Vol. Int. J. pp.216 FUZZY LOGIC AND NEURAL NETWORKS 14. Grossberg. 15. 17. 26(23).

e. Normally. 17. which act on two distinct subproblems.2.+ 0 ) 2 6 .2 HYBRID SYSTEMS Designation neurofuzzy has several different meanings.1.. The output of one technology becomes the input to another technology and so on. A simple sequential hybrid system is shown in Fig.g. Hybrid systems are those for which more than one technology is employed to solve the problem. on the hardware whereas fuzzy logic systems concentrate on software. 17. in signal processing) and fuzzy logic is utilized in the second subproblem (e. In the following light is tried to shed on the most common different interpretations. . This is one of the weakest forms of hybridization because an integrated combination of the technologies is not present..4 Hybrid Fuzzy Neural Networks 17 17. The hybrid systems are classified as: Sequential hybrids Auxiliary hybrids Embedded hybrids 17. Combining neural networks and fuzzy systems in one unified framework has become popular in the last few years. neural network is utilized in the first subproblem (e. i.1 Sequential Hybrid Systems Sequential hybrid systems make use of technologies in a pipeline-like structure.1 INTRODUCTION Neural networks and fuzzy systems try to emulate the operation of human brain.g. In that case. the link between these two soft computing methods is understood to be stronger. in reasoning task).. Sometimes neurofuzzy is associated to hybrid systems. Neural networks concentrate on the structure of human brain. when talking about the neurofuzzy systems.

218 FUZZY LOGIC AND NEURAL NETWORKS Inputs Technology A Technology B Outputs Fig.2. The embedded hybrid system is better than sequential and auxiliary hybrid systems.2. 17.3 illustrates an embedded hybrid system.2. An auxiliary hybrid system is shown in Fig.1 A sequential hybrid system. Fig. one technology calls the other as a subroutine to process or manipulate information by it. .3 Embedded Hybrid Systems In embedded hybrid systems. 17. The auxiliary hybrid system is better than the sequential hybrid system. The second technology processes the information provided by the first and hands it over for further use. Inputs Technology A Technology B Output Fig.2 Auxiliary Hybrid Systems In this.2 An auxiliary hybrid system. 17. The fusion is so complete that it would appear that no technology could be used without the others for solving the problem. 17. 17. 17. the technologies participating are integrated in such a manner that they appear intertwined.

Rules are of the type: Rule 1: Rule 13: Rule 25: IF(G o E is NB) AND (C o G o E is NB) THEN (C o LP is NS) IF(G o E is ZB) AND (C o G o E is ZE) THEN (C o LP is PS) IF(G o E is PB) AND (C o G o E is PB) THEN (C o LP is NS) where C o LP is change of learning parameter. 17. G o E is the gradient of the error surface.3 FUZZY LOGIC IN LEARNING ALGORITHMS A common approach is to use fuzzy logic in neural networks to improve the learning ability. positive small and positive big. (They also incorporated in rules information about the sign change of gradient and information about the momentum constant. 17. The purpose is to achieve a faster rate of convergence by controlling the learning rate parameter with fuzzy rules. MLP is multiplayer perceptron. C o G o E is change of G o E (approximation to second order gradient). PS and PB are fuzzy sets negative big.) Learning parameter performance Desired performance Error FLC MLP Output Actual performance Fig. negative small. 17. a fuzzy control of back-propagation is illustrated in Fig. ZE. 17. NB. .3. FLC is fuzzy logic controller.4 Learning rate control by fuzzy logic.HYBRID FUZZY NEURAL NETWORKS 219 Inputs Technology A Technology B Outputs Fig. For example. NS. zero equal.3 An embedded hybrid system.

. fuzzy input. t-norm (min) for addition and t-conorm (max) for product. and if they are chosen on the contrary. fuzzy weights.3) where g1(y) = ln y ln (1 y) is simply the inverse function of logistic sigmoid g(z) = 1/(1 + e z)..4 FUZZY NEURONS Definition 17. 1]. fuzzy input..2) corresponds to max-min composition known from the fuzzy logic. wd)) y = S (T(x1. RFNN) A regular fuzzy neural network is a network with fuzzy signals and/or fuzzy weights. fuzzy output 3. crisp input....1) . . The networks of this type are used in classification problems to map fuzzy input vectors to crisp classes. These combinations are again combined (i. t-conorm S(xi. 2.220 FUZZY LOGIC AND NEURAL NETWORKS Simulation results show that the convergence of fuzzy back propagation is faster than standard back propagation. fuzzy weights. fuzzy output (or crisp output by defuzzification).2)..1) corresponds to min-max composition and (17.(17. wi). Output of (17. . Activation function g can be any continuous function. (usually membership degrees) and real valued weights wi Î[0. If we choose linear activation function. We use the extension principle to calculate wixi Output fuzzy set Y is computed by the Extension principle Y(y) = R(w x + w x ) g S0 T 1 1 2 2 1( y ) if 0 £ y £ 1 otherwise ... w1). In addition there exists a type of network where the weights and targets are crisp and the inputs are fuzzy. wi).(17..1: Consider a simple network y = g(w1x1 + w2x2).. we get an OR fuzzy neuron (17. Input and weight are combined (i.2) T(xi. we get an AND fuzzy neuron (17.e. T (xd.. or some continuous operation. 1].. sigmoidal activation function and all the operations are defined by extension principle. S (xd. product is replaced) using t-norm y = T(S(x1. Definition 17.e. fuzzy output which can be used to implement fuzzy IF-THEN rules.. 17.1: (hybrid neural network) A hybrid neural network is a network with real valued inputs xi Î[0. Example 17. wd)) . where the inputs and weights are fuzzy numbers.2: (regular fuzzy neural network.. t-conorm.. Another way to implement fuzzy neuron is to extend weights and/or inputs and/or outputs (or targets) to fuzzy numbers. crisp weights. addition is replaced) using t-norm. The choices are as follows: 1. which means that the fuzzy neural nets based on the extension principle are universal approximators only for monotonic functions. The problem of regular fuzzy neural networks is that they are monotonic. or some other continuous operation. w1). .1).(17.

4) which means that regular fuzzy neural network is not a universal approximator.. the number of combinations is 16. wi are 1 2 i fuzzy numbers). The most common method to decrease the dimension of input space is the principal component analysis (PCA). wi).. compute the mean of inputs in data and subtract it off 2.. Activation function g can be any continuous function. This is a serious drawback for the networks of this type.. t-conorm S(xi.5 NEURAL NETWORKS AS PRE-PROCESSORS OR POSTPROCESSORS One of the biggest problems with fuzzy systems is the curse of dimensionality. if we have five inputs and each input space is partitioned into seven fuzzy sets.6) where mi is the number of fuzzy sets on axis i. The number of combinations of input terms (possibly also the number of rules) is Õm i d i . 17.HYBRID FUZZY NEURAL NETWORKS 221 Theorem 17. t-conorm. When the dimension of the problem increases the size of the fuzzy model (and the size of training set needed) grows exponentially. Therefore. Many researchers working with fuzzy neurons follow the basic principles described above. there is a strong need for data reduction.. then g(w1x1 + w2w2) Ì g(w1x¢ + w2 x¢ ) 1 2 Proof: Because min and max are increasing functions. and projecting the data onto this M-dimensional subspace. Selecting M attributes from d is equivalent to selecting M basis vectors which span the new subspace. i.3: (hybrid fuzzy neural network. This reduction is achieved via a set of linear transformations. wi).807. 1]. Input and weight are combined using t-norm T(xi.1: g is an increasing function of its arguments. Therefore. identifying principal reduce the dimensionality of a data in which there are large number of correlated variables and at the same time retaining as much as possible of the variation present in the data. or some continuous operation. then (w1x1 + w2x2) Ì (w1x¢ + w2 x¢ ) 1 2 . 1]. The smallest number of input variables should be used to explain a maximal amount of information. Definition 17. but there is no standard path to follow. and/or fuzzy valued weights wi Î [0. For example. HFNN) A hybrid fuzzy neural network is a network with fuzzy valued inputs xi Î [0..5) . The use of more than 4 inputs may be impractical. The algorithm goes as follows: 1. x¢. or some other continuous operation. if x1 Ì x¢ and x2 Ì x¢ (xi.(17. The main goal of identifying principal components is to preserve as much relevant information as possible.e. These combinations are again combined using t-norm. calculate covariance matrix and its eigenvectors and eigenvalues .(17.(17. which transform input variables to a new set of variables (uncorrelated principal components).

increase flexibility. is represented as a network. the network can be made to perform a non-linear principal component analysis.5.6 NEURAL NETWORKS AS TUNERS OF FUZZY LOGIC SYSTEMS The similarities between neural networks and fuzzy logic systems were noticed. A two-layer perceptron with linear output units (number of hidden units is M.5 Neurofuzzy network for back-propagation. some researchers began to represent fuzzy logic systems as feed forward networks. the fuzzy logic system with d inputs one output approximator. retain eigenvectors corresponding to the M largest eigenvalues 4. If two additional nonlinear hidden layers are allowed to put into the network. with M < d ) which is trained to map input vectors onto themselves by minimization of sum-of-squares error is able to perform a linear principal component analysis. The principle of the method is that it finds the derivatives of an error function with respect to the weights in the network. which led to the development of neurofuzzy systems. improve performance and decrease human intervention. centers and widths of membership functions. In Fig.222 FUZZY LOGIC AND NEURAL NETWORKS 3. Multivariate memb. 17. In the 1980s. a computationally efficient training algorithm for multi-layer neural networks was discovered. The error function can then be minimized by using gradient based optimization algorithms. 17. project input vectors onto the eigenvectors Neural networks may be used to perform the dimensionality reduction. If prior knowledge is unavailable and/or the plant is time-varying then the most sensible (possibly the only) solution is to utilize learning capabilities. The idea was to use the training algorithm to adjust weights. The original purpose of neurofuzzy systems was to incorporate learning (and classification) capability to fuzzy systems or alternatively to achieve similar transparency (intelligibility) in neural networks as in fuzzy systems. Since back-propagation can be applied to any feed forward network. functions Pm X1 * * l WM Xd * * Pm k 1 Pm 1 W1 Norma lize l l + Denominator = Gaussian membership function Fig. It was named error back-propagation. 17. . Learning is assumed to reduce design costs.

Other parameters (width and position of membership functions) have clear meaning 3. fuzzy logic systems and conventional models. different types of neural networks. The most common way to represent neurofuzzy architecture is shown in Figure 17. The performance of committee can be better than the performance of isolated networks. Initial weights can be chosen appropriately (linguistic rules) The drawback is the curse of dimensionality.g.6. 17.. e.Consequent lizer parameters Neurofuzzy classifier x1 A1 A2 B1 B2 Class 1 Class 2 Class 3 Class 4 Class 5 y x2 Linguistic variables Fig. Although it looks different from the network in Fig.5. 17. It can consist of networks with different architectures.6 And Or Max defuzzifier Neurofuzzy network for function approximation and classification problems. Only the way to illustrate network differs. Weights are the centers of THEN part fuzzy sets (clear meaning) 2.8 COMMITTEE OF NETWORKS The method of combining networks to form a committee has been used to improve the generalization ability of the networks.7 ADVANTAGES AND DRAWBACKS OF NEUROFUZZY SYSTEMS The advantages are as follows: 1. 17. 17. it is basically the same network.HYBRID FUZZY NEURAL NETWORKS 223 Neurofuzzy function approximator x1 A1 A2 B1 B2 S y x2 Linguistic Multivariables plier Norma. .

and jth neuron of the hidden layer. and III. 17. Input x1 Hidden Output y1 y2 x2 y3 Fig.224 FUZZY LOGIC AND NEURAL NETWORKS The committee prediction (output) is calculated as an average of the outputs of the q networks: yCOM(x) = 1 yi ( x ) q i =1 å q . He does not restrict the form of fuzzy system to be additive or SAM system. While defining a cost function for level sets of fuzzy outputs and fuzzy targets.7 shows this model fuzzy neural network. The only difference with (17. a-cuts) of fuzzy weights and fuzzy inputs. Fig. 17. we propose a learning algorithm from the cost function for adjusting three parameters of each strong L-R type fuzzy weight.. The total fuzzy input of jth neuron in the second layer is defined as: . neurons are organized into a number of layers and the signals flow in one direction. There are no interactions and feedback loops among the neurons of same layer. fuzzy weight and crisp inputs. crisp weight and fuzzy inputs.(17. fuzzy weight and fuzzy inputs. which provides a fuzzy weight of a connection between ith neuron of the input layer.7) The reduction of error arises from the reduced variance due to averaging.7) is that Kosko weights fuzzy system outputs yi(x) by credibilities wi Î [0. Kosko (1991) has proposed the use of weighted average to combine different fuzzy systems that try to predict the same input-output relation. Lastly we examine the ability of proposed fuzzy neural network implementing on fuzzy if-then rules.7 A three layered fuzzy neural network According to the type of inputs and weights we define three different kinds of fuzzy neural networks as follows: I. 17. Type (III) of fuzzy feed forward neural networks is presented here.. Next we define the strong L-R type fuzzy number. In the fuzzy neural networks based on BP. In this model.. 1]. the connections between the layers may be illustrated as a matrix of fuzzy weights wji. such that at least one system has nonzero credibility.e. and show its good properties in interval arithmetic.9 FNN ARCHITECTURE BASED ON BACK PROPAGATION The input-output relation is numerically calculated by interval arithmetic via level sets (i. II.

Tp) is obtained as ep = åe h ph .. and Tp = (Tp1.. we need to find out a type of fuzzy number to denote the fuzzy inputs. ....13) L e pkh d[T = h. fuzzy weights and fuzzy biases. this type fuzzy number has good property so that it can be easily adept to the interval arithmetic.. Next. The fuzzy output of the jth neuron is defined with the transfer function Opj = f(Neypj). a .. Tp NO) is NO dimensional fuzzy target vector corresponding to the fuzzy input vector xp.O ji i =1 N pj + Qj . 2. The cost function for the input-output pair (xp. j = 1.(17.(17.(17. Opi = xpi is the ith fuzzy input of that neuron. Netpj is the total fuzzy input of the jth neuron of hidden layer. the fuzzy output of the kth neuron of output layer is defined as follows: Netpk = .9) åw j =1 NH O kj . j = 1.(17... Tp) is the fuzzy input-output pairs..12) The cost function for the h-level sets of the fuzzy output vector Op and the fuzzy target vector are defined as eph = where L epkh = e pkh + eU pkh 2 åe k =1 NO pkh .. and Qj is fuzzy bias of the jth neuron.. fuzzy weights and fuzzy biases. NO The fuzzy output is numerically calculated for level sets (i. Furthermore. and put forward a FNN algorithm .10) .. pj + Qk .cut) of fuzzy inputs..11) Opk = f (Netpk). .. .(17. d[T = h.HYBRID FUZZY NEURAL NETWORKS 225 Netpj = å w . Tp2. let (xp.e...(17..8) Where.... L pk ]h L [Opk ]h i 2 [Opk ]U h 2 eU pkh U pk ]h i 2 Next section we introduce the strong L-R type fuzzy number. 2. NH Furthermore.

which is a wellestablished area of classical mathematics. 2.N. wg )LR kj kj kj Wji = (wa.R Representation of Fuzzy Numbers Definition 17.1 Strong L.9. qg )LR k k k .6: This kind of fuzzy number M is said to be an strong L-R type fuzzy number if L(1) = R(1) = 0. These are closed intervals of real numbers. M is the mean value of M. The strong L-R type is an important kind of fuzzy numbers..226 FUZZY LOGIC AND NEURAL NETWORKS based BP. Let (b x)| a = 1.F. qb.5: A fuzzy number M is said to be an L-R type fuzzy number if R LFG b x IJ |H a K | mM(x) = S |RFGH x b b IJK | T if x £ b a > 0 if x ³ b b > 0 .. the strong L-R type fuzzy number can be uniquely represented by three parameters.14) L is the left and R for right reference. such that the support of every fuzzy number is the interval (a. Those properties are essential for defining meaningful arithmetic operations on fuzzy numbers. wkj and fuzzy biases Qj. wg )LR ji ji ji Qk = (qa. We can write any strong L-R type fuzzy number symbolically as M = (a. Definition 17.) is a special class of the strong L-R type fuzzy number. + ¥ ]. Since each fuzzy set is uniquely represented by its a-cut. Qk by these parameters as Wkj = (wa . in other words. Definition 17. S is non-increasing on [0.. 3. L(b a) = L(a) = 0. S(0) =1. x = b a º a such that. v)LR. Accordingly we can adjust three parameters of each strong L-R type fuzzy weight and fuzzy biases. We express the strong L-R type fuzzy weight wji. we write M = (mab)LR.. S(x) = S (x). This kind of fuzzy number has properties as follows: 1. w b. v) of real numbers.4: A function. is a reference function of fuzzy numbers if 1. The triangular fuzzy number (T.15) . b. symbolically. These operations are the cornerstone of interval analysis. usually denoted L or R.(17. The a-cuts of every fuzzy number are closed intervals of real numbers 2. arithmetic operations on fuzzy numbers can be defined in terms of arithmetic operation on closed intervals of real numbers. 3. 17. We will utilize them in next section to define arithmetic operations on fuzzy numbers.(17. w b . a and b are called left and right spreads. same as R(b + b) = R(v) = 0. Fuzzy numbers are convex fuzzy sets.

18) y ¶wkj ¶e ph = ¶[ wkj ]a h .(17. q b. qk ..(17. ck = q gj qb q g qb j k k ..19) Since wkj is a strong L-R type fuzzy number. it is h-level and 0-level have relations as follows: [ wkj ]a = h y a wkj + ckj + wkj 1 + ckj y a wkj + ckj + wkj y [ wkj ]h = 1 + ckj LM (w w ) OP L (h) NM 1 + c QP L c (w w ) OP R (h) M NM 1 + c QP y kj a kj 1 . we can count the quantity of adjustment for each parameter by the cost function L Dwkj(t) = h L ¶wkj ¶e ph L + xDw k j (t 1) ..(17. w b .17) The derivates of above can be written as follows: a ¶wkj ¶e ph = ¶[ wkj ]a h ¶e ph ¶e ph y ¶[wkj ]a ¶e ph ¶[wkj ]h h + y a a ¶wkj ¶[ wkj ]h ¶wkj y ¶[wkj ]a ¶e ph ¶[wkj ]h h + y y y ¶wkj ¶[ wkj ]h ¶wkj .... cji = w g wb ji ji wb w a ji ji .. qk have same form as wkj.(17..20) kj kj y kj a kj 1 .21) kj Therefore.. wij . a ¶wkj ¶e ph = ¶e ph ¶[ wkj ]a h ¶e ph ¶[ wkj ]= h y ¶wkj ¶e ph = LM c NM1 + c LM c MN1 + c kj kj kj kj OP QP e L ( h) O PP + ¶[¶w 1+ c Q + 1 kj ¶e ph ckj ckj L1 ( h) + R 1 ( h) y 1 + ckj ¶[ wk j ] h 1 + ckj 1 + ckj ph y kj ]h LM NM LM 1 MN1 + c + kj ckj 1 + ckj R 1 OP QP O (h ) P PQ . Similar to Rumelhart...(17.16) U Dw kj(t) = h U ¶wkj ¶e ph + xDw Uj (t 1) k .(17.22) .(17.. We discuss how to learn the strong L-R type fuzzy weight wk j = (wa ..HYBRID FUZZY NEURAL NETWORKS 227 Qj = (q a.. qg)LR j j j Let ckj = g b wkj wkj b a wkj wkj . wv ) between the jth kj kj kj hidden unit and the kth output unit. and cj = b a b a qk qk qj qj Then b wkj = v a wkj + ckj + wkj 1 + ckj b b b b .23) ..(17..

If a pre-specified stopping condition (etc..e. Apn). Tp2... L = R + max (0. Back-propagation: Adjust the fuzzy weights and the fuzzy biases using the cost function cph. h2. m (m input-output pairs (xp. k..2 Simulation We consider a n-dimension classification problem.25) + 1) = g w kj(t) + Dw g kj We assume that n values of h (i. h1. the fuzzy weight is updated by the following rules: a a w kj(t + 1) = w a (t) + Dw kj kj g w kj(t . So we note the fuzzy input as Ap = (Ap1. hn 3. 1 |x|2) .. xpn) beling to Gp.(17. 4. Let (xp. . We define the error function: eph = max {(tp op)2/2|Op Î[yp]h} . It can be described by IF-THEN rules as follows: IF xp1 is Ap1 and. Repeat 3 for h = h1. THEN xp = (xp1... TpNo)whaere No dimensional fuzzy taget vector corresponding to the fuzy input vector xp... 2..228 FUZZY LOGIC AND NEURAL NETWORKS These relations explain how the error signals ¶e ph [ wk j ]a h and ¶e ph [ wk j ]a h for the h-levelset propagate to the 0-level of the strong L-R type fuzzy weight Wkj.. h2. and then.24) . and the target ouput Tp can be defined as follows: Tp = R1 S0 T if Ap Îclass 1 if Ap Îclass 2 .9.. . the learning algorithm of the fuzzy neural network can be defined as follows: 1. Ù. small etc.(17. hn) are used for the learning of the fuzzy neural network. . Initialize the fuzzy weights and the fuzzy biases. that is to say. It is easy to know that the error function become the classical error function e = å (t p =1 k p op )2 / 2 in BP algorithm when input vector Ap . . In this way.... .. we assume that Api is a symmetrical strong L-R type fuzzy number. 17. the total number of iterations) is not satisfied.27) We should train this network in order to make eph be minimum.. and Tp = (Tp1. for example: large.(17.. Tp) is the fuzzy input-output pairs.. 2.. for the convenience of computing. go to 2.26) according to the target output T and the real output O.(17.. and xpn is Apn.. Repeat the following procedure for p = 1.. Tp)). Forward calculation: Calculate the hlevel set of the fuzzy output vector Op corresponding to the fuzzy input vector xp.. Where p = 1.. We can solve the above problem by using the fuzzy neural network we discussed. . 2. .Api is lingistic term.

28) The result of the trained fuzzy neural network is shown in Fig.HYBRID FUZZY NEURAL NETWORKS 229 and yp are real numbers. 0. 17. We assume A1-A4 belong to class1...2..(17.8 A5 A6 A7 Class 2 A8 10 15 20 The result of fuzzy classification 17.10 ADAPTIVE NEURO-FUZZY INFERENCE SYSTEM (ANFIS) A fuzzy system can be considered to be a parameterized nonlinear map.. 0.29) l =1 i =1 Ai1 i where y¢is a place of output singleton if Mamdani reasoning is applied or a constant if Sugeno reasoning is applied.. 0.6. (17. the error function of the pth pair is: eph = å h max {(t p p o p ) 2 / 2| Op Î[ y p ]h} . . The membership function m A1 (xi) corresponds to the input x = (x1. 17. We train the fuzzy neural network with h-levelsets (h = 0...8). 20 18 16 0.4. and A5-A8 belong to class 2. The i and connective in the premise is carried out by a product and defuzzification by the center-of-gravity method. xn) of the rule l. F m ( x )I JK å GÕ H f (x) = F I å G Õ m ( x )J H K m yl n l =1 i =1 Ail i m n .5 14 A3 A4 8 6 Class 1 A2 A1 2 1 5 Fig. . we get a satisfied curve after 300 echoes.8. Using the proposed learning algorithm. called f. Lets write here explicitly the expression of f.

there exists a best approximating polynomial pnof degree £ n such that ||F pn||¥ £ ||F = p||¥ over all polynomials p of degree £ n..6: If FÎC[a. b].7: If F is a bounded function on [a. Wang: A course in fuzzy systems and control.. . xk) a set of points in [a. b].(17. then f can approximate F to any desired accuracy..31) Well-known theorems from Approximation theory for polynomials.. pn which minimizes å | F( x ) p( x )| i i i =1 k 2 .32) over all polynomials of degree £ n.5: Let F be a bounded function on [a.F.33) max |F(xi) p(xi)| .e. b].. Remark: The message of Theorem 2 is that polynomials are dense in the space of continuous functions in Hilbert space.g. The following theorems are found in R. Prentice Hall. Curtain and A..230 FUZZY LOGIC AND NEURAL NETWORKS This can be further written as f (x) = where wi = y¢ m å w b ( x) i i i =1 . which minimizes n 0£i£ k .. and E = (x1. then for any n ³ 0..(17.(17. We can also consider the simpler problem of approximating at finitely many points. and E = (x1. . 1997).(17.J.(17.... Theorem 17. xk) a set of points in [a..34) over all polynomials of degree £ n. . The same can also be said of trigonometric functions. nonlinear map on a compact set. Theorem 17. Then there exists a best approximating polynomial p k of degree £ n. Pritchard: Functional Analysis in Modern Applied Mathematics as Corollaries of Orthogonal Projection Theorem. b]. can be extended to fuzzy systems (e..30) bj(x) = åÕm l =1 i =1 i =1 m n Õm n Ail ( xi ) Ail ( xi ) If F is a continuous. b].. Theorem 17. i. F » fFS . Then k there exists the least squares polynomial of degree £ n...

38) Then f can be written as f = w1 f1 + w2 f2 x A1 x A2 B1 B2 P w2 N w2 x Fig.(17.1 ANFIS Structure Consider a Sugeno type of fuzzy system having the rule base 1. Evaluating the implication and the rule consequences gives f (x.39) y P w1 N w1 w1f1 S w2f2 y f y . 2.36) Or leaving the arguments out f= . then f2 = p2x + q2 y + r2 Let the membership functions of fuzzy sets Ai. be m Ai . i = 1.(17. Bi. y) = w1 ( x.. y) w1 ( x. In evaluating the rules. 1..9 ANFIS structure .. y) f2 ( x. If x is A1 and y is B1. Evaluating the rule premises results in wi = m Ai (x) m Bi (x). m Bi ..(17. 2 .(17..HYBRID FUZZY NEURAL NETWORKS 231 17. y) + w2 ( x. then f1 = p1x + q1 y + r1 2.37) This can be separated to phases by first defining wi = . y) f1 ( x.10.. 17.(17... If x is A2 and y is B2.. choose product for T-norm (logical and).. y) w1 f1 + w2 f2 w1 + w2 wi w1 + w2 . i = 1. y) + w2 ( x.35) 2.

5.232 FUZZY LOGIC AND NEURAL NETWORKS All computations can be presented in a diagram form. 3. 7. Define the following: Hybrid neural network Regular fuzzy neural network Hybrid fuzzy neural network 4. 2.M. H. What are the different types of hybrid systems? Explain them schematically. 1994. Macmillan. 3. Grossberg. What is committee of networks? Explain. Lee and B. Liu. QUESTION BANK. Proceedings of International conference on artificial neural networks. REFERENCES.A. Explain the use of fuzzy logic in neural networks to improve the learning ability. 21. S. Vol. Kosko. Understanding Neural Networks and Fuzzy Logic. NY. IEEE Computer Magazine. B. 2. S. 1. Kartalopoulos. 5. IEEE press. Prentice Hall. 1991. IEEE computer society press. Carpenter and S. 77-88. Fuzzy BP: a neural network model with fuzzy inference. Describe adaptive neuro-fuzzy inference system. 3. pp. Neural Network and Fuzzy Systems. Neural Networks: A Comprehensive Foundation. The art of adaptive pattern recognition by a self organizing neural network. pp. NJ. 1583-1588. Explain the role of neural networks as pre-processor or post-processor of data. Haykin. NY. 1. Describe FNN architecture based on back propagation. . Englewood.H. 6. No. G. 4.

flexible manufacturing systems (FMS) have been investigated as a tool for raising manufacturing productivity and product quality while decreasing production costs. One type of FMS is the Unmanned Flexible Manufacturing System (UFMS). In particular.2 TOOL BREAKAGE MONITORING SYSTEM FOR END MILLING In recent years. they cannot halt the process if the tool becomes damaged. a detecting technology for unexpected tool breakage is needed (Lan and Naerheim. and Tlusty. Materials costs increase and product quality suffers if a broken tool is used in production.1 INTRODUCTION 18 The hybrid fuzzy neural networks have a tremendous potential to solve engineering problems. the UFMS reduces direct labour costs and prevents personal oversights. To apply the UFMS effectively. In order to ensure efficiency within the system. Since CNC machines cannot detect tool conditions. 18. 1986). electronic sensors associated with a decision-making system must monitor the process.4 Hybrid Fuzzy Neural Networks Applications 18. computer numerical control (CNC) machines are not typically capable of tool breakage detection. The decision-making system analyzes information provided by sensors to make appropriate control actions.+ 0 ) 2 6 . manufacturers must confirm that the tool is in good condition in process. Hybridization is performed for the purpose of investing better methods of problem solving. To reduce costs of materials and prevent damaged tools from negatively affecting production. which has received a great deal of attention because it replaces human operators with robotic counterparts in manufacturing and assembly cells. In addition to performing the same function as the FMS. automatic and rapid detection of tool breakage is central to successful UFMS operation. However. global competition in industry has led to exploration of new means of more efficient production. since human operators are absent in these systems. . Yellowley. By themselves. 1988). It is improper to expect that if the individual technologies are good then hybridization of technologies should turn out to be even better. therefore. monitoring equipment and algorithms for the adaptation of the manufacturing process must be executed accurately (Altintas.

18. Han. meaning that each cutting tooth moving in the same direction generates a cyclic cutting force ranging from zero to maximum force. Chen and Black (1997) also introduced a fuzzy-nets system to distinguish tool conditions in milling operations.. Tarng and Lee (1993) proposed using the average and median force of each tooth in the milling operation. The time-series-based tooth period model technique (TPMT). FR. The machining parameters and average peak force were used to build the AR model and neural network. An appropriate threshold was built to analyze information and detect tool conditions. analyzing force signals and determining amplitude fluctuations allowed on-line tool breakage detection. In this study. feed rate. Jemielniak (1992) proposed that sudden changes in the average level of force signals could be due to catastrophic tool failure (CTF) in turning operations. The variation of dynamic cutting force was used to construct the fuzzy dispersion pattern needed to distinguish tool conditions. The principle of cutting force can be further defined as resultant force. Measured by sensors. The fuzzy-nets system was designed to build the rule-bank and solve conflicting rules with a computer. The common method of detecting tool breakage in process involves force signals resultant from tool processes on raw materials. Zhang. Fri.1) Fri is the resultant force of point I . and back to zero. Tae and Dong (1992) developed a fuzzy pattern recognition technique and a time-series AR model to detect tool wear in turning operations. generated from x and y directions. which used the fast a posterior error sequential technique (FAEST). and depth of cut selected as input factors. to measure force in milling operations. Lan and Naerheim (1986) proposed a time series auto regression (AR) model of force signals to detect tool breakage. and Chen (1995) used a telemeter technique.1 Methodology: Force Signals in the End Milling Cutting Process Milling is a fundamental machining process in the operation of CNC machines. Ko. was used in this experiment expressed as: Fri = Where Fx2i + Fy2 i . This cyclic force is graphed as a series of peaks. was applied by Tansel and McLaughlin (1993) to detect tool breakage in milling. and Jung (1994) introduced an unsupervised self-organized neural network combined with an adaptive time-series AR modeling algorithm to monitor tool breakage in milling operations. Variance of adjacent peak force was selected as an input parameter to train the system and build a rule-bank for detecting tool breakage.2. generated in x and y directions. Cho.234 FUZZY LOGIC AND NEURAL NETWORKS An in-process tool breakage detection system was developed in an end milling operation with cutting force and machining parameters of spindle speed.(18. the average and median forces of each tooth were used as input information. a battery-powered sensing force/torque cutter holder mounted on the spindle head with the transmitter. A dynamometer sensor is the main device used to measure force signals in different machining operations. The neural networks approach was employed as the decision-making system that judges tool conditions.. The application of neural networks and fuzzy logic in detecting tool breakage has also been studied in recent years. and milling operations can be of two varieties: peripheral and face milling. Milling is an interrupted cutting process. The resultant force.

Tool conditions and machining parameters affect the magnitude of resultant force. if a tooth is broken.. the prediction function was achieved via the weight information. Figure 18. as shown in Figure.3) åa W jk where. . was used to combine or aggregate inputs passing through the connections from other neurons.2) . Back propagation is intended for training layered (i. The cutting force and machining parameters. also called a summation or aggregated function. Comparatively. and depth of cut. 18. it generates a smaller peak force because it carries a smaller chip load. Finally.1. usually 1.2 Neural Networks In this work.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 235 Fxi is the force in X direction of point I Fyi is the force in Y direction of point i. and maximum peak force of a broken tool must be larger than that of a good tool. and from hidden to output neuron respectively. This approach involves supervised learning. the peak measurement of each tooths force should be roughly the same from tooth to tooth during one revolution of the cutting process.2. 18. feed rate. The propagation rule. 1 illustrates the diagram of undamaged and broken tools. maximum peak force in each revolution should differ between good and broken tools. j is a hidden neuron. and k is an output neuron. and Wji and Wji are the weight of bias. a back propagation neural network (BPNN) was chosen as the decision-making system because it is the most representative and commonly used algorithm. two main differences can be used to detect tool breakage: 1. Maximum variance force of adjacent peaks should differ between good and broken tools. therefore. the arcs joining nodes are unidirectional. It is relatively easy to apply.(18.. the tooth that follows a broken tooth generates a higher peak force as it extracts the chip that the broken tool could not. Applying the force principle.. the neural networks approach was used as a decision-making system using input from sensors to judge tool conditions. while a0 represents the bias. which requires a teacher that knows the correct output for any input. if the tool condition is good. nodes are grouped in layers). such as spindle speed. In this work. were selected as input factors. As the weights of the neural network were obtained. Wji and Wkj denote weight from input to hidden neuron.e. and uses gradient descent on the error provided by the teacher to train the weights. feed forward (i.(18.. i is an input neuron.. an in-process tool breakage detection system was developed in an end milling operation. It can be expressed as Si = Sk = åa W i k ji a0Wjo a0Wjo .. and there are no cycles) nets. and maximum variance force of adjacent peaks of broken tools must be larger than in undamaged tools. and has also proven successful in practical applications. has been proven effective in dealing with this kind of task. As a result.e. 2.

expressed as: 1 Oy = .(18..08 inch b b Fig.4) 1 + e ay where ay is a function of Sj and Sk respectively. feed = 15 ipm depth = 0. . the process is repeated until the error percentage falls into a reasonable range. and one is called the Sigmoid Function..1 The amplitude of cutting force of a good and broken tool. Many different transfer functions can be used to transfer data. is used to produce output based on level of activation. 18. Comparing actual output of neural networks to desired output.236 FUZZY LOGIC AND NEURAL NETWORKS N 1200 Fa 1000 800 600 400 200 0 –200 One revolution b a Good Tool N 1600 1400 1200 1000 800 600 Broken Tool Fa a One revolution 400 200 0 – 200 Key: a = force signal b = revolution signal Cutting Parameters: speed = 650 rmp. also called the output or squashing function. The transfer function.

2.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 237 18. .2.3 Experimental Design and System Development Experimental Design This experiment employed a CNC vertical machining center. 0. 600. one side of the tool was in proper working order and the other side was broken. 0. 18. and 3. The experimental setup was shown in Fig. In each cutter. 550. The cutting parameters were set as: 1. 500. The broken side of the tool possessed varying degrees of breakage (Fig.08. 0. 18 and 24 inch per minute).07. 18.3). A dynamometer was mounted on the table to measure cutting force.09 and 0. Five level of feed rate (6.2. A proximity sensor was built near the spindle to confirm data in each revolution. Four ¾-inch doubt-end four-flute high-speed steel cutters were used. and 650 revolution per minute). 12.2 The experimental setup.06. 18. VM C40 Proximity Sensor Workpiece Dynamometer Amplifier A/D Board DC Power Supply Fig. Five level of spindle speed (740.1 inches). Five levels of depth of cut (0.

**238 FUZZY LOGIC AND NEURAL NETWORKS
**

T1 T2 T3 T4

1.5

1.5

3

2.5

1.5 Unit of value: mm.

2

2

3

Fig. 18.3 Diagram of broken tool.

The cutters used to execute the experiment were selected randomly. Cutting force was measured in voltage by the Charge Amplifier and transformed to Newtons (N) via computer.

18.2.4

Neural Network-BP System Development

To develop back propagation of neural networks as a decision-making system, MATLAB software was applied to analyze data. Seven steps were conducted. In step one, prediction factors were determined in order to perform the training process. Step 2 was necessary to analyze differences between scaling data and unscaling data. Step 3 dealt with separating data into training and testing categories. From steps 4 through 6, parameters were developed for the training process, including the hidden layer/hidden neuron, learning rate, and momentum factor. Finally, in step 7, information from the training process was used to predict tool conditions. Step 1. Determine the factors Five input neurons were used for tool breakage prediction data: 1. Spindle speed; 2. Feed rate; 3. Depth of cut; 4. Maximum peak force; and 5. Maximum variance of adjacent peak force. Output neurons were either (1) Good, or (2) Broken. Three hundred data points were used in this work. Good tools collected half of these and broken tools collected the rest, and all data were randomized using MS Excel software. Step 2. Analyze unscaling and scaling data In order to avoid experimental errors resulting from bigger values of some data sets, some preprocessing was needed to obtain good training and prediction results. Since histograms of all data sets

HYBRID FUZZY NEURAL NETWORKS APPLICATIONS

239

were uniform or normal distributions, the Simple Linear Mapping method was employed for scaling. To compare the difference between two sample sizes, some parameters were first set and fixed. The number of hidden neurons was set at 4, the learning rate was set at 1, and the momentum item was 0.5. The number of training cycles was 2000, and the testing period was 5. Table 18.1 shows the comparison of the difference between scaling and unscaling data. As one can see, errors in scaling data are smaller than in unscaling data.

Table 18.1 Difference between scaling and non-scaling

Hidden layer Unscale Scale 1 1 Hidden neuron 4 4 Learn rate 1 1 Momentum factor 0.5 0.5 Train Error 0.505 0.040 Testing Error 0.550 0.160 RMS error of training 0.508 0.197 RMS error of testing 0.515 0.388

Step 3. Impact of the ratio of training and testing data The 300 original 300 data records were randomized and separated into three groups. The first group had 200 training data and 100 (200 ´ 100) testing data, the second had 225 training and 75 (225 ´ 75) testing data, and the third had 250 training and 50 testing (250 ´ 50) data. Table 18.2 shows the Back Propagation Neural Network (BPNN) with different sample sizes of training and testing data. The last four columns of Table 18.2 show training and testing errors. The training, testing, and RMS (root mean square) errors of training of the second group were smaller than in other groups. The RMS errors of testing data of the second group sample were larger than in the first; however, the RMS errors of each sample size were very similar. If samples had similar error percentages, the sample with the largest training sample size was selected because it provided sufficient information to predict testing data. From the experimental design, the ideal ratio between training and testing data was 3:1 for neural networks. The 225 ´ 75 sample size was employed in this analysis.

Table 18.2:

Tra*Tes 200* 100 225* 75 250* 50 Hidden layer 1 1 1 Hidden neuron 4 4 4

**Different sample size of training and testing data
**

Learn rate 1 1 1 Momentum factor 0.5 0.5 0.5 Train Error 0.040 0.036 0.044 Testing Error 0.160 0.093 0.106 RMS error of training 0.197 0.185 0.204 RMS error of testing 0.388 0.298 0.285

Step 4. Impact of the hidden layer and hidden neuron In the beginning, the number of hidden neurons was set at 5, and the hidden layer was set at 1. Different hidden neurons and layers were tested to determine which values would lead to the smallest error percentage. To this end, the hidden neurons were set at 4 and 6, and the hidden layers were set at 1 and 2. Table 18.3 shows the BPN with a different number of hidden neurons and layers. According to this data, the percentage error of the trial with 4 hidden neurons and 1 hidden layer was less than it was in all

**240 FUZZY LOGIC AND NEURAL NETWORKS
**

other trials. Thus, the configuration contained in the 4 hidden neuron/1 hidden layer experiment was chosen because it led to the best results. The formula, (input neurons + output neurons)/2, was useful for determining the number of hidden neurons at the beginning.

Table 18.3: Different number of hidden neurons and layers

Neuron in layer-1 3 4 5 3 4 4 4 Neuron in layer-2 ~ ~ ~ 3 2 3 4 Learn rate 1 1 1 1 1 1 1 Momentum factor 0.5 0.5 0.5 0.5 0.5 0.5 0.5 Train Error 0.080 0.036 0.049 0.267 0.489 0.316 0.164 Testing Error 0.200 0.093 0.093 0.320 0.453 0.333 0.227 RMS error of training 0.256 0.185 0.193 0.338 0.512 0.407 0.362 RMS error of testing 0.410 0.298 0.288 0.375 0.504 0.425 0.414

Step 5. Impact of the learning rate This step was necessary to determine the optimal learning rate. The initial learning rate was 1. Three additional learning rates, 0.5, 2, and 10, were used to compare with the initial. Table 18.4 shows the BPN with different learning rate values. Table 18.4 shows that the error percentage of the learning rates of 0.5 and 1 were the same, in addition to being lower than all other learning rates. To achieve the objective of finding the smallest error percentage, the learning rate of 1 was used, because the software originally recommended that value.

Table 18.4

Hidden layer 1 1 1 1 Hidden neuron 4 4 4 4 Learn rate 0.5 1 2 10

**Different learning rate values
**

Train Error 0.036 0.036 0.116 0.111 Testing Error 0.093 0.093 0.133 0.133 RMS error of training 0.185 0.185 0.306 0.286 RMS error of testing 0.298 0.298 0.319 0.317

Momentum factor 0.5 0.5 0.5 0.5

Step 6. Impact of the momentum factor The final step of data analysis was to change the value of the momentum item to obtain the configuration leading to the lowest error percentage. The initial value of the momentum item was 0.5. Another three values, 0.3, 0.6, and 0.8, were selected to compare with the initial value. Table18.5 shows the BPN with different values for the momentum item. Table 18.5 shows that the percentage of errors of momentum items of 0.3 and 0.5 are the same, and smaller than all others. To achieve the smallest error percentage, the 0.5 momentum item was used, because the software originally recommended that value. Step 7. Prediction After completing analysis and obtaining information about weight and input factors, equations to predict tool conditions were constructed. The variables a1, a2, ., and a5 represent 5 input factors, maximum peak force, spindle speed, feed rate, depth of cut, and maximum variance of adjacent peak force, respectively. By application of equations (18.5), the weighted value of hidden factors ah1, ah2, ah3, ah4 can be expressed as:

**HYBRID FUZZY NEURAL NETWORKS APPLICATIONS
**

Table 18.5

Hidden layer 1 1 1 1 Hidden neuron 4 4 4 4 Learn rate 1 1 1 1

241

**Percent error of momentum items
**

Train Error 0.036 0.036 0.049 0.116 Testing Error 0.093 0.093 0.093 0.133 RMS error of training 0.185 0.185 0.212 0.306 RMS error of testing 0.298 0.298 0.288 0.322

Momentum factor 0.3 0.5 0.6 0.8

ah1 = ah2 = ah3 = ah4 = aout1 = aout2 =

1 + exp 1 + exp

[ a1 ´ ( 4.652 ) + a2 ´ ( 0 .448 ) + a3 ´ ( 0.947 ) + a4 ´ (25.237 ) + a5 ´ ( 0 .853 ) (0.221 )]

1

...(18.5) ...(18.6)

[ a1 ´ ( 40 .457 ) + a2 ´ ( 39 .421 ) + a3 ´ (15.261 ) + a4 ´ ( 7.317 ) + a5 ´ (21.054 ) ( 44 .505 )]

1

1 1 + exp

[ a1 ´ (10.224 ) + a2 ´ (3.444 ) + a3 ´ ( 24 .252 ) + a4 ´ ( 3.449 ) + a5 ´ ( 4 .215) (1.289 )]

...(18.7)

1 + exp 1 + exp 1 + exp

[ a1 ´ (1.321 ) + a2 ´ (24.736 ) + a3 ´ ( 0.202 ) + a4 ´ ( 0.79 ) + a5 ´ ( 0.015 ) ( 0.829 )]

1

...(18.8) ...(18.9) ...(18.10)

[ ah1 ´ (11.697) + ah 2 ´ (16.977 ) + ah 3 ´ (12.295 ) + ah 4 ´ (11.807 ) ( 2.945 )]

1

[ ah1 ´ (11.697) + ah 2 ´ (16.977 ) + ah 3 ´ (12.295 ) + ah 4 ´ (11.807 ) (2.945 )]

1

Finally, the output information was used to judge the tool conditions: If aout1 > aout2, then the tool condition is used If aout1 < aout2, then the tool is broken.

18.2.5

Findings and Conclusions

To operate the UFMS successfully, in-process sensing techniques that relate to rapid-response decisionmaking systems were required. In this research, a neural networks model was developed to judge cutting force for accurate in-process tool breakage detection in milling operations. The neural networks were capable of detecting tool conditions accurately and in process. The accuracy of training data was 96.4%, and the accuracy of testing data was 90.7%. Partial results of training and testing data are shown in Tables 18.6 and Table 18.7.

**242 FUZZY LOGIC AND NEURAL NETWORKS
**

Table 18.6 Partial results of training data

Tool Conditions Good Broken Good Broken Good Broken Good Broken Good Broken a1 904.5 1220.54 634.64 780.14 674.36 847.06 1239.4 1677.92 1413.76 1861.72 Input factors a2 600 600 550 550 650 650 500 500 450 450 a3 12 12 10 10 15 15 18 18 15 15 a4 0.09 0.09 0.06 0.06 0.06 0.06 0.07 0.07 0.08 0.08 Output factors a5 45.8 954.04 274.64 368.64 248.64 537.54 225.2 1,159.64 300.56 1,250.92 aout1 1 0 0.98 0.03 1 .02 1 0 0 0.01 aout2 0 1 0.02 0.97 0 0.98 0 1 1 0.99 Good Broken Good Broken Good Broken Good Broken Broken Broken Prediction

**Table 18.7 Partial results of testing data
**

Tool Conditions Good Broken Good Broken Good Broken Good Broken Good Broken a1 711.94 1296.56 723.32 1215.96 1084.32 1542.92 1024.46 1253.28 1507.18 1876.74 Input factors a2 a3 550 550 550 550 550 550 600 600 450 450 15 15 12 12 18 18 18 18 20 20 a4 0.06 0.06 0.07 0.07 0.07 0.07 0.07 0.07 0.08 0.08 Output factors a5 aout1 177.02 481.66 311.52 1,042.50 192.5 1,303.92 172.22 580.3 550.06 1,062.02 1 0.01 0.35 0 0.99 0 0.98 0.03 0.98 0 Prediction aout2 0 0.99 0.65 1 0.01 1 0.02 0.97 0.02 1 Good Broken Broken Good Broken Good Broken Broken Broken Broken

The weights of hidden factors and output factors were generated from pre-trained neural networks, and a program was written to process these weights in order to respond to the tool conditions. Therefore, the in-process detection system demonstrated a very short response-time to tool conditions. Since tool conditions could be monitored in a real-time situation, the broken tool could be replaced immediately to prevent damage to the machine and mis-machining of the product. However, since the weights were obtained from the pre-trained process, they were fixed when they were put into the detection program. Therefore, the whole system does not have the adaptive ability to feed back information into the system. In this work, depth of cut was employed as one input factor. However, in actual industrial environments, the surface of work materials is often uneven, implying that the depth of cut set in the computer might differ from that used to cut the workpiece. Under the circumstances, the neural networks might generate a wrong decision and misjudge the tool conditions due to fluctuating depths of cut across machining.

homogeneity. It causes non-steady. Note that a circle indicates a fixed node whereas a square indicates an adaptive node (the parameters are changed during training). I is the degree of the membership of the input to the fuzzy membership function (MF) represented by the node: O1. i = mAi(x)i = 1. it predicts the flue gas properties. and oil). let us consider two-fuzzy rules based on a first order Sugeno model: Rule 1: Rule 2: if (x is A1) and (y is B1). In the following presentation OLi denotes the output of node i in a layer L. To present the ANFIS architecture. then (f1 = p1x + q1y + r1) if (x is A2) and (y is B2). then (f2 = p2x + q2y + r2) One possible ANFIS architecture to implement these two rules is shown in Fig. including the O2 content. if bell MF is used then.1 Adaptive Neuro-Fuzzy Inference System Fuzzy Logic Controllers (FLC) has played an important role in the design and enhancement of a vast number of applications. density. Adaptive Neuro-Fuzzy Inference Systems are fuzzy Sugeno models put in the framework of adaptive systems to facilitate learning and adaptation. 18.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 243 18. only their effects on the combustion. leading to increase in the emission level and variation of the generated heat flow.3 CONTROL OF COMBUSTION Beside the economical and environmental advantages. This topic presents an ANFIS system.3. Bio fuels and municipal wastes are very inhomogeneous. 2 O1.4. even if steady fuel feed volume is maintained. gas. which determines the amount of fuel fed to the combustion chamber. The properties (heat value. Those property variations are not predictable or directly measurable. 18. the type and the parameter of the fuzzy membership functions and rules are crucial for achieving the desired. 4 Ai and Bi can be any appropriate fuzzy sets in parameter form. moisture content. Layer 1: All the nodes in this layer are adaptive nodes. The combustion of those fuels or fuel-mixtures has different properties compared to the conventional fuels (coal. agitated combustion conditions. Such framework makes FLC more systematic and less relying on expert knowledge. there are several difficulties with burning bio fuels and municipal wastes. . For example. on the steam generation and on the power production can be observed through the O2 content of the flue gas. Combined with a stoichiometric model. The proper selection of the number. i = mBi 2(y)i = 3. mix ability) may vary in a large range.

.(18..(18. i = w i fi = w i ( pi x + qi y + ri ) i = 1. 18. i = w i = . i = mAi(x) mBi(y) i = 1.(18. The outputs of these nodes are given by: O2.. The output of each node in this layer is given by: O3. qi and ri are design parameters (consequent parameter since they deal with the then-part of the fuzzy rule) ... bi and ci are the parameters for the MF. Layer 3: Nodes in this layer are also fixed nodes.11) where ai... 2. The output of each node is simply the product of the normalized firing strength and a first order polynomial: O4. These are labelled M to indicate that they play the role of a simple multiplier. 2 .14) where pi . Y W1 M N W1 Y Y S M W2 Layer 2 N W2 Layer 3 Layer 4 Layer 5 mAi (x) = 1 LF x c I OP 1 + MG MNH a JK PQ 2 i i bi i = 1.12) wi i = 1. w1 + w2 ..244 FUZZY LOGIC AND NEURAL NETWORKS N A1 A A2 N B1 Y B2 Layer 1 Forwards Backwards Fig.(18. 2 The output of each node is this layer represents the firing strength of the rule.13) Layer 4: All the nodes in this layer are adaptive nodes. These are labelled N to indicate that these perform a normalization of the firing strength from previous layer.4 ANFIS. Layer 2: The nodes in this layer are fixed (not adaptive). 2 .

is the least square estimator: . In this ANFIS architecture. bi and ci) pertaining to the input MFs [13]..15) The ANFIS architecture is not unique. we can plug training data and obtain a matrix equation: AQ = y . These parameters are called premise parameters. slope and the center of the bell MFs.18) where Q contains the unknown parameters in S2.(18. 18.. and the solution for È.(18. For this observation. 2 . the output of the network becomes: f= w1 w2 f1 + f2 w1 + w2 w2 + w2 .17) . If these parameters are fixed. we can apply least square method to identify the consequent parameters. Layer 4 has also three modifiable parameters (pi..16) = w1 f1 + w2 f2 = w1 (p1x + q1y + r1) + w2 (p2x + q2y + r2) = w1 x p1 + w1 y q1 + w1 r1 + w x p2 + w2 y q2 + w 2 r2 2 b g b g b g b g b g b g This is a linear combination of the modifiable parameters.. Note here that ai. 15). 1). we can divide the parameter set S into two sets: S = S1 Å S2 S = set of total parameters S1 = set of premise (nonlinear) parameters S2 = set of consequent (linear) parameters Å = direct sum For the forward path (see Fig. there are two adaptive layers (12.. Some layers can be combined and still produce the same output. These parameters are called consequent parameters..(18. bi and ci describe the sigma.2 Learning Method of ANFIS The task of training algorithm for this architecture is tuning all the modifiable parameters to make the ANFIS output match the training data [14]. Layer 1 has three modifiable parameters (ai.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 245 Layer 5: This layer has only one node labelled S to indicate that is performs the function of a simple summer. This is a linear square problem. The output of this single node is given by: Oi. Now for a given set of values of S1. which is minimizes 2 A y È .. respectively.(18.3. 5 = f = å i wi fi åw f = åw i i i i i i = 1. qi and ri) pertaining to the first order polynomial...

The update of the parameters in the ith node in layer Lth layer can be written as $ QiL (k) = $ QiL (k 1) + h ¶ Q L ( k ) $ i ¶E ( k ) .246 FUZZY LOGIC AND NEURAL NETWORKS Q* = (AT A)1 AT y ..5). is to keep the O2 content around 3-5% [16].21) where h is the learning rate and the gradient vector $ ¶Z L . The efficiency depends on the completeness of burning and the waste heat taken away in the flue gas by the excess airflow.(18..4). it is a difficult task due to the inhomogeneous properties of the fuel. including the oxygen content. primary airflow Fp and secondary airflow Fs signals are calculated by the linearization model as a function of the reference of the combustion power such as: QHz = 0.. However.7207 .. from the efficiency point of view. Q)] i =1 N 2 .(18. is the back propagated error signal.. from the fuel screw QHz. The error signal form the oxygen content drives the PI controller of the flue screw signal. while combustion power is controlled by the primary airflow.(18. The structure of the PI controller is U(s) = Kpi + Kh 1 S i = 1. i being the nodes output and eL.20) in a recursive manner with respect È(S2). i. The premise parameters are updated by descent method [15]. calculates the combustion power (Pcomb) and flue gas components (Cf). excess air is required for ensuring complete burning. The combustion model. the error signals propagate backward. through minimizing the overall quadratic cost function J(Q) = 1 2 $ å [ y(k) y(k... The higher the burning rate and smaller the waste heat is the higher efficiency. signal primary airflow Fp. The O2 content of the flue gas is directly related to the amount of excess air. secondary airflow Fs.3.6) consists of two parallel PI controllers.. For the backward path (Fig. . (see Fig.3 Model of Combustion The role of the combustion process is to produce the required heat energy for steam generation at the possible highest combustion efficiency.(18.6) The oxygen and combustion power controller (Fig. The aim of the combustion control.22) 18..19) we can use also recursive least square estimator in case of on-line training.23) The reference signals for the fuel screw QHz. i $ $ ¶Q iL ¶Q iL $ ¶Z L. 18. 18. 2 . i ¶E = eL.2662Pcomb 9. In multi-fuel fired fluidised bed power plants (Fig.(18. utilising the ANFIS structure based on [20]. 18. 18..

Steam Boiler super heater drum Oxygen (4) Furnace waterwall Fuel feed Prim.(18. Selection corresponds to keeping the best members of the population to the next generation to preserve the individual with good performance (elite individuals) in fitness function. A general genetic algorithm contains. Crossover originates new members for the population.2662Pcomb 4.912 Fs = 0.0737Pcomb + 10..24) 18. usually.3. Fp = 0. These procedures are responsible for the global search minimization function without testing all the solutions. the next three procedures: selection. crossover and mutation.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS Steam Pressure Steam flow to network (2) Steam header Air fan (1) Furnace pressure (8) (9) Heater Induced draft fan X (3) Sec.5 Fluidized bed power plant. depending of the selected parents the growing of the fitness of the population is faster or . air (6) (5) I 247 Boiler drum Econ. 18.4 Optimization of the PI-Controllers Using Genetic Algorithms Standard genetic or genetic searching algorithms are used for numerical parameter optimization and are based on the principles of evolutionary genetics and the natural selection process [17]. air Prim..005 . by a process of mixing genetic information from both parents. air fan (7) Fig.

. and random[18]. an elitism of 2 individuals was used.248 FUZZY LOGIC AND NEURAL NETWORKS ANFIS fuel flow model and Stoichiometric combustion and fuel gas model Cf Linearlization model Pcomb QH2 FQ Fs Pcomb Oxygen and combustion power controllers Fig. The reference signal for the combustion power is taken from the measurement data.25) N where the k is a weighting factor. The individuals are randomly selected with equal opportunity to create the new population. by tournament. Among many other solutions. The performance of the controller based on the ANFIS model is compared to the performance of the real process. The mutation operator is a binary mask generated randomly according to a selected rate that is superposed to the existing binary codification of the population changing some of the bits [19]. lower.6 Control system of combustion process. the crossover of one site splicing is performed and all the members are subjected to mutation except the elite. always including the elite.. Mutation is a process by which a percentage of the genes are selected in a random fashion and changed. 18. In our case k = 2 to emphasize the importance of oxygen content which is directly related to the flue gas emissions.(18. In the implemented algorithm a small population of 20 individuals. Crossover is performed over half of the population. the parent selection can be done with the roulette method. The simulation shows that by applying the new controller structure together with the ANFIS model.. much smaller deviation in the oxygen content can be achieved while satisfying the same demand for combustion power. The fitness function is J= å 1 N $ ( ycomb ycomb ) N +k´ å (y 1 N O2 $ yO 2 ) .

5 6+ + Best Average Poorest 249 5.5 + + 3 2. 18.5 1 2 3 4 5 6 7 8 9 10 Generations + + + + + Fig.5 + 5 + + Fitness + 4. 18.5 + 4 + + + + + 3.8 PI combustion power controller optimization with GA.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 6.7 Fitness function by the generation of the GA Optimization of the Pl controller 120 110 100 90 80 70 60 50 40 0 100 200 300 400 500 Time [S] 600 700 800 900 1000 Setpoint Output Combustion power [MW] Fig. .

7 0 100 200 300 400 500 Time [S] 600 700 800 900 1000 Fig.95 3.10 Combustion power response: comparison of the achievement in real process and in the simulated control system.1 4. 18.15 4.9 3.85 3.18.05 Setpoint Output Oxygen content [%] 4 3.9 PI Oxygen content controller optimization.8 3. 120 110 100 Combustion power [MW] Combustion power measurement signal Setpoint Output 90 80 70 60 50 40 30 0 100 200 300 400 500 600 Time [S] 700 800 900 1000 Fig.2 4. .250 FUZZY LOGIC AND NEURAL NETWORKS Optimization of the Pl Controller 4.75 3.

A Fuzzy-Nets Tool-Breakage Detection System for End-Milling Operations.T. REFERENCES. Altintas. J. Lan. J.T. Chen. 2. Explain the application of hybrid fuzzy neural network for the tool breakage monitoring of end milling. 1996. and E. 1997. New Jersey: Prentice Hall. Sun.T.C. C. Y. Chen. Mizutani. 18. Jang. Vol. . 153-164. 1. Detection of Cutting Edge Breakage in Turning. 783-800. J. and J. A Fuzzy-Nets In-process (FNIP) System for Tool-Breakage Monitoring in End-Milling Operations. Tools manufacturing. 97-100. 5. 1988. Engineering for Industry. pp. Yellowley. M.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS Oxygen content measurement signal 9 8 7 Oxygen content [%] 251 Measurement Output Setpoint = 4% 6 5 4 3 2 1 0 0 100 200 300 400 500 600 700 800 900 1000 Time [S] Fig. Vol. 1997. Engineering for Industry. pp. Vol. and J. 2.S. The Detection of Tool Breakage in Milling Operations. 1986. 1992. 191-197. pp. 4. Explain the application of ANFIS to control the combustion process. 41. pp. 110. 1. Annals of the CIRP. Black. No. Vol. International Journal of Machining. Jemielniak. No. Tlusty. Naerheim. QUESTION BANK. I. 6.11 Oxygen content response: comparison of the achievement in real process and in the simulated control system. and Y.C. In-Process Detection of Tool Breakage in Milling. 271-277. 108. K. Neuro-Fuzzy and Soft Computing. 3. 12. 37. 6. 1. International Journal of Advanced Manufacturing Technology. J.

On Developing an adaptive neural-fuzzy control system. R. Tansel. Grindelwald. and D. pp. International Journal of Machining. Najim. 4. IASTED International Conference on Applied Modelling and Simulation. Vol.1997. Hímer.346. Neuro. Switzerland 2004. A Case Study of PID Contoller tunning by Genetic Algorithm Proccedings of IASTED International Conference on Modelling and Control. pp. and D. Nov. 259-269. Ikonen. Lee. F.A. Detection of Tool Breakage in Milling Operation-I. 17. On-line Detection of Tool Breakage Using Telemetering of Cutting Forces in Milling. May 1996. Vieria. E. 2004. 19-27. 36. Cho. 2001. Japan. Mizutani. 1990. Sun. and E.33. Water Gas Heater Nonlinear Physical Model: Oprimazation with Genetic Algorithms Proccedings of IASTED International Conference on Modelling Identification and Control.W. S. 143. pp. Kim. Y. McLanghlin. 1995.B. and C. Vol.T Jeon. Neuro-fuzzy and Soft Computation.H. I. A Sensor for the Detection of Tool Breakage in NC Milling. Wertz. pp. 16.fuzzy model of flue gas oxygen content. and H. and C. Vol. tools Manufacturing. Han. and K. USA. The Neural Network Approach. 4. Vol. Cambridge. MA. 2002. 259-272. T. and A. Kovacs.S. 1999. Proceeding of the 1999 IEEE.B. V. 1050-1055. Detection of Tool Breakage in Milling Operation-II. and A. 80-90. . International Journal of Machining. Fuzzy neural networks and application to the FBC process. 8. D. Z. 2. 19.N. 18. Sim. 14. 950957. Vol. Kim. Adaptation in Natural and Artificial System. J. 531-544. NJ. 35. J. No. No. 1993.Y. 13.Control Theory Applications. Proceedings of IASTED International Conference on Modeling Identification and Control. Proc. 545-588. Proc.252 FUZZY LOGIC AND NEURAL NETWORKS 7. 1. 33.Y. Hybrid model of oxygen content in flue gas. 10. On-line Monitoring of Tool Breakage in Face Milling: Using a Self-Organized Neural Network. July 1993. West. No. International Journal of Machining. Jung. 14. J.H. pp. J. International Conference on Control Applications. Tools Manufacturing. Journal of Manufacturing Systems. 1993. Kortela. C. Tools Manufacturing. 20. Holland. 12. K. Tarng. No. 1993.H. McLanghlin.L. Jang. USA. Tansel. Hawai. pp. Alturki.J. 341. 9. Switzerland. Y. MIT Press. Chen. Y. pp. pp. pp. Zhang.Yang and M. Abdennour. I. The Time Series Analysis Approach. Leppäkoski and J. D. Kovacs. Innsbruck. Vol. 1992. Grindelwald. and U.N. Neuro-fuzzy control of a steam boiler turbine unit. IEE Proc. and B. Journal of Materials Processing Technology. 15. IEEE/RSJ Conference on intelligent robots and systems Yokohama. Ko. Prentice Hall. Mota.S. K.W and M. 11.

142 Barto network 192 Bartos approach 192 Basic property 45 Bellman equations 196 Bias 127 Binary fuzzy relation 21 Biological neural network 121. 183.Index A A back propagation neural network 235 =-Cut 11 Activation function 124 Adaline 125. 121 B Back-propagation 3. 9 Characteristic of fuzzy systems 9 Classical modus ponens 44 Classical modus tollens 45 . 122 Boltzmann machines 165 Boolean logic 3 C Calssical modus ponens 45 Cart-Pole system 194 Cartesian product 24 Cells 122 Center of-gravity method 74 Center-of-area 87 Centroid defuzzification 90. 185 ART 2 211 Artificial network 122 Artificial neural 3 Artificial neural network 3. 129. 55 Asymmetric divergence 166 Auto-associator network 161 Auxiliary Hybrid Systems 218 Auxiliary hybrids 217 Average error 151 Averaging operators 58 Axon 3. 133 Adaptive Critic 194 Adaptive Critic Element 192 Adaptive linear element 133 Adaptive Resonance Theory 182 Andlike 62 Andness is 62 ANFIS 229 ANFIS Structure 231 Anti-Reflexivity 20 Anti-Symmetricity 20 Applications of fuzzy logic 95 Approximate reasoning 41 Arithmetic mean 59 Arithmetical mean 58 ART 1 169. 123 ASE-ACE combination 192 Associative learning 125 Associative Memory 163 Associative Search 193 Associative Search Element 192 Associativity 54. 139.

1 Fuzzy neuron 220 Fuzzy Number 12 Fuzzy Point 15 Fuzzy relations 19. 121 Dimensionality reduction 169 Direction set minimization method 148 Discrete membership function 10 Disjunction Rule 43 Dot product 170 Dubois and prade 55 Dynamic Programming 195 Dynamics 201 E Eigenvectors 181 Elman Network 159 Embedded Hybrid Systems 218 Embedded hybrids 217 Empty Fuzzy Set 15 End-effector positioning 201 End milling 233 End milling cutting process 234 Entailment Rule 43 Entropy 63 Equivalence 20 Error back-propagation 222 Error function 173 Euclidean distance 171 Evaluation network 194 Exclusive-or 135 Expressive power 151 Extremal conditions 58 F Feature extraction 169 Feed-forward network 139. 1.#" FUZZY LOGIC AND NEURAL NETWORKS Classical N-array relation 19 Clustering 169. 7 Fuzzy logic controller 82. 84 Fuzzy Logic Controllers 243 Fuzzy logic is 1. 103 Defuzzifier 68. 170 Committee 223 Committee of networks 223 Commutative 61 Commutativity 58 Compensatory 58 Competitive learning 170 Complement 17 Component extractor 181 Compositional rule 44 Conjugate gradient minimization 148 Conjunction rule 43 Contrast enhancement 187 Control of combustion 243 Control room temperature 100 Controller network 191 Convergence theorem 131 Convex fuzzy set 11 Cost function 172 Counter propagation 174 Critic 190 D Defuzzification 67. 86 Delta 131 Delta rule 134 Dendrites 3. 6. 21 Fuzzy rule-base system 71 Fuzzy set 9 Fuzzy singleton 63 Fuzzy systems 2 G Gaussian membership functions 89 Generalised delta rule 140 Generalized Modus Ponens 44 Geometric mean 59 Godel implication 32 Graded response 164 . 150 Feed-forward networks 125 Feedback control system 81 First-of-Maxima 87 Flexible manufacturing systems 233 FNN architecture 224 Follow-the-leader clustering algorithm 185 Forward Kinematics 200 Frank 55 Fuzzification 8 Fuzzy Approach 116 Fuzzy control 8 Fuzzy implication operator 32 Fuzzy implications 30 Fuzzy logic 1.

190 Least mean square 133. 21 Inverse Kinematics 200 J Jordan Network 158 K Kleene-Dienes implication 32 Kohonen network 177 Kullback information 166 L Larsen inference Mechanism 77 Larsen system 66 Law of the excluded middle 2 Laws of Thought 2 Learning 127 Learning Rate 144 Learning Samples 152. 122. 34 Linguistic variable truth 35 LMS 131 LMS rule 131 Local Minima 148 Long-term memory 183 M Mamdani inference Mechanism 73 Mamdani system 66 Mamdanis implication operator 32. 164 Non-fuzzy approach 112 Normal fuzzy set 11 Normalization 186 Number of layers 127 O Offset 127 One identy 54 Ordered weighted averaging 60 Original model 185 Orlike operator 62 Orness 62 Output units 123 P Paradigms of learning 125 Partial order 20 Perceptron 2. 121 Hybrid fuzzy neural network 221 Hybrid neural network 220 Hybrid systems 217 I Idempotency 58. 63 Material implication 29 Mathematical neuron network 122 Max-Criterion 88 Maximum 56 Measure of dispersion 63 Median 59 Membership function 10 Middle-of-Maxima 87 Milling 210 Minimum 54 Modifiers 33 Momentum 144 Monotonicity 54.INDEX Grading of apples 104 Gravity 87 H Hamacher 55. 55. 129 Perceptron learning rule 131 ## . 153 Hopfield network 161 Human brain 3. 56 Harmonic mean 59 Hebbian learning 126 Hebbian Rule 180 Height defuzzification 88 Hidden Units 123. 58. 125. 61 Multi-layer network 140 Multi-layer perceptrons 137 Multi-input-multi-output 82 N Negation rule 44 Network paralysis 148 Neural networks 2. 121 Neuro-fuzzy systems 8 Neuro-fuzzy-genetic systems 8 Neurofuzzy network 222 Neurons 3. 61 Inference mechanisms 72 Input units 123 Interpolation 42 Intersection 16. 135 Linear discriminant function 130 Linear threshold neural network 130 Linguistic variable 33.

22 Universal approximation theorem 142 Universal approximators 91 Universal fuzzy set 15 Unmanned flexible manufacturing system 233 Unsupervised learning 125 . 157 Reflexivity 20 Regular fuzzy neural network 220 Reinforcement learning 192 Reinforcement learning scheme 190 Representation 127 Road accidents 96 Robot arm dynamics 207 Robot control 200 S Self-organization 125 Self-organizing networks 169 Semi-linear 124 Sequential hybrids 217 Sgn function 124 Shadow of fuzzy relation 24 Short-term memory 183 Sigmod 124 Significance 7 Simplified fuzzy Reasoning 77 Single layer feed-forward network 129 Single layer network 129.#$ FUZZY LOGIC AND NEURAL NETWORKS Perceptron learning rule 131 Perceptrons 2 Precisiated natural language 8 Precision 7 Principal component analysis 221 Principle of incompatibility 101 Principle of optimality 195 Probabilistic 56 Processing Units 123 Product 55 Product fuzzy conjunction 89 Product fuzzy implication 89 Projection 23 Projection Rule 44 Q Q-learning 196 Quasi fuzzy number 12 R Recurrent networks 125. 56 t-conorm-based union 57 t-norm-based intersection 57 Taylor series 148 Test error rate 151 The linguistic variable truth 35 Threshold 127 Threshold (sgn) 130 Tool breakage 233 Total error 135 Total indeterminance 46 Total order 20 Traffic accidents and traffic safety 96 Trajectory generation 201 Transitivity 20 Translation rules 43 Trapezoidal fuzzy number 14 Triangular conorm 55 Triangular Fuzzy Number 13 Triangular norm 54 Tsukamoto inference mechanism 73 two layer feed-forward network 139 Two-input-single-output 82 U Union 16. 54 Symmetry 55 T T 54. 130. 134 Singleton fuzzifier 89 Soft computing 8 Standard Strict 32 Stochastic function 193 Strong 56 Subset 46 Subsethood 14 Sugeno Inference Mechanism 75 Summed squared error 135 Sup-Min Composition 26 Superset 47 Supervised learning 125 Support 11 Symmetricity 20.

INDEX V Vector quantisation 169. 174 W Weak 55 Winner Selection 170 Y Yager 55. 56 Z Zero identity 55 #% .

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue listening from where you left off, or restart the preview.

scribd