This action might not be possible to undo. Are you sure you want to continue?

Welcome to Scribd! Start your free trial and access books, documents and more.Find out more

This page intentionally left blank

This page intentionally left blank

To My Parents .

This page intentionally left blank .

Modeling and control of dynamic systems belong to the fields in which fuzzy set techniques have received considerable attention. The choice of describing engineering applications coincides with the Fuzzy Logic and Neural Network research interests of the readers. which provide little insight to help understand the underlying process. basic mathematics. aiming at . Some of the material in this book contains timely material and thus may heavily change throughout the ages. neural learning. The presentation reflects theoretical and practical issues in a balanced way. it is distinctly different from the fields of control systems or optimization where the terminology. and at the same time provide insight into the system that generated the data. the absence of any state-of-the-art (Indian Domain) textbook forced me to write this book. Thus. Recently. not only from the scientific community but also from industry. and the field is still developing rapidly. due to strongly non-linear behaviour. neural modeling. a great deal of research activity has focused on the development of methods to build or update fuzzy models from numerical data. education and research. especially with regard to their approximation accuracy. Attention is paid to the selection of appropriate model structures in terms of the dynamic properties. Neurofuzzy models can be regarded as black-box models. The field of neural networks has a history of some five decades but has found solid application only in the past fifteen years. This combination of fuzzy systems and neural networks enables a more effective use of optimization techniques for building fuzzy systems. Most approaches are based on neuro-fuzzy systems. The approach adopted in this book aims at the development of transparent rule-based fuzzy models which can accurately predict the quantities of interest. or due to the time varying characteristics. Back in 2000. which exploit the functional similarity between fuzzy reasoning systems and neural networks. Neural networks are useful for industry. which can facilitate the effective development of models. The orientation of the book is towards methodologies that in the authors experience proved to be practically useful. Many systems are not amenable to conventional modeling approaches due to the lack of precise. and neural memory. as well as the internal structure of the fuzzy rules. Fuzzy modeling along with other related techniques such as neural networks have been recognized as powerful tools. due to the high degree of uncertainty.Preface This book attempts to provide the reader with basic concepts and engineering applications of Fuzzy Logic and Neural Networks. and design procedures have been firmly established and applied for many years. This book is intended to cover widely primarily the topics on neural computing. formal knowledge about the system.

Chennakesava R. Examples are given throughout the text and six selected real-world applications are presented in detail. Alavala .viii PREFACE readership from the academic world and also from industrial practice.

2 Support 2.2 Neural Networks (NN) 1.1 Fuzzy Set 2.6 Fuzzy Number 2.6.3 Normal Fuzzy Set 2.6.12 Empty Fuzzy Set Fuzzy Logic 6-18 6 6 8 9 9 9 9 11 11 11 11 12 12 13 14 14 15 15 .6.2 What is Fuzzy Logic? 2.3 Historical Background 2.5 Convex Fuzzy Set 2.7 Quasi Fuzzy Number 2.6.8 Triangular Fuzzy Number 2.4 Applications Question Bank References LE 1-5 1 2 4 4 4 5 Part I: Chapter 2: Fuzzy Sets and Fuzzy Logic 2.1 Introduction 2.4 Characteristics of Fuzzy Logic 2.11 Equality of Fuzzy Sets 2.6.6.3 Similarities and Dissimilarities Between FL and NN 1.6.Contents 2HAB=?A Chapter 1: Introduction 1.6.6.6.10 Subsethood 2.5 Characteristics of Fuzzy Systems 2.6 Fuzzy Sets 2.1 Fuzzy Logic (FL) 1.6.6.9 Trapezoidal Fuzzy Number 2.4 =-Cut 2.

1 Introduction 3.6 Sup-Min Composition of Fuzzy Relations Question Bank References Chapter 4: Fuzzy Implications 4.2.3 Operations on Fuzzy Relations 3.13 Universal Fuzzy Set 2.6.2 Fuzzy Implications 4.2.3.2.2.4 Cartesian Product of Two Fuzzy Sets 3.1 Classical N-Array Relation 3.x CONTENTS 2.3.3.4 Symmetricity 3.3 Complement Question Bank References 15 15 16 16 16 17 17 18 Chapter 3: Fuzzy Relations 3.14 Fuzzy Point 2.1 Introduction 4.3.2 Fuzzy Relations 3.5 Anti-Symmetricity 3.6.3.9 Total Order 3.3 Projection 3.2.2 Union 3.3 Anti-Reflexivity 3.2.2 The Linguistic Variable Truth Question Bank References 19-28 19 19 19 20 20 20 20 20 20 20 20 21 21 21 22 23 24 24 26 27 27 29-40 29 30 33 34 35 38 39 .2.2.3.6 Transitivity 3.7.2 Union 2.5 Shadow of Fuzzy Relation 3.3.3 Modifiers 4.7.3.1 Intersection 2.1 Linguistic Variables 4.2 Reflexivity 3.7.2.8 Partial Order 3.2.7 Equivalence 3.10 Binary Fuzzy Relation 3.1 Intersection 3.7 Operations on Fuzzy Sets 2.

6 Averaging Operators 6.3.3 Rational Properties 5.1 An Averaging Operator is a Function 6.3 Disjunction Rule 5.3.8 Mamdani System 6.3.6 Compositional Rule of Inference 5.6.2.2 Fuzzy Rule-base System 7.4 Superset Question Bank.2 Translation Rules 5.4 t-norm-based Intersection 6.2. xi 41-53 41 43 43 43 43 44 44 44 45 45 46 46 47 50 51 Chapter 6: Fuzzy Rule-Based Systems 6.1 Introduction 5.3 Triangular Conorm 6.3 Sugeno Inference Mechanism 7.1 Mamdani inference Mechanism 7.3.CONTENTS Chapter 5: The Theory of Approximate Reasoning 5.3 Inference Mechanisms In Fuzzy Rule-base Systems 7.6.2 Ordered Weighted Averaging 6.2 Total Indeterminance 5.1 Introduction 7.4 Larsen inference Mechanism 54-70 54 54 55 57 57 58 58 60 63 66 66 67 68 68 71-80 71 71 72 73 73 75 77 . References.1 Introduction 6.3.7 Measure of Dispersion or Entropy of an Owa Vector 6.4 Projection Rule 5.2 Conjunction Rule 5.3.2.3.5 Negation Rule 5.5 t-conorm-Based Union 6.1 Entailment Rule 5.2 Triangular Norm 6.9 Larsen System 6.2.2.3.2.3 Subset 5.2 Tsukamoto inference Mechanism 7.1 Basic Property 5.10 Defuzzification Question Bank References Chapter 7: Fuzzy Reasoning Schemes 7.

1 Why use Fuzzy Logic? 9.5.4 Membership Functions 9.2 Fuzzification 9.4.5.1 Apple Defects Used in the Study 81-93 81 81 82 82 82 84 86 87 87 87 88 88 89 91 91 94-120 94 95 96 96 96 97 97 98 99 99 99 100 100 101 102 103 104 104 105 .3.4.2 Basic Feedback Control System 8.5 Fuzzy Logic Model to Control Room Temperature 9.4.5.4.4.4 Defuzzification Methods 8.4.6.1 Introduction 8.2 Fuzzy Logic Approach 9.5 Rule Base 9.3 Fuzzy Logic Control Systems 8.2 Mamdani Type of Fuzzy Logic Control 8.1 The Mechanics of Fuzzy Logic 9.2 First-of-Maxima 8.6 Fuzzy Logic Model for Grading of Apples 9.1 Center-of-Area/Gravity 8.5.4.4.5 Height defuzzification 8.6 Output 9.7 Conclusions 9.4.5.4 Max-Criterion 8.3.4 Fuzzy Logic Model for Prevention of Road Accidents 9.3 Application 9.2 Applications of Fuzzy Logic 9.4.4.3.xii CONTENTS 7.1 Two-Input-Single-Output (TISO) Fuzzy Systems 8.3 Rule Application 9.1 Traffic Accidents and Traffic Safety 9.5 Simplified Fuzzy Reasoning Question Bank References 77 79 79 Chapter 8: Fuzzy Logic Controllers 8.3.3 Middle-of-Maxima 8.4.3 Fuzzy Logic Controller 8.5 Effectivity Of Fuzzy Logic Control Systems Question Bank References Chapter 9: Fuzzy Logic-Applications 9.4 Defuzzification 9.3 When Not to use Fuzzy Logic? 9.5 Conclusions 9.

3 A Framework for Distributed Representation 10.6.5.4 Adaptive Linear Element (Adalime) 121-128 121 121 122 123 123 124 125 125 125 126 126 126 127 128 128 129-138 129 129 131 131 131 133 .2 The Fuzzy Approach 9.6.1 The Non-Fuzzy Approach 9.3.1 Introduction 10.3.6 Defuzzification 9.2 Terminology Question Bank References Chapter 11: Perceptron and Adaline 11.6.1 Notation 10.6.6.6.2 Convergence Theorem 11.5 Determination of Membership Functions 9.2 Networks with Threshold Activation Functions 11.7.1 Perceptron Learning Rule 11.7 An Introductory Example: Fuzzy v/s Non-fuzzy 9.3.7 Results and Discussion 9.7.6.3.5 Training of Artificial Neural Networks 10.8 Conclusion 9.6.1 Introduction 11.2 Connections between Units 10.3.2 Biological Neural Network 10.4 Network Topologies 10.2 Materials and Methods 9.7.CONTENTS 9.4 Fuzzy Rules 9.2 Modifying Patterns of Connectivity 10.1 Paradigms of Learning 10.6 Notation and Terminology 10.3 Some Observations Question Bank References xiii 105 106 108 109 110 111 112 112 112 116 117 118 118 Part II: Neural Networks Chapter 10: Neural Networks Fundamentals 10.5.1 Processing Units 10.3 Activation and Output Rules 10.3 Application of Fuzzy Logic 9.6.3 Perceptron Learning Rule and Convergence Theorem 11.

1 Understanding Back-Propagation 12.1 Description 13.8.6.1 The Jordan Network 13.2 Learning Rate and Momentum 12.5 The Delta Rule 11.8.7 Multi-layer Perceptrons Can do Everything Question Bank References Chapter 12: Back-Propagation 12.4 Hopfield networks for optimization problems 13.6.Rule In Recurrent Networks 13.1 The Effect of the Number of Learning Samples 12.4.3.8 How Good are Multi-layer Feed-forward Networks? 12.2.2 The Effect of the Number of Hidden Units 12.3.3 The Hopfield Network 13.2.Forward Networks 12.3.2 Multi .6 Exclusive-or Problem 11.2 The Generalised Delta .9 Applications Question Bank References 134 135 137 138 138 139-156 139 139 140 142 143 143 144 144 146 146 148 148 148 151 152 153 153 155 155 Chapter 13: Recurrent Networks 13.1 Introduction 13.4.7 Advanced Algorithms 12.4.2 The Elman Network 13.3 Back-Propagation in Fully Recurrent Networks 13.1 Network Paralysis 12.4 Boltzmann Machines Question Bank References 157-189 157 157 158 159 161 161 162 163 164 164 165 167 167 .1 Weight Adjustments with Sigmoid Activation Function 12.3 Neurons with graded response 13.2.2 Hopfield Network as Associative Memory 13.xiv CONTENTS 11.6 Deficiencies of Back-propagation 12.1 Introduction 12.4 Working with Back-propagation 12.2 Local Minima 12.3 The Generalised Delta Rule 12.3.3 Learning Per Pattern 12.3.5 Other Activation Functions 12.Layer Feed .

5 Reinforcement Learning Versus Optimal Control Question Bank References Chapter 16: Neural Networks Applications 16.6 Contrast enhancement Question Bank References Chapter 15: Reinforcement Learning 15.3 More eigenvectors 14.2.1 Introduction 16.2 ART1: The Simplified Neural Network Model 14.3 Operation 14.5 Normalization of the Original Model 14.4 Learning Vector Quantisation 14.4.2 Inverse Kinematics 16.2.5.4 Bartos Approach: The ASE-ACE Combination 15.4 Trajectory generation 190-198 190 190 191 192 193 194 194 195 197 197 199-215 199 200 200 200 201 201 .1 Background: Adaptive Resonance Theory 14.4.CONTENTS xv 169 169 170 170 174 174 176 177 179 180 181 181 182 182 183 184 185 186 187 188 188 Chapter 14: Self-Organising Networks 14.1 Associative Search 15.3 The Controller Network 15.4.3 Counter propagation 14.2.5.2.1 Introduction 14.4 Principal Component Networks 14.2 Robot Control 16.4.2.5.4 ART 1: The Original Model 14.5.2 Competitive Learning 14.1 Clustering 14.2.2 Principal Component Extractor 14.1 Normalized Hebbian Rule 14.2.2 Vector Quantisation 14.5 Adaptive Resonance Theory 14.2.5.3 Kohonen Network 14.5.3 The Cart-Pole System 15.4.2 Adaptive Critic 15.2 The Critic 15.1 Forward Kinematics 16.4.3 Dynamics 16.1 Introduction 15.

6 Camera-Robot Coordination in Function Approximation 16.2 Hybrid Systems 17.2.9 Fnn Architecture Based On Back Propagation 17.2.2.2.9.3 Embedded Hybrid Systems 17.1 ANFIS Structure Question Bank References Chapter 18: Hybrid Fuzzy Neural Networks Applications 18.1 Unsupervised Adaptive Resonance Theory (ART) Neural Networks 16.2.3 Experimental Design and System Development Experimental Design .2 Auxiliary Hybrid Systems 17.2.2.1 Introduction 17.7 Robot Arm Dynamics 16.2 Simulation 17.6 Neural Networks as Tuners of Fuzzy Logic Systems 17.1 Introduction 18.2.1 Strong L-R Representation of Fuzzy Numbers 17.1 Sequential Hybrid Systems 17.2 Results and Discussion Question Bank References 201 202 202 203 206 207 210 211 213 215 215 Part III: Hybrid Fuzzy Neural Networks 217-232 217 217 217 218 218 219 220 221 222 223 223 224 226 228 229 231 232 232 233-252 233 233 234 235 237 Chapter 17: Hybrid Fuzzy Neural Networks 17.xvi CONTENTS 16.10 Adaptive Neuro-fuzzy Inference System (ANFIS) 17.3.3 Fuzzy Logic in Learning Algorithms 17.2.5a Involvement of neural networks 16.2.5 Neural Networks as Pre-processors or Post-processors 17.8 Committee of Networks 17.4 Fuzzy Neurons 17.5 End-Effector Positioning 16.6b Approach 2: Topology conserving maps 16.2.2 Neural Networks 18.3.6a Approach-1: Feed-forward Networks 16.3 Detection of Tool Breakage in Milling Operations 16.1 Methodology: Force signals in the end milling cutting process 18.9.2.2 Tool Breakage Monitoring System for end Milling 18.7 Advantages and Drawbacks of Neurofuzzy Systems 17.10.

3.CONTENTS 18.3.4 Neural Network-BP System Development 18.4 Optimization of the PI-Controllers using Genetic Algorithms Question Bank References Index xvii 238 241 243 243 245 246 247 251 251 253 .2.3.5 Findings and Conclusions 18.3 Model of Combustion 18.3.3 Control of Combustion 18.2 Learning Method of ANFIS 18.1 Adaptive neuro-fuzzy inference system 18.2.

This page intentionally left blank .

neural networks have rooted in many application areas (expert systems. pattern recognition. and presented not as a control methodology.1 FUZZY LOGIC (FL) The concept of Fuzzy Logic was conceived by Lotfi A. follow more human-like reasoning paths than classical methods. Fuzzy logic is mainly associated to imprecision. but as a way of processing data by allowing partial set membership rather than crisp set membership or non-membership. numerical information input. imprecise input.4 Introduction Nowadays. utilize self-learning. These methods have in common that they are non-linear. If feedback controllers could be programmed to accept noisy. Basically.like the use of basis functions (fuzzy logic has membership functions and neural networks have activation functions) and the aim to estimate functions from sample data or heuristics. and neural networks to learning and curve fitting (also to classification).S. Unfortunately. Although these methodologies seem to be different. Zadeh. they have many common features . Professor Zadeh reasoned that people do not require precise. in order to apply a more human-like way of thinking in the programming of computers. .). Notions like rather tall or very fast can be formulated mathematically and processed by computers. a professor at the University of California at Berkley. high/low. This approach to set theory was not applied to control systems until the 70s due to insufficient small-computer capability prior to that time. approximate reasoning and computing with words. FL is a multivalued logic that allows intermediate values to be defined between conventional evaluations like true/false. and yet they are capable of highly adaptive control. system control. U.+ 0 ) 2 6 . yes/no. fuzzy logic. etc. etc. 1 1. have ability to deal with non-linearities. they would be much more effective and perhaps easier to implement. manufacturers have not been so quick to embrace this technology while the Europeans and Japanese have been aggressively building real products around it.

If the conventional techniques of system analysis cannot be successfully incorporated to the modeling or control problem. Fuzzy system performs better (uses less fuel. see Fig. The singlelayer networks. In the 1960s it was experimentally shown that perceptrons could solve many problems. Heraclitus proposed that things could be simultaneously True and not True.) there were strong and immediate objections: for example. Even when Parminedes proposed the first version of this law (around 400 B. missile guidance. + – Input Fuzzy Controller Control Plant to be controlled Output Fig. etc. Honda. These types of networks were called perceptrons.2 NEURAL NETWORKS (NN) The study of neural networks started by the publication of Mc Culloch and Pitts [1943]. wheel slip control. For example.2 FUZZY LOGIC AND NEURAL NETWORKS Fuzzy systems is an alternative to traditional notions of set membership and logic that has its origins in ancient Greek philosophy. for example washing machine controllers and autofocus cameras.1). Canon. with threshold activation functions. Samsung. Hitachi. complex aircraft engines and control surfaces. It has been made a rough guess that 90% of applications are in control (the main part deals with rather simple applications. drives smoother) when compared with a conventional PID controller. But it was Lukasiewicz who first proposed a systematic alternative to the bi-valued logic of Aristotle. Sharp. Japan. The result of this publication was that the neural networks lost their . Rockwell. Siemens. but many problems. states that every proposition must either be True or False. 1. The precision of mathematics owes its success in large part to the efforts of Aristotle and the philosophers who preceded him. the use of heuristic linguistic rules may be the most reasonable solution to the problem. Commercially most significant have been various household and entertainment electronics. Fuzzy control includes fans. Nissan. Omron. In their efforts to devise a concise theory of logic. the Law of the Excluded Middle. These limitations of one-layer perceptron were mathematically shown by Minsky and Papert in their book Perceptron [1969]. The most significant application area of FL has been in control field. It was Plato who laid the foundation for what would become fuzzy logic. the so-called Laws of Thought were posited. Humans and fuzzy systems can perform this nonlinear control task with relative ease by using practical and at the same time imprecise rules as If the trailer turns slightly left. there is no mathematical model for truck and trailer reversing problem. The most famous controller is the subway train controller in Sengai. One of these.C.1 Example of a control problem. industrial processes and so on. indicating that there was a third region (beyond True and False) where these opposites tumbled about. which did not seem to be more difficult could not be solved. 1. Fuji. were introduced by Rosenblatt [1959]. Companies that have fuzzy research are General Electric. in which the truck must be guided from an arbitrary initial position to a desired final position. automatic transmission. McDonnell Douglas. and later mathematics. then turn the wheel slightly left. helicopter control. Mitsubishi. 1.

by giving them a learning ability. Hinton. The simulating is based on the present knowledge of brain function. The operation of brain is believed to be based on simple basic elements called neurons. Neural networks offer nonlinearity. The significance of this new algorithm was that multiplayer networks could be trained by using it. The learning may be based on two mechanisms: the creation of new connections. This learning ability is achieved by presenting a training set of different examples to the network and using learning algorithm. 1. Each neuron has an activation level which. e. which are connected to each other with transmission lines called axons and receptive lines called dendrites (see Fig. Synapse Nucleus Axon dendrites 1 X1 X2 W0 W1 W2 S Summing threshold unit z Fig. input-output mapping. The result of this combination is then fed into a non-linear activation unit (activation function). which revived the study of neural networks. 1.2). which can in its simplest form be a threshold unit (See Fig. The difficulty is how to guarantee generalization and to determine when the network is sufficiently trained. In artificial neural networks the inputs of the neuron are combined in a linear way with different weights. it is not absolutely wrong to claim that artificial neural networks probably have no close relationship to operation of human brains. and Williams [1986]. and the modification of connections. So. Neural networks are often used to enhance and optimize fuzzy logic based systems.INTRODUCTION 3 interestingness for almost two decades. back-propagation algorithm was reported by Rumelhart. adaptivity and fault tolerance. ranges between some minimum and maximum value.. 1. The high connectivity of the network ensures that the influence of errors in a few terms will be minor. which changes the weights (or the parameters of activation functions) in such a way that the network will reproduce a correct output with the correct input values. NN makes an attempt to simulate human brain. . Nonlinearity is a desired property if the generator of input signal is inherently nonlinear.2). In the mid-1980s. and this knowledge is even at its best primitive. which ideally gives a high fault tolerance.2 Simple illustration of biological and artificial neuron (perceptron). in contrast to Boolean logic.g.

etc. The number of this kind of hybrid systems is growing. 3. In many cases. user-friendly application interfaces. What is the ancient philosophy of fuzzy logic? What are the various applications of fuzzy logic? What is the historical evolution of neural networks? What are the similarities and dissimilarities between fuzzy logic and neural networks? What are the various applications of neural networks? . fault diagnostics and information security. speech processing. 1. credit rating. Possible new application areas are programming languages. adaptive process control. 4. database management. Fuzzy logic may also be employed to improve the performance of optimization methods used with neural networks. Most neuro-fuzzy systems are fuzzy rule based systems in which techniques of neural networks are used for rule induction and calibration.4 FUZZY LOGIC AND NEURAL NETWORKS 1. good results have been achieved by combining both the methods. 5. 2.4 APPLICATIONS Applications can be found in signal processing. A very interesting combination is the neuro-fuzzy architecture. The main dissimilarity between fuzzy logic and neural network is that FL uses heuristic knowledge to form rules and tunes these rules using sample data.3 SIMILARITIES AND DISSIMILARITIES BETWEEN FL AND NN There are similarities between fuzzy logic and neural networks: estimate functions from sample data do not require mathematical model are dynamic systems can be expressed as a graph which is made up of nodes and edges convert numerical inputs to numerical outputs process inexact information inexactly have the same state space produce bounded signals a set of n neurons defines n-dimensional fuzzy sets learn some unknown probability function can act as associative memories can model any system provided the number of nodes is sufficient. robotics control. QUESTION BANK. automaticized programming. business forecasting. computer networks. whereas NN forms rules based entirely on data. in which the good properties of both methods are attempted to bring together. 1. quality assurance and industrial inspection. natural-language understanding. pattern recognition.

414. Lejewski.INTRODUCTION 5 REFERENCES. NY: 1967. Williams. 9. 1986. pp. 1969. Hinton and R. Europe Gets into Fuzzy Logic.S.S. 23-26. F. (November). Nov. 1959. McCulloch and W. Vol. 10. New York: Spartan Books. 104-107. Zadeh.. MacMillan. 6. L. 11. 338-353. S. 2. pp. MacMillan. 1994. Zadeh. Business Week. 5.417. pp. Encyclopedia of Philosophy. 1984. Making computers think like people. 8. Vol. Papert. W. 1943.A. 1. Perceptrons: An Introduction to Computational Geometry. Laws of thought. 1965. Vol. Information and Control.A. Nature. 5. Smith. 5. Why the Japanese are going in for this fuzzy logic. 39. 533-536. IEEE. Rumelhart. Machine Design. L. 323. pp. L. Vol. 115-133. Vol. A logical calculus of the ideas immanent in nervous activity. NY: 1967. Spectrum. The MTT Press. Minsky and S. Electronics Engineering Times. 4. 1990. Pitts. Zadeh. 20. Vol. Learning representations by backpropagating errors. Korner.E. 1991. IEEE Software. G.E. 8. pp. 12. M. C. 3. Rosenblatt. D. Soft computing and fuzzy logic. 48-56. Feb. . 7. T. Bulletin of Mathematical Biophysics. 11. pp. pp. Principles of Neurodynamics. 4. 26-32. Loses Focus on Fuzzy Logic. Jan Lukasiewicz. Fuzzy Sets. Vol.J. 8. pp.A. U. June 21. Encyclopedia of Philosophy. 1993.

Here is what some clever people have said in the past: Precision is not truth. Henri Matisse Sometimes the more measurable drives out the most important. Rene Dubos Vagueness is no more to be done away with in the world of logic than friction in mechanics.2 WHAT IS FUZZY LOGIC? Fuzzy logic is all about the relative importance of precision: How important is it to be exactly right when a rough answer will do? All books on fuzzy logic begin with a few good quotes on this very topic.4 Fuzzy Sets and Fuzzy Logic 2. 1 INTRODUCTION Fuzzy sets were introduced by L. Charles Sanders Peirce . such as thinking and reasoning. As a consequence.A Zadeh in 1965 to represent/manipulate data and information possessing nonstatistical uncertainties.+ 0 ) 2 6 . since such knowledge is by its nature both lexically imprecise and noncategorical. and this is no exception. The conventional approaches to knowledge representation lack the means for representing the meaning of fuzzy concepts. Fuzzy logic provides an inference morphology that enables approximate human reasoning capabilities to be applied to knowledge-based systems. the approaches based on first order logic and classical probability theory do not provide an appropriate conceptual framework for dealing with the representation of commonsense knowledge. It was specifically designed to mathematically represent uncertainty and vagueness and to provide formalized tools for dealing with the imprecision intrinsic to many problems. 2 2. The theory of fuzzy logic provides a mathematical strength to capture the uncertainties associated with human cognitive processes. The development of fuzzy logic was motivated in large measure by the need for a conceptual frame work which can address the issue of uncertainty and lexical imprecision.

What do I mean by mapping input space to output space? Here are a few examples: You tell me how good your service was at a restaurant. it seems almost surprising that no one attempted it sooner. but once you become acquainted with it. and hence I am opposed to every statement of positive truth and every man who makes it.something that humans have been managing for a very long time (Fig. You tell me how hot you want the water. You tell me how far away the subject of your photograph is.3 m/sec. Fuzzy logic sometimes appears exotic or intimidating to those unfamiliar with it. Fuzzy logic is a convenient way to map an input space to an output space. the concepts of fuzzy logic reach right down to our bones. 2.FUZZY SETS AND FUZZY LOGIC 7 I believe that nothing is unconditionally true. Dont be penny wise and pound foolish. This is the starting point for everything else. .1 Precision and significance. In this sense fuzzy logic is both old and new because. L. and Ill focus the lens for you. A Zadeh Some pearls of folk wisdom also echo these thoughts: Dont lose sight of the forest for the trees. You tell me how fast the car is going and how hard the motor is working. precise statements lose meaning and meaningful statements lose precision. LOOK OUT!! Precision Significance Fig. and Ill shift the gears for you. And so far as they are certain. they do not refer to reality. although the modern and methodical science of fuzzy logic is still young. 2.1). and Ill tell you what the tip should be. Fuzzy logic is a fascinating area of research because it does a good job of trading off between significance and precision . they are not certain. L. Albert Einstein As complexity rises. Mencken So far as the laws of mathematics refer to reality. H. Precision and significance in the real world A 1500 kg mass is approaching your head at 45. and the great emphasis here is on the word convenient. and Ill adjust the faucet valve to the right setting.

fuzzy arithmetic.. calculus of fuzzy if-then rules. design and utilization of intelligent systems. in a lattice. The basic tenet of soft computing is that. Where do we stand today? In viewing the evolution of fuzzy logic. set-theoretic in nature. that is. especially in Japan. knowledge representation. e. calculus of fuzzy quantifiers and related concepts and calculi. fuzzy mathematics. Perhaps the most striking development during the second phase of the evolution was the naissance and rapid growth of fuzzy control. in which fuzzy logic. and (b) a wide sense. probabilistic computing. 1973-1999. Soft computing came into existence in 1981. and the impact of soft computing is growing on both theoretical and applied levels. Basically.3 HISTORICAL BACKGROUND Almost forty years have passed since the publication of first paper on fuzzy sets. The first phase. and logic and reasoning were not at the center of the stage. abbreviated as FLn. with two-valued characteristic function generalized to a membership function taking values in the unit interval or. as they are at present. rough set theory and machine learning. The principal members of the coalition are: fuzzy logic. Fuzzy control applications proliferated but their dominance in the literature became less pronounced. are appearing.8 FUZZY LOGIC AND NEURAL NETWORKS 2. fuzzy arithmetic. fuzzy set theory. Basically. development of computing with words and perceptions brings together earlier strands of fuzzy logic and suggests that scientific theories should be based on fuzzy logic rather than on Aristotelian. evolutionary computing. in which fuzzy logic. the term fuzzy logic is used. or is allowed to be. cluster analysis. The distinguishing characteristic of FL is that in FL everything is. 1996 is the genesis of computing with words and the computational theory of perceptions. chaotic computing. from 1965 to 1973. The second phase. A combination which has attained wide visibility and importance is that of neuro-fuzzy systems. Other combinations. neurocomputing. Today. A key component of computing with words is the concept of Precisiated Natural Language (PNL). pattern recognition. Today. is a union of FLn. alongside the boom in fuzzy logic applications. An important development in the evolution of fuzzy logic. for the most part. fuzzy topology and. bivalent logic. better results can be obtained through the use of constituent methodologies of soft computing in combination rather than in a stand-alone mode. soft computing is a coalition of methodologies which collectively provide a foundation for conception. fuzzy mathematical programming. three principal phases may be discerned. a matter of degree. There were many other major developments in fuzzy-logic-related basic and applied theories.g. fuzzy logic is used in two different senses: (a) a narrow sense. decision analysis. neuro-fuzzy-genetic systems. in its wide sense. among them the genesis of possibility theory and possibilistic logic. abbreviated as FL. in general. PNL opens the door to a major enlargement of the role . Today. more generally. possibility theory. is a logical system which is a generalization of multivalued logic. The basic issues and applications which were addressed were. with the launching of BISC (Berkeley Initiative in Soft Computing) at UC Berkeley. more generally. with generalization of the concept of a set. and (b) the concept of a fuzzy if-then rule. marking the beginning of the third phase. two key concepts were introduced in this paper: (a) the concept of a linguistic variable. was concerned in the main with fuzzification. for the most part. almost all applications of fuzzy set theory and fuzzy logic involve the use of these concepts. The term fuzzy logic was used for the first time in 1974.

A fuzzy set A in X is characterized by its membership function (Fig. will be the Internet.(2. one of the most important application-areas of fuzzy logic. It may well turn out to be the case that. In fuzzy logic. Inference is viewed as a process of propagation of elastic constraints. exact reasoning is viewed as a limiting case of approximate reasoning. equivalently.5 CHARACTERISTICS OF FUZZY SYSTEMS There are two main characteristics of fuzzy systems that give them better performance for specific applications: Fuzzy systems are suitable for uncertain or approximate reasoning. But. 2. and µA (x) is interpreted as the degree of membership of element x in fuzzy set A for each x Î X.6. the word fuzzy is usually used in a pejorative sense. 1] . Any logical system can be fuzzified. It may take some time for this to happen.1 Fuzzy Set µA : X ® [0. From its inception. 2.6 FUZZY SETS 2.FUZZY SETS AND FUZZY LOGIC 9 of natural languages in scientific theories. everything is a matter of degree.2).. especially for the system with a mathematical model that is difficult to derive.2) . Fuzzy logic allows decision making with estimated values under incomplete or uncertain information. and especially PNL. 2.. fuzzy logic has been (and to some degree still is) an object of skepticism and controversy. skepticism about fuzzy logic is a reflection of the fact that. in coming years. knowledge is interpreted a collection of elastic or. in English. In part. for some fuzzy logic is hard to accept because by abandoning bivalence it breaks with centuries-old tradition of basing scientific theories on bivalent logic. In fuzzy logic. 2. centering on the conception and design of search engines and question-answering systems. fuzzy constraint on a collection of variables.4 CHARACTERISTICS OF FUZZY LOGIC Some of the essential characteristics of fuzzy logic relate to the following: In fuzzy logic.(2.. µA(u)) |u Î X} . more importantly.1) Let X be a nonempty set. but eventually abandonment of bivalence will be viewed as a logical development in the evolution of science and human thought.. It is clear that A is completely determined by the set of tuples A = {(u.

300000 cars are considered as cheap.. 300000 Rs. + µn/xn .1: defined as The membership function (Fig.. 1 –1 1 2 3 4 Fig.4). Example 2.3 A membership function for x is close to 1. 2..3) of the fuzzy set of real numbers "close to 1". 2.. 1 Rs.3) where the term µi / xi. . The family of all fuzzy sets in X is denoted by F(X). . i =1. and prices make no real difference to buyers eyes. Example 2. .2: Assume someone wants to buy a cheap car. .2 A discrete membership function for x is close to 1.4 Membership function of "cheap". 600000 Fig. For instance. Frequently we will write A(x) instead of µA(x). from the Figure cheap is roughly interpreted as follows: Below Rs. Cheap can be represented as a fuzzy set on a universe of prices..(2.. 450000 Rs. xn} is a finite set and A is a fuzzy set in X then we often use the notation A = µ1/x1 + . can be A(t) = exp ( b(t 1)2) where b is a positive real number.10 FUZZY LOGIC AND NEURAL NETWORKS 1 –2 –1 0 1 2 3 4 Fig.. 2. and depends on his purse (Fig.. 2.... If X = {x1.n signifies that µi is the grade of membership of xi in A and the plus sign represents the union. 2.

6 < a £ 1 2. 1.(2. 2} |{1} T if if if 0 £ a £ 0.6. 450000.4 =-Cut An a-level set of a fuzzy set A of X is a non-fuzzy set denoted by [A]a and is defined by [A]a = R{t e X | A(t ) ³ a} Scl(sup p A) T if if a>0 a=0 . Beyond Rs.5 Convex Fuzzy Set A fuzzy set A of X is called convex if [A]a is a convex subset of X"a Î [0. 0. 0.3: Assume X = { 2.3 Normal Fuzzy Set A fuzzy subset A of a classical set X is called normal if there exists an xÎX such that A(x) =1. or essentially bigger than 5000.6/2 + 0. Using the theory of fuzzy subsets we can represent these fuzzy numbers as fuzzy subsets of the set of real numbers. 300000 and Rs. More exactly. 1. supp(A) = {x ÎX | A(x) > 0}. Example 2. 1]. An a-cut of a triangular fuzzy number is shown in Fig. a variation in the price induces a weak preference in favor of the cheapest car. 1.6.6.. 2.5) 2.FUZZY SETS AND FUZZY LOGIC 11 Between Rs.6) where cl (supp A) denotes the closure of the support of A. about 5000.. 3. 2.6.2 Support Let A be a fuzzy subset of X. denoted supp (A). 2. 2.5. is the crisp subset of X whose elements all have nonzero membership grades in A. .3/ 1 + 0. For example.(2. Between Rs. 2. 450000and Rs. 1.. Otherwise A is subnormal.0/4.3 0. In this case [A] a R{-1. These are examples of what are called fuzzy numbers. the support of A. 4} and A = 0.6/0 + 1.3/3 + 0. In many situations people are only able to characterize numeric information imprecisely.0/1 + 0.0/ 2 + 0. 3} | = S{0. 600000 the costs are too high (out of consideration). people use terms such as.. near zero. a small variation in the price induces a clear preference in favor of the cheapest car.3 < a £ 0. . 600000.6 0.

8) In other words. 2..6) A is a fuzzy set of the real line with a normal. 1] ® Â is monotone increasing and lower semicontinuous.6. 2.10) . the left-hand side function a1 : [0. 2.(2.6.. 1] ® Â is monotone decreasing and upper semicontinuous. Then [A]g is a closed convex (compact) subset of Â for all g Î [0.(2.. 2.11) .1].5 An a-cut of a triangular fuzzy number.cut.(2. a2(g) max [A]g .9) . . (fuzzy) convex and continuous membership function of bounded support. 1 –2 –1 1 2 3 Fig. a1 (g) denotes the left-hand side and a2 (g) denotes the right-hand side of the g .7) Let Abe a fuzzy number.(2.6 Fuzzy number. Let us introduce the notations a1(g) = min [A]g. 2. fuzzy convex and continuous membership function satisfying the limit conditions 1® ¥ lim A(t) = 0 .6 Fuzzy Number A fuzzy number (Fig... and the right-hand side function a2 : [0. The family of fuzzy numbers will be denoted by F .12 FUZZY LOGIC AND NEURAL NETWORKS a a – cut Fig....(2...7 Quasi Fuzzy Number A quasi-fuzzy number A is a fuzzy set of the real line with a normal. It is easy to see that if a £ b then [A]a É [A]b Furthermore.

7 The support of A is [a1(0). The not fuzzy number is shown in Fig. 2. left width a > 0 and right width b > 0 if its membership function has the following form A(t) = R1 . a + (1 g)b]. a2(g)] The support of A is the open interval [a1 (0).9) with center a may be seen as a fuzzy quantity x is approximately equal to a. 1] such that [A]g is not a convex subset of R..1] .6.t | a |1 . 1 –3 –2 –1 1 2 3 Fig.8.a ... 2. 2.(2.t S b |0 | | T if a . .13) and we use the notation A = (a. A triangular fuzzy number (Fig. a2(0)].FUZZY SETS AND FUZZY LOGIC 13 We shall use the notation [A]g = [a1(g).a .12) a1(0) a2(0) Fig. "g Î[0. b + b). A 1 g a1(g) a2(g) .a £ t £ a if a £ t £ a + b otherwise .14) The support of A is (a a. It can easily be verified that [A]g = [a (1 g)a.8 Triangular Fuzzy Number A fuzzy set A is called triangular fuzzy number with peak (or center) a.(2. b). If A is not a fuzzy number then there exists an gÎ[0. 2. a2 (0)] and it is illustrated in Fig. a.. 2..8 Not fuzzy number.. 2.7.(2.

t | a |1 | A(t) = S t . 1 a–a a b b+b Fig. 2. 2.6. 2. A trapezoidal fuzzy number (Fig. b + (1 g)b]. a.10) may be seen as a fuzzy quantity x is approximately in the interval [a.9 Triangular fuzzy number. b + b). . b].. b]. b.14 FUZZY LOGIC AND NEURAL NETWORKS 1 a–a a a+b Fig.15) A = (a.. R1 . 2.9 Trapezoidal Fuzzy Number A fuzzy set A is called trapezoidal fuzzy number with tolerance interval [a. 2. We say that A is a subset of B if A(t) £ B(t).11. "t Î X. The subsethood is illustrated in Fig. b)...10 Subsethood Let A and B are fuzzy subsets of a classical set X.10 Trapezoidal fuzzy number.16) The support of is (a a.(2. "g Î[0. It can easily be shown that [A]g = [a (1 g)a.a . left width and right width b if its membership function has the following form.b |1 b |0 | T and we use the notation if a a £ t £ a if a £ t £ b if a £ t £ b + b otherwise . 1] .6. 2.(2.

1 1x 10 x Fig. It is easy to see that A Ì 1X holds for any fuzzy subset A of X.11 Equality of Fuzzy Sets Let A and B are fuzzy subsets of a classical set X.6. It is easy to see that Ø Ì A holds for any fuzzy subset A of X. We note that A = B if and only if A(x) = B(x) for x Î X.12 The graph of the universal fuzzy subset in X = [0. . "t Î X. 2.6. denoted by 1X. 2.13 Universal Fuzzy Set The largest fuzzy set in X. A and B are said to be equal. 2. 1 X0 X0 Fig.6.13) and we use the notation A = x0.13 Fuzzy point.6. is defined by 1X(t) = 1. then A is called a fuzzy point (Fig. if A Ì B and B Ì A. 2. 2.12) in X. 2. 2. denoted A = B. 10].11 A is a subset of B. called universal fuzzy set (Fig. If supp (A) = {x0}.12 Empty Fuzzy Set The empty fuzzy subset of X is defined as the fuzzy subset Ø of X such that Ø(x) = 0 for each x Î X.14 Fuzzy Point Let A be a fuzzy number.FUZZY SETS AND FUZZY LOGIC 15 B A Fig. 2. 2.

A B .18) The intersection of A and B is defined as The intersection of A and B is shown in Fig. 1}.2 Union A B Fig. For this reason.17) 2. It is easy to see that [A]g = [x0.19) Fig. when extending operations to fuzzy sets we use the same symbol as in set theory.(2. 1] . 2. 2. Let A and B are fuzzy subsets of a nonempty (crisp) set X.16 FUZZY LOGIC AND NEURAL NETWORKS Let A = x0 be a fuzzy point. 2. We note that all those operations which are extensions of crisp concepts reduce to their usual meaning when the fuzzy subsets have membership degrees that are drawn from {0. .1 Intersection (A Ç B)(t) = min {A(t).7..15 Union of two triangular fuzzy numbers.14.15.14 Intersection of two triangular fuzzy numbers.. "g Î [0.(2.7 OPERATIONS ON FUZZY SETS We extend the classical set theoretic operations from ordinary set theory to fuzzy sets..7... 2.. B(t)} = A(t) Ù B(t) for all t Î X . The union of A and B is defined as (A È B) (t) = max {A(t). B(t)} = A(t) Ú B(t) for all t Î X The union of two triangular numbers is shown in Fig.(2. x0] = {x0}. 2. 2.

List all a-cuts.. A(t)} = max {1 1/2.. 9. Given A ={a.7/x3 + 0.5/x1 + 0. 3.(2. 5. Given X = {1. What are the characteristics of fuzzy logic? What are the characteristics of fuzzy systems? What are the different fuzzy sets? Define them.22) It is clear that Ø1X = f and Øf = 1X. 6.. b. Let A(t) =1/2. 4. c. then it is easy to see that (ØA Ú A)(t) = max {ØA(t). Lemma2.4/x2 + 0. 6. (iii) universal fuzzy set. however. Find ØA.. What are the roles of a-cut in fuzzy set theory? What are the different fuzzy numbers? Define them. 2. 3. c}. Let A be a fuzzy set defined by A = 0. 11. Find A È B. 3..20) . 10. Let A(t) =1/2. fuzzy logic does satisfy De Morgans laws Ø(A Ù B) = ØA Ú ØB.FUZZY SETS AND FUZZY LOGIC 17 2. What are the operations on fuzzy sets? Explain with examples. then it is easy to see that (ØA Ú A)(t) = mix {ØA(t).3 Complement The complement of a fuzzy set A is defined as (ØA)(t) = 1 A(t) A Ú ØA = X and the law of non-contradiction principle A Ù ØA = f . 4. 12. 2. 1. 4. A(t)} = mix {1 1/2.8/x4 + 1/x5. Define the following: (i) equality of fuzzy sets.(2. 1. and A Ç B. 8. 1/2} = 1/2 ¹ 0 However. 7. "t Î R. 5. b. Ø(A Ú B = ØA Ù ØB) . 6}.} and A = {2.7.(2. the laws of excluded middle and noncontradiction are not satisfied in fuzzy logic..2: The law of non-contradiction is not valid. .1: The law of excluded middle is not valid. 2} and B = {1. What is fuzzy logic? Explain the evolution phases of fuzzy logic. "t Î R. 2. (ii) empty fuzzy set. 1/2} = 1/2 ¹ 1 Lemma2.21) A closely related pair of properties which hold in ordinary set theory are the law of excluded middle QUESTION BANK.

L. 1971. L. Computational Intelligence: Soft Computing and Fuzzy-Neuro Integration with Applications. 1. 1981. International Journal of Systems Science. 3. Zadeh. 16. IEEE Computer. 613-626.). 1965. Vol.10-37.A. 1973. 1978. Vol. pp. 3. Vol. 2. Fuzzy Sets. A note on fuzzy sets.A. SMC-3. pp. pp. pp. 1988.A. L. 1978. Prade. 2.A. Vol. Part 1.A. Soft computing and fuzzy logic. 8. Information Sciences. Albert. 1994.N. pp. 373-386. 7. No. 125-151.A. Fuzzy Sets and Systems. pp. 199-249. pp. Vol. 1972. Possibility theory and soft data analysis. L. Foundations of fuzzy sets. No. J. G. pp. 8. Brown. pp. 4. neutral networks and soft computing. Set theory for fuzzy sets of higher level. 77-84. Information and Control.A. No. Vol. Rudas. Turksen. 1998. pp. 12. A fuzzy-set-theoretic interpretation of linguistic hedges. 21. 1. A generalized fuzzy set theory. 40. 6. design and deployment of information/intelligent systems. A. . Fuzzy Sets and Systems. Operations on Fuzzy Numbers. No. 13. Vol. S. Journal of Mathematics Analysis and Applications. L. 103-111. 1984. Vol. L. Vol. IEEE Transactions. 15. Termini. 203-230. and I. Dubois. 1996. pp.A. edited by O. man and cybernetics. 756-759. IEEE Transactions on Fuzzy Systems. 5. pp. No. Making computers think like people. 2. 28-44. 8. L. Vol. 17. and H. L. pp. Information Sciences. IEEE Software. U. 12. Roles of soft computing and fuzzy logic in the conception. L. Zadeh. 19. 94-102. 2. 37. 1974. 8. Vol. 4-34. Fuzzy algorithms. L.4. 2. Mathematics Frontiers of the Social and Policy Sciences. pp. Hohle and L. 4.8. 1975. Gottwald. Vol. 2. Zadeh. Kaynak. Part 2.A. Outline of a new approach to the analysis of complex systems and decision process.257-296. Zadeh. L.Cobb and R. Watanable. 11.A. Zadeh. B. pp. Zadeh. L. Vol. 2. 3. Fuzzy logic. Information and Control. 1991.A. Fuzzy Sets and Systems. and S. 1972.A. 20. Zadeh. (November). No. 10. 1979. 8. Man and Cybernetics. pp. D. The concept of a linguistic variable and its application to approximate reasoning. J. 10. 18. 18. No. Springer-Verlag. 69-129. 2. 14. 9.M. 9. DeLuca. Berlin. Vol. Boulder. The algebra of fuzzy logic. No. No. 301-357. on systems. IEEE Transactions on systems. Fuzzy logiccomputing with words. IEEE Spectrum. pp. 1968. Communications of the ACM. Vol. Zadeh. Vol. 83-93. 48-56. No. L. Zadeh. 40. 338-353. Vol. Journal of Cybernetics. pp. L.A. Westview Press. 1. A. 1978. Zadeh. No. Fuzzy logic. Vol. 32-39. Zadeh. Stout. 1994. 1. L.18 FUZZY LOGIC AND NEURAL NETWORKS REFERENCES. Zadeh. pp. The concept of a linguistic variable and its application to approximate reasoning. P. Throll (eds. Information and Control. S. Zadeh. Algebraic properties of fuzzy sets. Zadeh. 26-32.

Charles. If X1 = . for example {(Charles. then R is called an n-ary relation in X. Diana). v).. The subsets of the Cartesian product X1 x x Xn are called n-ary relations.. then the relation married to on X ´ Y is. Rita.. v) = Example 3.2 R1 S0 T R1 S0 T if (u.(3. .+ 0 ) 2 6 .. Let R be a binary relation in Â. = Xn and R Ì X n. Example 3. Xn be classical sets..1) Consider the following relation (u. v)ÎR Û uÎ[a. v)ÎR otherwise . (James.1 Classical N-Array Relation Let X1. Then the characteristic function of R is defined as XR(u.2) if (u. c] ... A binary tuple is denoted by (u. where a tuple is an ordered pair. b]x[0.. v) = vÎ[0. Rita)} 3.4 Fuzzy Relations 3.2. Xn). b] and XR(u...1 INTRODUCTION 3 A classical relation can be considered as a set of tuples.2 FUZZY RELATIONS 3. James} and Y the domain of women {Diana. . v. an example of a ternary tuple is (u.. Eva).(3. v) Î[a .. Eva}. (John.1: Let X be the domain of man {John. c] otherwise . w) and an example of n-ary tuple is (X1.

.

v)} . 2) = 0.1 Intersection (R Ù S) (u. 3) = 1 R(1.8 0. R the domain of R is the whole Cartesian product X ´ Y.e.8 OP P 0. Let R be a binary fuzzy relation on R.3) The intersection of R and S is defined by Note that R: X ´ Y ® [0.. 1) = R(2. v) = S0. 3}. 3. . S(u.8 R(1. A fuzzy relation R is a fuzzy subset of X ´ Y.8P P 1 Q OPERATIONS ON FUZZY RELATIONS Fuzzy relations are very important because they can describe interactions between variables.8 |0.3 The membership function of R is given by R1 | R(u.3.3 0. 1) = 0.(3.. 3. 3) = R(3. 2) = R(3. Then R (u. 2) = R(2.10 Binary Fuzzy Relation Let X and Y be nonempty sets. In other words. v) in R.8 1 0. called approximately equal can be defined as R(1.5: A simple example of a binary fuzzy relation on U = {1.v| = 2 LM MM1 MN2 3 3.FUZZY RELATIONS 21 Consider the relation mod 3 on natural numbers {(m.2.3 T In matrix notation it can be represented as if u = v if |u .3 1 1 2 3 0. Example 3. v) = min {R(u. v). 1]. 2. 1) = R(2.v| = 1 if |u . R Î F (X ´ Y). If X = Y then we say that R is a binary fuzzy relation in X. Let R and S be two binary fuzzy relations on X ´ Y.3 0. i. 3) = R(3. v) is interpreted as the degree of membership of the ordered pair (u. n) | (n m) mod 3 º 0} This is an equivalence relation.

b] x [0.(3.7 . c] otherwise It is clear that the projection (or shadow) of R on the X-axis is the closed interval [a.9 0. 2 3 0 0.4 0.8 The intersection of R and S means that x is considerable larger than y and x is very close to y.9 0. S(u.8 0.8 R1 S0 T if (u. v ) Î[a ...6 0.6: Let us define two binary relations R = x is considerable larger than y .8 0 0.9 0. y3 y4 01 0. 0 0. v) = 1 2 3 y1 y2 y3 y4 0.8 01 .8Q OP PP P 0.5) S = x is very close to y 1 2 3 y1 y2 y3 y4 0. v)} Example 3.(3.5Q OP PP P 0.6 . c]. v) = min{R(u.7 OP P 0P P 0. y) = M MM x Nx Consider a classical relation R on Â. v).3.5 0.5 0.2 Union The union of R and S is defined by (R Ú S) (u.4 0 0. .5Q OP PP PQ .8 0 0.9 1 0.7 0. 1 2 3 y1 y2 y3 y4 0.7 0..8 0.4 0 01 0. y) = M MM x Nx LM x (R Ú S) (x.7 The union of R and S means that x is considerable larger than y or x is very close to y..4 0 0 0. LM x (R Ù S) (x.3 0 0. b] and its projection on the Y-axis is [0. R(u.3 0 0.9 0.9 1 0.7 0.4) LM MM x MN x x LM MM x MN x x 1 y1 y2 0.22 FUZZY LOGIC AND NEURAL NETWORKS 3.

i. y4).e. y2). 0 0.7) 3. (x1. x3 is assigned the highest membership degree from the tuples (x3.8. X Y Fig.(3. (x2.7 .7 OP PP P 0Q x1 is assigned the highest membership degree from the tuples (x1.e. which is the maximum of the first row. (x2.. y3)...8) LM x R = x is considerable larger than y = M MM x Nx then the projection on X means that 1 2 3 y1 y2 y3 y4 0.3..3 Projection Let R be a fuzzy binary fuzzy relation on X ´ Y.. Õx (x3) = 1. y3). (x2.(3.e. y4).8 0 0 0. y2). y) Î R} Õy = {y Î Y| $x Î X(x. 3. (x3.9 1 0.. (x1. i. then Õx = {x Î X| $y Î Y(x. (x1. Õx (x2) = 0.8 01 01 0.(3. i. .. .(3. x2 is assigned the highest membership degree from the tuples (x2. y4). y1). Õx (x1) = 1.6) . The projection of R on X is defined as Õx (x) = sup{R (x. . y) | x Î X} Example 3. y1). y1). (x3.9) . y2). which is the maximum of the second row. y) Î R} where Õx denotes projection on X and Õy denotes projection on Y.7: Consider the relation .2 Shadows of a fuzzy relation. y) | y Î Y} and the projection of R on Y is defined as Õy (y) = sup{R (x. (x3.. y3). which is the maximum of the third row.FUZZY RELATIONS 23 If R is a classical relation in X ´ Y.

5 Shadow of Fuzzy Relation (Co R) (y) = sup min{C(x). B(v)} . R(x..10) for all u Î X and v Î Y.(3. v) = min{A(u). Really. y)} . 3. y)| y} = sup {A(x) Ù B(y)| y} = min {A(x).. sup{B(y)}| y} = min {A(x)..3 Cartesian product of two fuzzy sets. . The composition of a fuzzy set C and a fuzzy relation R can be considered as the shadow of the relation R on the fuzzy set C (Fig..4 Cartesian Product of Two Fuzzy Sets The Cartesian product of A Î F (X) and B Î F (Y) is defined as (A ´ B) (u.(3.(3. 3.3.3) is a fuzzy relation in X ´ Y.4). 1} = A(x) .24 FUZZY LOGIC AND NEURAL NETWORKS 3.3. If A and B are normal. It is clear that the Cartesian product of two fuzzy sets (Fig. then Õy (A ´ B) = B and Õx (A ´ B) = A.. A B A´B Fig.11) 3. 3..12) The sup-min composition of a fuzzy set C Î F (X) and a fuzzy relation R Î F (X ´ Y) is defined as for all x Î X and y Î Y. Õx(x) = sup {A ´ B (x.

8/1 + ½ + 0. Assume that C = 0.2/1 + 1/2 + 0. y¢) (CoR) (y¢) X Y¢ R(x.8P P 1Q OP PP PQ Using the definition of sup-min composition we get LM 1 C o R = (0.3 0.8 0.8/3 0.4 Shadow of fuzzy relation 4 on the fuzzy set +.8: Let A and B be fuzzy numbers and let R=A´B a fuzzy relation.3 = 0.8 1 .8 1 0.3 0.8 0. Observe the following property of composition A o R = A o (A ´ B) = A B o R = B o (A ´ B) = B Example 3. 2.2/1 + 1/2 + 0. 3} and let R be a binary fuzzy relation in {1. 2.2/3) o M MM2 N3 1 2 3 1 0.8 1 0. 3}.8 0.9: Let C be a fuzzy set in the universe of discourse {1.3 0. y) Y Fig. Example 3.8 OP P 0.2/3 and LM 1 R= M MM2 N3 1 1 2 3 0.FUZZY RELATIONS 25 C(x) R(x. 3.

w)} for v Î Y It is clear that R o S is a binary fuzzy relation in X ´ Z.4 0 0..3.7 OP PP PQ .6 0.8 OP PP PQ z1 z2 z3 0.9 0.7 ..9 1 0.6 0. 1] 1+ y 2 3.7 0. Assume that C(x) = x and R(x.3 0 0.8 0. 1] and y Î[0.10: Let C be a fuzzy set in the universe of discourse [0.13) LM x R = x is considerable larger than y = M MM x Nx LM MM y S = y is very close to z = y MM y MN y Then their composition is 1 2 3 1 2 3 y1 y2 y3 y4 0.11: Consider two fuzzy relations . 0 0.5 0 0.8 0. Example 3. 1]. S(v. Using the definition of sup-min composition we get C o R(y) = sup min{x.5P Q LM x RoS= M MM x Nx 1 2 3 z1 z2 z3 0.6 Sup-Min Composition of Fuzzy Relations Let R Î F(X ´ Y ) and S Î F(Y ´ Z).8 0 0 0.4 0 0. y) = 1 |x y|.7 4 OP PP PP 0. The sup-min composition of R and S.(3. 1 |x y|} = for all x Î[0.8 01 01 0. . v).9 0. denoted by R o S is defined as (R o S) (u.4 0.26 FUZZY LOGIC AND NEURAL NETWORKS Example 3.5 0. w) = sup min{R(u.9 0.7 0. 1] and let R be a binary fuzzy relation in [0.

pp.7 0. (x2. Metrical properties of fuzzy relations.F. Find the Cartesian product of A and B. No. Structure of fuzzy binary relations. Kybernetes.4 0 = 1 x2 0. No. 1980. 2. LM MM x MN x x 1 y1 y2 y3 y4 0. Information Sciences. x3} and Y = {y1. 2. Zadeh. 3. Bladwin. R. but the classical product of the matrices R and S with the difference that instead of addition we use maximum and instead of multiplication we use minimum operator. Ovchinnikov. and N. NY.5 0. Explain the operations on the fuzzy relations.8Q M MN y 1 2 3 4 z1 z2 z3 0. 1982. Vol. Vol.5 0 0. y3. 1965. 2. Prade. S. B. Fuzzy Sets and Systems. . Frankl.6 0. (y2. d3. 0. the composition of R and S is nothing else. No.7 .7 0.8 0 0. 1981. 12. Bouchon. 389-396. .C.8 0. 4. 7. QUESTION BANK. L. d4} of the various diseases affecting the plants and Y = {y1. Some properties of fuzzy relationships.V.1). 0.9 0. Vol. and H.e. Problems of Control and Information Theory. pp. 3.A. (x3.R. x2. Given X = {x1.8 01 01 0.4 0. No. 1.7Q i.8 x3 0. 3. 11. G.7 2 3 OP LM y PP o MM y 0 P My 0. y4} be the common symptoms of the diseases. 0.5). 1971.FUZZY RELATIONS 27 formally. No.4 0. pp.4)} be the two fuzzy sets on the universes of discourse X = {x1. pp. Yager. Dubois. 1.9 OP P 0P P 0. 123-140.5 OP PP PP PQ LM MM MN z1 z2 z3 0. Similarity relations and fuzzy orderings. 0. What are the fuzzy relations? Explain them. x2. Guild. Vol. REFERENCES. d2. 177-200. 5. Zadeh. 1981. 169-195. x4} of four varieties of paddy plants.9 1 0. D.A. 2. 223-229. pp. Vol. D = {d1. 4. Cybernetics and Systems.3)} and B = {(y1. 6. Given any N-ary relation. J. Vol. Cohen and P. 0.3). 8. how many different projections of the relation can be taken? Given A = {(x1. 338-353. x3. 6. L. 0 0. Information and Control. Modelling Controllers Using Fuzzy Relations. 9. Find SUP-MIN composition.6 0.3 x 0 0. y2. 2. y2} respectively. 5.. Fuzzy Sets. 5. Academic Press.9 0. 2. 1980. Fuzzy Sets and Systems: Theory and Applications.F.

G. De Baets and E. Fuzzy Logic with Engineering Applications. Vol. Vol.1995. 331-341. B. Inc. Klir. No. 159-170. Yuan. Vol. 50. 17. On-new types of homomorphisms and congruences for partial algebraic structures and n-ary relations. No. 317-321. J. 149-157. 4.J. Vol. 69-79. Gupta. Kohout. International Journal of General Systems. 12.E. pp. Wiley. Zadeh. Upper Saddle River. New York. 13. Klawon.S. Vol. No. Prentice Hall. A new approach to the similarity in the fuzzy set theory. Fuzzy Sets and systems. 1994. R. pp. No. 75. 49. J. Vrba. Information Sciences. Operations on fuzzy graphs. 11. Vol.C. pp. pp. Faurous and J. Traces of fuzzy binary relations. 1. Fillard.X. L. 1994. pp. NY. 31-44. No. Foundations of Fuzzy Systems. Ross. 1992. 79. 213-221. Introduction to Fuzzy Arithmetic: Theory and Applications. No. 18. 4. 21. Fuzzy Sets and Systems. 9. Vol. 109-120. No. Kerre. 1988. and F. 15. U. 14. McGraw-Hill.J. Vol. 1993. Vol. 1986. 1. 134-146. No. Kaufman. T. 1993. Information Sciences. 1995. A. 54. 12. Quotients with respect to similarity relations. pp.P. Fuzzy logic. 19. NJ. W. W. P. . and B. 1993. pp. International Journal of General Systems. 83-93. J. pp. 60.. and C. 1988. Fuzzy Sets and Systems. 3. Gebhardt. J.A. Chichester. 307-315. 14. Fuzzy Sets and Systems. Bandler and L. Fuzzy Sets and Fuzzy Logic: Theory and Applications. Van Nostrand Reinhold. Kolodziejczyk. No. 3. 16. 1991. An upper bound on indices of finite fuzzy relations. J. 1. Li.J. pp. 20. 21. Mordeson. 3. Fodor. Vol. Hohle. IEEE Computer. Fuzzy relational compositions. pp. 3. 10. 2. General decomposition problem of fuzzy relations. 1991. pp.M. NY. Kruse.28 FUZZY LOGIC AND NEURAL NETWORKS 8. Peng. Decomposition problem of fuzzy relations: Further results. No. and M. 1988. Fuzzy Sets and Systems. 27.

3) R1 S0 T if t(p) £ t(q) otherwise p®q 1 1 1 0 The truth table for the material implication. p ® q = Øp Ú q p entails q means that it can never happen that p is true and q is not true. Example 4..2) 4 Let p = x is in A and q = y is in B are crisp propositions. The implication p ® q is interpreted as Ø (p Ù Øq).+ 0 ) 2 6 .. p ® q is true Û t(p) £ t(q) p®q= p 1 0 0 1 q 1 1 0 0 .(4.e.1 INTRODUCTION ..... where A and B are crisp sets for the moment.(4.(4.1) .. It is easy to see that The full interpretation of the material implication p ® q is that the degree of truth of p ® q quantifies to what extend q is at least as true as p.4 Fuzzy Implications 4.1: Let p = x is bigger than 10 and let q = x is bigger than 9.. because it can never happen that x is bigger than 10 and x is not bigger than 9.4) . i.(4. It is easy to see that p ® q is true. This property of material implication can be interpreted as: if X Ì Y then X ® Y .

6) The membership function of the fuzzy set B. 4.5) 4. for all x ³ 5 R1 | 5.. The membership function of the fuzzy set A.(4.25 4 is in the fuzzy set big pressure with grade of membership 0.75 x is in the fuzzy set big pressure with grade of membership 1.1 Membership function for big pressure. can be interpreted as (See Fig. illustrated in the Fig. 4. small volume.1 can be interpreted as 1 is in the fuzzy set big pressure with grade of membership 0 2 is in the fuzzy set big pressure with grade of membership 0. 1 1 5 X Fig.(4.2 FUZZY IMPLICATIONS Consider the implication statement.2 Membership function for small volume. big pressure. 4.30 FUZZY LOGIC AND NEURAL NETWORKS Other interpretation of the implication operator is X ® Y = sup{Z|X Ç Z Ì Y} . if pressure is high then volume is small. .2) 1 1 5 y Fig.u A(u) = S1 |0 4 T if u ³ 5 if 1 £ u £ 5 otherwise . 4....

7) where A is a fuzzy set. (A ® B)(u.. i. that is u is big pressure ® v is small volume º A(u) ® B(v) Remembering the full interpretation of the material implication p®q= . small volume then we define the fuzzy implication A ® B as a fuzzy relation.(4. for example.75 x is in the fuzzy set small volume with grade of membership 1.. for all x £1 R1 | v -1 B(v) = S1 |0 4 T If p is a proposition of the form x is A if v ³ 1 if 1 £ v £ 5 otherwise . v) should be defined pointwise and likewise.... B(v)) = A(u) ® B(v) In our interpretation A(u) is considered as the truth value of the proposition u is big pressure..8) R1 S0 T R1 S0 T if t( p) £ t(q ) otherwise . v) = I(A(u).FUZZY IMPLICATIONS 31 5 is in the fuzzy set small volume with grade of membership 0 4 is in the fuzzy set small volume with grade of membership 0.(4. That is (A ® B)(u. It is clear that (A ® B)(u.25 2 is in the fuzzy set small volume with grade of membership 0...(4.75 ® 1 = 1 .e. big pressure and q is a proposition of the form y is B for example.10) 4 is big pressure ® 1 is small volume A(4) ® B(1) = 0. and B(v) is considered as the truth value of the proposition v is small volume.9) One possible extension of material implication to implications with intermediate truth values can be A(u) ® B(v) = if t( p) £ t(q ) otherwise . v) depends only on A(u) and B(v).(4.

Then A(u) ® B(v) = 0.(4. A smoother extension of material implication operator can be derived from the equation X ® Y = sup {Z| X Ç Z Ì Y} That is A(u) ® B(v) = sup {z| min {A(u).7999.19) Standard Strict x ® y = Godel R1 S0 T R1 x®y= S Ty if x £ y otherwise if x £ y otherwise ..(4.13) This operator is called Godel implication. Other possibility is to extend the original definition. However...8 ® 0.15) It is easy to see this is not a correct extension of material implications.20) ..11) R1 S B (v ) T if A(u) £ B(v) otherwise .8 = 1 Suppose there is a small error of measurement in B(v). because 0 ® 0 yields zero. we are usually not interested in rules. it is easy to see that this fuzzy implication operator (called Standard Strict) sometimes is not appropriate for real-life applications.. our system is very sensitive to rounding errors of digital computation and small errors of measurement. B(v)} . in knowledge-based systems. y} .8..(4.(4.8 ® 0.16) .(4. A(u) ® B(v) = . Larsen Lukasiewiez Mamdani x ® y = xy x ® y = min{1. let A(u) = 0.. where the antecedent part is false. i.(4.(4.12) .14) This operator is called Kleene-Dienes implication.17) . and instead of 0.. B(v)} ..32 FUZZY LOGIC AND NEURAL NETWORKS However. This operator simply takes the minimum of truth values of fuzzy predicates A(u) ® B(v) = min {A(u)... Namely..(4... Then we have A(u) ® B(v) = 0.8 and B(v) = 0.. In many practical applications they use Mamdanis implication operator to model causal relationship between fuzzy variables.7999 = 0 This example shows that small changes in the input can cause a big deviation in the output.(4.8 we have 0..(4..18) ... ® q = Øp Ú q using the definition of negation and union A(u) ® B(v) = max {1 A(u). 1 x + y} x ® y = min{x. z} £ B(v)} so..e.

(more or less A)(x) = A( x) .(4.. . we can employ fuzzy sets to represent linguistic variables.. More or less old Old 30 60 Fig. y} Kleene-Dienes-Luk x ® y = 1 x + xy 4. A linguistic variable can be regarded either as a variable whose value is a fuzzy number or as a variable whose values are defined in linguistic terms.24) The use of fuzzy sets provides a basis for a systematic way for the manipulation of vague and imprecise concepts. 4.(4. In particular.3 MODIFIERS Let A be a fuzzy set in X..(4.21) .4 More or less old.(4..23) Kleene-Dienes x ® y = max {1 x. Then we can define the fuzzy sets very A and more or less A by (very A)(x) = A(x)2. Old Very old 30 60 Fig. 4...22) ..3 Very old.FUZZY IMPLICATIONS 33 Gains x®y= R1 S y/x T if x £ y otherwise ..

6 A possible partition of [ 1. 4. moderate as a speed close to 55 mph.(4.. 4. that is. We might interpret slow as a speed below about 40 mph.. more or less fast. then its term set T (speed) could be T = {slow. if speed is interpreted as a linguistic variable. the set of names of linguistic values of x with each value being a fuzzy number defined on U. moderate.. .5 Values of linguistic variable speed.} where each term in T (speed) is characterized by a fuzzy set in a universe of discourse U = [0.3.34 FUZZY LOGIC AND NEURAL NETWORKS 4.. very slow. For example. G is a syntactic rule for generating the names of values of x. M) .1 LINGUISTIC VARIABLES A linguistic variable is characterized by a quintuple (x. and fast as a speed above about 70 mph.6. T(x). These terms can be characterized as fuzzy sets whose membership functions are shown in Fig. NB NM NS ZE PS PM PB –1 1 Fig. 1 Slow Medium Fast 40 55 70 Speed Fig. G.5. In many practical applications we normalize the domain of inputs and use the type of fuzzy partition shown in Fig. 4. 1]. U.25) in which x is the name of variable. 4. fast. T(x) is the term set of x. 100]. and M is a semantic rule for associating with each value its meaning. .

Fairly true. [PS] Positive Small.7. Absolutely true}.7 Interpretation of absolutely false and absolutely true. . False.(4.. Truth True Absolutely true 1 Fig.(4. 4. 1].FUZZY IMPLICATIONS 35 Here we used the abbreviations NB Negative Big.3. 1].(4. False (u) = 1 u for each u Î [0.. Very true.. 4. True.(4... Absolutely false (u) = . [ZE] Zero.. [PB] Positive Big.2 The Linguistic Variable Truth Truth = {Absolutely false.26) R1 S0 T R1 Absolutely true (u) = S T0 1 False Absolutely false if u = 0 otherwise if u = 1 otherwise . [PM] Positive Medium.29) The interpolation if absolutely false and absolutely true are shown in Fig.28) . NS Negative Small. 4.27) . [NM] Negative Medium. Very false... One may define the membership function of linguistic terms of truth as True (u) = u for each u Î [0.

(4. Very false (u) = (1 u) 2 for each u Î [0.36 FUZZY LOGIC AND NEURAL NETWORKS The word Fairly is interpreted as more or less.. . Fairly false (u) = for each u Î [0.. Very true (u) = u2 for each u Î [0. 4..33) Very false 1 Fig. The word Fairly is interpreted as more or less. 1]. Let t be a term of linguistic variable Truth.31) Very true 1 Fig...(4.(4. 1].. 1].30) .u . 1].(4...(4.. 4. Truth Fairly true u . Then the statement x is A is t is interpreted as x is t o A. Truth Fairly false 1.32) .. Suppose we have the fuzzy statement x is A. Where (t o A)(u) = t(A(u)) for each u Î [0.34) . 1].8 Interpretation of fairly true and very true.9 Interpretation of fairly false and very false. Fairly true (u) = for each u Î [0..

where (t o A)(x) = R1 S0 T if A( x) = 1 otherwise .(4. let t = true..36) . Then x is A is true is defined by x is t o A = x is A because (t o A)(u) = t(A(u)) = A(u) for each u Î [0. where (t o A) (x) = R1 S0 T if A( x) = 0 otherwise .. It is why everything we write is considered to be true. Let t = absolutely true. 4. Then the statement x is A is Absolutely false is defined by x is t o A.35) 1 A is absolutely true a–a a b b–b Fig. 1]. 4.(4. 1 A = “A is true” a–a a b b–b Fig.. Then the statement x is A is Absolutely true is defined by x is t o A.11 Interpretation of A is absolutely true.. Let t = absolutely false.FUZZY IMPLICATIONS 37 For example.10 Interpretation of A is true.

What are the fuzzy implications? Explain with examples.38) 1 “A is very true” a–a a b b–b Fig... .14 Interpretation of A is very true.. 4. Let t = Very true. What are the linguistic variables? Give examples. Then the statement x is A is Fairly true is defined by x is t o A. 1. Let t = Fairly true.38 FUZZY LOGIC AND NEURAL NETWORKS 1 A is absolutely false a–a a b b–b Fig. Then the statement x is A is Fairly true is defined by x is t o A.12 Interpretation of A is absolutely false.. 4.37) 1 “A is fairly true” a–a a b b–b Fig. where (t o A)(x) = (A(x))2 .13 Interpretation of A is fairly true. where (t o A) (x) = A( x) . 4. QUESTION BANK. 2. 3. What are the fuzzy modifiers? Explain with an example.(4.(4.

On mode and implications in approximate reasoning. No. 4. S. 1182-1191. K. Dubois and H. International Journal of Approximate Reasoning. 1985. . pp. 8. W. 1.H. 115-134. negations and implications based on t-norms and t-conorms. Fuzzy Sets and Systems. Vol. In: M. 8. Kandel. 1987. Pilsworth.]. Mamdani. pp. 3. No. middle-aged and old. 7. 273-285. 1. 1976.FUZZY IMPLICATIONS 39 4. 22. W. Vol.J. pp. 1. No. No. IEEE Transactions on Systems. 4. 9. No.W. W. International Journal of Man-Machine Studies. 3. 12. pp. pp. Explain the linguistic variable TRUTH with examples. pp. 3. New York. Vol. Two fuzzier implication operators in the theory of fuzzy power sets. Kisska [Eds.J.E. pp. 199-249. Semantics of implication operators and fuzzy relational products. 6. Zadeh. Advances in the linguistic synthesis of fuzzy controllers. J. Trillas and L. No. Bandler and J. 11. 2. 1. Vol. D. Vol. 1977. J. Applications of fuzzy logic to approximate reasoning using linguistic systems. 669-678. 1983. 12. Part 1.W. pp. 8. 2. Kohout. The concept of a linguistic variable and its application to approximate reasoning. pp. Fuzzy Sets and Systems. 8. 1975. Vol. 1980. Vol. 1. 1980. A. 89-116. A general concept of fuzzy connectives. 1980. Properties of fuzzy implication operators. Vol. 3. 12. Stochastica. 5. Weber. 26. NorthHolland. 11.A. Baldwin and B. 6. No. L. Vol. Willmott. E. No. Bandler. Approximate Reasoning in Expert Systems. 1987. REFERENCES. Fuzzy Sets and Systems. Kohout. 157-166. and Cybernetics. 229-244. Axiomatic approach to implication for approximate reasoning with fuzzy logic. No. 193-219. A theorem on implication functions defined from triangular norms. 4. Ahlquist. Information Sciences. pp. 1984. M. Vol. Prade. 2. Vol. Oh and W. Bandler. Valverde.H. 5. Fuzzy power sets and fuzzy implication operators. pp. 13-30. R. Fuzzy Sets and Systems. Given the set 6 of people in the following age groups: 0 10 10 20 20 30 30 40 40 50 50 60 60 70 70 80 80 and above Represent graphically the membership functions of young. 10.B. Mamdani. 31-36. International Journal of Man-Machine Studies. E.F. 3. No. Man. E. Application of fuzzy implication to probe nonsymmetric relations: Part 1. pp. and L. Gupta. Bandler and L. 1980. 267-279. Fuzzy Sets and Systems.

4. W. 1. pp. De Cooman. Cappelle and F.C. 3. Smets and P. 14. E. International Journal of Approximate Reasoning. 20.M. Implication in fuzzy logic. 54. Vol. pp. pp. Fuzzy implication operators and generalized fuzzy method of cases. Kerre. No. 1. A. pp. Piskunov. 1993. Fuzzy Sets and Systems. Vol. 1. International Journal of Approximate Reasoning. 3.E Kerre. International Journal of Uncertainty. 2. The measure of the degree of truth and the grade of membership. 2. 67-72. 4. R. 15. Wu. B. No. Fuzziness and Knowledge-Based Systems. 333-341. D. 4. pp. Fodor. Vol. Magrez. pp.E. 3. J. 45. Trillas. 151-186. No. 25-35. Vol. G. Vol. Ruan and E.L. 1989. Castro. 10. Smets and P. Commutative implications on complete lattices. M. No. No. pp. Kandel. 1994. 42. Vol. No. 23-37. 327-347. Cao and A. pp. Vol. 17. 1992. 1990. 1. Inducing implication relations. Z. P. 1987. Fuzzy Sets and Systems. P. Delgado and E. No. 1988. 19. 21. International Journal of Approximate Reasoning. 235-250. No. 1994. 18. 1991. 293-300. . Da. 16. Magrez. Vanmassenhove. Vol. Vol.40 FUZZY LOGIC AND NEURAL NETWORKS 13. J. 307-318. 25. Fuzzy implication in fuzzy systems control. 31. Influence of the fuzzy implication operator on the method-of-cases inference rule. No. pp. Fuzzy Sets and Systems. Fuzzy Sets and Systems. On fuzzy implication operators. Applicability of Some Fuzzy Implication Operators.

This theory provides a powerful framework for reasoning in the face of imprecise and uncertain information. More often than not we do not know the complete causal link f between x and y. then y takes the value f (x1). " x Î X and we observe that x = x1. Central to this theory is the representation of propositions as statements assigning fuzzy sets as values to variables.+ 0 ) 2 6 . we know that y is a function of x y = f(x) Then we can make inferences easily Premise Fact Consequence y = f(x) x = x1 y = f(x1) This inference rule says that if we have y = f (x).1 INTRODUCTION In 1975 Zadeh introduced the theory of approximate reasoning. Namely.4 The Theory of Approximate Reasoning 5 5. Suppose we have two interactive variables x Î X and y Î Y and the causal relationship between x and y is completely known. only we know the values of f (x) for some particular values of x Â1 : also Â2 : If x = x2 then y = y2 If x = x1 then y = y1 .

. e. Suppose that we are given an x1ÎX and want to find an y1ÎY which corresponds to x1 under the rule-base.42 FUZZY LOGIC AND NEURAL NETWORKS also also Ân : If x = xn then y = yn y y = f (x) y =f (x¢) x = x¢ x Fig. Let x and y be linguistic variables. The basic problem of approximate reasoning is to find the membership function of the consequence C from the rule-base {Â1. Â1: also Â2 : If x is A2 then y is C2 If x is A1 then y is C1 If x = x2 then y = y2 If x = x1 then y = y1 . 5. Â1 : also Â2 : also also Ân : fact: If x = xn then y = yn x = x1 Consequence: y = y1 This problem is frequently quoted as interpolation.1 Simple crisp inference.g. . Ân} and the fact A. . . x is high and y is small.

THE THEORY OF APPROXIMATE REASONING 43 also also Ân : fact: If x is An then y is Cn x is A Consequence: y is c Zadeh introduced a number of translation rules.2 5.2. which allow us to represent some common linguistic statements in terms of propositions in our language. 5.1 Entailment Rule x is A AÌB x is B TRANSLATION RULES Menaka is very young very young Ì young Menaka is young 5.2.2 Conjunction Rule x is A x is B x is A Ç B Temperature is not very high Temperature is not very low Temperature is not very high and not very low 5.2.3 Disjunction Rule x is A or x is B x is A È B Temperature is not very high or Temperature is not very low Temperature is not very high or not very low .

y) have relation R y is Õy(R) (x.4 Projection Rule (x. y) is close to (3.6 Compositional Rule Of Inference premise fact consequence: if x is A then y is B x is A1 y is B1 .2.2.44 FUZZY LOGIC AND NEURAL NETWORKS 5. 2) y is close to 2 5.5 Negation Rule not (x is A) x is ØA not (x is high) x is not high In fuzzy logic and approximate reasoning. The fuzzy implication inference is based on the compositional rule of inference for approximate reasoning suggested by Zadeh. y) have relation R x is Õx (R) (x. the most important fuzzy implication inference rule is the Generalized Modus Ponens (GMP). y) is close to (3. 2) x is close to 3 (x. 5.2. The classical Modus Ponens inference rule says: premise fact consequence: if p then q p q This inference rule can be interpreted as: If p is true and p ® q is true then q is true.

1 Basic Property if x is A then y is B x is A y is B if pressure is big pressure is big volume is small then volume is small . The classical Modus Tollens inference rule says: If p ® q is true and q is false then p is false. which reduces to calssical modus ponens when A1 = A and B1 = B. premise fact consequence: if x is A then y is B y is B1 x is A1 which reduces to Modus Tollens when B = ¬B and A1 = ¬A. B and A1 are fuzzy numbers. The Generalized Modus Ponens should satisfy some rational properties.(5.THE THEORY OF APPROXIMATE REASONING 45 where the consequence B1 is determined as a composition of the fact and the fuzzy implication operator.. 5. especially in the realm of medical diagnosis. is closely related to the backward goaldriven inference which is commonly used in expert systems. B¢(v) = sup min {A¢(u). The Generalized Modus Tollens.2) The consequence B1 is nothing else but the shadow of A ® B on A1.. (A ® B) (u. is closely related to the forward data-driven inference which is particularly useful in the Fuzzy Logic Control...3 RATIONAL PROPERTIES Suppose that A.3.1) . v)}.(5. 5. The Generalized Modus Ponens. B1 = A1 o (A ® B) that is. v Î V uÎU .

5.2 Basic property.3. 5.3.3 Subset if x is A then y is B x is A1 Ì A y is B . 5.2 Total Indeterminance if x is A then y is B x is ¬A y is unknown if pressure is big pressure is not big volume is unknown then volume is small –A – A¢ –B – B¢ Fig. 5.46 FUZZY LOGIC AND NEURAL NETWORKS A¢ = A B¢ = B Fig.3 Total indeterminance.

THE THEORY OF APPROXIMATE REASONING 47 if pressure is big then pressure is very big volume is small volume is small B¢ = B –A – A¢ Fig. 5. We show that the Generalized Modus Ponens with Mamdanis implication operator does not satisfy all the four properties listed above. Example 5.3. 5. B and A1 are fuzzy numbers.4 Superset if x is A then y is B x is A1 y is B1 É B –A – A¢ –B – B¢ x Fig. 5. Suppose that A.1: (The GMP with Mamdani implication) if x is A then y is B x is A1 y is B1 .4 Subset property.5 Superset property.

B(y)} x = min B( y). sup A¢( x) x R S T U V W = min {B(y).A( x). min {A(x). Then we have B¢(y) = sup min {1 A(x).48 FUZZY LOGIC AND NEURAL NETWORKS where the membership function of the consequence B1 is defined by B¢(y) = sup {A¢(x) Ù A(x) Ù B(y) |x Î R}. 1} = B(y) So the subset is satisfied. 1} = B(y) So the basic property is satisfied. min {A(x). B(y)}} x = sup min {A(x). 1 .} x R S T U V W = min {B(y). sup A( x ) x x R S T U V W = min {B(y). Total indeterminance: Let A1 = ØA = 1 A and let y Î R be arbitrarily fixed. Then we have B¢(y) = sup min {A¢(x). B(y)}} x = sup min {A(x). y Î R Basic property: Let A1 = A and let y Î R be arbitrarily fixed. B(y)} x = sup min B( y ). Subset: Let A¢ Ì A and let y Î R be arbitrarily fixed. sup min { A( x). min {A(x). 1/2} = 1/2 B(y) < 1 This means that the total indeterminance property is not satisfied. . B(y)}} x = sup min {A(x). 1 A(x). B(y)} x = min B( y). A¢(x). Then we have B¢(y) = sup min {A(x).

. min {A(x).THE THEORY OF APPROXIMATE REASONING 49 Superset: Let y Î R be arbitrarily fixed. Total indeterminance: Let A1 = ØA = 1 A and let y Î R be arbitrarily fixed.6 The GMP with Mamdanis implication operator. 5. Then we have B¢(y) = sup min {1 A(x). A¢(x). B(y)} £ B(y) x So the superset property of GMP is not satisfied by Mamdanis implication operator. B(y)}} x = sup min {A(x). A(x) B(y)} x = B( y ) <1 1 + B( y ) This means that the total indeterminance property is not satisfied. Example 5. Then we have B¢(y) = sup min {A(x). Then we have B¢(y) = sup min {A¢(x). A(x) B(y) |x Î R} y Î R x Basic property: Let A1 = A and let y Î R be arbitrarily fixed. A –B – B¢ A(x) x Fig.2: (The GMP with Larsens product implication) if x is A then y is B x is A1 y is B1 where the membership function of the consequence B1 is defined by B¢(y) = sup min {A¢(x). A(x) B(y)} = B(y) x So the basic property is satisfied.

Explain generalized modus ponens with Larsens implication. the superset property is not satisfied. 2. 5. Explain generalized modus ponens with Mamdanis implication. A(x) B(y)} x = sup min {A(x). Given CÚD ~ H Þ (A Ù ~ B) CÚDÞ~H (A Ù ~ B) Þ (R Ú S) Can (R Ú S) be inferred from the above? . What are the rational properties? Explain them. A(x) B(y)} £ B(y) x So. Explain the theory of approximate reasoning. Superset: Let y Î R be arbitrarily fixed. A A¢ –B – B¢ x Fig.7 The GMP with Larsens implication operator. What are the translation rules? Explain them with examples. A¢(x) B(y)} x = B(y) So the subset property is satisfied. QUESTION BANK. Then we have B¢(y) = sup min {A¢(x).50 FUZZY LOGIC AND NEURAL NETWORKS Subset: Let A¢ Ì A and let y Î R be arbitrarily fixed. 4. Then we have B¢(y) = sup min {A¢(x). 6. 5. 3. 1.

IEEE Transactions on Systems. 1979. IEEE Transactions on Systems. No. 401-415. Mamdani. pp. R. L. IEEE Transactions on Systems. 199-228.R. 1986. J. 754-765. 3. Information Sciences. 13. L. Fuzzy Sets and Systems. 1987. 407-428. Giles. 2. Application of fuzzy logic to approximate reasoning using linguistic systems. 67-78. 12. 9. The concept of a linguistic variable and its application to approximate reasoningPart III. 301-357. Vol. International Journal of Man-Machine Studies. Zadeh. Yager. Vol. Syllogistic reasoning in fuzzy logic and its application to usually and reasoning with dispositions. Man and Cybernetics. pp. Mizumoto and H. pp. pp. B. pp. 18. approximate reasoning and dispositions. No. . E.R. pp. International Journal of Intelligent Systems.A.THE THEORY OF APPROXIMATE REASONING 51 REFERENCES. No. pp. 1182-1191. No. 11. 12. 1979. Man and Cybernetics. Zadeh. 309-325. 6. 199-251. 14. 8. Default and inexact reasoning with possibility degress.A. The concept of a linguistic variable and its application to approximate reasoningPart II. Strong truth and rules of inference in fuzzy logic and approximate reasoning. pp. Sugeno and T. Zimmermann. Gainess. 1977. H. Cybernetics and Systems. Vol.H. Vol. M. W. L. Syntheses. Fuzzy reasoning and fuzzy relational equations. pp. Vol. Vol. 623-668. 2. 1. 1983. 8. 26. 7. 1. R. Vol. 11. No. 1. Farreny and H. Prade. 10. No. Vol. Vol. IEEE Transactions on Pattern Analysis and Machine Intelligence. Baldwin.J. Fuzzy Sets and Systems. No. 2. Vol. 16. Fuzzy Sets and Systems. No. No. No. 1986. 260-283. 4. 1976. 17. 15. 16. H. Vol. 313-325.F.A. Zadeh. M. 1983. 30. 7.M. 1975.A. Medical Information. Zadeh. Comparison of fuzzy reasoning methods. 4. 2. Zadeh. The role of fuzzy logic in the management of uncertainty in expert systems. 5. 1983. pp. Information Sciences. 15. 1. No. L. Multi-dimensional Fuzzy Reasoning. Takagi. Prade. Zadeh. No. 270-276. 1985. 1975. 3. Vol.A. 1. 17. pp. 4. International Journal of Man-Machine Studies. 3. Vol. pp.A. Vol. No. 39-63. 3. Fuzzy logic and approximate reasoning. 1975. No. 6. Fuzzy Sets and Systems. A new approach to approximate reasoning using a fuzzy logic. L. pp. pp. 1975. 1985. 2. 23-63. Zadeh. L. pp. Linguistic variables. L. Vol. 8. Vol. Man and Cybernetics.A. A computational approach to approximate and plausible reasoning with applications to expert systems. Vol. A computational theory of dispositions. Wu. 43-80. The concept of a linguistic variable and its application to approximate reasoningPart I. Zadeh. Semantics for fuzzy reasoning. 4. 8. Information Sciences. 9. 309-325. Foundations of fuzzy reasoning. pp. pp. 8. L. 16. 20. Fuzzy Sets and Systems. Vol. 6. 173-186.A. 1985. 1982. pp. 9.

International Journal of Approximate Reasoning.52 FUZZY LOGIC AND NEURAL NETWORKS 19. A. pp. No. 1987. Luo and Z. X. pp. Unklesbay and N. Vol. pp. No. pp. No.B. Console. A.B. Vol. D. pp. A system for Reasoning with Imprecise Linguistic Information. 3. Concepts. Fuzzy reasoning in a multi-dimensional space of hypothesis. 1. Ying. 15-38. 1989. Prade. Fuzzy Sets and Systems. Keller. Qualitative models for reasoning under uncertainty in knowledgebased expert systems. Turksen. 181200. Kandel and L. 1990. 1991. pp. J. 2. No. 157-178. pp. 1991. Ruspini.M.P. D. Torasso and L. pp. Vol. Vol. T. No. 1991. 331-342. Li. N. No. Smets. 143-202. 1989. pp. Part 1: Inference with Possibility distributions. Approximate reasoning for production planning. 1991. 26. 30. 3. 281-293. Fuzzy Sets and Systems. 1990. Fuzzy Sets and Systems. Vol. 1. R. No. Quantitative evaluation of university teaching quality An application of fuzzy set and approximate reasoning. Vol. 33. R. Gupta and J. 311-325. pp. Vol. pp. 1. M. Schwartz. 1990. 31.Z. 3. D. Information Sciences. No. Kandel and P. 26. 23. 1988. S. 57. M. Approximate Reasoning Models. 1991. Feng. Vol. 202-244. 194-205. 431-450. M. Wang.G. Fuzzy sets in approximate reasoning. 307-330. 1989. 1. Wang. approximate reasoning and prototypical knowledge. International Journal of Approximate Reasoning. An approximate reasoning technique for recognition in color images of beef steaks. 463-488. International Journal of Intelligent Systems.Z. Part 2: Logical approaches. 1-11. Fuzzy sets in approximate reasoning. 77-81. Qi. . A method of inference in approximate reasoning based on interval-valued fuzzy sets. pp. Ellis Horwood. Z. 21. 5. 47-68. P. No. 1990. Vol. Some notes on multi-dimensional fuzzy reasoning. No. 28. 2. 29. 19. 40. 35. Vol. 1988. No. pp. pp. Unklesbay. Vol. International Journal of General Systems. 5. 297-317. pp.M. K. Fuzzy Sets and Systems. No. 1990. Prade. No. Fuzzy Sets and Systems. future. D. International Journal of Approximate Reasoning. approximate spatial reasoning: Integrating qualitative and quantitative constraints. 23-37. Vol. pp. 27. Cao. 24. Gorzalczany. 4. Vol.S. Vol. 4. Dubois and H. Approximate reasoning: past. Vol. Man and Cybernetics. Kruse and E. pp. C. 25. Fuzzy Sets and Systems. 1. 4. Fuzzy Sets and Systems. 40. Subhangkasen. 1. No. 34. 16. International Journal of Intelligent Systems. 21. Dehnad. Fuzzy Sets and Systems. 1991. C. 1991. Vol. 36. rules and fuzzy reasoning: A factor space approach. 3. Dutta. No. No. 1-17. No. Fuzzy modus ponens: A new model suitable for applications in knowledge-based systems. 4. 21. 37. 1. Schwecke. pp. 36.S. Lopez de Mantaras. 1. 40.H. 22. Magrez and P. Chischster. 37. present. Lee. Theory of t-norms and fuzzy inference methods. 32. 4. 5. IEEE Transactions on Systems. Peng.L Grize and K. Vol. Representation of compositional relations in fuzzy reasoning. Dubois and H. 36. 20. Cybernetics and Systems. I. 1. A new model of fuzzy reasoning. Vol. Y. P. E. 1987.

7.G.T. Vol. Bien and M. 3. 2. Metric truth as a basis for fuzzy linguistic reasoning. 49. An inference network for bi-directional approximate reasoning based on an equality measure. M. 51. No. 1-25. 1. 5. Koczy and K. Chen. No. pp. No. International Journal of Approximate Reasoning. pp. 1993. 1992.L. pp. Chun. 1994. International Journal of Pattern Recognition and Artificial Intelligence. pp. Hirota. No. 1993. No. Z. Vol. 1994.F. Rule-based Fuzzy Classification for Software Quality Control. 29-53. P. 1992. 51. Reddy and M. 3. 1993. Vol. 1994. 48. 41.L. pp. pp. 1993. Approximate reasoning by linear rule interpolation and general approximation. pp. 1992. Vol. 197-225. 9. 2. on Systems. Analogy between approximate reasoning and the method of interpolation.S. An improved algorithm for inexact reasoning based on extended fuzzy production rules. No. International Journal of Approximate Reasoning. 229-250. Lano. 42. 463-481. C. No.M. 50. 1. 1992. 3. 39. Fuzzy Sets and Systems. Vol.S. Vol. Vol. 259-266. 3. . No. 177-180. 131-146. Fuzzy Sets and Systems. Yager. The use of fuzzy relational thesauri for classificatory problem solving in information retrieval and expert systems. J. 43. 44. 23. Ray. 349358. Zhang. Vol. pp. 315-326. Anderson. An approximate reasoning system. No. 57. Elbert. No. 2.V. 71-79. 11. Z. pp. Raha and K. 23. K. L. On measuring the specificity of IF-THEN rules. 52. 46. International Journal of Intelligent Systems. 45.E. An approximate reasoning system: Design and implementation. Zhao and B. Vol. 1992. Kacprzyk. 1. No. pp. Babu. D. V. 1993. Approximate reasoning with IF-THEN-UNLESS rule in a medical expert system. pp. Cohen and M. Formal frameworks for approximate reasoning. Fuzzy Sets and Systems. 47. pp. Vol. Cybernetics and Systems. Larsen and R. International journal of approximate reasoning. Vol. S. 7. Hudson. Vol. Niskanen. IEEE Transactions. H. IEEE Transactions on Fuzzy Systems. pp. S. Some methods of reasoning for fuzzy conditional propositions. 1. Li. 431-440. 40. 9. No. Fuzzy Sets and Systems.THE THEORY OF APPROXIMATE REASONING 53 38. 63. Q. Man and Cybernetics.A. Vol. No. 4.R. 3. 31-41.

0} . z)) = T(T(x.1) . T(y. 1] T(x.(6..(6.. 1) = x.. the smallest fuzzy set among the fuzzy sets produced by all possible fuzzy unions (t-conorms). "x Î [0... y Î [0. for all a Î [0. y) £ T(x¢. In fuzzy sets theory triangular norm are extensively used to model the logical connective and. That is. b) = max {a + b. 1] . y) = T(y.(6.3) .. x).1 INTRODUCTION Triangular norms were introduced by Schweizer and Sklar to model the distances in probabilistic metric spaces.. 1] is a triangular norm (t-norm for short) if it is symmetric. y.(6.4) These axioms attempt to capture the basic properties of set intersection. In other words. 1) = a. Functions that qualify as fuzzy intersections and fuzzy unions are usually referred to in the literature as t-norms and t-conorms. non-decreasing in each argument and T(a. y). The standard fuzzy union (max operator) produces. Furthermore. 1] ´ [0.+ 0 ) 2 6 . 1] T(x. while the standard fuzzy union is the strongest fuzzy union. 6. T(x.2) ..4 Fuzzy Rule-Based Systems 6 6. the standard fuzzy operations occupy specific positions in the whole spectrum of fuzzy operations: the standard fuzzy intersection is the weakest fuzzy intersection. on the contrary.. "x. z Î [0.(6...5) . the standard fuzzy intersection (min operator) produces for any given fuzzy sets the largest fuzzy set from among those produced by all possible fuzzy intersections (t-norms). The basic t-norms are: minimum : min (a. y¢) if x £ x¢ and y £ y¢. b} . "x. associative. 1]. b) = min {a.6) £ukasiewicz: TL(a.. respectively. 1] ® [0.2 TRIANGULAR NORM A mapping T: [0.1. z).(6. any t-norm T satisfies the properties: Symmetricity : Associativity : Monotonicity : One identy : T(x.

.. through associativity. "x Î [0.... 1] ® [0. an) = a1 X a2 X ..FUZZY RULE-BASED SYSTEMS 55 product : weak : TP(a.(6..(6. b} S0 T if max {a . b) ï a b é ù ï1 . a2. 6..(6. z) S(x.13) .. y). 0) = x. to n > 2 arguments. b) = ab Tw(a.. X an TL (a1. b} = 1 otherwise ..17) . y) = S(y.(6..18) Associativity : Monotonicity : Zero identity : . 1) max {a .12) All t-norms may be extended.. b) = íTL (a. y) £ S(x¢. { } p>0 . a Î (0.... associative.(6..... 0U |å | S V | | T W n i i =1 A t-norm T is called strict if T is strictly increasing in each argument. for all a Î [0.. is a triangular co-norm (t-conorm) if it is symmetric.. any t-conorm S satisfies the properties: Symmetry : S(x. 0) = a.. y¢) if x £ x¢ and y £ y¢ S(x.(6. a2. b) = Hg(a. 1]. b) ï P ï Fl(a.. x) S(x. 1] ´ [0.ab) ab . b} ïT (a. Triangular co-norms are extensively used to model logical connectives or.7) Rmin {a.3 TRIANGULAR CONORM A mapping S : [0..n + 1.. 1].1) ú ï l -1 ê ú ë û î if l = 0 if l = 1 if l = ¥ otherwise . b) = Da(a.1) (l .(6. In other words. 1] .g )(a + b .11) Frank : ì min {a.log l ê1 + (l .16) .. S(y..15) . non-decreasing in each argument and S(a. z)) = S(S(x.(6... an) = max .(6... b) = . g³0 g + (1 .(6..b) p ] . b) = 1 min 1. a} Yp(a.(6.a ) p + (1 .14) R a .8) ..9) . The minimum t-norm is automatically extended and TP (a1.(6. b. p [(1 .10) Hamacher : Dubois and Prade : Yager : ab ..

1] From monotonicity.g )ab .25) YORP (a. y}. a) = a for any aÎ[0. b) = 0 otherwise ..2: Proof: Let S be a t-conorm. Lemma 6. Then the following statement holds max{a.24) . b) = min (a. . x) £ T (y. 0) ³ x S(x. a}. x) ³ S(y..g )ab . b) = a + b. 1 b) defines a t-conorm and we say that S is derived from T. From commutativity of T it follows that a = T(a. We can obtain the following expression using monotonicity of T a = T(a..23) Hamacher Yager Lemma 6. symmetricity and the extremal condition we get S(x.. y) £ min {x. b) = max {a.(1 . y Î[0. b) = . Then the following statement holds Tw(x. b} S1 T if min (a ....56 FUZZY LOGIC AND NEURAL NETWORKS If T is a t-norm then the equality S(a. y) ³ S(y..3: T(a. y) = T (y. Lemma 6.(6. a) = a holds obviously.(2 . b} £ s (a. b Î [0. 1].. a) = a holds for any aÎ[0. p a p + b p . y}. y}. a) £ T(a. 1] From monotonicity. 1].(6.ab STRONG (a.. y) ³ S(x. b) £ STRONG (a.(6. y) £ T(x. b} for any a Î [0.20) . b}. 0) ³ y This means that S(x. y) £ min {x. g³0 1 . Proof: If T(a. 1} SP(a. These equations show that T(a. 1] if and only if T is the minimum norm...22) £ukasiewicz : probabilistic : strong : .(6. "a.21) . b) = a + b .1: Proof: : : HORg (a.. b} SL(a. y) £ T(x.(6. b) = min {a.19) Rmax {a. a) £ min {b. b) £ min {a. The basic t-conorms are: maximum : max (a.. P > 0 { } Let T be a t-norm. y) ³ max{x. b) = min {a + b. b) = 1 T(1 a. b).(6. symmetricity and the external condition we get T(x. a) £ T(b. and a £ b £ 1. "x.(6. 1) £ x T(x. 1) £ y This means that T(x. b) = min 1. Suppose T(a. b) then T(a.

x5.0/x7 B = 0.(6.4 J-NORM-BASED INTERSECTION .0/x5 + 0.9/x3 + 1..26) Let T be a t-norm.2/x7 Then A È B has the following form A È B = 0. If we are given an operator C such that min {a.5 J-CONORM-BASED UNION .0/x7 B = 0.1] then we say that C is a compensatory operator. Example 6.28) . c Î[0. y) = { x + y 1.0/x4 + 0. 1].0/x4 + 0. x3.0/x2 + 0.3/x6 + 0.3/x6 + 0. x4. B(t)) for all t Î X.0/x5 + 0..2/x7.6/x5 + 0.0/x4 + 1. x2.2/x7. x6. b}. x3.0/x6 + 0. c) = max {T(a. y)= Ð AND (x. b Î[0. Then we have (A È B) (t) = min {A(t).4: The distributive law of t-norm T on the max operator holds for any a.27) Let S be a t-conorm.0/x4 + 1. Let A and B be fuzzy subsets of X = {x1. B(t)) for all t Î X. b) £ max {a. b}.3/x6 + 0.3/x2 + 0.6/x5 + 0. The S-union of A and B is defined as (A Ç B) (t) = S(A(t). c). .0/x3 +1. 1: Let T(x.. The operation union can be defined by the help of triangular conorms.FUZZY RULE-BASED SYSTEMS 57 Lemma 6. Then A Ç B has the following form A Ç B = 0.5/x3 +1.0/x1 + 0. x6. be the £ukasiewicz t-norm. 0} (A Ç B) (t) = max {A(t) + B(t 1.3/x2 + 0. x7} and be defined by A = 0.3/x2 + 0. 6.2/x7. B(t)1} for all t Î X.6/x3 + 1. Example 6. x2.6/x6 + 0. x4.. 1}be the £ukasiewicz t-conorm.9/x3 + 1.0/x1 + 0.6/x2 + 1.3/x6 + 0. T(b. Let A and B be fuzzy subsets of X = {x1..0/x4 + 1. 2: Let (S(x.(6. The T-intersection of A and B is defined as (A Ç B) (t) T (A(t).1/x1 + 0.1/x1 + 0. y) = LOR (x.6/x3 + 1.6/x5 + 0.0/x5 + 0.. b} £ C(a. b. x7} and be defined by A = 0. x5. y) = min {x + y.0/x4 + 0. c)}.3/x2 + 0.0/x1 + 0. 0)} for all t Î X.1/x1 + 0. "a.(6. Then we have 6. T(max{a.

In a decision process the idea of trade-offs corresponds to viewing the global evaluation of an action as lying between the worst and the best local ratings.34) M is continuous.(6. y}.(6..(6. the global evaluation of an action will lie between the worst and the best local ratings: Lemma 6. 1] Extremal conditions M(0.. . y) £ max {x.58 FUZZY LOGIC AND NEURAL NETWORKS 6.33) . 1] ´ [0.. Intersection connectives produce a high output only when all of the inputs have high values.30) .6 AVERAGING OPERATORS A typical compensatory operator is the arithmetical mean defined as MEAN (a. "x. M(1. 1] satisfying the following properties Idempotency M(x. This occurs in the presence of conflicting goals. "x.32) .. 1]. "x Î[0.. These connectives can be categorized into the following three classes union. y} £ M(x. 6. 0) = 0. M.29) Fuzzy set theory provides a host of attractive aggregation connectives for integrating membership values representing uncertain information. Averaging operators realize trade-offs between objectives.1 An Averaging Operator is a Function M : [0. We prove that whatever is the particular definition of an averaging operator. Union produces a high output whenever any one of the input values representing degrees of satisfaction of different features or criteria is high.. when a compensation between the corresponding compabilities is allowed.(6..6. 1] ® [0. x)...5: If M is an averaging operator then min {x.(6. x) = x. Compensative connectives have the property that a higher degree of satisfaction of one of the criteria can compensate for a lower degree of satisfaction of another criteria to a certain extent.. by allowing a positive compensation between ratings. if x £ x¢ and y £ y¢ . 1] Commutativity M(x. Averaging operators represent a wide class of aggregation operators. y) £ M(x¢.(6. intersection and compensation connectives.. y¢). b) = a+b 2 . 1) = 1 Monotonicity M(x. union connectives provide full compensation and intersection connectives provide no compensation.31) . y Î[0. In the sense.. y Î[0.y) = M(y.

Averaging operators have the following interesting properties: Property 1. y. max {x. y) 2xy ( x + y ) xy (x + y ) 2 1 .. min {x.(6.1 Name Harmonic mean Geometric mean Arithmetic mean Dual of geometric mean Dual of harmonic mean Median Generalized p-mean Mean operators M(x. y.y ) ( x + y . aÎ(0. y.35) where aÎ(0. y).. y}) £ M(x. the quasi-arithmetic mean of a1 and a2 is defined by M(a1. p³1 . y} £ M(max {x. y}. The only associative averaging operators are defined by Ry | M(x. Table 6.an) = f 1 F1 I GH n å f (a )JK n 1 i =1 This family has been characterized by Kolmogorov as being the class of all decomposable continuous averaging operators. For example... y}) = max {x..2 xy ) (2 .(1 .. 1) e(x p + y p ) 2j 1/p . a) = Sa |x T if if if x£ y£a x£a£ y a£x£y . a) = med (a.x )(1 . Property 2. A strictly increasing averaging operator cannot be associative. a).FUZZY RULE-BASED SYSTEMS 59 Proof: From idempotency and monotonicity of M it follows that min {x. y} = M(min {x.(6.x .y ) med (x. 1) An important family of averaging operators is formed by quasi-arithmetic means M(a1. a2) = f 1 FG f (a ) + f (a ) IJ H 2 K 1 2 . y) and M{x.36) The next table shows the most often used mean operators.. y} Which ends the proof.

3 ´ 0.42) ... 1.... an} .(6....an) = w1b1 + w2b2 +... a2. It is noted that different OWA operators are distinguished by their weighting function.7 + 0.7.1 ´ 0. an) £ max {a1.(6.. an). in particular an aggregate ai is not associated with a particular weight wi but rather a weight is associated with a particular ordered position of aggregate.. .4.(6..2 = 0.. 0... then F(0.. We shall now discuss some of these. . an) £ F(a1.(6.wn)T such as wi Î[0...... 1]. Furthermore F(a1... In 1988 Yager introduced a new aggregation technique based on the ordered weighted averaging (OWA) operators.. a2. a2. an) = max {a1.1)T.3: Assume W = (0.+ wn = 1. . an} £ F(a1.. expert systems and multi-criteria decision aids. Example 6...75. a2.4 ´ 1 + 0. 0. . an} F*: In this case W = W* = (1.60 FUZZY LOGIC AND NEURAL NETWORKS 6. an) = T T å wjbj j =1 n . a2.. An OWA operator of dimension n is mapping F: Ân ® Â. that has an associated weighting vector W = (w1. a2. . 0. 0) and F*(a1. . When we view the OWA weights as a column vector we shall find it convenient to refer to the weights with the low indices as weights at the top and those with the higher indices with weights at the bottom.... an) = min {a1.3...41) Thus the upper an lower star OWA operator are its boundaries.... In 1988 Yager pointed out three important special cases of OWA aggregations: F*: In this case W = W* = (1. an) £ F *(a1.. 0. a2..37) . 0. + an n .38) ..2..... ..2 ´ 0. One sees aggregation in neural networks. . w2...0)T and F*(a1. fuzzy logic controllers.39) a1 + ... .2 Ordered Weighted Averaging The process of information aggregation appears in many applications related to the development of intelligent systems.. . For any OWA operator F holds F*(a1..6) = 0.. a2. 1/n) and FA(a1......2.6. A fundamental aspect of this operator is the re-ordering step.. a2.6 + 0...(6. 0.(6..... From the above it becomes clear that for any F min {a1... a2.. a2... . an) .. 0... 1 £ i £ n and w1 + w2 +. an} FA: In this case W = WA = (1/n .40) A number of important properties can be associated with the OWA operators. vision systems.+ wnbn = where bj is the j-th largest element of the bag (a1. a2.

FUZZY RULE-BASED SYSTEMS 61 The OWA operator can be seen to be commutative. a2. Let (a1.. 0)T. . n such that for each i... a measure of orness.43) A third characteristic associated with these operators is monotonicity.. 1/n)T.(6. 0)T and WA = (1/n. ... If ai = a for all i then for any OWA operator F(a1. d2.. .1 Window type OWA operator.. an) be a bag of aggregates and let {d1.. .45) From the above we can see the OWA operators have the basic properties associated with an averaging operator. Then F(a1.... ai ³ ci..... cn) ... an) = F(d1. In order to classify OWA operators in regard to their location between and and or. the closer is to zero. associated with any vector W is introduce by Yager as follows orness (W) = 1 n-1 å (n i)wi i =1 n It is easy to see that for anyW the orness(W) is always in the unit interval.. 0. Then for any OWA operator F(a1. ... . a2. note that the nearer W is to an or. Assume ai and ci are a collection of aggregates. For this class of operators we have R0 |1 w = S m |0 T i if if if i<k k £i £k +m i³ k +m 1/m 1 k k+m–1 n Fig. . Furthermore.. while the nearer it is to an and.. c2.(6. Example 6.... . i = 1. Another characteristic associated with these operators is idempotency... 0...(6.. 6..... . an) = a. dn} be any permutation of the ai...44) where F is some fixed weight OWA operator.6: Let us consider the vectors W * = (1. . the closer its measure is to none. dn) . a2. 4: A window type OWA operator takes the average of the m arguments around the center.. W* = (1. Lemma 6. . . an) ³ (c1.. .

. a2.. defined by F(a1.5..5..1 å ( n .5 and when much of the weights are non-zero near the bottom.6 and 3 andness (W) = 1 orness (W) = 1 0.. 1 e(k j) n-1 .(6.2b2 + 0. j < k. orness (W*) = 0 and orness (WA) = 0.j ) e . Theorem 6.. an OWA opeartor with much of non-zero weights near the top will be an orlike operator. orness (W) ³ 0.i ) w¢ = n . andness (W) ³ 0..i ) w + ( n .2. the OWA operator will be andlike...62 FUZZY LOGIC AND NEURAL NETWORKS Then it can easily be shown that orness (W*) = 1. a3) = 0. Generally. A measure of andness is defined as andness (W) = 1 orness (W). a3) is an or like aggregation.4 This means that the OWA operator.5: Let W = (0. 0. while moving weight down causes us to decrease orness(W). Then orness (W) = 1 (2 ´ 0.0)T.1: and W ¢ = (w1. .2) = 0.. The following theorem shows that as we move weight up the vector we increase the orness.. wj + e. wn)T orness (W ¢) = 1 n-1 å ( n . 0.k ) e i 1 i i 1 orness (W¢ ) = orness (W ) + since k > j. wn)T where e > 0. Then orness (W¢ ) > orness (W) Proof: From the definition of the measure of orness we get .8.6 = 0.0b3 where bj is the j-th largest element of the bag (a1.. . that is. Example 6. orness (W ¢ ) > orness (W ).8 + 0.46) (Yager. 1993) Assume W and W ¢ are two n-dimensional OWA vectors such that W = (w1. wk e.8b1 + 0.( n . a2. . that is..

the supremum turns into a simple minimum C(w) = min { x0 (x0) Ù A1 (x0) Ù C1(w)} = min {1 Ù A1(x0) Ù C1(w)} = min {A1(x0).. C1(w)} for all w. min {A1(u). We can see when using the OWA operator as an averaging operator Disp(W) measures the degree to which we use all the aggregates equally.2 Fuzzy singleton. 1 x0 X0 Fig. Then the process of computation of the membership function of the consequence becomes very simple. For example.FUZZY RULE-BASED SYSTEMS 63 6. Observing that x0 (u) = 0.49) . 6. w)} = sup min { x0 (u). C1(w)}} u u . Suppose now that the fact of the GMP is given by a fuzzy singleton.7 MEASURE OF DISPERSION OR ENTROPY OF AN OWA VECTOR In 1988 Yager defined the measure of dispersion (or entropy) of an OWA vector by disp (W) = å w ln w i i i . if we use Mamdanis implication operator in the GMP then Rule 1: Fact: consequence: if x is A1 then z is C1 x is x0 z is C where the membership function of the consequence C is computed as C(w) = sup min { x0 (u).. .. (A1 ® C1) (u.48) for all w. "u ¹ x0.(6.(6..(6..47) We can see when using the OWA operator as an averaging operator Disp (W) measures the degree to which we use all the aggregates equally..

6. .. 6. W Rule 1: Fact Consequence: if x is A1 then z is C1 x is x0 z is C . w)} = A1(x0) ® C1(w) u .. (A1 ® C1) (u.64 FUZZY LOGIC AND NEURAL NETWORKS A1 C1 C A1(x0) X0 U W Fig. w)} = A1(x0) ® C1(w) u for all w. If we use Godel implication operator in the GMP.(6.. (A1 ® C1) (u.52) where the membership function of the consequence C is computed as C(w) = sup min { x0 (u). then C(w) = sup min { x0 (u).51) C A1 C1 X0 Fig. 3: Inference with Mamdanis implication operator.(6.. C(w) = R1 SC (w) T 1 if A1 ( x0 ) £ C1 ( w) otherwise .(6. So.50) for all w..4 u Inference with Godel implication operator..

.. n.56) .(6.55) ..54) .. È x0 o Rn 1 i =1 n if x is A1 then z is C1 if x is A2 then z is C2 if x is An then z is Cn x is x0 z is C ..58) C(w) = A1(x0) ® C1(w) Ú ..(6.(6......... ...57) .. w) = Ai(u) ® Ci(w) for i = 1. Find C from the input x0 and from the rule base R = {R1.. C¢1 is called the output of the i-th rule C ¢ (w) = Ai(x0) ® Ci(w) 1 for each w....53) .(6.(6.. Then combine the C¢1 component wise into C¢ by some aggregation operator: C = U C¢ = x0 o R1 È .. Rn} Interpretation of sentence connective also implication operator then compositional operator o We first compose x0 with each Ri producing intermediate result C¢1 = x0 o Ri for i = 1.(6. w) = (Ai ® Ci)(u..FUZZY RULE-BASED SYSTEMS 65 Consider a block of fuzzy IF-THEN rules R1 : also R2: also also Rn : fact: Consequence: The i-th fuzzy rule from this rule-base Ri : if x is Ai then z is Ci is implemented by a fuzzy implication Ri and is defined as Ri(u. n.. Ú An(x0) ® Cn(w) ...

66) . the inference process is the following input to the system is x0 fuzzified input is x0 firing strength of the i-th rule is Ai(x0) the i-th individual rule output is C¢1(w): = A1(x0) ® C1(w) overall system output (action) is C = C¢ È .64) .5) system MAMDANI SYSTEM .6) system (a ® b = ab) input to the system is x0 fuzzified input is x0 firing strength of the i-th rule is Ai(x0) the i-th individual rule output is C ¢ (w) = Ai(x0) Ci(w) 1 overall system output (action) is C(w) = V Ai(x0) Ci(w) i =1 n ....(6.61) (a ® b = a Ù b) input to the system is x0 fuzzified input is x0 firing strength of the i-th rule is Ai(x0) the i-th individual rule output is C¢ (w) = Ai(x0) Ù Ci(w) 1 overall system output (action) is C(w) = V Ai(x0) Ù Ci(w) i =1 n .60) .(6.62) . 6..66 FUZZY LOGIC AND NEURAL NETWORKS So..(6.. ÈC¢ 1 n Overall system output = union of the individual rule outputs.(6...... ...8 For the Mamdani (Fig.63) 6.(6.. 6.(6....9 LARSEN SYSTEM For the Larsen (Fig..59) 6.(6.65) .(6.

6. A1 A1(X0) C1 C ¢1 A2 A2(X0) X0 C2 C ¢2 C = C ¢2 Fig. In the on-line control. 6. a non-fuzzy (crisp) control action is usually required.5 Illustration of Mamdani system. specifying a possibility distribution of the (control) action. 6. one must defuzzify the fuzzy control action (output) inferred from the fuzzy reasoning algorithm. Consequently.6 Illustration of Larsen system.FUZZY RULE-BASED SYSTEMS 67 A1 C1 C ¢1 Degree of match X0 Individual rule output A2 C2 = C ¢2 Degree of match X0 Individual rule output Overall system output Fig. namely: .10 DEFUZZIFICATION The output of the inference process so far is a fuzzy set.

17. 69-81. Statistical metric spaces. B. pp. "x. Sklar. QUESTION BANK.67) where z0 is the crisp action and defuzzifier is the defuzzification operator. 16. 169-186. Debrecen.. 10. 1. What is t-norm? What are the properties to be satisfied by a t-norm? What are the various basic t-norms? What is t-conorm? What are the properties to be satisfied by a t-conorm? What are the various basic t-conorms? Let T be a t-norm. Publication Mathematics. b) £ STRONG (a. y}. 6. 5. Schwartz and A. Sklar. Pacific Journal of Mathematics. Prove the following statement TW (x. Explain the Measure of dispersion. 2. Prove the following statement: max {a. Vol. 14. Vol. 12. 19. What is defuzzification? REFERENCES. 15. Publication Mathematics. 1961. y) £ min{x. What is t-norm based intersection? Explain with an example. Vol. 1. Debrecen. Schwartz and A. 18. 3. 20. Defuzzification is a process to select a representative element from the fuzzy output C inferred from the fuzzy control algorithm. 1960. y Î[0. 1]. " a. Sklar. What is entropy of an ordered weighted averaging (OWA) vector? Explain the inference with Mamdanis implication operator. 2. B. b). Let S be a t-conorm. What is t-conorm based union? Explain with an example. 10. 9. 1963.. Associative functions and statistical triangle inequalities. . Explain Larsen rule-based system. 3. Explain Mamdani rule-based system. y) £ T(x. b Î [0. Explain the inference with Godels implication operator. pp. 8. B.68 FUZZY LOGIC AND NEURAL NETWORKS z0 = defuzzifier (C) .(6. b} £ (S(a. 11. 313-334. Associative functions and abstract semigroups. 1] 8. 10. Schwartz and A. 13. pp. 7. What are the averaging operators? What are the important properties of averaging operators? Explain order weighted averaging with an example. 4.

F. Measures of fuzziness based on t-norms. 4. Filev and R. 1982. Pedrucz. M. 10. Theory of t-norms and fuzzy inference methods. 3. 3. Fuzzy Sets and Systems. Krishanpuram. Vol. Qi. pp. Vol. pp. 1993. International Journal of Man-machine Studies. Vol. 8. 1991. Use of rule-based system for process control. pp. pp. Rhee and R. 1988. Fodor. 40. pp. 1990. 289-296. 61. Applied Intellignece. No. Vol. A. A general approach to rule aggregation in fuzzy logic control.E. Coben and M. 275-29358. Novak and W.J.P. R. 6. 207-229. No. R.C. A fuzzy logic rule-based automatic target recognition. Fuzzy Sets and Systems. 6. Fuzzy control rules and their natural control laws. 1993. Pedrycz. 1985. No. No. 7.R. D.M. pp. Cybernetics and Systems. No. pp.R. Bukley. Cao. IEEE Expert. 5. 21. 3-13. Vol. 295-312. 1992. 1991. Vol. Implementing fuzzy rule-based systems on silicon chips. Vol. 14. 195199. A remark on constructing t-norms. 7. Fuzzy rule generation methods for high-level computer vision. Dubois and H. 1991. 48. Vol. On computation of the compositional rule of inference under triangular norms. 6. M. and J. 1992. No. Approximate reasoning with IF-THEN-UNLESS rule in a medical expert system. Fuller and H. Vol. Peng. pp. 1988. 22.R. No. 1982. 11. 3. Nafarich and J. 16. B. L. Fuzzy rule generation for fuzzy control. 1990. 1992.M. Prade. 1. Vol. Vol. No. 8. Vol. No. Takefuji. Anderson.A. 1. D. Lim and T. pp. Wang. No. 1. 1. R. International Journal of Intelligent Systems. E. 2. Bouslama and A Ichikawa. Fuzzy Sets and Systems. pp. 1. Yagar. Yagar.T. 15. pp. 1992. 1991. 22. Vol. 1992. 13. A general theory of uncertainty based on t-conorms. Fuzzy Sets and Systems. J. 71-79. X. Vol. 6. International Journal of Intelligent Systems.J. 267-275. 103-122. No. Fuzzy Sets and Systems. 83-89. pp. 1. No. Gupta and J. 335-351. No. 3. Fuzzy sets and t-norms in the light of fuzzy logic. pp. 23-63. 12. pp. International Journal of Intelligent Systems. M. Yagar. Fuzzy Sets and Systems.F. R. Mendel. 1992. IEEE Transactions on Systems. 16. 18. Gradual inference rules in approximate reasoning. 13. pp.L. Generating rules for fuzzy logic controllers by functions.X. pp. 9.C. No. . 3. Information Sciences. 1. 20. D. 48. 7. Zimmerman. 1414-1427. Keller. 51. IEEE Control Systems Magazine. J. 6. A generalized defuzzification method via bad distributions. No. V. 1992. Generating fuzzy rules by learning through examples. Yagar. Man and Cybernetics.R. 687-687. Stochastica. 5. Vol. No. 29. 1. Cybernetics and Systems. No. 2. 23. Input-output mathematical model with t-fuzzy sets. 1. 41. Vol. Vol. 3. pp. 113-127.H.H. Bernard. Hudson. No. 65-86. pp.M. 17. F. pp. 431-450. 59. Fuzzy Sets and Systems. No. Strong truth and rules of inference in fuzzy logic and approximate reasoning. Vol. pp. 60. 245-258. Fuzzy Sets and Systems.FUZZY RULE-BASED SYSTEMS 69 4. 19. 31-45. Vol. No. 15-23. 5. J. Czogala and W. Vol. 36.

1993. 259-270. Driankov and H. Sudkamp. 3. pp. 11. No. 3. No. Vol. 1.P. A rule-based method to calculate exactly the widest solutions sets of a max-min fuzzy relations inequality. 33. 1. 31. . Dutta and P. 39-58. 1. 27. 58. 64. C. No. Vol. Rule-based fuzzy classification for software quality control. E. 3. Tano. Fuzzy Sets and Systems. International Journal of Approximate Reasoning. Miki and S. 26. Patterns of fuzzy-rule based interference. On measuring the specificity of IF-THEN rules. Vol. 1993. No. Doherry. 30. Pedrycz. 59. pp. Cross and T. 1993. Vol. 1994. No. Vol. pp. pp. Why triangular membership functions? Fuzzy Sets and Systems. pp. Sudkamp.B. Y. Fuzzy rule-based simple interpolation algorithm for discrete signal. Arnould and S. 167-182. Bonissone. pp. Vol. International Journal of Approximate Reasoning. 2. pp. Ebert. pp. International Journal of Uncertainity. 1993. Fuzzy Sets and Systems. 63. Vol. Fuzziness and Knowledge-based Systems. Hellendoom. 349-358. pp. 1. W. 1. J. No. Similarity. 1994. 1. No. Vol. 3-40. Vol. 29-53. Kacprzyk. 64. 73-86. 1994. Nakamura. T. No. No. Combination of rules or their consequences in fuzzy expert systems. Uchino. 1994. 32. 1994. 8. T. Vol. 1993.Tian and I. 11. pp. No. 235-255. 29. Fuzzy IF-THEN-UNLESS rules and their implementation. International Journal of Approximate Reasoning. Yamakawa. T. 28. T. interpolation and fuzzy rule construction.70 FUZZY LOGIC AND NEURAL NETWORKS 24. Fuzzy Sets and Systems. P. Turksen. V. Fuzzy Sets and Systems. 163-204. P. 25. Fuzzy Sets and Systems. 3. 58. S. 21-30. Integrating case and rule-based reasoning.

since only the rules leading to the objective need to be evaluated. these data are found either in the knowledge base. Similarly.1 INTRODUCTION This chapter focuses different inference mechanisms in fuzzy rule-based systems with examples.. 7.+ 0 ) 2 6 ... it is commonly called forward chaining. Since the data-driven method proceeds from IF clauses to THEN clauses in the chain through the production rules. which then uses them to evaluate relevant production rules and draw all possible conclusions... the expert system searches for data specified in the IF clauses of production rules that will lead to the objective. available data are supplied to the expert system. There exist two approaches to evaluating relevant production rules. or by querying the user... Rn : if x is An and y is Bn then z is Cn z is C x is x0 and y is y0 if x is A1 and y is B1 then z is C1 if x is A2 and y is B2 then z is C2 . The inference engine of a fuzzy expert system operates on a series of production rules and makes fuzzy inferences.2 FUZZY RULE-BASE SYSTEM R1 : R2 : . it is commonly called backward chaining. In this case.4 Fuzzy Reasoning Schemes 7 7. since the goal-driven method proceeds backward from THEN clauses to the IF clauses. The first is data-driven and is exemplified by the generalized modus ponens.. in the THEN clauses of other production rules.. Backward chaining has the advantage of speed. An alternative method of evaluation is goal-driven... Here. it is exemplified by the generalized modus tollens form of logical inference. in its search for the required data.

.(7..3) ...(7.... v. Find C from the input x0 and from the rule base R = {R1. w) = (Ai ´ Bi ® Ci)(u.. È x0 ´ y0 o Rn 1 i =1 n .3 INFERENCE MECHANISMS IN FUZZY RULE-BASE SYSTEMS We present five well-known inference mechanisms in fuzzy rule-based systems.. n..2) .(7.. i n overall system output = union of the individual rule outputs. Rn} Interpretation of logical connective and sentence connective also implication operator then compositional operator o We first compose x0 ´ y0 with each Ri producing intermediate result C¢ = x0 ´ y0 o Ri 1 for i = 1. w) = [Ai(u) Ù Bi(v)] ® Ci(w) for i = 1.. y0) fuzzified input is ( x0 .(7. .5) 7. .4) C(w) = Ai(x0) ´ Bi(y0) ® C1(w) Ú .... Ú An(x0) ´ Bn(y0) ® Cn(w) input to the system is (x0... Here C¢ is called the output of the i-th rule 1 C¢(w) = [Ai(x0) Ù Bi(y0)] ® Ci(w) 1 for each w... n. . .72 FUZZY LOGIC AND NEURAL NETWORKS The i-th fuzzy rule from this rule-base Ri : if x is Ai and y is Bi then z is Ci is implemented by a fuzzy relation Ri and is defined as Ri(u. ÈC¢ ...1) . y0 ) firing strength of the i-th rule is Ai(x0) Ù Bi(y0) the i-th individual rule output is C¢ (w): = A1(x0) Ù B1(x0) ® C1(w) 1 overall system output is C = C¢ È .. For simplicity we assume that we have two fuzzy IF-THEN rules of the form .(7. Then combine the C¢ component wise into C¢ by some aggregation operator: 1 C = U C¢ = x0 ´ y0 o R1È ....

C 2 (w) = (a2 Ù C2(w)) ¢ ¢ Then the overall system output is computed by oring the individual rule outputs C(w) = C¢ (w) Ú C 2 (w) = (a1 Ù C1(w)) Ú (a2 Ù C2(w)) ¢ 1 Finally.FUZZY REASONING SCHEMES 73 R1 : also R2 : fact: Consequence: if x is A1 and y is B1 then z is C1 if x is A2 and y is B2 then z is C2 x is x0 and y is y0 z is C 7.1 Mamdani Inference Mechanism The fuzzy implication is modelled by Mamdanis minimum operator and the sentence connective also is interpreted as oring the propositions and defined by max operator..7) . 2. to obtain a deterministic control action.9) ..(7. are computed by a1 = A1(x0) Ù B1(y0).(7. The firing levels of the rules. a2 = A2 (x0) Ù B2(y0) The individual rule outputs are obtained by C 1 (w) = (a1 Ù C1(w)). i = 1.. 7.. a2 = A2 (x0) Ù B2(y0) . 7...2 Tsukamoto Inference Mechanism All linguistic terms are supposed to have monotonic membership functions. The firing levels of the rules.8) .3. 2.. denoted by ai.(7. . denoted by ai. we employ any defuzzification strategy.(7.1 Inference with Mamdanis implication operator..6) A1 B1 C1 u B2 v C2 w A2 x0 u y0 v Min w Fig. are computed by a1 = A1(x0) Ù B1(y0).3. i = 1.

.3 + 4 ´ 0.. z0 is computed by the discrete Center of-Gravity method. n Example 7.12) i where ai is the firing level and zi is the (crisp) output of the i-th rule.3 Therefore. B1(y0) = 0.6)/(0.(7..8 It follows that the firing level of the second rule is a2 = min{A2(x0).. the firing level of the first rule is a1 = min{A1(x0).6) = 6 . B1(y0)} = min{0.e...7.10) a1z1 + a 2 z2 a1 + a 2 .7.(7.8} = 0.6 The individual rule outputs z1 = 8 and z2 = 4 are derived from the equations C1(z1) = 0. If we have n rules in our rule-base then the crisp control action is computed as n z0 = åaz i =1 n i i åa i =1 .6.11) i.74 FUZZY LOGIC AND NEURAL NETWORKS In this mode of reasoning the individual crisp control actions z1 and z2 are computed from the equations a1 = C1(z1). 0..(7..3 + 0. B2(y0) = 0.3 and from A2(x0) = 0.3} = 0..1: We illustrate Tsukamotos reasoning method by the following simple example R1 : also R2 : fact: Consequence: if x is A2 and y is B2 then z is C2 x is x0 and y is y0 z is C if x is A1 and y is B1 then z is C1 Then according to the figure we see that A1(x0) = 0.6. i = 1. C2(z2) = 0. B2(y0)} = min{0.3..6 and the crisp control action is z0 = (8 ´ 0. 0. a2 = C2(z2) and the overall crisp control action is expressed as z0 = .

14) if x is A1 and y is B1 then z1 = a1x + b1y Sugeno and Takagi use the following architecture The firing levels of the rules are computed by a1 = A1(x0) Ù B1(y0). a2 = A2(x0) Ù B2(y0) then the individual rule outputs are derived from the relationships z* = a1x0 + b1y0.3 u B1 0.6 X0 u 0.. n.(7...8 Y0 v Min 0... .7 0. .(7... z * = a2x0 + b2y0 1 2 and the crisp control action is expressed as z0 = * * a1z1 + a 2 z2 a1 + a 2 .. 7. 7.3.16) i where ai denotes the firing level of the i-th rule.(7.3 Sugeno Inference Mechanism R1 : also R2 : fact: Consequence: if x is A2 and y is B2 then z2 = a2x + b2y x is x0 and y is y0 z0 .6 Z2 = 4 w Fig.(7.15) If we have n rules in our rule-base then the crisp control action is computed as z0 = åa z i =1 n n * 1 i åa i =1 ..13) . i = 1.FUZZY REASONING SCHEMES 75 A1 0..3 v Z1 = 8 C1 w A2 B2 C2 0.2 Tsukamotos inference mechanism..

8 mSMALL (y0) = mSMALL (2) = 0.76 FUZZY LOGIC AND NEURAL NETWORKS A1 A2 u v a1 a1x + b1y B1 B2 a2 x u y v Min a2x + b2y Fig. mBIG (y0) = mBIG (2) = 0.6) = 4.6)/(0.8.2 and from mMEDIUM (x0) = mMEDIUM (3) = 0. mBIG (y0)} = min {0.9} = 0. the firing level of the first rule is a1 = min {mBIG (x0).3 Sugenos inference mechanism. Example 7.6.6 The individual rule outputs are computed as z * = x0 + y0 = 3 + 2 = 5.2 + 0.2 Therefore. 0.9 It follows that the firing level of the second rule is a2 = min {mMEDIUM (x0).6.25 .2} = 0. 0. z* = 2x0 y0 = 2 ´ 3 2 = 4 1 2 So the crisp control action is z0 = (5 ´ 0. 7.2: We illustrate Sugenos reasoning method by the following simple example R1 : also R2 : fact : Consequence: if x is MEDIUM and y is BIG then z2 = 2x y x is 3 and y is 2 z0 if x is BIG and y is SMALL then z1 = x + y Then according to the figure we see that mBIG (x0) = mBIG (3) = 0. mSMALL (y0)} = min {0.2 + 4 ´ 0.

(7..... 7.(7.5 Simplified Fuzzy Reasoning R1 : also R2 : fact: Consequence: if x is A2 and y is B2 then z2 = C2 x is x0 and y is y0 z0 if x is A1 and y is B1 then z1 = C1 .2 x+y=5 1 0. If we have n rules in our rule-base then the consequence C is computed as C(w) = V Ú (ai Ci(w)) i =1 n .3. Let us denote ai the firing level of the i-th rule.FUZZY REASONING SCHEMES 77 1 0.18) .17) . ..4 Larsen Inference Mechanism The fuzzy implication is modeled by Larsens product operator and the sentence connective also is interpreted as oring the propositions and defined by max operator.4 Example of Sugenos inference mechanism.. n 7.2 u v a1 = 0..8 0. 7.(7.3..9 3 u 2 v Min a2 = 0.6 2x – y = 4 Fig.6 0..19) where ai denotes the firing level of the i-th rule. i = 1. a2 = A2(x0) Ù B2(y0) Then membership function of the inferred consequence C is pointwise given by C(w) = (a1C1(w)) Ú (a2C2(w)) To obtain a deterministic control action. we employ any defuzzification strategy. 2 a1 = A1 (x0) Ù B1(y0). i = 1.

21) If we have n rules in our rule-base then the crisp control action is computed as z0 = åa C i =1 n i i åa i =1 .20) a1c1 + a 2 c2 a1 + a 2 n . .78 FUZZY LOGIC AND NEURAL NETWORKS A1 B1 C1 u A2 B2 v C2 w X0 Fig.6 Simplified fuzzy reasoning...(7.(7. . L1 H2 L3 a1 C1 M1 M2 M3 a2 C2 H1 H2 H3 a3 Min Z3 Fig.(7... 7. n. 7. i = 1.. and the crisp control action is expressed as z0 = ....22) i where ai denotes the firing level of the i-th rule. a2 = A2(x0) Ù B2(y0) then the individual rule outputs are c1 and c2. The firing levels of the rules are computed by a1 = A1(x0) Ù B1(y0)..5 u Y0 v Min w Inference with Larsens product operation rule.

Vol. . 15.A. Baldwin. pp.A. 6. Gorzalczany. H. 1986. Explain Larsen inference mechanism. W. No.K. Fuzzy logic and approximate reasoning. 1. Explain Sugeno inference mechanism. No. Mamdani and B. 1. Foundations of fuzzy reasoning. 13. Pedrycz. 6. REFERENCES. Vol. 1979. pp. 16. 4. 1975. pp. 1988. International Journal of Man-machine Studies. 4. Vol. Information Sciences. Vol. 6. 5. 16. 407-428. 7. No. 1. 1976. 2. Approximate Reasoning in Intelligent Systems. 301-357.A. Zadeh. 9.R. 2. J. 2. The concept of a linguistic variable and its application to approximate reasoning I. pp. E. 1983. pp. Vol. Approximate reasoning for production planning. 1987. Decision and Control. Four methods of approximate reasoning with interval-valued fuzzy sets. 1975. Vol. 270-276. Synthese. Explain Mamdani inference mechanism. London. pp. L. 26. 9. Turksen. Multidi-mensional fuzzy reasoning. 8. I.B.A. 10. E. What are the different approaches to evaluating relevant production rules? Explain them. 1. Mamdani.A. Applications of fuzzy relational equations for methods of reasoning in presence of fuzzy data. E. 12. IEEE Transactions on Systems. 199-251. L. International Journal of Approximate Reasoning. 5. Academic Press. Default and inexact reasoning with possibility degrees. Fuzzy Sets and Systems. Vol. 1977. Prade. No. Vol. Turksen. 1-17. 14. 3. 163-175. No. Applications of fuzzy logic to approximate reasoning using linguistic systems. Takagi. 43-80. 2. Gaines. 11. 313-325. 3.FUZZY REASONING SCHEMES 79 QUESTION BANK. Vol. Information Sciences. Zadeh. 9. Sanchez and L. U. 1975. Zadeh. 11. The concept of a linguistic variable and its application to approximate reasoning III. Vol. pp. 8. pp. No. 8. 3. M. No. Oxford. 1989.H. Man and Cybernetics. Fuzzy Sets and Systems. B. 3. Fuzzy Reasoning and Its Applications. pp. 12. 2. Farreny and H. Zadeh. Fuzzy logic and reasoning.B.B. 1182-1191. M. 1. 21. Fuzzy Sets and Systems. Gaines. L. Vol. No. 26. 4. Explain Tsukamoto inference mechanism.H. No. Information sciences. No. Man and Cybernetics. 30. 465-480. Vol. Fuzzy Sets and Systems. 121-142. I. 23-37. A method of inference in approximate reasoning based on interval-valued fuzzy sets. 8. 1981. International Journal of Man-machine Studies. 1987. Sugeno and T. IEEE Transactions on Systems. 1985.R. pp. 1975. Vol. L. Zadeh. Explain simplified reasoning scheme. pp.F. pp. pp. The concept of a linguistic variable and its application to approximate reasoning II. 623-668. Pergamon Press.

Approximate reasoning with IF-THEN-UNLESS rule in a medical expert system. Fuzzy Sets and Systems. Fuzzy Sets and Systems. 77-81. A review and comparison of six reasoning methods. No. IEEE Transactions on Fuzzy Systems. 311-325. 1991. 71-79. Information Sciences. No. 4. A. pp. No. Vol. Vol. 1992. Bien and M. Fuzzy Sets and Systems. 2. Dutta.Z. 5. man and cybernetics. 20. D. Dutta. 143-202. Pawlak. Cybernetics and Systems. No. No. pp. 36. 57. 27. H. pp. 177-180. 3. Rough sets: Theoretical aspects of reasoning about data. An inference network for bidirectional approximate reasoning based on an equality measure.F. 25. 756-770.L. pp. 47-68. E. Part I: Inference with possibility distributions. S.G. Kluwer. C. pp. 3. Sugeno.B. Nakanishi. A. D. pp. 40. Vol. Reasoning with imprecise knowledge to enhance intelligent decision support. Z. present. No. 1991. pp.H. Vol. 57. . 18. 19. No. 2. Vol. Fuzzy Sets and Systems. 26. Vol. 1. A new improved algorithm for inexact reasoning based on extended fuzzy production rules. Z. 4. I. 36. A new model for fuzzy reasoning. Cao. Coben and M. Representation of compositional relations in fuzzy reasoning. Chen. Vol. 1993. Vol. 7. 1989. Vol. 257-294. Bostan. Fuzzy sets in approximate reasoning. 409-420. 17. No.E. 22. 297-317. Hudson. 1. S. 1991. Approximate reasoning: past. Z. 1. pp.P. Prade. 23. Fuzzy reasoning in a multidimensional space of hypotheses. 23. 1992. Kruse and E. Ruspini. 3. M. International Journal of Intelligent Systems. Chun. Vol. 1990. 19. International Journal of Approximate Reasoning. R.M. 1990. 5. Vol. Li. Kandel and L. Basu and A. pp. 21.80 FUZZY LOGIC AND NEURAL NETWORKS 16. Turksen and M. pp. 1991. Anderson. 307-330. Luo and Z. No. IEEE Transactions on Systems. Wang. 1994. No. pp. future. 1. Schwecke. Dubois and H. International Journal of Approximate Reasoning. 24. 1990. Approximate spatial reasoning: Integrating qualitative and quantitative constraints.

parameter t defines the order of the controller.1 INTRODUCTION Conventional controllers are derived from control theory techniques based on mathematical models of the open-loop process.. called system.1 A basic feedback control system.2 BASIC FEEDBACK CONTROL SYSTEM . e(k t).. . The purpose of the feedback controller is to guarantee a desired response of the output y. f is in general a non-linear function. despite the presence disturbances of the system parameters. e represents the error between the desired set point y* and the output of the system y.. is called regulation.. . and noise measurements. e(k t)) providing a control action that describes the relationship between the input and the output of the controller.+ 0 ) 2 6 ... . 8.(8.1) The general form of the discrete-time control law is u(k) = f(e(k).. 8. The process of keeping the output y close to the set point (reference input) y*.4 Fuzzy Logic Controllers 8 8.. e(k 1). u(k 1). to be controlled. y* e Controller u System y Fig. The output of the controller (which is the input of the system) is the control action u.

When this is the case. equivalently.(8.A.... The expert knowledge is usually of the form IF (a set of conditions are satisfied) THEN (a set of consequences can be inferred). e(k 4). e(k t)....3 FUZZY LOGIC CONTROLLER L. z is the control variable. . also Rn : if x is An and y is Bn then z is Cn if x is A2 and y is B2 then z is C2 if x is A1 and y is B1 then z is C1 For example. The knowledge-based nature of FLC dictates a limited usage of the past values of the error e and control u because it is rather unreasonable to expect meaningful linguistic statements for e(k 3).. and Ci are linguistic values of the linguistic variables x. they are often called fuzzy conditional statements. . . e(k 1). the system will be referred to as a multi-input-multioutput (MIMO) fuzzy system... in the case of two-input-single-output fuzzy systems... y and z in the universes of discourse U.2) where the function F is described by a fuzzy rule base. However. In our terminology.82 FUZZY LOGIC AND NEURAL NETWORKS 8. it does not mean that the FLC is a kind of transfer function or difference equation.2 Mamdani Type of Fuzzy Logic Control We can represent the FLC in a form similar to the conventional control law u(k) = f(e(k). 8.. Furthermore. fuzzy control rules have the form.. the dynamic behaviour of a fuzzy system is characterized by a set of linguistic description rules based on expert knowledge. a fuzzy control rule is a fuzzy conditional statement in which the antecedent is a condition in its application domain and the consequent is a control action for the system under control.3. Bi. 8. u(k 1). Zadeh (1973) was introduced the idea of formulating the control algorithm by logical rules.1 Two-Input-Single-Output (TISO) Fuzzy Systems R1 : also R2 : also . and W.3. V. In a fuzzy logic controller (FLC). Basically. where x and y are the process state variables. Ai. Since the antecedents and the consequents of these IF-THEN rules are associated with fuzzy concepts (linguistic terms). respectively. . a rule-base. several linguistic variables might be involved in the antecedents and the conclusions of these rules. e(k t) . e(k t). fuzzy control rules provide a convenient way for expressing control policy and domain knowledge. and an implicit sentence connective also links the rules into a rule set or.

.. A prototypical rule-base of a simple FLC realizing the control law above is listed in the following R1 : R2 : R3 : R4 : R5 : if e is "positive" and De is "near zero" then Du is "positive" if e is "negative" and De is "near zero" then Du is "negative" if e is "near zero" and De is "near zero" then Du is "near zero" if e is "near zero" and De is "positive" then Du is "positive" if e is "near zero" and De is "negative" then Du is "negative" . .. 8. our task is the find a crisp control action z0 from the fuzzy rule-base and from the actual crisp inputs x0 and y0: R1 : also R2 : also ....4) .6) This type of controller was suggested originally by Mamdani and Assilian in 1975 and is called the Mamdani type FLC.5) and is a manifestation of the general FLC expression with t = 1... So.. and the error e(k) and its change De(k) = e(k) e(k 1) On the other hand.2 Membership functions for the error. The actual output of the controller u(k) is obtained from the previous value of control u(k 1) that is updated by Du(k) u(k) = u(k 1) + Du(k)..(8. such control law can be formalized as Du(k) = F(e(k).(8.(8.3) N Error ZE P Fig.FUZZY LOGIC CONTROLLERS 83 A typical FLC describes the relationship between the changes of the control Du(k) = u(k) u(k 1) On the one hand.(8.. De(k)) . if x is A2 and y is B2 then z is C2 if x is A1 and y is B1 then z is C1 .

84 FUZZY LOGIC AND NEURAL NETWORKS also Rn : input output if x is An and y is Bn then z is Cn x is x0 and y is y0 z0 Of course. . Furthermore. and therefore.3 Fuzzy logic controller.3. 8. 8. Fuzzy rulebase.3) usually consist of four major parts: Fuzzification interface.. A fuzzification operator has the effect of transforming crisp data into fuzzy sets. 1 X0 . In most of the cases we use fuzzy singletons as fuzzifiers fuzzifier (x0): = x0 where x0 is a crisp input value from a process. we have to fuzzify the crisp inputs. 8..3 Fuzzy Logic Control Systems Fuzzy logic control systems (Figure 8. Crisp x in U Fuzzifier Fuzzy set in U Fuzzy rule base Fuzzy inference engine Fuzzy set in V Crisp y in V Defuzzifier Fig. Fuzzy inference machine and Defuzzification interface. and therefore to get crisp value we have to defuzzify it.7) X0 Fig.(8. the inputs of fuzzy rule-based systems should be given by fuzzy sets. the output of a fuzzy system is always a fuzzy set.4 Fuzzy singleton as fuzzifier.

Symbolically.. if we have the collection of rules R1 : also R2 : also . we can use any t-norm to model the logical connective and. the overall behavior of a fuzzy system is characterized by these fuzzy relations. In other words. Since each fuzzy control rule is represented by a fuzzy relation... A fuzzy control rule Ri : if (x is Ai and y is Bi then (z is Ci) is implemented by a fuzzy implication Ri and is defined as R(u.. Fuzzy control rules are combined by using the sentence connective also.e. y and fuzzy relations Ri. Aggregate the individual rule outputs to obtain the overall system output. i. [Ai(u) and Bi(v)] ® Ci(w) = [Ai (u) ´ Bi(v)] ® Ci(w) = min {[Ai(u).FUZZY LOGIC CONTROLLERS 85 Suppose now that we have two input variables x and y.. we apply the compositional rule of inference: R1 : if x is A1 and y is B1 then z is C1 also R2 : if x is A2 and y is B2 then z is C2 also .. also Rn : if x is A1 and y is B1 then z is C1 if x is A2 and y is B2 then z is C2 . To infer the output z from the given process states x. also Rn : if x is An and y is Bn then z is Cn input x is x0 and y is y0 z is C Consequence : . a fuzzy system can be characterized by a single fuzzy relation which is the combination in question involves the sentence connective also.(8. Find the output of each of the rules.8) if x is An and y is Bn then z is Cn The procedure for obtaining the fuzzy output of such a knowledge base consists from the following three steps: Find the firing level of each of the rules.9) . Bi(v)] ® Ci(w)} Of course.. w) = [Ai(u) and Bi(v)] ® Ci(w) where the logical connective and is implemented by the minimum operator.(8.v..

15) 8....... fact o Rn) That is.V An(x0) ´ Bn (y0) ® Cn(w) for all w Î W.....17) Example 8. is obtained from the individual rule outputs C i by C(w) = Agg {C¢ .1: If the sentence connective also is interpreted as oring the rules by using minimum-norm then the membership function of the consequence is computed as C = ( x0 ´ y0 o R1 È..10) x0 (u) = 0..(8. .(8. C. .(8..(8.(8. x0 ´ y0 o Rn) taking into consideration that .4 DEFUZZIFICATION METHODS The output of the inference process so far is a fuzzy set. v ¹ y0 The computation of the membership function of C is very simple: ..... An(x0) ´ Bn(y0) ® Cn(w)} for all w Î W.12) .13) .. In the on-line control. C = Agg ( x0 ´ y0 o R1 ... specifying a possibility distribution of control action. u ¹ x0 and y0 (v) = 0.11) .14) C(w) = Agg {A1(x0) ´ B1(y0) ® C1(w)..(8.È x0 ´ y0 o Rn) That is C(w) = A1(x0) ´ B1(y0) ® C1(w) V. namely: z0 = defuzzifier (C) where z0 is the nonfuzzy control output and defuzzifier is the defuzzification operator. one must defuzzify the fuzzy control action (output) inferred from the fuzzy control algorithm.. Consequently. a nonfuzzy (crisp) control action is usually required......18) . 1 n . ....86 FUZZY LOGIC AND NEURAL NETWORKS Where the consequence is computed by consequence = Agg (fact o R1. . The procedure for obtaining the fuzzy output of such a knowledge base can be formulated as The firing level of the I-th rule is determined by Ai(x0) ´ Bi(y0) The output of the I-th rule is calculated by C¢ (w) = Ai(x0) ´ Bi(y0) ® Ci(w) for all w Î W 1 The overall system output... ..16) .(8. C¢ } for all w Î W..(8.(8. .

(8..20) 8.FUZZY LOGIC CONTROLLERS 87 Defuzzification is a process to select a representative element from the fuzzy output C inferred from the fuzzy control algorithm..e.22) .5 First-of-maxima defuzzification method.19) w The calculation of the Center-of-Area defuzzified value is simplified if we consider finite universe of discourse W and thus discrete membership function C (w) z0 = å z C(z )dz å c(z ) j j j . i. having maximal membership grades z0 = 1 N åz j =1 n j .1 Center-of-Area/Gravity The defuzzified value of a fuzzy set C is defined as its fuzzy centroid: z0 = z z w zC( z ) dz c( z) dz . The most often used defuzzification operators are: 8.4.2 First-of-Maxima The defuzzified value of a fuzzy set C is its smallest maximizing element....3 Middle-of-Maxima The defuzzified value of a discrete fuzzy set C is defined as a mean of all values of the universe of discourse.(8. 8.(8... 8.4.21) Z0 Fig.(8..4. z0 = min z C ( z ) = max C ( w) u R S T U V W .

. 8.4 Max-Criterion This method chooses an arbitrary value.(8.. Example 8. 8. z0 Î z C( z ) = max C( w) w R S T U V W . Z0 Fig. .e... i.4. If an obstacle occurs right ahead. from the set of maximizing elements of C.88 FUZZY LOGIC AND NEURAL NETWORKS where {z1..(8.6 Middle-of-maxima defuzzification method.5 Height Defuzzification The elements of the universe of discourse W that have membership grades lower than a certain level a are completely discounted and the defuzzified value z0 is calculated by the application of the Center-ofArea method on those elements of W that have membership grades not less than a: z0 = [C ] [ C ]a z z zC( z)dz c( z )dz . ...2: Consider a fuzzy controller steering a car in a way to avoid obstacles.(8.. the plausible control action depicted in Figure could be interpreted as turn right or left Both Center-of-Area and Middle-of-Maxima defuzzification methods result in a control action driveahead straightforward which causes an accident.25) a where [C]a denotes the a-level set of C as usually.24) 8.4.23) G where G denotes the set of maximizing element of C. zN} is the set of elements of the universe W which attain the maximum value of C. If C is not discrete then defuzzified value of a fuzzy set C is defined as z0 = z z G zdz dz ..

.a I OP B (u) = exp M. .29) .. fuzzifier (y): = y Product fuzzy conjunction [Ai(u) and Bi(v)] = Ai(u) Bi(v) Product fuzzy implication (Larsen implication) [Ai(u) and Bi(v)] ® Ci(w) = Ai(u) Bi(v) Ci(w) (8.(8.. 8. n with Gaussian membership functions LM 1 F u . A suitable defuzzification method would have to choose between different control actions (choose one of two triangles in the Figure) and then transform the fuzzy set into a crisp value.27) . 8.(8.5 EFFECTIVITY OF FUZZY LOGIC CONTROL SYSTEMS Using the Stone-Weierstrass theorem.G MN 2 H b JK PQ L 1 F w .a I OP C (w) = exp M.FUZZY LOGIC CONTROLLERS 89 C Z0 Fig.28) .7 Undesired result by Center-of-Area and Middle-of-Maxima defuzzification methods.26) i2 2 i i3 i3 Singleton fuzzifier fuzzifier (x): = x .. Wang (1992) showed that fuzzy logic control systems of the form Ri: if x is Ai and y is Bi then z is Ci..(8. i = 1..a I OP MN 2 GH b JK PQ L 1 F v .G MN 2 H b JK PQ 2 Ai(u) = exp - i1 i1 2 i i2 .

......32) i i i i i i Singleton fuzzier fuzzifier (x0): = x0 Minimum norm fuzzy conjunction [Ai(u) and Bi(v)] = min {Ai(u)Bi(v)} Minimum norm fuzzy implication [Ai(u) and Bi(v)] ® Ci(W) = min {Ai(u).90 FUZZY LOGIC AND NEURAL NETWORKS Centroid defuzzification method z= åa i =1 n i =1 n i 3 Ai ( x ) Bi ( y ) å A ( x) B ( y ) i i .|c .w| £ g i otherwise . . ..33) .. (8.(8.v| b B (v) = exp S T0 R1 .31) Castro in 1995 showed that Mamdanis fuzzy logic controllers Ri : if x is Ai and y is Bi then z is Ci.v| £ bi otherwise if |ci ....35) ... are universal approximators. Rn} .|a . he proved the following theorem Theorem 8.(8. Ci(W)} Maximum t-conorm rule aggregation Agg (R1. R2. R2.u| a S0 T R1 . i. Bi(v).w| g C (w) = exp S T0 Ai(u) = exp i i if |ai ..u| £ a i otherwise if |bi .|b .. n with Symmetric triangular membership functions R1 .34) . .. there exists a fuzzy logic control system with output function f such that sup ||g(x) f(x)|| £ e x ÎU . Namely..36) .. (8..e.1 For a given real-valued continuous function g on the compact set U and arbitrary e > 0. i = 1.(8.30) where ai3 is the center of Ci.(8.(8. Rn ) = max {R1. they can approximate any continuous function on a compact set to arbitrary accuracy..

7. Vol. 5. 1. 3. 3. Vol. L. Vol. Czogala and W. Pedrycz. Czogala and W. No. E. Analysis of a fuzzy logic controller.H.J. 3. No. 1976. E. E.37) where ci is the center of Ci are also universal approximators.FUZZY LOGIC CONTROLLERS 91 Centroid defuzzification method z= å c min { A ( x) B ( y)} i i i i =1 n n å min { A ( x) B ( y)} i i i =1 . Rutherford. Vol. 7. Vol. No.C. 2. 6. Mamdani. 1. Brase and D. pp. 3. Explain Mamdani type of fuzzy logic controller. Selection of parameters for a fuzzy logic controller. pp.(8. Fuzzy sets and systems. . C. Mamdani and S. 1982. Kickert and E. 185-199. pp. Fuzzy Sets and Systems. No. pp. Mamdani. Mamdani. Vol. Automatica. 1. 13. 29-44. 6. 1-13. Vol. 235-242. 5.H. Measurement and Control. Pedrycz. Machine Studies. What are the various defuzification methods? Explain them. 257-274.. International Journal of Man. 8. E. 13. An experiment in linguistic synthesis with a fuzzy logic controller. M. What are the various parts of fuzzy logic control system? Explain them.H. 3. Advances in the linguistic synthesis of fuzzy controllers. 1977.H. pp. 9. 1. 4. 1982. No. 94. 1979. a rationale for fuzzy control. QUESTION BANK. 669-678. No. Selection of parameters for a fuzzy logic controller. 1. King and E. pp. 1975. The application of fuzzy control systems to industrial process. 4. pp.A. Zadeh. 3-4. 2. Cybernetics and Systems.A. What is fuzzy logic controller? Explain two-input-single-output fuzzy system. 8.. W. 6.J. 2. 1978. pp. 3. No. 1971. Vol. P. Lee. 1979. 185-199. Fuzzy Sets and Systems. What is the effectivity of fuzzy logic control systems? REFERENCES. Vol. 3. Journal of dynamical systems. 7. Assilian. Control problems in fuzzy systems. International Journal of Machine Studies. No. pp. 2. No. Fuzzy Sets and Systems. 1.M. Fuzzy rule generation for fuzzy control. 275-293.

B. G. Newell. 638-656. pp. 1825-1848. C. pp. 16. Man and Cybernetics. Gupta. 1985. J. Boullama and A. Dutta Majumdar. IEEE Transactions on Systems. 99-111. 1993. No. Kiszks and G. Peng. 3. No.R. Control and Cybernetics. 14. pp. Vol. Buckley.H. 83-89. 1988. Fuzzy v/s non-fuzzy controllers. Vol. No. Chen. 5. 18. 1-14. M. 3-13. 23. Wong. 22. Vol. Vol. IEEE Transactions on Systems. pp. Man and Cybernetics. 249-258. No. K. 25. Generating rules for fuzzy logic controllers by functions. 45. Applied Intelligence. 11. M. 149-158. 2. International Journal of Systems Sciences. Vol. A.M. Vol. 1. 1992. C.1.F. 925-960. 1. Fuzzy logic control.Q. No. B. Man and Cybernetics. Sugeno. 16. Ragot and M. Chou and D. Fuzzy inference and its applicability to control systems. Lamotte. T.S. 1993. 21. 127130. No. Vol. Multivariable structure of fuzzy control systems. Sugeno.M. Oh. Yager. Theory of the fuzzy controller: an introduction. Cao. Fuzzy Sets and Systems. No. pp. Chang. X. J.A. 5983. pp. IEEE Control Systems Magazine. Studies on the output of fuzzy controller with multiple inputs. 18. Fuzzy Sets and Systems. J. Vol. pp. Control of dynamic systems using fuzzy learning algorithm.H.H. 1992. 21. 20. No. 255-273. 1992.T. No . 3. 1991. 1. L. 2. 28. Vol.Q.J.B. Use of rule-based system for process control. Tanaka and M. 8. Buckley. pp. J. 27. Li and Z. 223-229. 12. Ichikawa. R. Fuzzy Sets and Systems. F. 65-86. Fuzzy Sets and Systems. Cheung. pp. 48. K. 9. 57. pp. Infromation Sciences. 13. pp. 1988. Vol.Y. 2. Bernard. 1992. No.F.J. 2. Kybernets. 15. 1984. 14. 1993. 10. 17. Vol. 1. Fuzzy identification and control of a liquid level rig. Study on stability of fuzzy closed-loop control systems. 19. Modeling controllers using fuzzy relations. IEEE Transactions on Systems. Man and Cybernetics. No. 1990. pp. F. No. Huang and J. 1992. 26. Vol. 57. Kandel . Fuzzy control rules and their natural control laws. Ray and D. Vol. 3. pp. 2. No. A general approach to rule aggregation in fuzzy logic control. 159-168. 15.P. Vol. 1986. Mon. 1989. 51. 116-132. Chen and L. 1991. 1993. No. Vol. Guild. Chung and J. Vol. 5. Bladwin and N. Design of a fuzzy controller using input and output mapping factors. Fuzzy Sets and Systems. Vol. 24. 26. No.J. Trojan. pp. Application of circle criteria for stability analysis of linear SISO and MIMO systems associated with fuzzy logic controller. pp. Graham and R. An introductory survey of fuzzy control. 4. Fuzzy Sets and Systems. No. 5. 1985. pp. Vol. pp. IEEE Transactions of on Systems. 36. C. No. J. Fuzzy Sets and Systems. 1. Sugeno. 2. 36. . R. 24. pp. Takagi and M. 335-351. Fuzzy Sets and Systems. Fuzzy identification of systems and its applications to modeling and control.M. No. 345-349. Fuzzy Sets and Systems. pp. B. 48. 135-156. J. Stability analysis and design of fuzzy control systems. 59.92 FUZZY LOGIC AND NEURAL NETWORKS 10. Abdelnour. Vol. No.C.

1993. Steffess and E. S.B. 1993. Cybernetics and Systems. 24. Kiupel and P. Ruiz.J. Asia-Pacific Engineering Journal. Barro and R. 1. 277-285. 2. 91-114. Vol. pp. No. . H. No. R. Bugarin.Y. Vol. Yagar. Arend.M. Altrock. Three models of fuzzy logic controllers. Journal of Systems Science. Fuzzy control of steam turbines. 3. 61.R. 35. 1994. Krause.1905-1914. Mc Murray. No. 24.FUZZY LOGIC CONTROLLERS 93 29. Yager. 29-36. Fuzzy control architectures. 30. 1. A. Adaptive fuzzy control applied to home heating system. 1993. Vol.125-146. and D. Vol. Han and V.V. pp. Filev. 1994. Journal of Intelligent and Fuzzy Systems. No. pp. Frank. pp. No. John Wiley.P. 1-32. Essentials of Fuzzy Modeling and Control. Fuzzy Sets and Systems. Rommler. Pedrycz. N. 10. Vol. C. Fuzzy controllers: Principles and architectures. Filev and R. 1993. Two-layer multiple-variable Fuzzy logic controller. J.R. pp. No. 23. 33. D. Man and Cybernetics. B. 34. W. Vol. 1. 1994. IEEE Transactions of Systems. 31. New York.P. 32.2. pp. C.2.O.

Fuzzy logic is conceptually easy to understand. You can create a fuzzy system to match any set of input-output data. The basis for fuzzy logic is the basis for human communication. impenetrable models. Everything is imprecise if you look closely enough. With any given system. Fuzzy logic can model nonlinear functions of arbitrary complexity. This process is made particularly easy by adaptive techniques like ANFIS (Adaptive Neuro-Fuzzy Inference Systems). 6. which take training data and generate opaque.C H A P T E R Fuzzy Logic Applications 9 9. Fuzzy logic is based on natural language. In direct contrast to neural networks. 7. 5. Fuzzy logic is flexible. 4. Fuzzy logic can be blended with conventional control techniques. it’s easy to massage it or layer more functionality on top of it without starting again from scratch. fuzzy logic lets you rely on the experience of people who already understand your system. . The mathematical concepts behind fuzzy reasoning are very simple. Fuzzy systems don’t necessarily replace conventional control methods. This observation underpins many of the other statements about fuzzy logic. 3. most things are imprecise even on careful inspection. Fuzzy reasoning builds this understanding into the process rather than tacking it onto the end. 2. Fuzzy logic is tolerant of imprecise data. Fuzzy logic can be built on top of the experience of experts. but more than that. which are available in the Fuzzy Logic Toolbox. What makes fuzzy nice is the “naturalness” of its approach and not its far-reaching complexity. In many cases fuzzy systems augment themand simplify their implementation.1 WHY USE FUZZY LOGIC? Here is a list of general observations about fuzzy logic: 1.

of course.FUZZY LOGIC APPLICATIONS 95 The last statement is perhaps the most important one and deserves more discussion. 9. that which is used by ordinary people on a daily basis. Control engineers also use it in applications where the on-board computing is very limited and adequate control is enough. Natural language. Sentences written in ordinary language represent a triumph of efficient communication. has been shaped by thousands of years of human history to be convenient and efficient.2 APPLICATIONS OF FUZZY LOGIC Fuzzy logic deals with uncertainty in engineering by attaching degrees of certainty to the answer to a logical question. A cross section of applications that have successfully used fuzzy control includes: 1. Fuzzy logic is not the answer to all technical problems. The control system will not be optimal but it can be acceptable. Domestic Goods • Washing Machines/Dryers • Vacuum Cleaners • Toasters • Microwave Ovens • Refrigerators 3. Exposure and Anti-shake • Hi-Fi Systems 4. In the right application fuzzy logic systems are simple to design. Automotive Systems • Vehicle Climate Control • Automatic Gearboxes • Four-wheel Steering • Seat/Mirror Control Systems . In most cases someone with a intermediate technical background can design a fuzzy logic controller. Consumer Electronics • Television • Photocopiers • Still and Video Cameras – Auto-focus. Since fuzzy logic is built. and can be understood and implemented by nonspecialists in control theory. Commercially. We are generally unaware of this because ordinary language is. Environmental • Air Conditioners • Humidifiers 2. but for control problems where simplicity and speed of implementation is important then fuzzy logic is a strong candidate. something we use every day. Why should this be useful? The answer is commercial and practical. fuzzy logic has been used with great success to control machines and consumer products.

and to increase safety for drivers. One way to reduce the number of accidents is to reduce average speeds. agree that this rate is higher in India since many traffic accidents are not recorded. but also through physical obstacles on the roads. you will see it can be a very powerful tool for dealing quickly and efficiently with imprecision and nonlinearity. Cost of traffic accident is roughly 3% of gross national product. The major objectives with ITS are to achieve traffic efficiency. The program is partly based on the assumption that high speed contributes to accidents. In this study. However. However. road humps. Using velocity of vehicle and pursuit distance that can be measured with a sensor on vehicle a model has been established to brake pedal (slowing down) by fuzzy logic. when there is a certain degree of freedom of choice. might also be reflected in a higher acceptance of other measures. try something else. and in winter these obstacles can reduce access for snow clearing vehicles. by for instance redirecting traffic. use it. These obstacles can cause damages to cars. lack of infrastructure.1 Traffic Accidents And Traffic Safety The general goal of traffic safety policy is to eliminate the number of deaths and casualties in traffic. 9. using fuzzy logic method. Many controllers. Many reasons can contribute these results. Fuzzy logic is the codification of common sense-use common sense when you implement it and you will probably make the right decision. An alternative to these physical measures is different applications of Intelligent Transportation Systems (ITS). Many researchers support the idea of a positive correlation between speed and traffic accidents. . which has increasing usage area in Intelligent Transportation Systems (ITS).96 FUZZY LOGIC AND NEURAL NETWORKS 9. environment.4. 9. weather conditions etc. and whether acceptance of safety measures is also reflected in their perception of road traffic. do a fine job without using fuzzy logic. Obstacles such as flower pots. and might reduce dangerous behaviour in traffic. If you find it is not convenient. if you take the time to become familiar with fuzzy logic. many people died or injured because of traffic accidents all over the world. cyclists and other traffic groups. for example single vehicle accidents or some accidents without injury or fatality. Speed reduction can be accomplished by police surveillance. physical measures are not always appreciated by drivers. they can cause difficulties for emergency vehicles. When statistics are investigated India is the most dangerous country in terms of number of traffic accidents among Asian countries. If a simpler solution already exists. which are mainly driver fault. pedestrians. However. literacy. When should you not use fuzzy logic? Fuzzy logic is a convenient way to map an input space to an output space. a model was developed which would obtain to prevent the vehicle pursuit distance automatically. This goal forms the background for the present traffic safety program. small circulation points and elevated pedestrian crossings are frequently found in many residential areas around India.4 FUZZY LOGIC MODEL FOR PREVENTION OF ROAD ACCIDENTS Traffic accidents are rare and random. However. Another aspect is whether the individual’s acceptance.3 WHEN NOT TO USE FUZZY LOGIC? Fuzzy logic is not a cure-all. for example. One important aspect when planning and implementing traffic safety programs is therefore drivers’ acceptance of different safety measures aimed at speed reduction.

9. inference engine. the rule base is formed with the assistance of human experts. Based on this fact we can infer another fact that is called a conclusion or consequent (the fact following “Then”). During defuzzification. fuzzifier. In this case. A large number of different inferential procedures are found in the literature. rules are extracted from numerical data in the first step.2.4. The general structure of the model is shown in Fig. Models based on fuzzy logic consist of “If-Then” rules. An interesting case appears when a combination of numerical information obtained from measurements and linguistic information obtained from human experts is used to form the fuzzy rule base. one value is chosen for the output variable. as shown in Figure 9. A typical “If-Then” rule would be: If the ratio between the flow intensity and capacity of an arterial road is SMALL Then vehicle speed in the flow is BIG The fact following “If” is called a premise or hypothesis or antecedent. A set of a large number of rules of the type: If premise Then conclusion is called a fuzzy rule base. The literature also contains a large number of different defuzzification procedures. 9.1.FUZZY LOGIC APPLICATIONS 97 9.3 Application In the study. minimum inference or product inference is used. In the next step this fuzzy rule base can (but need not) be supplemented with the rules collected from human experts. The task of the fuzzifier is to map crisp numbers into fuzzy sets (cases are also encountered where inputs are fuzzy variables described by fuzzy membership functions).1 Basic elements of a fuzzy logic. In fuzzy rule-based systems. numerical data has been used as well as through a combination of numerical data-human experts. The final value chosen is most often either the value corresponding to the highest grade of membership or the coordinate of the center of gravity. In most papers and practical engineering applications.4. The inference engine of the fuzzy logic maps fuzzy sets onto fuzzy sets. a model was established which estimates brake rate using fuzzy logic.2 Fuzzy Logic Approach The basic elements of each fuzzy logic system are. 9. and defuzzifier. Input Fuzzifier Defuzzifier Crips output Rules Inference Fig. Input data are most often crisp values. rules. recently. .

9. 9.4 Membership function of distance.3. different membership functions were formed for speed. 1 Low Medium High 0. For maximum allowable car speed (in motorways) in India.5 0 0 50 100 150 Fig.4 Membership Functions In the established model.4. 9. and 9. Because of the fact that current distance sensors perceive approximately 100-150 m distance. Low Medium High 1 0. Brake rate membership function is used 0-100 scale for expressing percent type.2 General structure of fuzzy logic model. Membership functions are given in Figures 9. . 9.3 Membership function of speed. distance and brake rate.5 0 0 20 40 60 80 100 120 Fig.5. speed scale selected as 0-120 km/h on its membership function.4. 9. distance membership function is used 0-150 m scale.98 FUZZY LOGIC AND NEURAL NETWORKS Speed Rule base Distance Brake rate Fig.

6 Output Fuzzy logic is also an estimation algorithm. 9.5 Rule Base We need a rule base to run the fuzzy model. it can be said that this fuzzy logic approach can be effectively used for reduce to traffic accident rate. and brake rate. 9. 9. In this study. Fig.7 Conclusions Many people die or injure because of traffic accidents in India. Many reasons can contribute these results for example mainly driver fault.4.1: Speed LOW LOW LOW MEDIUM MEDIUM MEDIUM HIGH HIGH HIGH Fuzzy allocation map of the model Distance LOW MEDIUM HIGH LOW MEDIUM HIGH LOW MEDIUM HIGH Brake rate LOW LOW MEDIUM MEDIUM LOW LOW HIGH MEDIUM LOW 9.4. various alternatives are able to crossexamine using the developed model. So. lack of infrastructure. Table 9. weather conditions etc. a model was established for estimation of brake rate using fuzzy logic approach.FUZZY LOGIC APPLICATIONS 99 Low Medium High 1 0. Fuzzy Allocation Map (rules) of the model was constituted for membership functions whose figures are given on Table-9.5 Membership function of brake rate. 9. . Figure 6 shows that the relationship between inputs.5 0 0 10 20 30 40 50 60 70 80 90 100 Fig.6 is an example for such the case. environment.1. Car brake rate is estimated using the developed model from speed and distance data. This model can be adapted to vehicles. speed and distance. It is important that the rules were not completely written for all probability.4. For this model.

‘cold’ and ‘comfortable’ have been defined by membership distributions over a range of actual temperatures. Numerical models provide high precision. 9. 9. which can take one or more fuzzy values. The form of the control model also determines the appropriate level of precision in the result obtained. Three fuzzy sets: ‘hot’. This implies a crisp . Much of the fuzzy literature uses set theory notation. Here the process is described in common language. In these cases. an object (in this case a temperature value) is either a member of a set or it is not a member. The power of a fuzzy model is the overlap between the fuzzy values. which describe the control model. Mamdani showed that fuzzy logic could be used to develop operational automatic control systems. which obscures the ease of the formulation of a fuzzy controller. The room temperature is the variable shown in Fig. 9.7. A fuzzy variable is one of the parameters of a fuzzy model. each represented by a fuzzy set and a word descriptor.1 The Mechanics of Fuzzy Logic The mechanics of fuzzy mathematics involve the manipulation of fuzzy variables through a set of linguistic equations.6 Relationship between inputs and brake rate. which can take the form of if–then rules.5. The outline of fuzzy operations will be shown here through the design of a familiar room thermostat. linguistic models provide an alternative. but the complexity or non-linearity of a process may make a numerical model unfeasible. quantitative observations are often required to make quantitative control decisions. A single temperature value at an instant in time can be a member of both of the overlapping sets. The linguistic model is built from a set of if-then rules.5 FUZZY LOGIC MODEL TO CONTROL ROOM TEMPERATURE Although the behaviour of complex or nonlinear systems is difficult or impossible to describe using numerical models. Although Zadeh was attempting to model human activities. In conventional set theory. These decisions could be the determination of a flow rate for a chemical process or a drug dosage in medical practice. the proof of stability and other validations remain important topics. 9.100 FUZZY LOGIC AND NEURAL NETWORKS 80 Brake rate 60 40 0 20 0 50 50 Distance 100 150 100 Speed Fig. Although the controllers are simple to construct.

boundary between the sets. the boundaries between sets are blurred. 9.6 Cold 0.2 0. There are no constraints on the specification of the form of the membership distribution.4 0. usually in the form of if-then rules.33 Comfortable Hot 0. Finally the resulting fuzzy output is converted back into physical values through a defuzzfication process. Guidance for these choices is provided by Zadeh’s Principle of Incompatibility: As the complexity of a system increases.67 0. 9. 0.7 Room temperature.7. By admitting multiple possibilities in the model.7 are triangular. 9. but the triangular form is commonly chosen. our ability to make precise and yet significant statements about its behaviour diminishes until a threshold is reached beyond which precision and significance (or relevance) become almost mutually exclusive characteristics. 0. as shown in Fig. Finer resolution is possible with additional sets. a temperature of 15°C becomes three fuzzy values. Thus. The number of values and the range of actual values covered by each one are also arbitrary. The blurred set boundaries give fuzzy logic its name. The Gaussian form from statistics has been used. where measurements are converted into memberships in the fuzzy sets.2 1.33 ‘comfortable’ and 0.8 0. In fuzzy logic.2 Fuzzification For a single measured value. but the computation cost increases. The membership functions are used to calculate the memberships in all of the fuzzy sets.0 Membership value 0. as its computation is simple. The membership functions defining the three fuzzy sets shown in Fig. In the overlap region. an object can be a partial member of each of the overlapping sets. The operation of a fuzzy controller proceeds in three steps. the fuzzification process is simple. the linguistic imprecision is taken into account. . The first is fuzzification.FUZZY LOGIC APPLICATIONS 101 1. The second step is the application of the linguistic model. 9.0 0 5 10 15 20 25 30 35 Temperature (Degrees C) 40 45 50 Fig.5.66 ‘cold’.00 ‘hot’.

00.8 0. The rules needed to describe a process are often obtained through consultation with workers who have expert knowledge of the process operation.2 0. The membership of the histogram in ‘cold’. In Fig. the rule antecedents. The fuzzy inference is extended to include the uncertainty due to measurement error as well as the vagueness in the linguistic descriptions. Thus.2 1.4 Comfortable Hot 0.8. By similar operations. 9.8. the simplicity of the rules trades off against the number of rules. The membership of the histogram in ‘cold’ is given by: max {min [mcold(T). For complex systems the number of rules required may be very large. These experts include the process designers.6 Cold 0. but more . there must be a rule to cover every possible combination of fuzzy input values.0 0 5 10 15 20 25 30 35 Temperature (Degrees C) 40 45 50 Fig. 9. is 0.0 and it can be used as a fuzzy set.00. 9. to estimate the extent of control action.8 the measurement data histogram is normalized so that its peak is a membership value of 1. 1. Although each rule is simple.then rules.40 and 0.3 Rule Application The linguistic model of a process is commonly made of a series of if . 9. indicated by the arrow in Fig.73. the membership of the histogram in ‘comfortable’ and ‘hot’ are 0.5.8 Fuzzification with measurement noise. It is interesting to note that there is no requirement that the sum of all memberships be 1. the rule consequents. The minimum operation yields the overlap region of the two sets and the maximum operation gives the highest membership in the overlap.102 FUZZY LOGIC AND NEURAL NETWORKS A series of measurements are collected in the form of a histogram and use this as the fuzzy input as shown in Fig. mhistogram(T)]} where the maximum and minimum operations are taken using the membership values at each point T over the temperature range of the two distributions. 9.0 Membership value 0. These use the measured state of the process.

33 and 0. but gives a distribution corresponding to the overlap between A and its adjacent sets. Here the method becomes indecisive and does not produce a satisfactory result. if humidity was to be included in the room temperature control example. 9. The first is the maximum membership method. For very complex systems.5. Rules may also be generated from operating data by searching for clusters in the input data space. Two methods are commonly used. The result for the three rules is then 0. one physical value of a controller output must be chosen from multiple recommendations. (Temperature is Cold) is the membership value of the actual temperature in the ‘cold’ set. where mA and mB are membership values in sets A and B respectively.66 membership in ‘on’ and 0. heater on and heater off). In automatic control. The rules presented in the above example are simple yet effective. This method fails when there are two or more equal maximum membership values for different recommendations.4 Defuzzification The results of rule application are membership values in each of the consequent or output sets. In the above rule.66 membership in the heater setting ‘on’. In decision support systems. When several rules give membership values for the same output set. This gives the interesting result that A AND NOT (A) does not vanish. rules of the form: IF (Temperature is Cold) AND (Humidity is High) THEN (Heater is ON) might be used. Rule 1 transfers the 0. the process operators. 9.66 membership in ‘cold’ to become 0. the membership in ‘on’ will be the minimum of the two antecedent membership values. Zadeh defined the logical operators as AND = Min (mA.33 membership in ‘off’. The rules can include both the normal operation of the process as well as the experience obtained through upsets and other abnormal conditions. compound rules may be formulated. Similar values from rules 2 and 3 are 0. the experts may not be able to identify their thought processes in sufficient detail for rule creation. Zadeh also defined the NOT operator by assuming that complete membership in the set A is given by mA = 1. These can be used directly where the membership values are viewed as the strength of the recommendations provided by the rules. Exception handling is a particular strength of fuzzy control systems. For example.00 in the ‘off’ setting for the heater. All of the output membership functions are combined using the OR operator and the position of the highest membership value in the range of the output variable is used as the controller output. To extend these to more complex control models. mB) and OR = Max (mA.FUZZY LOGIC APPLICATIONS 103 importantly. Defuzzification is the process for converting fuzzy output values to a single value or final decision.7: Rule 1 : Rule 2 : IF (Temperature is Cold) THEN (Heater is On) IF (Temperature is Comfortable) THEN (Heater is Off) Rule 3 : IF (Temperature is Hot) THEN (Heater is Off) In Rule 1. The membership in NOT (A) is then given by m NOT (A) = 1 – mA. It is possible that several outputs are recommended and some may be contradictory (e. Mamdani used the maximum of the membership values.g. . there must be a consistent method to resolve conflict and define an appropriate compromise. A simple temperature control model can be constructed from the example of Fig. mB).

. There are two main differences. With singletons.67 and mOFF = 0. The rules are applied using formalized operations to yield memberships in output sets. ‘off’ at 0% power and ‘on’ at 100% power. labor requirements. Finally. etc. The output values used in the thermostat example are singletons. Singletons are fuzzy values with a membership of 1.40. self-learning techniques such as neural networks (NN) and fuzzy logic (FL) seem to represent a good approach.104 FUZZY LOGIC AND NEURAL NETWORKS The second method uses the center of gravity of the combined output distribution to resolve this potential conflict and to consider all recommendations based on the strengths of their membership values. availability. cost and inconsistency.6 FUZZY LOGIC MODEL FOR GRADING OF APPLES Agricultural produce is subject to quality inspection for optimum evaluation in the consumption cycle. such as plants and commodities. conflicting rules essentially cancel and a fair weighting is obtained. the operating procedures for the calculations are well set out. Therefore. the agricultural environment is highly variable. 9. a heater power of 65%. However. The center of gravity is given by XF = z z x ( x ) dx ( x ) dx where x is a point in the output range and XF is the final control value.5 Conclusions Linguistic descriptions in the form of membership functions and rules make up the model. Defuzzifying these gives a control output of 67% power. Although only two singleton output functions were used. First. The rules are generated a priori from expert knowledge or from data through system identification methods. soil. Techniques used in industrial applications. By taking the center of gravity. in terms of weather.00 at a single value rather than a membership function between 0 and 1 defined over an interval of values.33. with center of gravity defuzzification. display high variation due to their inherent morphological diversity. biological materials. the center of gravity equation integrals become a simple weighted average. Applying the rules gave mON = 0. Center of gravity defuzzification gave. such as template matching and fixed object modeling are unlikely to produce satisfactory results in the classification or control of input from agricultural products.5. applying the same rules gave mON = 0. Efforts to develop automated fruit classification systems have been increasing recently due to the drawbacks of manual grading such as subjectivity. Measurement data are converted to memberships through fuzzification procedures. Output membership functions can be initially set. In the histogram input case.73 and mOFF = 0. These integrals are taken over the entire range of the output. tediousness. in this case. Second. In the example there were two. Input membership functions are based on estimates of the vagueness of the descriptors used. but can be revised for controller tuning. 9. The sum of the membership functions was normalized by the denominator of the center of gravity calculation. applying automation in agriculture is not as simple as automating the industrial operations. the heater power decreases smoothly between fully on and fully off as the temperature increases between 10°C and 25°C. Once these are defined. these are combined through defuzzification to give a final control output.

where L is the lightness factor and a and b are the chromaticity coordinates. Maximum circumference measurement was performed using a Cranton circumference measuring device. russeting. assuming that the same measurements can be done using a sensor fusion system in which measurements of features are collected and controlled automatically. Programming for fuzzy membership functions.6. Sizes of surface defects (natural and bruises) on apples were determined using a special figure template. Size defects were determined measuring the maximum and minimum heights of apples using a Mitutoya electronic caliper. To compare the classification results from the FL approach and from sensory evaluation by a human expert.FUZZY LOGIC APPLICATIONS 105 Fuzzy logic can handle uncertainty. In addition to these defects. Grading of apples was performed in terms of characteristics such as color. Weight was measured using an electronic scale. Color was measured using a CR-200 Minolta colorimeter in the domain of L. Readings of these properties were obtained from different measurement apparatuses. 9. To establish a multi-sensor measuring system for quality features in the long term.6. weight and size. 3. which consisted of a number of holes of different diameters. to manage crop production. Scars. shape. were measured. a and b. fuzzification and defuzzification was done in Matlab. weight and size. to control the start-up and shutdown of food extrusion processes. Fuzzy logic is a nonparametric classification procedure. to predict corn breakage. It provides a means of translating qualitative and imprecise information into quantitative (linguistic) terms. to steer a sprayer automatically. . 9. color. maintaining flexibility in making decisions even on complex biological systems. ignoring their age. 2.1 Apple Defects Used in the Study No defect formation practices by applying forces on apples were performed. ambiguity and vagueness. external defects. a size defect (lopsidedness) was also measured by taking the ratio of maximum height of the apple to the minimum height. to manage a food supply and to predict peanut maturity. to reduce grain losses from a combine. defect. Fuzzy logic was successfully used to determine field trafficability. Only defects occurring naturally or forcedly on apple surfaces during the growing season and handling operations were accounted for in terms of number and size. to predict the yield for precision farming. to decide the transfer of dairy cows between feeding groups. which can infer with nonlinear relations between input and output categories. leaf roller. The following objectives were included in this study: 1. shape. bitter pit. The main purpose of this study was to investigate the applicability of fuzzy logic to constructing and tuning fuzzy membership functions and to compare the accuracies of predictions of apple quality by a human expert and the proposed fuzzy logic model.2 Materials and Methods Five quality features. To design a FL technique to classify apples according to their external features developing effective fuzzy membership functions and fuzzy rules for input and output variables based on quality standards and expert expectations. punctures and bruises were among the defects encountered on the surfaces of Golden Delicious apples.

and fuzzy inference. 1976). input variables were reduced to 3 defect. The Hue angle (tan-1(b/a))..11). medium and good. Similarly. the apples were graded by the human expert into three quality groups. expectations and USDA standards (USDA. Fuzzy logic techniques were applied to classify apples after measuring the quality features.9 and 9. Although it was measured at the beginning. as it was difficult for the human expert to quantify it nondestructively. In addition. After the combinations of features given in the above equations. 9.3 Application of Fuzzy Logic Three main operations were applied in the fuzzy logic decision making process: selection of fuzzy inputs and outputs. size and color.1) where B is the amount of bruising. as total area (normalized). natural defects.(9.10). circumference. A total of 181 golden delicious apples were graded first by a human expert and then by the proposed fuzzy logic approach. R is the total area of russeting defect (normalized) and SD is the normalized size defect. The USDA standards for apple quality explicitly define the quality criteria so that it is quite straightforward for an expert to follow up and apply them.. 1976). formation of fuzzy rules. such as scars and leaf roller. blush (reddish spots on the cheek of an apple) percentage and weight were combined under “Size” using the same procedure as with “Defect” Size = 5 ¥ C + 3 ¥ W + 5 ¥ BL . defects were collected under a single numerical value. an exponential function with the base of the irrational number e was used to simulate the inclination of the human expert in grading apples in terms of size (Fig. W is weight (normalized) and BL is the normalized blush percentage.2) where C is the circumference of the apple (normalized). Eighty of the apples were kept at room temperature for 4 days while another 80 were kept in a cooler (at about 3°C) for the same period to create color variation on the surfaces of apples. which was used to represent the color of apples. Extremely large or small apples were already excluded by the handling personnel.. 21 of the apples were harvested before the others and kept for 15 days at room temperature for the same purpose of creating a variation in the appearance of the apples to be tested. Although triangular and trapezoidal functions were used in establishing membership functions for defects and color (Fig. Along with the measurements of features. medium and good). depending on the expert’s experience. “defect” after normalizing each defect component such as bruises. medium and bad apple groups defined by USDA standards (USDA. 1976). . A trial and error approach was used to develop membership functions. The grading performance of fuzzy logic proposed was determined by comparing the classification results from FL and the expert. was shown to be the best representation of human recognition of color. based on the expert’s expectations and USDA standards (USDA. 9. firmness was excluded from the evaluation.106 FUZZY LOGIC AND NEURAL NETWORKS The number of apples used was determined based on the availability of apples with quality features of the 3 quality groups (bad. To simplify the problem. ND is the amount of natural defects. russetting and size defects (lopsidedness). 9.3 ¥ SD .(9.6. The expert was trained on the external quality criteria for good. Coefficients used in the above equations were subjectively selected.. Defect = 10 ¥ B + 5 ¥ ND + 3 ¥ R + 0. bad.

9. 1 Yellow Greenish-yellow Green 90 95 100 104.13 7. .3) .5 7.FUZZY LOGIC APPLICATIONS 107 1 Low Medium High 0.1 1.11 Membership functions for the size feature..6 Fig..05 6.10 Membership functions for the color feature.4 Defects 4.05 Size 11.10 7.0 2.15 11.5 106 Hue values 114 116 117 Fig.2 1.7 2.9 Membership functions for the defect feature. 9. 1 Small Medium Big 6.71828 and x is the value of size feature.(9.27 Fig.80 8. 9. Size = ex where e is approximately 2.

. is well formed size (desired).2 C1 + S3 Q2.17 Where. then quality is very good (rule Q1. C2 is greenish-yellow color quality medium).4 Fuzzy Rules At this stage. mD} mC » mD = (mC ⁄ mD) = max {mC. Table 9.4 C2 + S1 Q1.3 Q2.5 Q3. and it is a badly formed (small) apple. S1. The minimum method chooses the most certain output among all the membership degrees.17 in Table 9.9 Q3. C1 is the greenish color quality (desired). An example of the fuzzy AND (the minimum method) used in if-then rules to form the Q11 quality group in Table 9. OR.(9. S3 is badly formed size (bad). respectively. Finally..1 C 1 + S2 Q1.2).1 Q3.5) was used to combine the membership degrees from each rule established. mD} complement = = 1 – mD . The second subscript of Q shows the number of rules for the particular quality group. while D2 and D3 represent moderate (medium) and high (bad) amounts of defects.2: Fuzzy rule tabulation C1 + S1 D1 D2 D3 Q1. human linguistic expressions were involved in fuzzy rules. Two of the rules used to evaluate the quality of Golden Delicious apples are given below: If the color is greenish. For quality groups represented with “Q” in Table 1.13 Q3. respectively. A fuzzy set is defined by the expression below: C D = {X.15 Q3.7) .2 Q2.6 Q3. Degree of membership for any set ranges from 0 to1.1 Q2.6) .1 in Table 9. and it is a well formed large apple. m0(x))| x Œ X} m0(x): Æ [0.4 Q3. then quality is very bad (rule Q3. Three primary set operations in fuzzy logic are AND. there is no defect.3 Q3. on the other hand.5 C2 + S2 Q2.8 Q3. D1 represents a low amount of defects (desired).2). which ranges from 1 to 17 for the bad quality group. which are given as follows AND: OR: mC Ÿ mD = min {mC .2. A value of 1.5) ..14 C 3 + S3 Q3. D is a fuzzy subset in X and μD(x) is the membership function of fuzzy set D.2 is given as follows.11 Q3. while 2 and 3 stand for the moderate and bad quality groups.4) The minimum method given by equation (9.. the first subscript 1 stands for the best quality group. and C3 is yellow color quality (bad). S2 is moderately formed size (medium). then three memberships are required to express the size values in a fuzzy rule. If there are three subgroups of size.(9. 1] where X represents the universal set. ..10 C3 + S1 Q2.108 FUZZY LOGIC AND NEURAL NETWORKS 9.12 C 2 + S2 Q2.7 Q3.(9..3 Q3..6 Q3. If the color is pure yellow (overripe).7 C2 + S3 Q3.2 Q3. The rules used in the evaluations of apple quality are given in Table 9. there are a lot of defects.6. and the Complement.16 Q3.0 represents a 100% membership while a value of 0 means 0% membership..(9.

1.75 m(D3) = ( x (1) – 1. when x(1) ≥ 4. Q3. Q2.FUZZY LOGIC APPLICATIONS 109 Q11 = (C1 Ÿ S1 Ÿ D1) = min {C1.14) m(D3) = 1.. for instance. Q3. Q3.8) On the other hand...2 .(9. Q3.. D1} .77 .7 . Q1.10 . max (k1) = (Q1. S1. 3) = max {Q1.4 . when defect innput x(1) < 0. Q2.76 classes. 2. Q3. color]. 1 ⁄ Q1.(9.5 . Q3.2. These functions can be defined either by linguistic terms or numerical ranges. Q3.12) 9. If. for example.15) . Q3..1 .4. Q3. Q3. then the membership function for the class of a high amount of defects (D3) is m(D3) = 0. Q2.16 . was formed as given below: If the input vector x is given as x = [defects. the fuzzy OR (the maximum method) rule was used in evaluating the results of the fuzzy rules given in Table 9.6 m(D2) = ..9 .75 £ x(1) £ 4.6.2 .9 through 9..5 Determination of Membership Functions ( using intuition and qualitative assessment of the Membership functions are in general developed by x (1) – 0.(9. size.(9..75) .10) .12 .24 or x (1) > 7. Q2. The membership function for high amounts of defects.8 ..(9. the membership function is m(D2) = 0. Q3. Q3. determination of the quality group that an apple would belong to. when 1. was done by calculating the most likely membership degree using equations 9.9) . Q3.. 1.52 For a medium amount of defects (D2).6 ) k3 = (Q3. for instance.6 .13..3 Q3. The membership function used in this study for defect quality in general is given in equation 9. Q3.(9. when 0. . 2 ⁄ Q1.14 .3 Q2 . k1 = (Q1.11 . Q1. or both.5 .(9.52 or 2.15 .24) relations between the input variable(s) and output 1. Q1.24 £ x (1) £ 2 . 4 ... In the existence of more than one membership function that is actually in the nature of the fuzzy logic approach.2 .1 .17 ) .3 ) k2 = (Q2.. equation 11 produces the membership degree for the best class (Lee. 1990).. the challenge is to assign input data into one or more of the overlapping membership functions.(9.11) where k is the quality output group that contains different class membership degrees and the output vector y given in equation 10 below determines the probabilities of belonging to a quality group for an input sample before defuzzification: y = [max (k1) max (k2) max (k3)] where. when x (1) < 1.13) . 3} then..13 . Q3. Q1.

.

.

defects and size are three important criteria in apple classification. 9.05 0 0 2 4 6 Service 8 10 Fig.1 The Non-Fuzzy Approach Let’s start with the simplest possible relationship (Fig. is 15%. providing good flexibility in reflecting the expert’s expectations and grading standards into the results.25 0.112 FUZZY LOGIC AND NEURAL NETWORKS 9. the same system is solved using fuzzy logic.2 0. 9. . First the problem is solved using the conventional (non-fuzzy) method. It was also seen that color. fuzzy and non-fuzzy approaches are applied to the same problem. Grading results obtained from fuzzy logic showed a good general agreement with the results from the human expert.13 Constant tipping.S. internal defects and some other sensory evaluations. tip = 0. variables such as firmness. writing MATLAB commands that spell out linear and piecewise-linear relations.7. could increase the efficiency of decisions made regarding apple quality. An average tip for a meal in the U.6. Suppose that the tip always equals 15% of the total bill.15 0. However. though the actual amount may vary depending on the quality of the service provided. 9.15 Tip 0. what should the tip be? This problem is based on tipping as it is typically practiced in the United States. Consider the tipping problem: what is the “right” amount to tip your waitperson? Given a number between 0 and 10 that represents the quality of service at a restaurant (where 10 is excellent). Then. in addition to the features mentioned earlier.8 Conclusion Fuzzy logic was successfully applied to serve as a decision support technique in grading apples.13).1 0.7 AN INTRODUCTORY EXAMPLE: FUZZY V/S NON-FUZZY To illustrate the value of fuzzy logic. 9.

15 0. we might have the tip go linearly from 5% if the service is bad to 25% if the service is excellent (Fig. 9.05 10 5 Food 0 0 5 Service 10 Fig. The formula does what we want it to do. 9.1 0.20/10 * service + 0. 9.FUZZY LOGIC APPLICATIONS 113 This does not really take into account the quality of the service.20/20 ¥ (service + food) + 0. and it is pretty straight forward. Now our relation looks like this: tip = 0. so we need to add a new term to the equation.25 0. 14 Linear tipping.2 Tip 0. we may want the tip to reflect the quality of the food as well. .05 0 2 4 6 Service 8 10 Fig.1 0. Suppose we try: tip = 0.25 0. what should the tip be? Let’s see how the formula will be affected now that we’ve added another variable (Fig.05 0.15). Since service is rated on a scale of 0 to 10.15 Tipping depend on service and quality of food.05 0. 9. However. This extension of the problem is defined as follows: Given two sets of numbers between 0 and 10 (where 10 is excellent) that respectively represent the quality of the service and the quality of the food at a restaurant.2 Tip 0.15 0.14).

means that those nice linear mappings no longer apply. but when you look at them closely.e. 9. Suppose you want the service to be a more important factor than the food quality. you want to give a 15% tip in general.8.20/10 ¥ food + 0.15 0. We can still salvage things by using a piecewise linear construction (Fig. 9. else if service < =10.05 10 5 Food 0 0 5 Service 10 Fig.20/10 ¥ service + 0. they do not seem quite right.05). Suppose you want more of a flat response in the middle. i. tip = (0.15.17). tip = (0. You can string together a simple conditional statement using breakpoints like this: if service < 3.. The response is still somehow too uniformly linear. Let’s say that the service will account for 80% of the overall tipping “grade” and the food will make up the other 20%. end .16 Tipping based on the service to be a more important factor than the food quality. Let’s return to the one-dimensional problem of just considering the service. the results look pretty. tip = 0.10/3) ¥ (service –7) + 0. in turn.25 0. This. Try: servRatio = 0.15. tip= servRatio ¥ (0. 0.05.10/3) ¥ service + 0.16). else if service < 7 . 9. and will depart from this plateau only if the service is exceptionally good or bad (Fig.05) + (1– servRatio) ¥ (0.114 FUZZY LOGIC AND NEURAL NETWORKS In this case.2 Tip 0.1 0.

9. tip = ((0.15 0. tip = (0.05) ¥ servRatio + (1 – servRatio) ¥ (0.05). tip = ((0.18 Tipping with two-dimensional variation. 17 Tipping using a piecewise linear construction.05).25 0. . If we extend this to two dimensions (Fig.2 Tip 0.1 0.10/3) ¥ (service – 7) + 0. where we take food into account again. 9.20/10 ¥ food + 0. 9.20/10 ¥ food + 0.1 0. else if service < 7.FUZZY LOGIC APPLICATIONS 115 0.05).15 0.15) ¥ servRatio + (1 – servRatio) ¥ (0.2 Tip 0.20/10 ¥ food + 0.05 10 5 Food 0 0 5 Service 10 Fig. if service < 3.10/3) ¥ service + 0.18). something like this result: servRatio = 0. else. end 0.05 0 2 4 6 Service 8 10 Fig.15) ¥ servRatio + (1 – servRatio) ¥ (0.8.25 0.

then tip is generous The order in which the rules are presented here is arbitrary. .05 10 5 Food 0 0 5 Service 10 Fig. we might add the following two rules: 4.116 FUZZY LOGIC AND NEURAL NETWORKS The plot looks good. then tip is generous In fact. it is even less apparent how the algorithm works to someone who did not witness the original design process.25 0. If food is delicious.2 Tip 0. What matters is what we have shown in this preliminary exposition: fuzzy is adaptable. If service is excellent or food is delicious.1 0. things like: • How are the rules all combined? • How do I define mathematically what an “average” tip is? The details of the method do not really change much from problem to problem . If food is rancid. Moreover. If service is good. then tip is cheap 5. 9. we can combine the two different lists of rules into one tight list of three rules like so: 1. then tip is cheap 2. And coincidentally. leaving aside all the factors that could be arbitrary.19 Tipping using fuzzy logic.the mechanics of fuzzy logic are not terribly complex.7. If service is good. then tip is average 3. then tip is average 3. for example?) we would have a complete fuzzy inference system. we might end up with the following rule descriptions: 1. simple. If service is excellent. and it is definitely not easy to modify this code in the future. If we make a list of what really matters in this problem. Now if we give mathematical meaning to the linguistic variables (what is an “average” tip. 9. there’s a lot left to the methodology of fuzzy logic that we’re not mentioning right now. It was a little tricky to code this correctly.15 0. then tip is cheap 2.2 The Fuzzy Approach It would be nice if we could just capture the essentials of this problem. we have just defined the rules for a fuzzy logic system. Of course. If service is poor or the food is rancid. If we wanted to include the food’s effect on the tip. but the function is surprisingly complicated. then tip is generous These three rules are the core of our solution. and easily applied. It does not matter which rules come first. 0. If service is poor.

here is the piecewise linear tipping problem slightly rewritten to make it more generic. We found a piecewise linear relation that solved the problem.7.. we were able to add two more rules to the bottom of the list that influenced the shape of the overall output without needing to undo what had already been done. tipRange=highTip–lowTip. but it was something of a nuisance to derive.25. the maintenance of the structure of the algorithm decouples along fairly clean lines. .. greatService=10. badService=0.05. the tip should be average. and once we wrote it down as code. badFood=0.15. highTip=0. *service+lowTip)*servRatio + . On the other hand. For example. but there is a greater likelihood that recalibration will not be so quick and simple. You can recalibrate the method quickly by simply shifting the fuzzy set that defines average without rewriting the fuzzy rules. (tipRange/foodRange*food+lowTip). serviceRange=greatService–badService.. by using fuzzy logic rules.3 Some Observations Here are some observations about the example so far. tip=averTip*servRatio + (1–servRatio)* .. okayService=3. Moreover. foodRange=greatFood–badFood. averTip=0. The picture above was generated by the three rules above. greatFood=10. city to city. only now the constants can be easily changed. In other words.. the subsequent modification was pretty easy.FUZZY LOGIC APPLICATIONS 117 Here is the picture associated with the fuzzy system that solves this problem (Fig. tip=(((averTip–lowTip)/(okayService–badService)) . Also. tip is cheap if service<okayService. tip is average elseif service<goodService. % If service is poor or food is rancid. the fuzzy system is based on some “common sense” statements. 9. % If service is good. 9.. You can do this sort of thing with lists of piecewise linear functions. goodService=7. % Establish constants lowTip=0. (1–servRatio)*(tipRange/foodRange*food+lowTip). but the underlying logic the same: if the service is good. country to country. It performs the same function as before. it was not very easy to interpret. It worked.19). The notion of an average tip might change from day to day.

. United States Standards for Grades of Apples. Kickert and H. Vol. True. Automatica. end Notice the tendency here. 4. Washington. think how much more likely your code is to have comments! Fuzzy logic lets the language that’s clearest to you.J. Information and Control.118 FUZZY LOGIC AND NEURAL NETWORKS % If service is excellent or food is delicious. pp. C. Application of a fuzzy controller in a warm water plat. 1977. as with a fuzzy system.R. tip is generous If.. 1976. 10.. The truly fascinating thing to notice is that if we remove everything except for three comments. 1965.H. USDA Agricultural Marketing Service. as with all code.. W. Why use fuzzy logic? What are the applications of fuzzy logic? When not use fuzzy logic? Compare non-fuzzy logic and fuzzy logic approaches. or perhaps by trying to rewrite it in slightly more self-evident ways. D. Mamdani. pp. 4. IEEE Transactions on Systems. which is why it is a very successful technique for bridging the gap between people and machines.M. What we are doing here is not that complicated.A. (greatService–goodService))* .. for creeping generality to render the algorithm more and more opaque. Man and Cybernetics. pp.. QUESTION BANK. what remain are exactly the fuzzy rules we wrote down before: % If service is poor or food is rancid.. tip is cheap % If service is good. L. (service–goodService)+averTip)*servRatio + . 1. 338-353. 3. 1976. we can fight this tendency to be obscure by adding still more comments. tip=(((highTip–averTip)/ . 3. 1. REFERENCES. (1–servRatio)*(tipRange/foodRange*food+lowTip). 2.C. 301-308. 707-717. Vol. 4.P. A fuzzy logic controller for a traffic junction. No. the comment is identical with the code. 8. high level comments. . 7. Zadeh. Van Nauta Lemke. No. threatening eventually to obscure it completely. tip is average % If service is excellent or food is delicious. tip is generous else. Fuzzy sets. Pappis and E. 2. 12. also have meaning to the machine. Vol. but the medium is not on our side.

121-133. 255-273. Vol. Fuzzy identification and control of a liquid level rig. Vol. Transportation Research Part C. Kikuchi. pp. Newell.H. No. V. Computer. 26. Colvin and D. Modeling of driver anxiety during signal change intervals. U. 279-298. Hogans. Peleg and P. Vol. A fuzzy logic expert system for dairy cow transfer between feeding groups. Vol. 1985. pp. Vol. A fuzzy logic yield simulator for prescription farming. No. Chakroborty and H. Hanafusa. No. Vol. Fuzzy Sets and Systems. Fuzzy logic in control systems: Fuzzy logic controller. 2. No.C. 2. Lee. 27-35. 3. pp. pp. International Journal of Approximate Reasoning.G. International Journal of Systems Science. S.FUZZY LOGIC APPLICATIONS 119 5. M. Chen. pp. P. A practical application of fuzzy control for an air-conditioning system. S. 12. pp. 2272. 1999-2009. MI.M. 17. pp. 1339. 9. IEEE Transactions on Systems. Vol. Transactions of the ASAE. Takahasgi. Maltz. 2. P. Fuzzy controller robot arm trajectory. 5. 18.L.S. 1. Man and Cybernetics. Vol. 9. pp. 16.2278. Perincherry. 69-83. Vol. 1991. 28. Vol.O. 961-968. Park. 5. Westin. 1988. Electron. 1999. 1993. 7. No. 5. Colvin. T. 21. Marell and K. 2. 1994. Evaluation of cabbage seedling quality by fuzzy logic. 1993. 1992. ASAE Paper No. Fuzzy control of steam turbines. 1994. Man and Cybernetics.B. Fuzzy Sets and Systems. 13.R. 1993. 23. Palm. No. Sayyarodsari and J. B.Part I and Part II.L.C. 19. Gutman. 24. B. Song and S. 8. Hofaifar. Classification of apple surface features using machine vision and neural networks. 331-348. R. Y.S. 1990. Vol. Transactions of the ASAE. pp.S. 37. 1905-1914. Joseph. Transportation research record. J. 45. Czogala and T. 6. 23. No. pp. Fuzzy Sets and Systems. Wu. 31. pp. Karlen. E. 1994. pp. 267-276. Fuzzy control of model car. 6. Information Sciences: Applications. 1993.J. .E. Sugeno and M. 131-139. N. 10. 1992. Modeling of a fuzzy controller with application to the control of biological processes. A fuzzy multi-criteria decision making method for technology transfer strategy selection in biotechnology. Classification of fruits by a Boltzman perceptron neural network. Control of a redundant manipulator using fuzzy rules. Thangavadivelu and T. 1994. K. 15. Kiupel and P.P. Trafficability determination using fuzzy set theory. Liu and J. A Fuzzy Dynamic Learning Controller for Chemical Process Control. and No. 63. 103-113. No. Fuzzy Sets and Systems. Ben-Hannan. 943028. Ambuel. Vol. 20: 404-435. Vol. 3. pp. T. 20. Tobi and T. 131147. Vol. Kahn and E. Yang. Transactions of the ASAE. Graham and R. Nishida. 37. pp. 1. Chang and Y. T. Vol. 34. Roger. 13-22. IEEE Transactions on Systems. J. Frank. No. 7 pp. 11. No. 1993.C. Edan. Rawlik. A model for rider-motorcycle system using fuzzy control. Vol. Chen and E. Q. Intelligent Transportation System and Traffic Safety Drivers Perception and Acceptance of Electronic Speed Checkers. 2. 54. A. No. P. 16. pp. 1994. Agriculture. 1647-1654. 3. E. C. 1991. Grinspan. Fuzzy Sets and Systems. 10. A. No. 14. 1-12. No. 22. 1989. Automatica. S. USA. St. pp.

Elvik. Verma. 2000. 1999. pp. USA. pp: 849-851. 26. 33. Teodorovic. 25. Vol. and E. Fuzzy Logic Systems for Transportation Engineering: The State Of The Art. D. Tollner. . Accident Analysis and Prevention. R. Vol. Shahin.120 FUZZY LOGIC AND NEURAL NETWORKS 24. Volume: 32.A. Transportation Research Part A. M.P. 483-490. Fuzzy logic model for predicting peanut maturity. B. Transactions of the ASAE. 337-364. pp. How much do road accidents cost the national economy. 2. No. 43.W. 2000.

1 INTRODUCTION The artificial neural networks. become equal or surpass their threshold values.4 Neural Networks Fundamentals 10 10.2). received by neuron. the neurons or the processing units may have several input paths corresponding to the dendrites. Learning strategies as a basis for an adaptive system will be presented in the last section. The human brain consists of nearly 1011 neurons (nerve cells) of different types. The weighted value is passed to the neuron. In this way. it triggers sending an electric signal of constant level and duration through axon. which transmits. one can find nucleus with which the connections with other neurons are made through a network of fibres called dendrites. where it is modified by threshold function such as sigmoid function. In this chapter we first discuss these processing units and discuss different network topologies. which perform the processing. In a typical neuron. The architecture of each network is based on very similar building blocks. 10. . which we describe in this course. that is.+ 0 ) 2 6 . The modified value is directly presented to the next neuron.1). the weighted values of these paths (Fig.2 BIOLOGICAL NEURAL NETWORK The term neural network comes from the intended analogy with the functioning of the human brain adopting simplified models of biological neural network. 10. are all variations on the parallel distributed processing (PDP) idea. the message is transferred from one neuron to the other. When signals. Extending out from the nucleus is the axon. In the neural network. 10. electric potentials to the neurons. The units combine usually by a simple summation. by means of complex chemical process. with which the axon is connected to (Fig.

122 FUZZY LOGIC AND NEURAL NETWORKS Dendrite Cell body Nucleus Myelin sheath Axon Nerve ending Synapse Fig. 10.. cells). 10. . which determines the new level of activation based on the effective input sk(t) and the current activation yk(t) (i.e. X1 X2 Wij W2j 1 2 Wij Xi Xn Fig. the update). a state of activation yk for every unit.2 Wnj Schematic representation of mathematical neuron network. connections between the units. which determines the effective input sk of a unit from its external inputs. an activation function Fk. 10. which is equivalent to the output of the unit. which communicate by sending signals to each other over a large number of weighted connections. a propagation rule. Generally each connection is defined by a weight wjk which determines the effect which the signal of unit j has on unit k.3 A FRAMEWORK FOR DISTRIBUTED REPRESENTATION An artificial network consists of a pool of simple processing units.1 Schematic representation of biological neuron network. A set of major aspects of a parallel distributed model can be distinguished as: a set of processing units (neurons.

output units (indicated by an index o) which send data out of the neural network.. In some cases the latter model has some advantages.3 The basic components of an artificial neural network. units can be updated either synchronously or asynchronously. The total input to unit k is simply the weighted sum of the separate outputs from each of the connected units plus a bias or offset term qk: sk(t) = åw j jk (t) yj(t) + qk(t) . providing input signals and if necessary error signals. each unit has a (usually fixed) probability of updating its activation at a time t. a second task is the adjustment of the weights. 10. Within neural systems it is useful to distinguish three types of units: input units (indicated by an index i) which receive data from outside the neural network. which is propagated to other units. a method for information gathering (the learning rule).(10. some of which will be discussed in the next sections. and hidden units (indicated by an index h) whose input and output signals remain within the neural network.NEURAL NETWORKS FUNDAMENTALS 123 an external input (aka bias. and usually only one unit will be able to do this at a time. Apart from this processing.3.3. an environment within which the system must operate. The propagation rule used here is the standard weighted summation.3 illustrates these basics.1 Processing Units Each unit performs a relatively simple job: receive input from neighbors or external sources and use this to compute an output signal. with asynchronous updating. The system is inherently parallel in the sense that many units can carry out their computations at the same time. w w wjk w j yj qk Sk = k å j w jk y j + q k Fk yk Fig.2 Connections between Units In most cases we assume that each unit provides an additive contribution to the input of the unit with which it is connected. During operation. offset) Gk for each unit. all units update their activation simultaneously.1) . 10. With synchronous updating. Figure 10.. 10.

In some applications a hyperbolic tangent is used.3) F w GH å j jk (t ) y j (t ) + q k (t ) I JK .4 Various activation functions for a unit.(10. introduced by Feldman and Ballard. some sort of threshold function is used: a hard limiting threshold function (a sgn function). yk = F(sk) = 1 1 + e sk . as well as implementation of lookup tables. which gives the effect of the total input on the activation of the unit.124 FUZZY LOGIC AND NEURAL NETWORKS The contribution for positive wjk is considered as an excitation and for negative wjk as inhibition.3. Although these units are not frequently used. sk (t)). We need a function Fk which takes the total input sk(t) and the current activation yk(t) and produces a new value of the activation of the unit k: yk(t + 1) = Fk ( yk(t).(10.(10... but the neuron input determines the probability p that a neuron get a high activation value: . 10.5) is used. or a linear or semi-linear function.1) sigma units.(10. the output of a unit can be a stochastic function of the total input of the unit. 10. the yjm are weighted before multiplication. We call units with a propagation rule (10. Generally..2) Often.. 10. the activation function is a non-decreasing function of the total input of the unit: yk(t + 1) = Fk ( sk(t)) = Fk . or a smoothly limiting threshold (see Fig.3 Activation and Output Rules We also need a rule. they have their value for gating of input... In some cases more complex rules for combining inputs are used. In that case the activation is not deterministically determined by the neuron input.4) although activation functions are not restricted to non-decreasing functions. In some cases.. yielding output values in the range [1.4). in which a distinction is made between excitatory and inhibitory inputs. is known as the propagation rule for the sigma-pi unit is given by sk(t) = åw j jk (t) Õy m jm (t) + qk(t) . Often. A different propagation rule. For this smoothly limiting function often a sigmoid (S-shaped) function like Sgn i Semi-linear i Sigmoid i Fig.. +1].

. such that the dynamical behavior constitutes the output of the network. and Hopfield and will be discussed in subsequent chapters. where the data flow from input to output units is strictly feed-forward. In some cases. the activation values of the units undergo a relaxation process such that the network will evolve to a stable state in which these activations do not change anymore. Kohonen. As for this pattern of connections. or by the system. using a priori knowledge. These are: · Supervised learning or Associative learning in which the network is trained by providing it with input and matching output patterns. Various methods to set the strengths of the connections exist. The data processing can extend over multiple (layers of) units. connections extending from outputs of units to inputs of units in the same layer or previous layers.1 Paradigms of Learning We can categorize the learning situations in two distinct sorts. the dynamical properties of the network are important. One way is to set the weights explicitly. 10. which contains the network (self-supervised). the main distinction we can make is between: Feed-forward networks. 10.4 NETWORK TOPOLOGIES In the previous section we discussed the properties of the basic processing unit in an artificial neural network. These input-output pairs can be provided by an external teacher.5 TRAINING OF ARTIFICIAL NEURAL NETWORKS A neural network has to be configured such that the application of a set of inputs produces (either direct or via a relaxation process) the desired set of outputs. the change of the activation values of the output neurons are significant. In other applications. but no feedback connections are present.(10.6) in which T (temperature) is a parameter which determines the slope of the probability function. Examples of recurrent networks have been presented by Anderson. In this paradigm the system is supposed to discover . Contrary to feed-forward networks. In all networks we consider that the output of a neuron is to be identical to its activation level. Another way is to train the neural network by feeding it teaching patterns and letting it change its weights according to some learning rule.NEURAL NETWORKS FUNDAMENTALS 125 p(yk ¬ 1) = 1 1 + e sk / T . Recurrent networks that do contain feedback connections.5.. which will be discussed in the next chapter. This section focuses on the pattern of connections between the units and the propagation of data. that is. 10. Classical examples of feed-forward networks are the Perceptron and Adaline. · Unsupervised learning or Self-organization in which an (output) unit is trained to respond to clusters of pattern within the input.

g. their interconnection must be strengthened. often: the input of the network by clamping input pattern vector p.7) where g is a positive constant of proportionality representing the learning rate. presented to the network). o an output unit. . p is often not necessary) or added (e. and will be discussed in the next chapter. the simplest version of Hebbian learning prescribes to modify the weight wjk with Dwjk = g yj yk .. h a hidden unit. Note that not all symbols are meaningful for all networks. i an input unit. d p the jth element of the desired output of the network when input pattern vector p was input to the j network. If j receives input from k. In the next chapters some of these update rules will be discussed. The basic idea is that if two units j and k are active simultaneously.126 FUZZY LOGIC AND NEURAL NETWORKS statistically salient features of the input population. d p the desired output of the network when input pattern vector p was input to the network. k. Virtually all learning rules for models of this type can be considered as a variant of the Hebbian learning rule.2 Modifying Patterns of Connectivity Both learning paradigms discussed above result in an adjustment of the weights of the connections between units. 10.. there is no a priori set of categories into which the patterns are to be classified rather the system must develop its own representation of the input stimuli. k. Many variants (often very exotic ones) have been published the last few years. x p the jth element of the pth input pattern vector.. 10.. Vectors are indicated with a bold non-slanted font: j. This is often called the Widrow-Hoff rule or the delta rule.1 Notation NOTATION AND TERMINOLOGY We use the following notation in our formulae. contrariwise to the notation below. and that in some cases subscripts or superscripts may be left out (e..6. .(10. Another common rule uses not the actual activation of unit k but the difference between the actual and desired activation for adjusting the weights: Dwjk = g yj (dk yk) . j s p the input to a set of neurons when input pattern vector p is clamped (i.e. . have indices) where necessary. the unit j. x p the pth input pattern vector. Unlike the supervised learning paradigm.5.8) in which dk is the desired activation provided by a teacher.(10. vectors can. according to some modification rule.6 10.g...

They may be used interchangeably. This convention is widely though not yet universally used. Furthermore. That is. q the biases to the units. the output of each neuron equals its activation value. threshold: These terms all refer to a constant (i. and even for an optimal set of weights the approximation error is not zero. Bias. Since there is no need to do otherwise. in most cases the network will only approximate the desired function. activation of a unit. W the matrix of connection weights. Number of layers: In a feed-forward network.e. Because a neural network is built from a set of standard functions. the second one is the learning algorithm. one hidden layer. gjk the learning rate associated with weight wjk . and one output layer is referred to as a network with two layers. y p the activation values of element j of the network when input pattern vector p was input to the j network.2 Terminology Output vs.. we consider the output and the activation value of a unit to be one and the same thing. Representation vs. Given that there exist a set of optimal weights in the network. wjk the weight of the connection from unit j to unit k.6. offset. The first one is the representational power of the network. learning: When using a neural network one has to distinguish two issues which influence the performance of the system. although the latter two terms are often envisaged as a property of the activation function. Ep the error in the output of the network when input pattern vector p is input. this external input is usually implemented (and can be written) as a weight from a unit with activation value 1. 10. A the energy of the network. is there a procedure to (iteratively) find this set of weights? . the inputs perform no computation and their layer is therefore not counted. The representational power of a neural network refers to the ability of a neural network to represent a desired function.NEURAL NETWORKS FUNDAMENTALS 127 y p the activation values of the network when input pattern vector p was input to the network. Fj the activation function associated with unit j. independent of the network input but adapted by the learning rule) term which is input to a unit. Uj the threshold of unit j in Fj . wj the weights of the connections which feed into unit j. qj the bias input to unit j. Thus a network with one input layer. The second issue is the learning algorithm.

The MIT Press. 3. 9. D. No.H. pp.C: Spartan Books. 1. What are the paradigms of neural network learning? REFERENCES. The Organization of Behaviour. 5. 6. pp. Washington. G. J. Widrow. 6. Neural networks and physical systems with emergent collective computational abilities. Mel. 1989. T. 1986. Basic Processes in Reading Perception and Comprehension Models (pp. Hebb. What are the basic components of artificial neural network? What are the network topologies? What are the various activation function? Explain them schematically.J. Connectionist Robot Motion Planning. 1977. Hopfield. New York: Wiley. Jovitz. McClelland. 5. 205-254. 1990.A. Vol. 2554-2558. 263-269. 79. B.O. Associative Memory: A System-Theoretical Approach. 2. Ballard. LaBerge and S. 2.C. CA: Academic Press. Connectionist models and their properties. 1982. 27-90). What are the major aspects of parallel distributed model? Explain the biological neural network. Vol. . D. Generalization and Information Storage in Networks of Adaline Neurons. 4.T. Anderson. 7. 1. Hillsdale. 435-461. M. in Self Organizing Systems 1962. Proceedings of the National Academy of Sciences. G.J.128 FUZZY LOGIC AND NEURAL NETWORKS QUESTION BANK. 1982. J. 1. 1977.). pp. Samuels (Eds. 6. Goldstein. San Diego. J.A. Neural models with cognitive implications. Feldman.. and D. 4. B. Kohonen. Cognitive Science. In D. Rumelhart and J. 1962.E.A. 3. NJ: Erlbaum. D.W. ed. Vol. 1949. 2. 8. Pearlmutter.L. PP. Neural Computation. Jacobi. Springer-Verlag. B. Learning state space trajectories in recurrent neural networks. Parallel Distributed Processing: Explorations in the Microstructure of Cognition.

1 (we leave the output index o out).1 Single layer network with one output and two inputs. 11. which is some function of the input: X1 W1 y X2 W2 q +1 Fig. as sketched in Fig. Two classical models will be described in the first part of the chapter: the Perceptron. including some of the classical approaches to the neural computing and learning problem. In the second part we will discuss the representational limitations of single layer networks. proposed by Rosenblatt and the Adaline.2 NETWORKS WITH THRESHOLD ACTIVATION FUNCTIONS A single layer feed-forward network consists of one or more output neurons o. each of which is connected with a weighting factor wio to all of the inputs i.4 Perceptron and Adaline 11 11.+ 0 ) 2 6 .1 INTRODUCTION This chapter describes single layer neural networks. 11. The output of the network is formed by the activation of the output neuron. In the first part of this chapter we discuss the representational power of the single layer networks and their learning algorithms and will give some examples of using the networks. In the simplest case the network has only two inputs and a single output. presented by Widrow and Hoff. The input of the neuron is the weighted sum of the inputs plus the bias term. . 11.

In this section we consider the threshold (sgn) function: F(s) = R+1 S1 T if s > 0 otherwise . the pattern will be assigned to class +1...2 Geometric representation of the discriminant function and the weights. A geometrical representation of the linear threshold neural network is given in Fig..e.130 FUZZY LOGIC AND NEURAL NETWORKS y=F F I GH å w x + G JK 2 i i i =1 .. . The network can now be used for a classification task: it can decide whether an input pattern belongs to one of two classes. depending on the input.3) The single layer network represents a linear discriminant function..(11. if the total input is negative.. Note that also the weights can be plotted in the input space: the weight x2 + + w1 + w2 –q ||W|| x1 + + + + Fig. Equation (11.. given by the equation: w1x1 + w2 x2 + q = 0 . The separation between the two classes in this case is a straight line. the sample will be assigned to class +1.(11. i. If the total input is positive.(11.2) The output of the network thus is either +1 or 1..2. 11.(11.3) can be written as x2 = w1 q x1 w2 w2 . how far the line is from the origin.1) The activation function F can be linear so that we have a linear network.4) and we see that the weights determine the slope of the line and the bias determines the offset. 11. or non-linear.

3.. the perceptron learning rule will converge to some solution (which may or may not be the same as w*) in a finite number of steps for any initial choice of the weights. Note that the procedure is very similar to the Hebb rule. 2. A learning sample is presented to the network.PERCEPTRON AND ADALINE 131 vector is always perpendicular to the discriminant function.(11. Besides modifying the weights.. The perceptron learning rule is very simple and can be stated as follows: 1. Select an input vector x from the set of training samples. Go back to 2. will be greater than 0 or: there exists a d > 0 such that |w* o x| > d for all inputs x..(11. . 3. modify all connections wi according to: Dwi = d(x)xi. For each weight the new value is computed by adding a correction to the old value.2 Convergence Theorem For the learning rule there exists a convergence theorem.. we come to the second issue: how do we learn the weights and biases in the network? We will describe two learning methods for these types of networks: the perceptron learning rule and the `delta or `LMS rule..3 PERCEPTRON LEARNING RULE AND CONVERGENCE THEOREM 11.5) . the value |w* o x|.1 Perceptron Learning Rule Suppose we have a set of learning samples consisting of an input vector x and a desired output d(x). If y ¹ d(x) (the perceptron gives an incorrect response).3. For a classification task the d(x) is usually +1 or 1. The threshold is updated in a same way: wi(t + 1) = wi(t) + Dwi(t) q(t + 1) = q(t) + Dq(t) . Start with random weights for the connections. no connection weights are modified. This q is considered as a connection w0 between the output neuron and a dummy predicate unit which is always on: x0 = 1. this threshold is modified according to: Dq = R0 Sd ( x) T if the perceptron responds correctly otherwise . Because w*is a correct solution.(11.7) 11. Now that we have shown the representational power of the single layer network with linear threshold units. we take ||w*|| = 1. when the network responds correctly. Both methods are iterative procedures that adjust the weights. 4. which states the following: If there exists a set of connection weights w* which is able to perform the transformation y = d(x). Given the perceptron learning rule as stated above. the only difference is that.. we must also modify the threshold q. where o denotes dot or inner product.6) The learning problem can now be formulated as: how do we compute Dwi(t) and Dq(t) in order to classify the larning patterns correctly? 11. Proof: Given the fact that the length of the vector w* does not play a role (because of the sgn operation).

.3. If we start with connections w = 0..(11.132 FUZZY LOGIC AND NEURAL NETWORKS Now define cos a = w o w* .1: A perceptron is initialized with the following weights: w1 = 1. tmax = M d2 . .8) Example 11. || w|| When according to the perceptron learning rule. q = 2. after maximally tmax modifications of the weights the perceptron is correctly performing the mapping. we know that Dw = d(x)x. the system modifies its connections only a limited number of times. From this it follows that: w¢ o w* = w o w* + d(x) o w* o x = w o w* + sgn(w* o x) w* o x > w o w* + d ||w¢||2 = ||w + d(x)x||2 = w2 + 2d (x) w o x + x2 < w2 + x2 =w +M After t modifications we have: w(t) o w* > w o w* + td ||w(t)||2 < w2 + tM such that cos a(t) = 2 (because d (x) = sgn [w o x]) w* o w(t ) || w(t )|| w* o w + td w2 + tM > From this follows that limt®¥ cos a(t) = limt®¥ d M t = ¥ while cos a £ 1. w2 = 2. The conclusion is that there must be an upper limit tmax for t. The perceptron learning rule is used to learn a correct discriminant function for a number of samples. 11. connection weights are modified at a given input x. sketched in Fig. tmax will be reached when cos a = 1. In other words. and the weight after modification is w¢ = w + Dw.

1) it can be calculated that the network output is +1. 11. which can sum up currents caused by the input voltage signals. In Fig. developed by Widrow and Hoff. In a simple physical implementation (Fig. 11. The same is the case for point B. According to the perceptron learning rule. The delta-rule uses the net output without further mapping into output values 1 or +1. with values x = (0:5. i = 0. When presenting point C with values x = (0:5.. Dw2 = 0:5. respectively. and the input and output signals by xi and y. n. and sample C is classified correctly. From equation (11. also known as the delta rule. then the output of the central block is defined to be . The learning rule was applied to the adaptive linear element. The first sample A. also named Adaline. 11. The perceptron learning rule uses the output of the threshold function (either 1 or +1) for learning. The main functional di_erence with the perceptron training rule is the way the output of the system is used in the learning rule. with values x = (0:5. 0:5) the network output will be 1. while the target value d(x) = +1. so no change. so no weights are adjusted.3 the discriminant function before and after this weight update is shown. which outputs either +1 or 1. 11..4 ADAPTIVE LINEAR ELEMENT (Adaline) An important generalisation of the perceptron training algorithm was presented by Widrow and Hoff as the least mean square (LMS) learning procedure. 0:5) and target value d(x) = -1. q = 1. depending on the polarity of the sum. it may be clear that a system with many parallel outputs is directly implementable by multiple units of the above kind. q = 1. the weight changes are: Dw1 = 0:5. is also followed by a quantiser. If the input conductances are denoted by wi. Although the adaptive process is here exemplified in a case when there is only one output. Usually the central block.. the network output is negative.3 Discriminant function before and after weight update.PERCEPTRON AND ADALINE 133 x2 2 + A 1 B +C 1 2 x1 Original discriminant function After weight update + Fig. w2 = 2:5..4) this device consists of a set of controllable resistors connected to a circuit. 1:5) and target value d(x) = +1 is presented to the network. the summer. The new weights are now: w1 = 1:5. 1.

but here we focus on the linear relationship and use the network for a function approximation task. usually iteratively. to attain the correct values.9) where q = w0. By thresholding the output value.. .. in the sense of least squares. An adaptive operation means that there exists a mechanism by which the wi can be adjusted. . The problem is to determine the coeficients wi. n. for instance. 11..(11.10) Such a simple network is able to represent a linear relationship between the value of the output unit and the value of the input units. i = 0. The purpose of this device is to yield a given value y = d p at its output when the set of values xip i = 1. Suppose we want to train the network such that a hyperplane is fitted as well as possible to a set of training samples consisting of input values x p and desired (or target) output values d p.. n Level w0 Output y= åw x i =1 i i +q . 2. a classifier can be constructed (such as Adaline). in such a way that the input-output response is correct for a large number of arbitrarily chosen signal sets. In high dimensional input spaces the network represents a (hyper) plane and it will be clear that also multiple output units may be defined.5 THE DELTA RULE For a single layer network with an output unit with a linear activation function the output is simply given by y= åw x j j j +q . Widrow introduced the delta rule to adjust the weights. . is applied at the inputs.134 FUZZY LOGIC AND NEURAL NETWORKS +1 –1 +1 w1 w2 w3 Gains Input pattern switches S – Summer Error S Quantizer + – 1 +1 Reference switch Fig. For every given input sample. n. the average error must be minimised. The delta-rule now uses a cost-or error-function based on these differences to adjust the weights. where y p is the actual output for this pattern. 11. the output of the network differs from the target value d p by (d p y p).4 The adaline. For the Adaline. 1. If an exact mapping is not possible.(11. .

. These characteristics have opened up a wealth of new applications.16) where d = d y is the difference between the target output and the actual output for pattern p...14) ¶E p = (d p y p) ¶y p such that Dp wj = g d p xj p p p .. The delta rule modifies weight appropriately for target and actual outputs of either polarity and for both continuous and binary input and output units.6 EXCLUSIVE-OR PROBLEM In the previous sections we have discussed two learning algorithms for single layer networks. the total error E is defined to be E= åE p p = 1 2 å (d p p y p )2 . as indicated by the name least mean square. (11.(11.. The idea is to make a change in the weight proportional to the negative of the derivative of the error as measured on the current pattern with respect to each weight: Dp wj = g ¶E p ¶w j . .13) ¶y p = xj ¶w j and .11) where the index p ranges over the set of input patterns and E p represents the error on pattern p.15) .(11..12) where g is a constant of proportionality.(11. That is.10). The LMS procedure finds the values of all the weights that minimize the error function by a method called gradient descent.. but we have not discussed the limitations on the representation of these networks. eq. ... is the summed squared error.(11. 11.(11. ¶w j ¶y p ¶w j Because of the linear units.. The derivative is ¶E p ¶E p ¶y p = ..(11..PERCEPTRON AND ADALINE 135 The error function.

5. 1) cannot be separated by a straight line from the two open circles at (1.6a demonstrates that the four input points are now embedded in a three-dimensional space defined by the two inputs plus the single hidden unit. – 1) OR (1.1 Exclusive-or truth table. To see that such a solution cannot be found. the output of the perceptron is zero when s is negative and equal to one when s is positive. take a loot at Fig. 1) and (1... 11. the problem can be solved. x1 (– 1.5 Geometric representation of input space For the specific XOR problem we geometrically show that by introducing hidden units.1).17) According to eq.1. the net input is equal to: s = w1x1 + w2x2 + q . The obvious question to ask is: How can this problem be overcome? Minsky and Papert prove that for binary inputs. as depicted in Fig. as desired. 11. any transformation can be carried out by adding a layer of predicates which are connected to all inputs. – 1) XOR Fig. Table 3. The input space consists of four points. 1). 11. (11. the output of the perceptron is equal to one on one side of the dividing line which is defined by: w1x1 + w2x2 = q . The proof is given in the next section. This simple example demonstrates that adding hidden units increases the class of .136 FUZZY LOGIC AND NEURAL NETWORKS Table 11. thereby extending the network to a multi-layer perceptron.(11. 1) x1 x2 x2 ? ? x2 And (– 1.5 a geometrical representation of the input domain is given.. N 1 1 1 1 N 1 1 1 1 @ 1 1 1 1 One of Minsky and Paperts most discouraging results shows that a single layer perceptron cannot represent a simple exclusive-or function.. 11.(11.18) and equal to zero on the other side of this line. For a constant q. 1) and (1. In a simple network with two inputs and one output. 1) x1 (1. These four points are now easily separated by a linear manifold (plane) into two groups. and the two solid circles at (1. 11.1 shows the desired relationships between inputs and output units for this function. Fig. In Fig.

clearly.5 a. the XOR problem can be solved. – 1.like networks. 11. For binary units.(11. (1.. we can divide the set of all possible input vectors into two classes: X + = {x|d(x) = 1} and X = {x|d(x) 1} .. 1. Similarly.7 Solution of the XOR problem. separation (by a linear manifold) into the required groups is now possible..6 onto the four points indicated here. one can prove that this architecture is able to perform any transformation given the correct connections and weights. For a given transformation y = d(x).(11. perceptron..1 with an extra hidden unit. The most primitive is the next one..PERCEPTRON AND ADALINE 137 problems that are soluble by feed-forward. – 0.20) is equal to 1 for xp = wh only. 1) 1 1 1 1 (– 1. For every x p Î X+ a hidden unit h can be reserved of which the activation yh is 1 if and only if the specific pattern p is present at the input: we can choose its weights wih equal to the specific pattern xp and the bias qh equal to 1 . With the indicated values of the weights wij (next to the connecting lines) and the thresholds qi (in the circles) this perceptron solves the XOR problem.21) . the weights to the output neuron can be chosen such that the output is one as soon as one of the M predicate neurons is one: p y o = sgn F y GH å M h =1 h +M 1 2 I JK . by this generalization of the basic architecture we have also incurred a serious loss: we no longer have a learning rule to determine the optimal weights.5 – 1 – 0. (b) This is accomplished by mapping the four points of Fig. Fig..19) Since there are N input units. (a) The perceptron of Fig. However. 11. 11.7 MULTI-LAYER PERCEPTRONS CAN DO EVERYTHING In the previous section we showed that by adding an extra hidden unit. the total number of possible input vectors x is 2N. – 1) b. 11.N such that p y h = sgn F wx GH å i p ih i N+ 1 2 I JK .(11.

1. 1949. 3. 1969. Widrow. Papert. Rosenblatt. which is equal to the number of patterns in X +. 5. Explain Adaline neural network. The MIT Press.E. The Organization of Behaviour. . Describe the perceptron learning rule. Explain the delta rule used to adjust the weights of Adaline network. 7. which is maximally 2N-1. Hebb.O. 1960. F. 2. A more elegant proof is given by Minsky and Papert. Adaptive Switching Circuits. Derive the convergence theorem for perceptron learning rule. 3. and we will always take the minimal number of mask units. D. B. Explain single layer neural network with one output and two inputs. What are the advantages of multiplayer perceptron over single layer perceptron? REFERENCES. New York: Spartan Books. Perceptrons: An Introduction to Computational Geometry. Hoff. 4. 4. M. Single layer perceptron cannot represent exclusive-OR. Minsky. which is maximally 2N. 6. The problem is the large number of predicate units. and M. Justify this statement. Dunno. Principles of Neurodynamics. 2. Of course we can do the same trick for X . and S. 1. QUESTION BANK.138 FUZZY LOGIC AND NEURAL NETWORKS This perceptron will give y0 = 1 only if x Î X+: it performs the desired mapping. New York: Wiley. but the point is that for complex transformations the number of required units in the hidden layer is exponential in N. 1959. In 1960 Ire Wescon Convention Record.

1985). 1989. Back-propagation can also be considered as a generalization of the delta rule for non-linear activation functions and multilayer networks. no processing takes place in these units. 1 INTRODUCTION As we have seen in the previous chapter. The Ni inputs are fed into the first layer of Nh. & Kowalski. The input units are merely fan-out units. until the last layer of hidden units. of which the outputs are fed into a layer of No output units (see Fig. 12. . which receive their input from units from a layer directly below and send their output to units in a layer directly above the unit. 12. Funahashi. For this reason the method is often called the back-propagation learning rule. Hornik. a single-layer network has severe restrictions: the class of tasks that can be accomplished is very limited. 1985. Keeler. but did not present a solution to the problem of how to adjust the weights from input to hidden units. (10.2 MULTI . 1989. provided the activation functions of the hidden units are non-linear (the universal approximation theorem).1). Hartman. 1990) that only one layer of hidden units suffices to approximate any function with finitely many discontinuities to arbitrary precision. Minsky and Papert showed in 1969 that a two layer feed-forward network can overcome many restrictions.LAYER FEED . as given in eq. & White. Although back-propagation can be applied to networks with any number of layers. Cun. An answer to this question was presented by Rumelhart.FORWARD NETWORKS A feed-forward network has a layered structure. 1 hidden units. In this chapter we will focus on feed forward networks with layers of processing units. Each layer consists of units. The central idea behind this solution is that the errors for the units of the hidden layer are determined by back-propagating the errors of the units of the output layer. Stinchcombe. Hinton and Williams in 1986. and similar solutions appeared to have been published earlier (Parker. The output of the hidden units is distributed over the next layer of Nh. 1989. There are no connections within a layer. The activation of a hidden unit is a function Fi of the weighted inputs plus a bias.4).7) it has been shown (Cybenko. 2 hidden units. just as for networks with binary units (section 11.C H A P T E R Back-Propagation 12 12.

(12. In most applications a feed-forward network with a single layer of hidden units is used with a sigmoid activation function for the units.. (12.. given by p p yk = F(Sk) ..4) .. 12.3 THE GENERALISED DELTA RULE Since we are now using units with nonlinear activation functions. we must set Dpwjk = g ¶E p ¶w jk .140 FUZZY LOGIC AND NEURAL NETWORKS h Ni o N0 Nh.3) The error E p is defined as the total quadratic error for pattern p at the output units: Ep = 1 o p p do yo 2 o =1 åd N i 2 . we have to generalise the delta rule.. The activation is a differentiable function of the total input..1) in which p sk = åw j p jk yk + qk .1 A multi-layer network withlayers of units. 12.(12.1 Nh1–1 Nh1–2 Fig...(12.2) To get the correct generalization of the delta rule as presented in the previous chapter. which was presented in chapter 11 for linear functions to the set of non-linear activation functions.

. it follows from the definition of E p that ¶E p p p p = (do yo) ¶yo . evaluated at the net input Sk to that unit.10) p which is simply the derivative of the squashing function F for the kth unit.6) ¶E p ¶Skp .11) which is the same result as we obtained with the standard delta rule.. (12.1) we see that ¶ykp p = F(Sk) p ¶S k ....9). we get .8) p The trick is to figure out what dk should be for each unit k in the network.(12.(12... one factor reflecting the change in error as a function of the output of the unit and one reflecting the change in the output as a function of changes in the input. we have p dk = ¶E p ¶E p ¶ykp = ¶Skp ¶ykp ¶S kp .(12. is that there is a simple recursive computation of these ds which can be implemented by propagating error signals backward through the network.. In this case.BACK-PROPAGATION 141 p where do is the desired output for unit 0 when pattern p is clamped.7) we will get an update rule which is equivalent to the delta rule as described in the previous chapter.10) in equation (12.. We further set E = åE p p as the summed squared error.9).(12..2) we see that the second factor is . By equation (12.. Substituting this and equation (12.... Thus. We can write ¶E p ¶E p ¶Skp = ¶w jk ¶Skp ¶w jk By equation (12. which we now derive. To compute the first factor of equation (12.(12. p To compute dk we apply the chain rule to write this partial derivative as the product of two factors. assume that unit k is an output unit k = o of the network. resulting in a gradient descent on the error surface if we make the weight changes according to: p Dpwjk = gdk yjp . The interesting result. First.5) ¶Skp = yjp ¶w jk When we define p dk = .9) Let us compute the second factor.(12. we consider two cases.

we do not readily know the contribution of the unit to the output error of the network. Differently put.) and we use the chain rule to write ¶E p p = ¶yh ¶E p ¶Sop = p ¶Shp o = 1 ¶So å No ¶E p ¶ p p ¶yh o = 1 ¶So å No å No wko y jp j =1 ¶E p = w = p ho j = 1 ¶So å No åd j =1 No p o who . in order to reduce an error.14) give a recursive procedure for computing the ds for all units in the network.142 FUZZY LOGIC AND NEURAL NETWORKS p p p do = (do yo ) Fo' (S p ) o . the weights from input to hidden units are never changed.12) and (12... When a learning pattern is clamped. In this case. a hidden unit h receives a delta from each output unit o equal to the delta of that output unit weighted with (= multiplied by) the weight of the . But it alone is not enough: when we only apply this rule.. which are then used to compute the weight changes according to equation (12.. next time around. but what do they actually mean? Is there a way of understanding back-propagation other than reciting the necessary equations? The answer is.. the whole back-propagation process is intuitively very clear.. yes. we usually end up with an error in each of the output units.. and the actual network output is compared with the desired output values. The simplest method to do this is the greedy method: we strive to change the connections in the neural network in such a way that. the error eo will be zero for this particular pattern.13) Substituting this in equation (12.1 Understanding Back-Propagation The equations derived in the previous section may be mathematically correct. if k is not an output unit but a hidden unit k = h.14) Equations (12. This is solved by the chain rule which does the following: distribute the error of an output unit o to all the hidden units that is it connected to.15) That is step one. (12. What happens in the above equations is the following. We know from the delta rule that.. we have to adapt its incoming weights according to Dwho = (d° y°) yh .(12.. s2 . weighted by this connection. of course.12) for any output unit o.. we again want to apply the delta rule..3. 12. and we do not have the full representational power of the feed-forward network as promised by the universal approximation theorem. This procedure constitutes the generalized delta rule for a feed-forward network of non-linear units.. sjp.. In fact. Lets call this error eo for a particular output unit o.. the activation values are propagated to the output units. (12. we do not have a value for d for the hidden units.. However.(12. We have to bring eo to zero. Secondly.8).9) yields p d h = F(Shp ) å@ j =1 No p o who . In order to adapt the weights from input to hidden units. however. the error measure can be written p p as a function of the net inputs from hidden to output layer Ep = Ep (s1.

12.(12.18) In this case the derivative is equal to ¶ 1 1 F ¢(S ) = p = e ¶S p 1 + e s 1 + e s p e j 2 e e j = sp 1 e e j p sp e1 + e j e s 2 1+ e sp j = y p(1 y p ) . before the back-propagation process can continue.(12. In symbols: dh = åd w 0 0 ho Well.16) ..17) Take as the activation function F the sigmoid function as defined in chapter 2: y p = F(S p ) = 1 1 + e s p ..20) The error signal for a hidden unit is determined recursively in terms of error signals of the units to which it directly connects and the weights of those connections.(12.(12.4. on the unit k receiving the input and the output of the unit j sending this signal along the connection: Dpwkj = gdkp yjp If the unit is an output unit.(12... the error signal is given by p p p p do = (do y o ) Fo' (S o ) . resulting in an error signal do for each output unit.. This output is compared with its desired value do.4 WORKING WITH BACK-PROPAGATION The application of the generalised delta rule thus involves two phases: During the first phase the input p x is presented and propagated forward through the network to compute the output values y o for each p output unit. not exactly: we forgot the activation function of the hidden unit..19) such that the error signal for an output unit can be written as: p p p p p do = (do yo ) y o o (1 yo ) .. For the sigmoid activation function: .. The second phase involves a backward pass through the network during which the error signal is passed to each unit in the network and appropriate weight changes are calculated.. 12.BACK-PROPAGATION 143 connection between those units.1 Weight Adjustments with Sigmoid Activation Function The results from the previous section can be summarised in three equations: The weight of a connection is adjusted by an amount proportional to the product of an error signal d. F ¢ has to be applied to the delta..

and (c) with large learning rate and momentum term added.. i. 12... when using the same sequence over and over again the network may become focused on the first few patterns.3 Learning Per Pattern Although. however. True gradient descent ¶w requires that infinitesimal steps are taken. . .(12. (b) for large learning rate: note the oscillations. The role of the momentum term is shown in Fig. the back-propagation algorithm performs gradient descent on the total error only if the weights are adjusted after the full set of learning patterns has been presented. it takes a long time before the minimum has been reached with a low learning rate. Care has to be taken. When adding the momentum term. 12. more often than not the learning rule is applied to each pattern separately. the minimum will be reached faster..4. E p is calculated. 2. 12. The constant of proportionality is the learning rate g.e. whereas for high learning rates the minimum is never reached because of the oscillations. For example. P). There exists empirical indication that this results in faster convergence.2 Learning Rate And Momentum The learning procedure requires that the change in weight is proportional to ¶E p .2 The descent in weight space. (a) for small learning rate. theoretically.21) 12.2. b a c Fig. One way to avoid oscillation at large. a pattern p is applied. with the order in which the patterns are taught. is to make the change in weight dependent of the past weight change by adding a momentum term: p Dwjk (t + 1) = gdk yjp+ aDwjk (t) . When no momentum term is used.4.144 FUZZY LOGIC AND NEURAL NETWORKS p p d h = F ¢(S h ) å No p p dop who = y h(1 y h ) j =1 åd j =1 No p o who .. and the weights are adapted (p = 1. This problem can be overcome by using a permuted training method.(12. For practical purposes we choose a learning rate that is as large as possible without leading to oscillation.22) where t indexes the presentation number and a is a constant which determines the effect of the previous weight change.

.000 learning iterations with the back-propagation training rule. 12.3 Example of function approximation with a feed forward network. The relationship between x and d as represented by the network is shown in Fig. The input of the system is given by the two-dimensional vector x and the output is given by the one-dimensional vector d. Check for yourself how equation (4. Bottom left: The function which generated the learning samples. described in the previous section. We see that the error is higher at the edges of the region within which the learning samples were generated. 10 hidden units with sigmoid activation function and an output unit with a linear activation function.1: A feed-forward network can be used to approximate a function from examples. while the function which generated the learning samples is given in Fig.3 (top right). We want to estimate the relationship d = f(x) from 80 examples {x p. A feed-forward network was programmed with two inputs. Bottom right: The error in the approximation. d p} as depicted in Fig. 12. 1 1 0 0 –1 1 1 0 – 1 –1 0 –1 1 0 0 –1 –1 1 1 1 0 0 –1 1 1 0 0 –1 –1 –1 1 1 0 –1 –1 0 Fig. The network weights are initialized to small values and the network is trained for 5. The approximation error is depicted in Fig.20) should be adapted for the linear instead of sigmoid activation function. The network is considerably better at interpolation than extrapolation. 12. 12. Top left: The original learning samples.3 (bottom left).3 (top left). Suppose we have a system (for example a chemical process or a financial market) of which we want to know the characteristics.3 (bottom right). Top right: The approximation with the network. 12.BACK-PROPAGATION 145 Example 12.

The same function (albeit with other learning points) is learned with a network with eight sigmoid hidden units (see Figure 12. 12. In some cases this leads to a formula. whereas in the back-propagation approach these weights can take any value and are typically learning using a learning heuristic. To illustrate the use of other activation functions we have trained a feed-forward network with one output unit. as will be discussed in the next section. A lot of advanced algorithms based on back-propagation learning have some optimized method to adapt this learning rate.146 FUZZY LOGIC AND NEURAL NETWORKS 12. other functions can be used as well.5 OTHER ACTIVATION FUNCTIONS Although sigmoid functions are quite often used as activation functions. The basic difference between the Fourier approach and the back-propagation approach is that the in the Fourier approach the weights between the input and the hidden units (these are the factors n) are fixed integer numbers which are analytically determined. This can be a result of a non-optimum learning rate and momentum. The factor a0 corresponds with the bias of the output unit. For example. which is known from traditional function approximation theories. From the figures it is clear that it pays off to use as much knowledge of the problem at hand as possible.(12.. The result is depicted in Fig.. a single output unit for f (x) and hidden units with an activation function F = sin (s). Most troublesome is the long training process. and one input with ten patterns drawn from the function f (x) = sin(2x) sin(x). the phase factor qn corresponds with the bias term of the hidden units and the factor n corresponds with the weights between the input and hidden layer. . Outright training failures generally arise from two sources: network paralysis and local minima.23) We can rewrite this as a summation of sine terms f(x) = a0 + . the factors cn correspond with the weighs from hidden to output unit.5).(12. from Fourier analysis it is known that any periodic function can be written as a infinite sum of sine and cosine terms (Fourier series): f(x) = n=0 å(an cos nx+ bn sin nx) åcn sin (nx + qn) n =1 ¥ ¥ .4.. which make the algorithm not guaranteed to be universally useful.24) 2 2 with cn = an + bn and qn = arctan (b/a).. there are some aspects. 12. four hidden units.6 DEFICIENCIES OF BACK-PROPAGATION Despite the apparent success of the back-propagation learning algorithm. This can be seen as a feed-forward network with a single input unit for x.

. 12.4 The periodic function B(N) = sin (2N) sin (N) approximated with sine activation functions. +1 –4 2 4 6 –1 Fig. 12.5 Fig.5 The periodic function B(N) = sin (2N) sin (N) approximated with sigmoid activation functions.BACK-PROPAGATION 147 +1 –4 –2 2 4 6 8 0.

Because of the gradient descent..(12. Instead of following the gradient at every step.21). and because of the sigmoid activation function the unit will have an activation very close to zero or very close to one. Teukolsky. As is p clear from equations (12.7 ADVANCED ALGORITHMS Many researchers have devised improvements of and extensions to the basic back-propagation algorithm described above.. again results in the system being trapped in local minima. and the training process can come to a virtual standstill. It is too early for a full evaluation: some of these techniques may prove to be fundamental.2 Local Minima The error surface of a complex network is full of hills and valleys. the directions are non-interfering. the weight adjustments which are proportional to ykp (1 yk ) will be close to zero.148 FUZZY LOGIC AND NEURAL NETWORKS 12..» xT Ax bT x + c 2 . e. 12. when exceeded. the weights can be adjusted to very large values. which directly minimizes in the direction of the steepest descent (Press. Another suggested possibility is to increase the number of hidden units. a set of n directions is constructed which are all conjugate to each other such that minimization along one of these directions uj does not spoil the minimization along one of the earlier directions ui. The total input of a hidden unit or output unit can therefore reach very high (either positive or negative) values. A few methods are discussed in this section. 12.20) and (12. Probabilistic methods can help to avoid this trap. Suppose the function to be minimized is approximated by its Taylor series f (x) = f (p) + where T denotes transpose. Although this will work because of the higher dimensionality of the error space.e.g. it appears that there is some upper limit of the number of hidden units which. but they tend to be slow. 1986).1 Network Paralysis As the network trains... and the chance to get trapped is smaller. This is different from gradient descent. and c º f (p) å ¶x i ¶f i p xi + 1 2 å ¶x ¶x i.25) . conjugate gradient minimization. Note that minimization along a direction u brings the function f at a place where its gradient is perpendicular to u (otherwise minimization along u is not complete).. such that n minimizations in a system with n degrees of freedom bring this system to a minimum (provided the system is quadratic). the network can get trapped in a local minimum when there is a much deeper minimum nearby. others may simply fade away. j i ¶2 f j p 1 xi xj + . Thus one minimization in the direction of ui suffices.6.6. Flannery. & Vetterling. May be the most obvious improvement is to replace the rather primitive steepest descent method with a direction set minimization method. i.

BACK-PROPAGATION 149

b º Ñ f

¶2 f ¶xi ¶x j

p

[A]ij =

...(12.26)

p

A is a symmetric positive definite n ´ n matrix, the Hessian of f at p. The gradient of f is Ñf = Ax b such that a change of x results in a change of the gradient as d(Ñf ) = A(dx) ...(12.28) Now suppose f was minimized along a direction ui to a point where the gradient gi+ 1of f is perpendicular to ui, i.e., u iTgi + 1 = 0 ...(12.29) and a new direction ui+1is sought. In order to make sure that moving along ui+1 does not spoil minimization along ui we require that the gradient of f remain perpendicular to ui, i.e., u iTgi + 2 = 0 otherwise we would once more have to minimise in a direction which has a component of ui. Combining (12.29) and (12.30), we get 0 = u iT(gi+1 gi+2) = u iTd(Ñf) = u iTAui+1 When eq. (12.31) holds for two vectors ui and ui + 1 they are said to be conjugate. Now, starting at some point p0, the first minimization direction u0 is taken equal to g0 = Ñf (p0), resulting in a new point p1. For i ³ 0, calculate the directions ui+1 = gi +1 + giui where gi is chosen to make u iT Aui 1 and the successive gradients perpendicular, i.e.,

giT+ 1 gi + 1 giT gi

...(12.30)

...(12.27)

...(12.31)

...(12.32)

gi =

with gk = Ñf |pk for all k ³ 0

...(12.33)

Next, calculate pi+2 = pi+1 + li+1 ui+1where li+1 is chosen so as to minimize f(Pi + 2 )3. It can be shown that the us thus constructed are all mutually conjugate (e.g., see (Stoer & Bulirsch, 1980)). The process described above is known as the Fletcher-Reeves method, but there are many variants, which work more or less the same (Hestenes & Stiefel, 1952; Polak, 1971; Powell, 1977). Although only n iterations are needed for a quadratic system with n degrees of freedom, due to the fact that we are not minimizing quadratic systems, as well as a result of round-off errors, the n directions have to be followed several times (see Fig. 12.6). Powell introduced some improvements to correct for behaviour in non-quadratic systems. The resulting cost is O(n) which is significantly better than the linear convergence 4 of steepest descent.

150 FUZZY LOGIC AND NEURAL NETWORKS

Gradient ut +l

ut

**A very slow approximation
**

Fig. 12.6 Slow decrease with conjugate gradient in non-quadratic systems. [The hills on the left are very steep, resulting in a large search vector KE. When the quadratic portion is entered the new search direction is constructed from the previous direction and the gradient, resulting in a spiraling minimization. This problem can be overcome by detecting such spiraling minimizations and restarting the algorithm with K0 = ÑB ].

Some improvements on back-propagation have been presented based on an independent adaptive arning rate parameter for each weight. Van den Boomgaard and Smeulders (Boomgaard & Smeulders, 1989) show that for a feed-forward network without hidden units an incremental procedure to find the optimal weight matrix W needs an adjustment of the weights with Dw(t + 1) = g(t + 1) [d(t + 1) w(t) ´ (t + 1)] ´ (t + 1) ...(12.34) in which g is not a constant but an variable (Ni + 1) ´ (Ni + 1) matrix which depends on the input vector. By using a priori knowledge about the input signal, the storage requirements for can be reduced. Silva and Almeida (Silva & Almeida, 1990) also show the advantages of an independent step size for each weight in the network. In their algorithm the learning rate is adapted after every learning pattern:

RuC | | g (t + 1) = S |dC | T

jk

jk ( t ) jk ( t )

¶E (t + 1) ¶E (t ) and have the same signs ¶w jk ¶w jk ¶E (t + 1) ¶E (t ) if and have the opposite signs ¶w jk ¶w jk if

...(12.35)

BACK-PROPAGATION 151

where u and d are positive constants with values slightly above and below unity, respectively. The idea is to decrease the learning rate in case of oscillations.

**12.8 HOW GOOD ARE MULTI-LAYER FEED-FORWARD NETWORKS?
**

From the example shown in Fig. 12.3 is clear that the approximation of the network is not perfect. The resulting approximation error is influenced by: 1. The learning algorithm and number of iterations. This determines how good the error on the training set is minimized. 2. The number of learning samples. This determines how good the training samples represent the actual function. 3. The number of hidden units. This determines the expressive power of the network. For smooth functions only a few number of hidden units are needed, for wildly fluctuating functions more hidden units will be needed. In the previous sections we discussed the learning rules such as back-propagation and the other gradient based learning algorithms, and the problem of finding the minimum error. In this section we particularly address the effect of the number of learning samples and the effect of the number of hidden units. We first have to define an adequate error measure. All neural network training algorithms try to minimize the error of the set of learning samples which are available for training the network. The average error per learning sample is defined as the learning error rate error rate: Elearning =

1 Plearning

Plearning p=1

åE

p

...(12.36)

in which Ep is the difference between the desired output value and the actual network output for the learning samples: Ep =

1 2

å (d

0=1

No

p o

p yo )

...(12.37)

This is the error, which is measurable during the training process. It is obvious that the actual error of the network will differ from the error at the locations of the training samples. The difference between the desired output value and the actual network output should be integrated over the entire input domain to give a more realistic error measure. This integral can be estimated if we have a large set of samples. We now define the test error rate as the average error of the test set:

1 Etest = Ptest

åE

p=1

Ptest

p

...(12.38)

In the following subsections we will see how these error measures depend on learning set size and number of hidden units.

152 FUZZY LOGIC AND NEURAL NETWORKS

**12.8.1 The Effect Of the Number of Learning Samples
**

A simple problem is used as example: a function y = f(x) has to be approximated with a feed-forward neural network. A neural network is created with an input, 5 hidden units with sigmoid activation function and a linear output unit. Suppose we have only a small number of learning samples (e.g., 4) and the networks is trained with these samples. Training is stopped when the error does not decrease anymore. The original (desired) function is shown in Fig. 4.7A as a dashed line. The learning samples and the approximation of the network are shown in the same figure. We see that in this case Elearning is small (the network output goes perfectly through the learning samples) but Etest is large: the test error of the network is large. The approximation obtained from 20 learning samples is shown in Fig. 12.7B. The Elearning is larger than in the case of 5 learning samples, but the Etest is smaller.

1 A 1 B

0.8

0.8

0.6

y y

0.6

0.4

0.4

0.2

0.2

0

0

0.5 X

1

0

0

0.5 X

1

Fig. 12.7

Effect of the learning set size on the generalization. The dashed line gives the desired function, the learning samples are depicted as circles and the approximation by the network is shown by the drawn line. 5 hidden units are used. a) 4 learning samples. b) 20 learning samples.

This experiment was carried out with other learning set sizes, where for each learning set size the experiment was repeated 10 times. The average learning and test error rates as a function of the learning set size are given in Fig. 12.8. Note that the learning error increases with an increasing learning set size, and the test error decreases with increasing learning set size. A low learning error on the (small) learning set is no guarantee for a good network performance! With increasing number of learning samples the two error rates converge to the same value. This value depends on the representational power of the network: given the optimal weights, how good is the approximation. This error depends on the number of hidden units and the activation function. If the learning error rate does not converge to the test error rate the learning procedure has not found a global minimum.

**BACK-PROPAGATION 153
**

Error rate

Test set

Learning set

**Number of learning samples
**

Fig. 12.8 Effect of the learning set size on the error rate. The average error rate and the average test error rate are as a function of the number of learning samples.

12.8.2

The Effect of the Number of Hidden Units

The same function as in the previous subsection is used, but now the number of hidden units is varied. The original (desired) function, learning samples and network approximation is shown in Fig. 4.9A for 5 hidden units and in Fig. 4.9B for 20 hidden units. The effect visible in Fig. 4.9B is called over training. The network fits exactly with the learning samples, but because of the large number of hidden units the function which is actually represented by the network is far more wild than the original one. Particularly in case of learning samples which contain a certain amount of noise (which all real-world data have), the network will fit the noise of the learning samples instead of making a smooth approximation. This example shows that a large number of hidden units leads to a small error on the training set but not necessarily leads to a small error on the test set. Adding hidden units will always lead to a reduction of the Elearning. However, adding hidden units will first lead to a reduction of the Etest, but then lead to an increase of Etest. This effect is called the peaking effect. The average learning and test error rates as a function of the learning set size are given in Fig. 12.10.

12.9 APPLICATIONS

Back-propagation has been applied to a wide variety of research applications. Sejnowski and Rosenberg (1986) produced a spectacular success with NETtalk, a system that converts printed English text into highly intelligible speech. · A feed-forward network with one layer of hidden units has been described by Gorman and Sejnowski (1988) as a classification machine for sonar signals.

9 Effect of the number of hidden units on the network performance.10 The average learning error rate and the average test error rate as a function of the number of hidden units.2 0 0 0. who used a two-layer feed-forward network with back-propagation learning to perform the inverse kinematic transform which is needed by a robot arm controller. An example is the work of Josin (1988). 12. · A multi-layer feed-forward network with a back-propagation training algorithm is used to learn an unknown function between input and output signals from the presentation of examples. .8 0. 12. It is hoped that the network is able to generalize correctly.6 0.4 0.6 y y 0.154 FUZZY LOGIC AND NEURAL NETWORKS 1 A 1 B 0. a) 5 hidden units. 12 learning samples are used. Error rate Test set Learning set Number of hidden units Fig. The dashed line gives the desired function.2 0. the circles denote the learning samples and the drawn line gives the approximation by the network. b) 20 hidden units.5 X 1 Fig. so that input values which are not presented as learning patterns will result in correct output values.8 0.4 0.5 X 1 0 0 0.

and Systems. Williams. MA: Massachusetts Institute of Technology. Keeler. 1989. pp. 2.E.L.H. 10.D. Minsky. 4. L. and H. 359-366. G. 9. 303-314. 1989. Cun. Y. No. 1980. 10. 3. Vol. . 599-604. pp. Rep. Funahashi. 2. Hartman. 85. 1985. 1985. Perceptrons: An Introduction to Computational Geometry. Layered neural networks with Gaussian hidden units as universal approximations. 5. Center for Computational Research in Economics and Management Science. pp. 6. 1. Explain the effect of the number of hidden links in multi-layer feed forward networks. Vol. Vol. K. 11. 1. 1969.BACK-PROPAGATION 155 QUESTION BANK. Y.Berlin: Springer-Verlag.J. and J. Mathematics of Control. 2. Explain the sine activation function with an example. M. Rumelhart. Neural Networks. Une procedure dapprentissage pour reseau a seuil assymetrique. pp. 4. Neural Networks. Introduction to Numerical Analysis. K. Vol. Hornik. Neural Computation.A. 8. Press. 2. What are the deficiencies of back-propagation algorithm? Explain various methods employed to overcome the deficiencies of back-propagation algorithm. Learning representations by back-propagating errors. TR (47). G. Vol. 1986. Explain learning rate and momentum with back-propagation with an example. 1989. 12. Vol. Hinton. D. Describe the generalized delta rule. Parker. 7. No. 3. 2. 533-536. pp. Explain the effect of the number of learning samples in multi-layer feed forward networks. No. 1990. and W. and R.P. S. Vetterling. 5. 193-192. 3. Nos. What is back-propagation algorithm? Explain.I. New York-Heidelberg. J. Cybenko. and S. M. 8.M. Approximation by superpositions of a sigmoidal function. White. Cambridge.E. Vol. How the weights are adjusted with sigmoid activation function? Explain with an example. 323. Numerical Recipes: The Art of Scientific Computing. No.B. 4. W. The MIT Press. 1986. 9. 2. 7. Explain the multi-layer feed forward networks. 5. 210-215. D. On the approximate realization of continuous mappings by neural networks. B. Bulirsch. What are the applications of back-propagation algorithm? REFERENCES. Flannery. Kowalski. E.T. Cambridge: Cambridge University Press. Proceedings of Cognitiva. How good are multi-layer feed forward networks? Explain. 6. Signals. Papert. Learning-Logic (Tech. J.J. Stinchcombe. and R. Multilayer feed forward networks are universal approximates. Stoer. Nature. 2. Teukolsky.

Journal of National Bureau of Standards. Gorman.J. 1.J.R. 1952.156 FUZZY LOGIC AND NEURAL NETWORKS 11. New York: Academic Press. Sejnowski. and E. Neural-space generalization of a topological transformation. pp. Vol. M. Nos. Josin. and C. Analysis of hidden units in a layered network trained to classify sonar targets. 15. 75-89. No. 1971. Methods of conjugate gradients for solving linear systems. NETtalk: A Parallel Network that Learns to Read Aloud (Tech. Polak. JHU/EECS-86/01). 14. 12. 1986. 49. pp.D. Restart procedures for the conjugate gradient method. Hestenes. 59. 241-254. pp. Mathematical Programming. Sejnowski.J. G. 1977. Rosenberg. 283-290. Neural Networks. Vol. M. 1988. T. . E.P. R. Biological Cybernetics. and T. Powell. The John Hopkins University Electrical Engineering and Computer Science Department. 16. 409-436.R. Vol. 13. Computational Methods in Optimization. 1. pp. Vol. 1988. Rep. Stiefel. 12.

Also some special recurrent networks will be discussed: the Hopfield network. Before we will consider this general case. In such networks. But what happens when we introduce a cycle? For instance. therewith introducing stochasticity in neural computation. network size. we can connect a hidden unit with itself over a weighted connection. however. it is possible to continue propagating activation values until a stable point (attractor) is reached. when one is considering a recurrent network.RULE IN RECURRENT NETWORKS The back-propagation learning rule. but there are also recurrent networks where the learning rule is used after each propagation (where an activation value is transversed over each weight only once). i..e. while external inputs are included in each propagation. the approximation capabilities of such networks do not increase. The theory of the dynamics of recurrent networks extends beyond the scope of a one-semester course on neural networks. 13. introduced in chapter 12. As we will see in the sequel. recurrent extensions to the feed-forward network will be discussed. we may obtain decreased complexity. we will first describe networks . 1 INTRODUCTION The learning algorithms discussed in the previous chapter were applied to feed-forward networks: all data flows in a network in which no cycles are present.+ 0 ) 2 6 . Yet the basics of these networks will be discussed. which are attractor based. etc. can be easily used for training patterns in recurrent networks.2 THE GENERALISED DELTA . An important question we have to consider is the following: what do we want to learn in a recurrent network? After all. subsequently we touch upon Boltzmann machines. the recurrent connections can be regarded as extra inputs to the network (the values of which are computed by the network itself).4 Recurrent Networks 13 13. or even connect all units with each other. the activation values in the network are repeatedly updated until a stable point is reached after which the weights are adapted. there exist recurrent network. which can be used for the representation of binary patterns. Although. as we know from the previous chapter. connect hidden units to input units. to solve the same problem. In this chapter.

Output activation values are fed back to the input layer. derivatives. . . The disadvantage is. . etc. 2. Thus a time window of the input vector is input to the network.1.158 FUZZY LOGIC AND NEURAL NETWORKS where some of the hidden unit activation values are fed back to an extra set of input units (the Elman network). of course. In the Jordan network.1 The Jordan Network One of the earliest recurrent neural networks was the Jordan network. Create inputs x1. Suppose we have to construct a network that must generate a control command depending on an external input. second. x(t 1). Besides only inputting x(t). The Jordan and Elman networks provide a solution to this problem. Naturally. x¢. x¢¢.1 The Jordan network. 13. we also input its first. which is slow and difficult to train. 13. Due to the recurrent connections. or where output values are fed back into hidden units (the Jordan network). xn which constitute the last n values of the input vector. a window of inputs need not be input anymore. instead. the network is supposed to learn the influence of the previous time steps itself. the activation values of the output units are fed back into the Input units h o State units Fig. to a set of extra neurons called the state units.2. An example of this network is shown in Fig. A typical application of such a network is the following. x(t 2). leading to a very large network. 13. Create inputs x. which is a time series x(t). x2. With a feed-forward network there are two possible approaches: 1. computation of these derivatives is not a trivial task for higher-order derivatives. . that the input dimensionality of the feed-forward network is multiplied with n.

the forward calculations are performed once. With this network. The schematic structure of this network is shown in Fig. except that (1) the hidden units instead of the output units are fed back. Example 13. the Jordan and Elman networks can be used to train a network on reproducing time sequences. There are as many state units as there are output units in the network.2. . The connections between the output and state units have a fixed weight of +1. Learning is done as follows: 1.RECURRENT NETWORKS 159 input layer through a set of extra input units called the state units. forces F must be applied. we trained an Elman network on controlling an object moving in 1 D. 13. since the object suffers from friction and perhaps other external forces. Thus all the learning rules derived for the multi-layer perceptron can be used to train this network. to a set of extra neurons called the context units. The idea of the recurrent connections is that the network is able to remember the previous states of the input values. 3. t = 1 2. go to 2. To control the object. and (2) the extra input units have no self-connections.1: As we mentioned above. which are extra input units whose activation values are fed back from the hidden units. t ¬ t + 1. 13. 4. This object has to follow a pre-specified trajectory xd. Pattern xt is clamped. Output layer Hidden layer Input layer Fig. learning takes place only in the connections between input and hidden units as well as hidden and output units. Again the hidden units are connected to the context units with a fixed weight of value +1. the hidden unit activation values are fed back to the input layer. As an example. The back-propagation learning rule is applied. The context units are set to 0. Thus the network is very similar to the Jordan network. 13.2.2 Context layer The Elman network.2 The Elman Network In the Elman network a set of context units are introduced. The context units at step t thus always have the activation value of the hidden units at step t 1.

. The hidden units are connected to three context units. The third line is the error. In total. 13. The same test can be done with an ordinary feedforward network with sliding window input. and three hidden units.4 Training a feed-forward network to control an object.3 Training an Elman network to control an object. The third line is the error. We tested this with a network with five inputs. 4 2 0 100 200 300 400 500 –2 –4 Fig.160 FUZZY LOGIC AND NEURAL NETWORKS To tackle this problem. the dashed line the realized trajectory. four of 4 2 0 100 200 300 400 500 –2 –4 Fig. the dashed line the realized trajectory. The results of training are shown in Fig. The solid line depicts the desired trajectory N@. 13. 13. one output F. The solid line depicts the desired trajectory N@. we use an Elman net with inputs x and xd. five units feed into the hidden layer.3.

a pattern is clamped. 1989. and the output of the network consists of the new activation values of the neurons. All connections are weighted. can be used to train a multi-layer perceptron to follow trajectories in its activation values.RECURRENT NETWORKS 161 which constituted the sliding window x3. which can be used for training attractor networks. a learning method can be used: back-propagation through time (Pearlmutter. independently of each other Pineda (1987) and Almeida (1987) discovered that error back-propagation is in fact a special case of a more general gradient learning method. For instance. The disappointing observation is that the results are actually better with the ordinary feed-forward network. and one the desired next position of the object.5 The auto-associator network.. Results are shown in Fig. 13. the discussion of which extents beyond the scope of our course.2. which has the same complexity as the Elman network. This learning method.4. i ¹ j (see Fig. Hopfiled (1982) brings together several earlier ideas concerning these networks and presents a complete mathematical analysis. However. 1990). . x1 and x0. It consists of a pool of neurons with connections between each unit i and j.3 THE HOPFIELD NETWORK One of the earliest recurrent neural networks reported in literature was the auto-associator independently described by Anderson (1977) and Kohonen (1977). Fig. All neurons are both input and output neurons. 15. 13.5). the network iterates to a stable state. 13. also when a network does not reach a fixed point.e. 13. i.3 Back-Propagation in Fully Recurrent Networks More complex schemes than the above are possible. x2.

2). but using values +1 and 1 presents some advantages discussed below. 10. When the extra restriction wjk = wkj is made. The net input Sk(t + 1) of a neuron k at cycle t + 1 is a weighted sum Sk(t + 1) = å yj (t)wjk + qk j¹k ... The activation values are binary. since the yk are bounded from below and the wjk and qk are constant. but this is of course not essential. when the network is in state a. Originally.1) and (13. (13... All neurons are both input and output neurons..162 FUZZY LOGIC AND NEURAL NETWORKS 13.3) A state a is called stable if.(13. We will therefore adhere to the latter convention.(13.3. Proof: First. because De = Dyk F yw GH å j j¹k jk + qk I JK . Secondly.(13.1: A recurrent network with connections wjk = wkj in which the neurons are updated using rule (13.. e is monotonically decreasing when state changes occur.2) is applied to the net input to obtain the new activation value yi(t + 1) at time t + 1: R+ 1 | y (t + 1) = S. when xp is clamped. all neurons are stable.2). (13. Hopfield chose activation values of 1 and 0. yk(t) = sgn (Sk(t 1)) ..(13.2) i.e.2) has stable limit points.1) A simple threshold function (Fig.5) is always negative when yk changes according to eqs. 13..4) is bounded from below. in accordance with equations (13. all neurons are stable.1 | y (t ) T k k if Sk (t + 1) > U k if Sk (t + 1) < U k otherwise ..1 Description The Hopfield network consists of a set of N interconnected neurons (Fig. A neuron k in the Hopfield network is called stable at time t if.(13. .. A pattern xp is called stable if.5). yk (t + 1) = sgn (Sk (t + 1)) For simplicity we henceforth choose Uk = 0.4) Theorem 13. The state of the system is given by the activation values Y = y(k). which update their activation values asynchronously and independently of other neurons. the behavior of the system can be described with an energy function e= 1 2 j¹k åå y j yk w jk - åq k k yk .1) and (13.. note that the energy expressed in eq.

.. 1983). the pattern 00 00 is always stable. Similarly. that the network gets saturated very quickly.e. & Wallace.2 Hopfield Network as Associative Memory A primary application of the Hopfield network is an associative memory. whereas in the 1/0 model this is not always true (as an example. when some pattern x is stable. weights only increase). 2. which is in some sense near to the cued pattern. It appears that. it will render the incorrect or missing data by iterating to a stable state.7) Now modify wjk by Dwjk = yj yk(ej +ek) if j ¹ k. in practice.1: Given a starting weight matrix W = [wjk]. Feinstein. in the original Hebb rule.3.6) p i. It appears.(13. 13. (Bruce. but 11 11 need not be). 1986). Removing the restriction of bidirectional connections (i. wjk is increased. otherwise decreased by one (note that.(13. the weights of the connections between the neurons have to be thus set that the states of the system corresponding with the patterns which are to be stored in the network are stable. where the algorithm remains oscillatory (try to find one)! The second problem stated above can be alleviated by applying the Hebb rule in reverse to the spurious stable state. wjk = wkj) results in a system that is not guaranteed to settle to a stable state... There exist cases. There are two problems associated with storing too many patterns: 1. however. Gardner. for each pattern x p to be stored and each element x kp in x p define a correction ek such that Ak = R0 S1 T if yk is stable and x p is clamped otherwise . Algorithm 13.RECURRENT NETWORKS 163 The advantage of a + 1/1 model over a 1/0 model then is symmetry of the states of the network. The Hebb rule can be used to store P patterns: wjk R xx | = Så |0 T p p =1 p p j k if j ¹ k otherwise .e.e. Spurious stable states appear (i. Forrest. For. The stored patterns become unstable. Repeat this procedure until all patterns are stable. In this case. too. but with a low learning factor (Hopfield. however. and that about 0:15N memories can be stored before recall errors become severe. When the network is cued with a noisy or incomplete test pattern. this algorithm usually converges. Thus these patterns are weakly unstored and will become unstable again. Canning.. These states can be seen as dips in energy space. & Palmer. The first of these two problems can be solved by an algorithm proposed by Bruce et al. its inverse is stable. . both a pattern and its inverse have the same energy in the +1/1 model.. stable states which do not correspond with stored patterns). if xjp and xk are equal..

(13. (Wilson and Pawley. 1984). When the network is settled. the applicability is limited..9) is added to the energy. the threshold activation function is replaced by a sigmoid.3.and end-points are the same.8) are zero if and only if there is a maximum of one active neuron in each row and column. j .nJ K 2 . the following energy must be minimized: A e= 2 ååå y X j k¹ j Xj y Xk B + 2 åå å j X C y Xj yYj + 2 X ¹Y F y GH å å X j Xj I . each neuron has an external bias input Cn. B. 13. Although this application is interesting from a theoretical point of view. and C are constants. the subscripts are defined modulo n. An energy function describing this problem can be set up as follows. respectively. Here. Finally. a path of minimal distance must be found between n cities. where dXY is the distance between cities X and Y and D is a constant. other reports show less encouraging results. whereas each column represents the position in the tour. For example.(13. j+1 + dk.3.10) ..2) with a sigmoid activation function between 0 and 1. The activation value yxj = 1 indicates that city X occupies the jth place in the tour. 1988) find that in only 15% of the runs a valid result is obtained. In this problem. The neurons are updated using rule (13. The last term is zero if and only if there are exactly n active neurons.. The weights are set as follows: wXJ. j1) data term where djk = 1if j = k and 0 otherwise. indicating a specific city occupying a specific position in the tour. j +1 + yY . Each row in the matrix represents a city.. an extra term e= D 2 å ååd X Y¹X j XY y Xj ( yY . As before. The first and second terms in equation (13.4 Hopfield Networks for Optimization Problems An interesting application of the Hopfield network with graded response arises in a heuristic solution to the NP-complete traveling salesman problem (Garey & Johnson. 1979). this system can be proved to be stable when a symmetric weight matrix is used (Hopfield. For convenience.3.1 ) .3 Neurons with Graded Response The network described in section 13..(13. few of which lead .8) where A. Hopfield and Tank (1985) use a network with n ´ n neurons. such that the begin. each row and each column should have one and only one active neuron. Yk = AdXY (1 djk) inhibitory connections within each row = Bdjk(1 dXY) inhibitory connections within each column = C global inhibition = DdXY(dk.1 can be generalized by allowing continuous activation values. To ensure a correct solution.. Whereas Hopfield and Tank state that the network converges to a valid solution in 16 out of 20 trials while 50% of the solutions are optimal. To minimise the distance of the tour.164 FUZZY LOGIC AND NEURAL NETWORKS 13.

The weights are still symmetric.. The competition between the degenerate tours often leads to solutions which are piecewise optimal but globally inefficient. it will perform a search of the coarse overall structure of the space of global states. p(yk ¬ + 1) = 1 + e . each of which may be traversed in two directions as well as started in N points. the number of different tours is N!/2N. The main problem is the lack of global information.(13. This is a process whereby a material is heated and then cooled very. A good way to beat this trade-off is to start at a high temperature and gradually reduce it. Note that at thermal equilibrium the units still change state. such that all but one of the final 2N configurations are redundant. As the temperature is lowered.11) where T is a parameter comparable with the (synthetic) temperature of the system. without any impurities. . very slowly to a freezing point.RECURRENT NETWORKS 165 to an optimal or near-optimal solution.2) in a stochastic update. there are N! possible tours. the network will eventually reach thermal equilibrium and the relative probability of two global states a and b will follow the Boltzmann distribution Pa . for an N-city problem. In accordance with a physical system obeying a Boltzmann distribution. At low temperatures there is a strong bias in favor of states with low energy. The degenerate solutions occur evenly within the hypercube. Differently put..12) where Pa is the probability of being in the ath global state. At high temperatures.( e . as first described by Ackley.4 BOLTZMANN MACHINES The Boltzmann machine. but the time required to reach equilibrium may be long. it will begin to respond to smaller energy differences and will find one of the better minima within the coarse-scale minimum it discovered at high temperature..e )/ T = e a b Pb . The operation of the network is based on the physics principle of annealing. In doing so.(13. Since. This stochastic activation function is not to be confused with neurons having a sigmoid deterministic activation function.De k / T 1 . in which a neuron becomes active with a probability p. In the Boltzmann machine this system is mimicked by changing the deterministic update of equation (13. the N-dimensional hypercube in which the solutions are situated is 2N degenerate. and will find a good minimum at that coarse level. and with a stochastic instead of deterministic update rule. and ea is the energy of that state.. 13. Hinton. and Sejnowski in 1985 is a neural network that can be seen as an extension to Hopfield networks to include hidden units. the crystal lattice will be highly ordered. such that the system is in a state of very low energy. At higher temperatures the bias is not so favorable but equilibrium is reached faster. the network will ignore small energy differences and will rapidly approach equilibrium. As a result. but the probability of finding the network in any global state remains constant.

**166 FUZZY LOGIC AND NEURAL NETWORKS
**

As multi-layer perceptions, the Boltzmann machine consists of a non-empty set of visible and a possibly empty set of hidden units. Here, however, the units are binary-valued and are updated stochastically and asynchronously. The simplicity of the Boltzmann distribution leads to a simple learning procedure, which adjusts the weights so as to use the hidden units in an optimal way (Ackley et al., 1985). This algorithm works as follows: First, the input and output vectors are clamped. The network is then annealed until it approaches thermal equilibrium at a temperature of 0. It then runs for a fixed time at equilibrium and each connection measures the fraction of the time during which both the units it connects are active. This is repeated for all input-output pairs so that each connection can measure (yj yk)clamped, the expected probability, averaged over all cases, that units j and k are simultaneously active at thermal equilibrium when the input and output vectors are clamped. Similarly, (yj yk)free is measured when the output units are not clamped but determined by the network. In order to determine optimal weights in the network, an error function must be determined. Now, the probability Pfree(Y p) that the visible units are in state Y pwhen the system is running freely can be measured. Also, the desired probability Pclamped(Y p)that the visible units are in state (Y p) is determined by clamping the visible units and letting the network run. Now, if the weights in the network are correctly set, both probabilities are equal to each other, and the error E in the network must be 0. Otherwise, the error must have a positive value measuring the discrepancy between the networks internal mode and the environment. For this effect, the asymmetric divergence or Kullback information is used: E=

åP

p

clamped

(Y p ) log

P clamped (Y p ) P free (Y P )

...(13.13)

Now, in order to minimize E using gradient descent, we must change the weights according to Dwjk = g It is not difficult to show that

¶E ¶w jk

...(13.14)

**¶E 1 = ( y j yk ) clamped - ( y j yk ) free ¶w jk T
**

Therefore, each weight is updated by Dwjk = g ( y j yk ) clamped - ( y j yk ) free

...(13.15)

...(13.16)

RECURRENT NETWORKS 167

QUESTION BANK.

1. 2. 3. 4. 5. 6. 7. 8. 9. What happens when a cyclic data is introduced to feed forward networks? Explain the generalized delta-rule in recurrent networks. Describe the Jordan network with an example. Describe Elman network with an example. Describe the Hopfield network. Describe the Hopfield network as associative memory. Describe Hopfield network for optimization problems. Describe the Boltzman machine. What are the problems resulted while storing too many patterns using associative memory? How these problems can be solved?

REFERENCES.

1. M.I. Jordan, Attractor dynamics and parallelism in a connectionist sequential machine, In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, Hillsdale, NJ: Erlbaum, pp. 531-546, 1986. 2. M.I. Jordan, Serial Order: A Parallel Distributed Processing Approach (Tech. Rep. No. 8604). San Diego, La Jolla, CA: Institute for Cognitive Science, University of California, 1986. 3. J.L. Elman, Finding structure in time. Cognitive Science, Vol. 14, pp. 179-211, 1990. 4. F. Pineda, Generalization of back-propagation to recurrent neural networks, Physical Review Letters, Vol. 19, and pp. 2229-2232, 1987. 5. L.B. Almeida, A learning rule for asynchronous perceptrons with feedback in a combinatorial environment, In Proceedings of the First International Conference on Neural Networks, Vol. 2, pp. 609-618,1987. 6. B.A. Pearlmutter, Learning state space trajectories in recurrent neural networks, Neural Computation, Vol. 1, No. 2, pp. 263-269, 1989. 7. B.A. Pearlmutter, Dynamic Recurrent Neural Networks (Tech. Rep. Nos. CMU-CS-90-196), Pittsburgh, PA 15213: School of Computer Science, Carnegie Mellon University, 1990. 8. J.A. Anderson, Neural Models with Cognitive Implications. In D. LaBerge and S.J. Samuels (Eds.), Basic Processes in Reading Perception and Comprehension Models, Hillsdale, NJ: Erlbaum, pp. 27-90, 1977. 9. T. Kohonen, Associative Memory: A System-Theoretical Approach, Springer-Verlag, 1977. 10. J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences, Vol. 79, pp. 2554-2558, 1982. 11. A.D. Bruce, A. Canning, B. Forrest, E. Gardner, and D.J. Wallace, Learning and memory properties in fully connected networks, In J.S. Denker (Ed.), AIP Conference Proceedings 151, Neural Networks for Computing, pp. 65-70, DUNNO, 1986.

**168 FUZZY LOGIC AND NEURAL NETWORKS
**

12. J.J. Hopfield, D.I. Feinstein, and R.G. Palmer, unlearning has a stabilizing effect in collective memories, Nature, Vol. 304, pp. 159-159, 1983. 13. J.J. Hopfield, Neurons with graded response have collective computational properties like those of two-state neurons, Proceedings of the National Academy of Sciences, Vol. 81, pp. 3088-3092, 1984. 14. M.R. Garey, and D.S. Johnson, Computers and Intractability. New York: W.H. Freeman, 1979. 15. J.J. Hopfield, and D.W. Tank, neural computation of decisions in optimization problems, Biological Cybernetics, Vol. 52, pp. 141-152, 1985. 16. G.V. Wilson, and G.S. Pawley, On the stability of the traveling salesman problem algorithm of Hopfield and tank, Biological Cybernetics, Vol. 58, pp. 63-70, 1988. 17. D.H. Ackley, G.E. Hinton, and T.J. Sejnowski, (1985). A learning algorithm for Boltzmann machines, Cognitive Science, Vol. 9, No. 1, pp. 147-169, 1985.

+ 0 ) 2 6 - 4

Self-Organizing Networks

14

14. 1

INTRODUCTION

In the previous chapters we discussed a number of networks, which were trained to perform a mapping F: Â¢¢ ® Âm by presenting the network examples (x p, d p) with d p = F(x p) of this mapping. However, problems exist where such training data, consisting of input and desired output pairs are not available, but where the only information is provided by a set of input patterns x p. In these cases the relevant information has to be found within the (redundant) training samples x p. Some examples of such problems are: Clustering: the input data may be grouped in clusters and the data processing system has to find these inherent clusters in the input data. The output of the system should give the cluster label of the input pattern (discrete output); Vector quantisation: this problem occurs when a continuous space has to be discretized. The input of the system is the n-dimensional vector x, the output is a discrete representation of the input space. The system has to find optimal discretization of the input space; Dimensionality reduction: the input data are grouped in a subspace, which has lower dimensionality than the dimensionality of the data. The system has to learn an optimal mapping, such that most of the variance in the input data is preserved in the output data; Feature extraction: the system has to extract features from the input signal. This often means a dimensionality reduction as described above. In this chapter we discuss a number of neuro-computational approaches for these kinds of problems. Training is done without the presence of an external teacher. The unsupervised weight adapting algorithms are usually based on some form of global competition between the neurons. There are very many types of self-organizing networks, applicable to a wide area of problems. One of the most basic schemes is competitive learning as proposed by Rumelhart and Zipser (1985). A very similar network but with different emergent properties is the topology-conserving map devised by Kohonen. Other self-organizing networks are ART, proposed by Carpenter and Grossberg (1987), and Fukushima (1975).

170 FUZZY LOGIC AND NEURAL NETWORKS

14.2 14.2.1 Clustering

COMPETITIVE LEARNING

Competitive learning is a learning procedure that divides a set of input patterns in clusters that are inherent to the input data. A competitive learning network is provided only with input vectors x and thus implements an unsupervised learning procedure. We will show its equivalence to a class of traditional clustering algorithms shortly. Another important use of these networks is vector quantisation. An example of a competitive learning network is shown in Fig. 14.1. All output units o are connected to all input units i with weights wio. When an input pattern x is presented, only a single output unit of the network (the winner) will be activated. In a correctly trained network, all x in one cluster will have the same winner. For the determination of the winner and the corresponding learning rule, two methods exist.

O

wio i

Fig. 14.1 A simple competitive learning network. Each of the four outputs o is connected to all inputs i.

**Winner Selection: Dot Product
**

For the time being, we assume that both input vectors x and weight vectors wo are normalized to unit length. Each output unit o calculates its activation value yo according to the dot product of input and weight vector: yo =

å

i

wio xi = w T x o

...(14.1)

In a next pass, output neuron k is selected with maximum activation "o ¹ k : yo £ yk ...(14.2) Activations are reset such that yk = 1 and yo ¹ k = 0. This is the competitive aspect of the network, and we refer to the output layer as the winner-take-all layer. The winner-take-all layer is usually implemented in software by simply selecting the output neuron with highest activation value. This function can also be performed by a neural network known as MAXNET (Lippmann, 1989). In MAXNET, all neurons o are connected to other units o0 with inhibitory links and to itself with an excitatory link: wo, o =

R- e S+ 1 T

if o ¹ o¢ otherwise

...(14.3)

the weight update must be changed to implement a shift towards the input: wk(t + 1) = wk(t) + g (x(t) wk(t)) . Instead of rotating the weight vector towards the input as performed by equation (14.1) gives a biological plausible solution. Each time an input x is presented.1) and (14.. 14. weight vectors are rotated towards those areas where many inputs appear: the clusters in the input.5) reduces to (14.(14. the winning neuron k is selected with its weight vector wk closest to the input pattern x. which all lie on the unity sphere. Winner selection: Euclidean distance Previously it was assumed that both inputs x and weight vectors w were normalized.(14. The weight update given in equation (14. 14.. From now on.5) It is easily checked that equation (14.SELF-ORGANIZING NETWORKS 171 It can be shown that this network converges to a situation where only the neuron with highest initial activation survives.3 it is shown how the algorithm would fail if normalized vectors were to be used.(14.wk (t )) || wk (t ) + g ( x(t ) . To this end. Using the activation function given in equation (14. whereas the activations of all other neurons converge to zero. using the Euclidean distance measure: k: ||wk x||£||wo x|| "o . The three weight vectors are rotated towards the centers of gravity of the three different input clusters.wk (t ))|| . Once the winner k has been selected.6) .2) if all vectors are normalized.4) where the divisor ensures that all weight vectors w are normalized. the weights are updated according to: wk(t + 1) = wk (t ) + g ( x(t ) . Note that only the weights of winner k are updated.. the weight vector closest to this input is selected and is subsequently rotated towards the input. we will simply assume a winner k is selected without being concerned which algorithm is used.1) and (14.. 14.2.2).4). The Euclidean distance norm is therefore a more general case of equations (14. In Fig. This procedure is visualized in Fig.4) effectively rotates the weight vector wo towards the input vector x. Naturally one would like to accommodate the algorithm for normalized input data. Consequently.. Weight vector Pattern vector w1 w3 w2 Fig.2 Example of clustering in 3D with normalized vectors..

Therefore. The more often it wins.. Similarity is measured by a distance function on the input vectors. A common criterion to measure the quality of a given clustering is the square error criterion.3 Determining the winner in a competitive learning network. i.. Another more thorough approach that avoids these and other problems in competitive learning is called leaky learning. given by E= å ||wk x p||2 p . a.. Krishnamurthy. & Melton.e.. it is customary to initialize weight vectors to a set of input patterns {x} drawn from the input set at random. In this algorithm. the less sensitive it becomes to competition. . as discussed before. vectors x and w1 are nearest to each other. It is not difficult to show that competitive learning indeed seeks to find a minimum for this square error by the negative gradient of the error-function..7) with g ¢ < g the leaky learning rate. This is implemented by expanding the weight update given in equation (14.(14. Again only the weights of the winner are updated. 14. neurons that consistently fail to win increase their chances of being selected winner. input patterns are divided in disjoint clusters such that similarities between input patterns in the same cluster are much bigger than similarities between inputs in different clusters. In b. and their dot product xTw1 = |x||w1| cos a is larger than the dot product of x and w2. The three vectors having the same directions as in a.6) with wl(t + 1) = wl(t) + g ¢(x(t) wl(t)) "l ¹ k . In a.. it is not beyond imagination that a randomly initialized weight vector wo will never be chosen as the winner and will thus never be moved and never be used. but with different lengths. Chen.. however. the dot product xTw1 is still larger than xTw2. However. Conversely. Especially if the input vectors are drawn from a large or high-dimensional input space.8) where k is the winning neuron when input xp is presented. the pattern and weight vectors are not normalized.172 FUZZY LOGIC AND NEURAL NETWORKS W1 W1 X W2 X W2 a b Fig.(14. b. The weights w are interpreted as cluster centres. that a competitive network performs a clustering process on the input data. 1990). each neuron records the number of times it is selected winner. A point of attention in these recursive clustering techniques is the initialization.. Cost function: Earlier it was claimed. A somewhat similar method is known as frequency sensitive competitive learning (Ahalt. Three normalized vectors. and in this case w2 should be considered the winner when x is applied.

14.(14.4 ++ + ++ + + ++ + + + + + + + 0 0.5 Fig. . (14.1: In Fig. we calculate the effect of a weight change on the error function. Now..12) Example 14. (3.8 0.4.4 0.1 0 – 0.. The positions of the weight vectors after 500 iterations is given by o. The network was trained with g = 0:1 and a g ¢ = 0:001 and the positions of the weights after 500 iterations are shown. (14.7 + 0.6).xip ¶E p = ¶wio 0 such that R S T if unit o wins otherwise .12).10) where g is a constant of proportionality. Proof: that As in eq..(14. . The data are given by +.. 1 0..SELF-ORGANIZING NETWORKS 173 Theorem 14.6) written down for one element of wo.1: The error function for pattern x p Ep = å ||wk x p ||2 p ..5 0.(14.8) is minimized by repeated weight updates using eq. we have to determine the partial derivative of E p: wio .11) p Dpwio = g (wio x ip) = g (x o wio) which is eq.9) where k is the winning unit.6).5 1 Competitive learning for clustering data. 14..6 0. Therefore..(14.9 0. eq. A competitive learning network using Euclidean distance to select the winner was initialized with all weight vectors wo = 0. is minimised by the weight update rule in eq. (14.2 0. 8 clusters of each 6 data points are depicted. (14. So we have DpWio = C ¶E p ¶wio .3 0.

but more in quantising the entire input space. five neurons discretized the input space into five smaller subspaces. the upper part of the input space is divided into two large separate regions. networks that perform vector quantisation are combined with another type of network in order to perform function approximation. The lower part.3 Counter Propagation In a large number of applications. competitive learning can be used in applications where data has to be compressed such as telecommunication or storage. index k of the winning neuron). Vector quantisation through competitive learning results in a more fine-grained discretization in those areas of the input space where most input have occurred in the past. where many more inputs have occurred.2. The difference with clustering is that we are not so much interested in finding clusters of similar data.174 FUZZY LOGIC AND NEURAL NETWORKS 14. only few (in this case two) neurons are used to discretized the input space. A vector quantisation scheme divides the input space in a number of disjoint subspaces and represents each input vector x by the label of the subspace it falls into (i. x2 x1 Input pattern Weight vector Fig. Thus. the weight vectors also lie in Â2.e. The quantisation performed by the competitive learning network is said to track the input probability density function: the density of neurons and thus subspaces is highest in those areas where inputs are most likely to appear. In the areas where inputs are scarce. 14. An example of such a network is given in . and be applied to function approximation problems or classification problems. An example of tracking the input density is sketched in Figure 14.2.5.2 Vector Quantisation Another important use of competitive learning networks is found in vector quantisation. the upper part of the figure. The input patterns are drawn from Â2. whereas a more coarse quantisation is obtained in those areas where inputs are scarce. 14. In this way. However. however. We will describe two examples: the counter propagation method and the learning vector quantisation.5 This figure visualizes the tracking of the input density. competitive learning has also be used in combination with supervised learning methods..

Update the weights wih with equation (14. 2. This network can approximate a function f : Ân ® Âm by associating with each neuron o a function value [w1o. calculate the distance from its weight vector to the input pattern and find winner k. or one can choose to learn the quantisation and the approximation layer simultaneously. wmo]T which is somehow representative for the function values f(x) of inputs x represented by o.. 14. k) = h . . 14. 1988).SELF-ORGANIZING NETWORKS 175 Fig.. w2o. As an example of the latter. 14. and the function value [w1k. Depending on the application. h)dx . This network can be used to approximate functions from Â2 to Â2. A well-known example of such a network is the Counter propagation network (Hecht-Nielsen.. one can choose to perform the vector quantisation before learning the function approximation. Vector quantisation h i o y Wih Fig.6.6 can be supervisedly trained in the following way: 1. This way of approximating a function effectively implements a look-up table: an input x is assigned to a table entry k with "o ¹ k: ||x wk||£||x wo||.14) It can be shown that this learning procedure converges to who = z yog (x.15) . k) as: g(x.6 Feedforward Who A network combining a vector quantisation layer with a 1-layer feed-forward neural network..(14.. If we define a function g(x. wmk]T in this table entry is taken as an approximation of f(x).(14. 3. Perform the supervised approximation step: wko(t + 1) = wko(t) + g (do wko(t)) This is simply the d rule with yo = output is given by d = f(x)..6). Present the network with both input x and function value d = f (x). For each weight vector. w2k. the network presented in Fig.13) å yhwho = wko when k is the winning neuron and the desired R1 S0 T Ân if k is winner otherwise . Perform the unsupervised quantisation step. . the input space Â2 is discretized in 5 disjoint subspaces.(14.

Olshen. each decision could. The latter could be replaced by a reinforcement learning procedure (see chapter 15). the combination of quantisation and approximation is not uncommon and probably very efficient.. y k2 are compared with d p. using the following strategy: ||xp wk1||<||x p wk2||<||x p wi|| "o ¹ k1. which results in a better approximation of the function in those areas where input is most likely to appear.2.4 Learning Vector Quantisation It is an unpleasant habit in neural network literature. The quantisation layer can be replaced by various other quantisation schemes. to also cover Learning Vector Quantisation (LVQ) methods in chapters on unsupervised clustering. but also the second best k2: p p 4. They are all based on the following basic algorithm: 1. Granted that these methods also perform a clustering or quantisation task and use similar learning rules. e. extended with the possibility to have the approximation layer influence the quantisation layer (e. The weight update rule given in equation (6. a class label (or decision of some other kind) yo is associated. the quantisation scheme tracks the input probability density function. In fact. 14. a simple identity or combinations of sines and cosines are much better approximated by multilayer back-propagation networks if the activation functions are chosen appropriately. As we have seen before. k2 and ||x p wk2|| ||x p wk1|| < e then wk2(t + 1) = wk2(t) + g (x wk2(t)) and wk1(t + 1) = wk1(t) + g (x wk1(t)) p p if y k1 ¹ d p and d p = y k2 .6) is used selectively based on this comparison. p 2. 1991) are based on this very idea. such as Kohonen networks or octree methods (Jansen. A rather large number of slightly different LVQ methods is appearing in recent literature. they are trained supervisedly and perform discriminant analysis rather than unsupervised clustering.. An example of the last step is given by the LVQ2 algorithm by Kohonen (1977). not only the winner k1 is determined. Friedman. and Groen. 1984.g. The labels y k1.176 FUZZY LOGIC AND NEURAL NETWORKS i. Smagt.. These networks attempt to define decision boundaries in the input space. Using distance measures between weight vectors wo and input vector x . With each output neuron o. Not all functions are represented accurately by this combination of quantisation and approximation layers. various modern statistical function approximation methods (Breiman. be a correct class label. However. to obtain a better or locally more fine-grained quantisation).e. p 3. given a large set of exemplary decisions (the training set). A learning sample consists of input vector xp together with its correct class label y o. each table entry converges to the mean function value over all inputs in the subspace represented by that table entry.g. and Stone. e. if we expect our input to be (a subspace of) a high dimensional input space <n and we expect our function f to be discontinuous at numerous points. Of course this combination extends itself much further than the presented combination of the presented single layer competitive learning network and the single layer feed-forward network.. Friedman. 1994).g.

1. which represents a discretization of the input space. The topology-conserving quality of this network has many counterparts in biological brains..e.e. in the visual system. for g( ) a Gaussian function can be used. wk2 with the correct label is moved towards the input vector. The brain is organized in many places so that aspects of the sensory environment are represented in the form of two-dimensional maps.. the weights to this winning unit as well as its neighbours are adapted using the learning rule wo(t + 1) = wo(t) + gg(o.. the neurons in the network are folded in the input space. 14. Now. which can for example be used for visualization of the data. the winning unit k is determined. k)(x(t) wo(t)) . In the Kohonen network.16) Here. g(o. the neurons in S. Next. e.9. although this is chronologically incorrect.. For example. determines which output neurons are neighbours. Using the same formulas as in section 6.(14. how many next-best winners are to be determined. will be mapped on neighbouring neurons. There are organized mappings of the body surface .7). For example: data on a two. the Kohonen network has a different set of applications. This means that learning patterns which are near to each other in the input space (where near is determined by the distance measure used in finding the winning unit) must be mapped on output units. the weights to the output units are thus adapted such that the order present in the input space Â2 is preserved in the output. the dimensionality of S must be at least N. although this is application-dependent. Also. However. such as depicted in Fig. Thus the topology inherently present in the input signals will be preserved in the mapping. input signals. the learning patterns are random samples from ÂN. i. how to adapt the number of output neurons i and how to selectively use the weight update rule.g. which are also near to each other. a sample x(t) is generated and presented to the network. which are near to each other. The new LVQ algorithms that are emerging all use different implementations of these different steps. when learning patterns are presented to the network. the output units in S are ordered in some fashion. If the intrinsic dimensionality of S is less than N. k) = 1. which is chosen by the user. such as depicted in Fig. k) = exp ( (o k)2) (see Fig.e. 14. i.SELF-ORGANIZING NETWORKS 177 i. The ordering. the same or neighboring units. The mapping.. Usually. is said to be topology preserving. if inputs are uniformly distributed in ÂN and the order must be preserved. 1984) can be seen as an extension to the competitive learning network. if the inputs are restricted to a subspace of ÂN. how to define class labels yo. For example.dimensional manifold in a high dimensional input space can be mapped onto a two-dimensional Kohonen network. a Kohonen network can be used of lower dimensionality. there are several topographic mappings of visual space onto the surface of the visual cortex.3 KOHONEN NETWORK The Kohonen network (1982. Thus. while wk1 with the incorrect label is moved away from it. 14. 14. often in a twodimensional grid or array.. k) is a decreasing function of the grid-distance between units o and k. such that (in one dimension!) g(o. Due to this collective learning scheme. At time t. such that g(k.8.

9 The mapping of a two-dimensional input space on a one-dimensional Kohonen network. Iteration 0 Iteration 200 Iteration 600 Iteration 1900 Fig.75 0.5 0. .(o1 + 1. Fig.178 FUZZY LOGIC AND NEURAL NETWORKS 1 0. 14. In this case. 14.( i1 i 2 + 1) . 14.8 A topology-conserving map converging. the rightmost when the map is almost completely formed. g( ) is shown for a two-dimensional grid because it looks nice. A line in each figure connects weight wi . The leftmost figure shows the initial weights.7 Gaussian neuron distance function g( ).25 0 –2 –1 0 1 1 2 –2 –1 2 1 0 Fig. The weight vectors of a network with two inputs and 8 x 8 output neurons arranged in a planar grid are shown.(o1 o2 ) with weights wi . o2 ) and wi .

14. The use of topographic representations. To explain the plausibility of a similar structure in biological networks. and tonotopic mappings of frequency in the auditory cortex. Kohonen remarks that the lateral inhibition between the neurons could be obtained via efferent connections between those neurons. This can be intuitively verified by comparing the spreads (d x1. After the rotation. such that if we know x1 we can make a reasonable prediction of x2 and vice versa since the points are centered around the line x1 = x2. Makisara. The weights are adjusted in such a way that they could be considered as prototype vectors (vectorial means) for the input patterns for which the competing neuron wins. that already many applications have been devised of the Kohonen topology-conserving maps. The two dimensional samples (x1. the network has been used to merge sensory data from different kinds of sensors. de2 ) in the figures. Here the conditional prediction has no use because the points have uncorrelated coordinates. e2) axis as plotted in the figure. such as auditory and visual. . is so common that it obviously serves an important information processing function. It does not come as a surprise. and Gisbergen. An example is shown in Figure 14. inhibition to farther off neurons.11. If we rotate the axes over p/4 we get the (e1. the variance of the samples is large along the e1 axis and small along the e2 axis. In one dimension.10). and Saramaki. Excitation Lateral distance Fig. x2) are plotted in the figure. therefore. which map an input vector to a number of binary output elements or neurons. 1991). Another property of this rotation is that the variance or energy of the transformed patterns is maximized on a lower dimension. Kohonen himself has successfully used the network for phonemerecognition (Kohonen. 1984). It can be easily seen that x1 and x2 are related. The self-organizing transform described in this section rotates the input space in such a way that the values of the output neurons are as uncorrelated as possible and the energy or variances of the patterns is mainly concentrated in a few output neurons. Krommenhoek. Also. where some important aspect of a sensory modality is related to the physical locations of the cells on a surface.SELF-ORGANIZING NETWORKS 179 onto the cortex in both motor and somatosensory areas. 14.10 Mexican hat. d x2 ) and (de1 . looking at the same scene (Gielen. those connection strengths form a Mexican hat (see Figure 14. Lateral interaction around the winning neuron as a function of distance: excitation to nearby neurons.4 PRINCIPAL COMPONENT NETWORKS The networks presented in the previous sections can be seen as (nonlinear) vector transformations.

18) where L() indicates an operator which returns the vector length. but which scales the vector length to unity..180 FUZZY LOGIC AND NEURAL NETWORKS x2 dx2 e1 de2 de1 e2 x1 dx1 Fig. Now the operator which computes the vector length. and g is a small learning parameter.11 Distribution of input samples. The output yo(t) of this neuron is given by the usual inner product of its weight w and the input vector x: yo(t) = w(t)T x(t) . here the standard Hebb rule is. However. There the delta rule was normalized. the norm of the vector. 14. extending the theory in the last section to multidimensional outputs... the basic Hebbian rule would make the weights grow uninhibitedly if there were correlation in the input patterns.4.(14.(14. This transform is very closely related to the eigenvector transformation known from image processing where the image has to be coded or transformed to a lower dimension and reconstructed again by another transform as well as possible. The next section describes a learning rule which acts as a Hebbian learning rule.19) . 14.17) As seen in the previous sections. Compare this learning rule with the normalized learning rule of competitive learning.. all models are based on a kind of Hebbian learning.(14. can be approximated by a Taylor expansion around g = 0: L(w(t)) + gy(t) x(t) = 1 + g ¶L + O(g2) ¶g g = 0 . This can be overcome by normalising the weight vector to a fixed length.1 Normalized Hebbian Rule The model considered here consists of one linear neuron with input weights w.. which leads to the following learning rule w(t + 1) = w(t ) + gy(t ) x(t ) L( w(t ) + gy(t ) x(t )) . In the subsequent section we will see that a linear neuron with a normalised Hebbian learning rule acts as such a transform. typically 1..

..(14.3 More Eigenvectors In the previous section it was shown that a single neurons weight converges to the eigenvector of the correlation matrix with maximum eigenvalue. it resolves for small g(t ).20) Since dL = y(t)2 discarding the higher order terms of g leads to dg g =0 w(t + 1) = w(t) + gy(t) x(t)(x(t) y(t)w(t)) .24) Substituting this in the differential equation and concluding the theorem is left as an exercise..4.21) we see that the expectation of the weights for the Oja learning rule equals E(w(t + 1)|w(t)) = w(t) + g (Rw(t) (w(t)TRw(t))w(t)) which has a continuous counterpart .2 Principal Component Extractor Remember probability theory? Consider an N-dimensional signal x(t) with Mean m = E(x(t)). Proof: 1 Since the eigenvectors of R span the N-dimensional space.. From equation (6.(14..22) d w(t) = Rw(t) (w(t)T Rw(t)) w(t) dt . With equation (6.21) which is called the Oja learning rule (Oja..18).2: Let the eigenvectors ei of R be ordered with descending associated eigenvalues li such that l1 > l2 > ..23) Theorem 14. 1982). the weight vector can be decomposed as w(t) = å i N bi (t)ei ..(14.23) the weights w(t) will converge to ± e1. 14..g F GGH ¶L + o( g 2 ) ¶g g = 0 I JJK .(14. Correlation matrix R = E((x(t) m) (x(t) m)T). but normalizes its weight vector directly by the second product term yo(t) x(t) w(t).SELF-ORGANIZING NETWORKS 181 When we substitute this expression for the vector length in equation (6. > lN . i. the weight of the neuron is directed in the direction .. In the following we assume the signal mean to be zero.. What exactly does this learning rule do with the weight vector? 14.4.(14. so m = 0. the first product terms is the Hebb rule yo(t) x(t)... This learning rule thus modifies the weight in the usual Hebbian sense.e. 2 w(t + 1) = (w(t) + gy(t) x(t)) 1 .

so according to this definition in the limit we will find e2. then its weights will lie in the direction of the remaining eigenvector with the highest eigenvalue. 14. x If now a second neuron is taught on this signal ~.29) å i N aiei = ai . x= å aiei i N . from the signal x ~ =xa e x 1 1 . the coefficient a1 = 0.28) . Grossberg introduced a model for explaining biological phenomena. Consider the signal x which can be decomposed into the basis of eigenvectors ei of its correlation matrix R. Since the deflation removed the component in the direction of the first eigenvector.25) If we now subtract the component in the direction of e1. the data is not only fed forward but also back from output to input units.(14.(14.. as with networks in the next chapter. We call ~ the deflation of x.. The model has three crucial properties: .5.1 Background: Adaptive Resonance Theory In 1976. 14..(14.27) ~ =xy w x o The term subtracted from the input vector can be interpreted as a kind of a back-projection or expectation. We can continue this strategy and find all the N eigenvectors belonging to the signal x... In the previous section we ordered the eigenvalues in magnitude. the direction in which the signal has the most energy. We can write the deflation in neural network terms if we see that yo = wT x = e T 1 since w = e1 ~ equals So that the deflated vector x . Here we tackle the question of how to find the remaining eigenvectors of the correlation matrix given the first found eigenvector.182 FUZZY LOGIC AND NEURAL NETWORKS of highest energy or variance of the input patterns.... the weight will converge to the remaining eigenvector with maximum eigenvalue.5 ADAPTIVE RESONANCE THEORY The last unsupervised learning network we discuss differs from the previous networks in that it is recurrent..26) we are sure that when we again decompose ~x into the eigenvector basis.(14. simply x because we just subtracted it.. Compare this to ART described in the next section.(14.

it must be stored in the short-term memory. called F1 (the comparison layer) and F2 (the recognition layer) (see Fig.. Fig. 3. Each neuron in the comparison layer receives three inputs: a component of the input pattern. Contrast enhancement of input patterns. Distinguishing a hiding panther from a resting one makes all the difference in the world.12). Before the input pattern can be decoded. The classification is compared to the expectation of the network. and a reset module. whereas the STM is used to cause gradual changes in the LTM. which resides in the LTM weights from F2 to F1. The expectations. which are connected to each other via the LTM (see Category representation field STM activity pattern LTM STM activity pattern Feature representation field F1 LTM F2 Input Fig. the expectations are strengthened otherwise the classification is rejected. A normalization of the total network activity.12 The ART architecture.13). Short-term memory (STM) storage of the contrast-enhanced pattern. The long-term memory (LTM) implements an arousal mechanism (i. a component of the feedback pattern.e. A neuron outputs a 1 if and only if at least three of these inputs are high: the two-thirds rule.2 ART1: The Simplified Neural Network Model The ART1 simplified model consists of two layers of binary neurons (with values 1 and 0). Each neuron in F1 is connected to all neurons in F2 via the continuous-valued forward long term memory (LTM) W f. whereas classification takes place in F2. The mechanism used here is contrast enhancement. and a gain G1. residing in the LTM connections. F1 and F2.SELF-ORGANIZING NETWORKS 183 1. If there is a match. The awareness of subtle differences in input patterns can mean a lot in terms of survival. First a characterization takes place by means of extracting features. 14. As mentioned before. the input is not directly classified. 14. the classification). The other modules are gain 1 and 2 (G1 and G2). giving rise to activation in the feature representation field. and vice versa via the binary-valued backward LTM W b. the human eye can adapt itself to large variations in light intensities. . 14. Biological systems are usually very adaptive to large changes in their environment.5. 2. For example. The system consists of two layers. The input pattern is received at F1. translate the input pattern to a categorization in the category representation field. 14.

14. the reset signal will inhibit the neuron in F2 and the process is repeated.3 Operation The pattern is sent to F2. If there is a substantial mismatch between the two patterns. and only the neurons in F1 which receive a one from both x and F2 remain active. Gain 2 is the logical or of all the elements in the input pattern x. Gain 1 equals gain 2. The winning neuron then inhibits all the other neurons via lateral inhibition. Initialization: wb (0) = 1 ji f w ji = 1 1+ N . we use the notation employed by Lippmann (1987): 1. and in F2 one neuron becomes active. Gain 1 is inhibited. which reproduces a binary pattern at F1.13 The ART 1 neural network. Instead of following Carpenter and Grossbergs description of the system using differential equations.184 FUZZY LOGIC AND NEURAL NETWORKS F2 M neurons + G2 + j + W b +f W F1 N neurons – + G1 + i – + Reset Input Fig. the reset signal is sent to the active neuron in F2 if the input vector x and the output of F1 differ by more than some vigilance level.5. except when the feedback pattern from F2 contains any 1. The neurons in the recognition layer each compute the inner product of their incoming (continuousvalued) weights and the pattern sent over these connections. then it is forced to zero. Finally. 14. This signal is then sent back over the backward LTM.

choose the vigilance threshold r.5. 7.e.. Select the winning neuron k(0 £ k < M).14 shows exemplar behaviour of the network. Fig. Note that wb o x essentially is the k inner product x* o x. 5.SELF-ORGANIZING NETWORKS 185 where N is the number of neurons in F1. We will only discuss the first model.30) 4. The network incorporates a follow-the-leader clustering algorithm (Hartigan.4 ART 1: The Original Model In later work. which will be large if x* and x are near to each other.(14. ART 1. the distance between the new pattern and all existing classes exceeds some threshold. Re-enable all neurons in F2 and go to step 2. 3. Compute the activation values of the neurons in F2: y¢ = i åw j =1 N f ij (t)x1 . o £ l < N: wb (t + 1) = wb (t)xl kl kl f w lk (t + 1) = b wkl (t ) xl N 1 + 2 åw i =1 b ki ( t ) xi 8.propagation network. and 0 £ j < M. else go to step 6.... Also. 14. 0 £ i < N. The novelty in this approach is that the network is able to adapt to new incoming patterns. . If no matching class can be found. By changing the structure of the network rather than the weights. while the previous memory is not corrupted. 1975). 0 £ r £ 1. ART1 overcomes this problem. Vigilance test: if b wk (t ) o x >r xox .31) where o denotes inner product. 6. M the number of neurons in F2. Set for all l. all patterns must be taught sequentially. Apply the new input pattern x. go to step 7.(14. a new class is created containing the new pattern. Carpenter and Grossberg (1987) present several neural network models to incorporate parts of the complete theory.. In most neural networks. 14. the teaching of a new pattern might corrupt the weights for all previously learned patterns. Neuron k is disabled from further activity. i. 2. such as the back. This algorithm tries to fit each new input pattern in an existing class. Go to step 3.

14 An example of the behaviour of the Carpenter Grossberg network for letter patterns. 14.5. the weights of W b for the first four output units) are shown..e.5 Normalization of the Original Model We will refer to a cell in F1 or F2 with k. we set I = intensity Qk = sk I 1. the surroundings of each cell have a negative influence on the cell . In order to introduce normalization in the model. . So we have a model in which the change of the response yk of an input at a certain cell k depends inhibitorily on all other inputs and the sensitivity of the cell. i.186 FUZZY LOGIC AND NEURAL NETWORKS Backward LTM from Input pattern Output 1 Output 2 Output 3 Output 4 Not active Not active Not active Not active Not active Not active Not active Fig. On the right the stored patterns (i.yk ås k and let the relative input ås . 14. The binary input patterns on the left were applied sequentially..e. Each cell k in F1 or F2 receives an input sk and respond with an activation level yk. l l¹k has an excitatory response as far as the input at the cell is concerned +Bsk.

We can show that eq. If we set B £ (n 1)C or C/(B + C) ³ 1/n.. Here.37) Now. At equilibrium.35) åy k never exceeds B: it is normalized... we will revert to the simplified model as presented by Lippmann. (14.(14.. when dyk = 0... then all the yk are zero: the effect of C is enhancing differences. This can be done by adding an extra inhibitory input proportional to the inputs from the other cells with a factor C: dy k sl = Ayk + (B yk)sk (yk + C) dt l¹k å . has a decay Ayk .33) Because of the definition of Qk = sk I 1 we get yk = Qk BI A+ I . we have yk = 1 nCI Qk A+ I n FG H IJ K . The description of ART1 continues by defining the differential equations for the LTM. then more of the input shall be chopped off...32) does not suffice anymore. at equilibrium yk is proportional to Qk. since BI £B A+ I The total activity ytotal = .34) Therefore.(14.(14. we chop off all the equal fractions (uniform parts) in F1 or F2. when an input in which all the sk are equal is given. and with I = dt yk(A + 1) = Bsk ås we have that . contrast enhancement is applied: the contrasts between the neuronal values in a layer are amplified.6 Contrast Enhancement In order to make F2 react better on differences in neuron values in F1 (or vice versa)..SELF-ORGANIZING NETWORKS 187 has an inhibitory response for normalization yk sk.(14. The differential equation for the neurons in F1 and F2 now is dyk sl = Ayk + (B yk)sk . A and B are constants. . 14.yk dt l¹k å k ..36) At equilibrium. and.(14.(14. In order to enhance the contrasts.5. Instead of following Carpenter and Grossbergs description..32) with 0 £ yk (0) £ B because the inhibitory effect of an input can never exceed the excitatory input. when we set B = (n 1) C where n is the number of neurons..

Neural Networks. 1-141. Explain ART 1 neural network. A. J. 8. 10. 1.E. 7. 54115. K. 1989. pp. 37. 23. R. What are the advantages of self-organizing networks? What is competitive learning network? Explain various methods of determining the winner and the corresponding learning rule. 19. G. P. Vol. In A.H. Vol. 9. 2. R. S. Competitive learning algorithms for vector quantisation. and Image Processing. Nested networks for robot control. R.P. 20. Friedman.). 1991. Grossberg. Melton. 119-130. pp. 1994.F. Explain normalized Hebbian rule. Adaptive pattern classification and universal recoding I & II. Friedman. Krishnamurthy. 3. 187-202. 1988. 1.A. Neural Networks. Vol. Describe the vector quantisation scheme. Biological Cybernetics.C. Classification and Regression Trees. 7. K. Van der. . D. Grossberg. 4. 4. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Rumelhart.H. 10. Murray (Ed.A. 3. 5.H. Nielsen.A. Chen. Computer Vision. Cognitron: A self-organizing multilayered neural network. and S. 1976. No. Kluwer Academic Publishers. Breiman. 1990. 1985. 277-290. 121-136. 1. 9. 3. 1975. 121-134.K. pp.P. Olshen. S. 2. 6. What is Kohonen network? Explain. 1987. Describe the square error criterion to measure the quality of a given clustering. D. REFERENCES. 75-112. J. 9.Vol. Carpenter. Vol. Describe the normalization of ART 1. pp. and D. and F. 5. Neural Computation. 11. pp. 11. Vol. pp. Annals of Statistics. 6. Counterpropagation networks. Fukushima. Multivariate adaptive regression splines. Cognitive Science. 131-139. Zipser. Lippmann. Vol. Graphics. Describe the learning vector quantisation method. P. 1-38. Explain the counter propagation network. Smagt. Vol. A massively parallel architecture for a self-organizing neural pattern recognition machine. 12. Describe adaptive resonance theory. Neural Network Applications. Review of neural networks for speech recognition. Neural Networks. and C. Feature discovery by competitive learning. pp. Groen. 1988. 1. L. Fukushima. Ahalt. 1984. Vol. Biological Cybernetics.J. P. A. 8. Wadsworth and Broks/Cole. Stone. 1.188 FUZZY LOGIC AND NEURAL NETWORKS QUESTION BANK. Jansen. pp.C.

Speech. 16. Gielen. and T. and J. Hartigan. 17. G. 4919-4930. Hertzberger (Eds. 37. 1987. 121-134. Groen. Clustering Algorithms. 26(23).A.A. 1982. Berlin: Springer-Verlag. K. 54115. Biological Cybernetics. A procedure for self-organized sensor-fusion in topologically ordered maps. Vol. Carpenter. 4-22. C. T. 1984. 15. 13. In T. Kohonen. Springer-Verlag.4. Kohonen. and S. M.A. Saramaki. Phonotopic maps|insightful representation of phonological features for speech recognition. F. A simplified neuron model as a principal component analyzer. 1987. 20. T.SELF-ORGANIZING NETWORKS 189 12. S. 18. 2 No. Krommenhoek. 43. 1991. DUNNO. Associative Memory: A System-Theoretical Approach. pp. Lippmann. An introduction to computing with neural nets. 1982. Biological Cybernetics. T.). In Proceedings of the 7th IEEE International Conference on Pattern Recognition. Vol. 19. Grossberg. Elsevier Science Publishers. Applied Optics. Vol. Vol. J.P. . Self-organized formation of topologically correct feature maps. and Signal Processing. pp. IEEE Transactions on Acoustics. T. 21. G. 1977.A. Self-Organization and Associative Memory. Computer Vision. and Image Processing. Carpenter. 187-202. Oja. Kanade. Kohonen. Makisara. 59-69. 23. Graphics. A massively parallel architecture for a self-organizing neural pattern recognition machine. and S.C. 267-273. 417-423. E. 22. R. 1984.O. 1987. and L. 1976. 14. New York: John Wiley & Sons. Adaptive pattern classification and universal recoding I & II. ART 2: Self-organization of stable category recognition codes for analog input patterns. Journal of Mathematical Biology. Grossberg. 1975. Grossberg. pp. 15. pp. Gisbergen. Proceedings of the Second International Conference on Autonomous Systems. Kohonen.

On the other hand. Reinforcement learning involves two subproblems.2 THE CRITIC The first problem is how to construct a critic. which indicates how well the neural network is performing. existing of input and desired output values. 15. The second problem is to Find a learning procedure which adapts the weights of the neural network such that a mapping is established which minimizes J. not always such a set of learning examples is available. which is able to evaluate system performance. However.+ 0 ) 2 6 .4 Reinforcement Learning 15 15. Fig.1 shows a reinforcement-learning network interacting with a system. respectively. learning controller Fig. 15. This temporal credit assignment problem is solved by learning a critic network which represents a cost function J predicting future reinforcement. If the objective of the network is to minimize a direct measurable quantity r. 15. The two problems are discussed in the next paragraphs.1 Reinforcement learning scheme.1 INTRODUCTION In the previous chapters a number of supervised training methods have been described in which the weight adjustments are calculated using a set of learning samples. Often the only information is a scalar evaluation r. Critic $ J Reinforcement signal u System x Reinf. how is current behaviour to be evaluated . The First is that the reinforcement signal r is often delayed since it is a result of network outputs in the past. performance feedback is straightforward and a critic is not required.

REINFORCEMENT LEARNING 191

if the objective concerns future system performance? The performance may for instance be measured by the cumulative or future error. Most reinforcement learning methods (Barto, Sutton and Anderson (1983) use the temporal difference (TD) algorithm (Sutton, 1988) to train the critic. Suppose the immediate cost of the system at time step k are measured by r(xk, uk, k), as a function of system states xk and control actions (network outputs) uk. The immediate measure r is often called the external reinforcement signal in contrast to the internal reinforcement signal in Fig. 7.1. Define the performance measure J(xk, uk, k)of the system as a discounted cumulative of future cost. The task of the critic is to predict the performance measure: J(xk, uk, k) =

i=k

åy

¥

ik

r (xk, uk, k)

...(15.1)

in which g Î [0, 1] is a discount factor (usually » 0.95). The relation between two successive prediction can easily be derived: J(xk, uk, k) = r(xk, uk, k) + rJ(xk+1, uk+1, k + 1) be: ...(15.2)

**$ If the network is correctly trained, the relation between two successive network outputs J should $ $ J (xk, uk, k) = r(xk, uk, k) + r J (xk+1, uk+1, k + 1)
**

...(15.3)

If the network is not correctly trained, the temporal difference d(k) between two successive predictions is used to adapt the critic network:

$ $ d(k) = [r(xk, uk, k) + r J (xk+1, uk+1, k + 1)] J (xk, uk, k)

2

...(15.4)

A learning rule for the weights of the critic network wc(k), based on minimizing d (k) can be derived: Dwc(k) = ae(k) in which a is the learning rate.

$ ¶J ( xk , uk , k ) ¶wc ( k )

...(15.5)

15.3

THE CONTROLLER NETWORK

If the critic is capable of providing an immediate evaluation of performance, the controller network can be adapted such that the optimal relation between system states and control actions is found. Three approaches are distinguished: 1. In case of a finite set of actions U, all actions may virtually be executed. The action which decreases the performance criterion most is selected:

$ uk = min J (xk, uk, k)

u ÎU

...(15.6)

**192 FUZZY LOGIC AND NEURAL NETWORKS
**

The RL-method with this controller is called Q-learning (Watkins & Dayan, 1992). The method approximates dynamic programming which will be discussed in the next section. 2. If the performance measure J(xk, uk, k) is accurately predicted, then the gradient with respect to the controller command uk can be calculated, assuming that the critic network is differentiable. If the measure is to be minimized, the weights of the controller wr are adjusted in the direction of the negative gradient: Dwr(k) = b

$ ¶J ( xk , uk k ) ¶u(k ) ¶u( k ) ¶wr (k )

...(15.7)

with b being the learning rate. Werbos (1992) has discussed some of these gradient based algorithms in detail. Sofge and White (1992) applied one of the gradient based methods to optimize a manufacturing process. 3. A direct approach to adapt the controller is to use the difference between the predicted and the true performance measure as expressed in equation 15.3. Suppose that the performance measure is to be minimized. Control actions that result in negative differences, i.e. the true performance is better than was expected, then the controller has to be rewarded. On the other hand, in case of a positive difference, then the control action has to be penalized. The idea is to explore the set of possible actions during learning and incorporate the beneficial ones into the controller. Learning in this way is related to trial-and-error learning studied by psychologists in which behavior is selected according to its consequences. Generally, the algorithms select probabilistically actions from a set of possible actions and update action probabilities on basis of the evaluation feedback. Most of the algorithms are based on a look-up table representation of the mapping from system states to actions (Barto et al., 1983). Each table entry has to learn which control action is best when that entry is accessed. It may be also possible to use a parametric mapping from systems states to action probabilities. Gullapalli (1990) adapted the weights of a single layer network.

15.4

BARTOS APPROACH: THE ASE-ACE COMBINATION

Barto, Sutton and Anderson (1983) have formulated reinforcement learning as a learning strategy, which does not need a set of examples provided by a teacher. The system described by Barto explores the space of alternative input-output mappings and uses an evaluative feedback (reinforcement signal) on the consequences of the control signal (network output) on the environment. It has been shown that such reinforcement learning algorithms are implementing an on-line, incremental approximation to the dynamic programming method for optimal control, and are also called heuristic dynamic programming (Werbos, 1990). The basic building blocks in the Barto network are an Associative Search Element (ASE) which uses a stochastic method to determine the correct relation between input and output and an Adaptive Critic Element (ACE) which learns to give a correct prediction of future reward or punishment (Fig. 15.2). The external reinforcement signal r can be generated by a special sensor (for example a collision sensor of a mobile robot) or be derived from the state vector. For example, in control applications, where the state s of a system should remain in a certain part A of the control space, reinforcement is given by:

**REINFORCEMENT LEARNING 193
**

Reinforcement T ACE WC1 WC2 WCn

$ T Internal

Reinforcement detector

reinforcement

Decoder

WS1 WS2 ASE WSn

yo

System

State vector

Fig. 15.2:

Architecture of a reinforcement learning scheme with critic element.

r=

R0 S1 T

if s Î A otherwise

...(15.8)

15.4.1

Associative Search

In its most elementary form the ASE gives a binary output value yo(t) Î{0, 1}; as a stochastic function of an input vector. The total input of the ASE is, similar to the neuron presented in chapter 2, the weighted sum of the inputs, with the exception that the bias input in this case is a stochastic variable N with mean zero normal distribution: s(t) =

åw

j =1

N

sj x j

(t) + Nj

...(15.9)

The activation function F is a threshold such that yo(t) = y(t)

R1 S0 T

if s(t ) > 0 otherwise

...(15.10)

For updating the weights, a Hebbian type of learning rule is used. However, the update is weighted with the reinforcement signal r(t) and an eligibility ej is defined instead of the product y0(t) xj(t) of input and output: wsj(t + 1) = wsj(t) + ar(t) ej(t) where a is a learning factor. The eligibility ej is given by ... (15.11)

**194 FUZZY LOGIC AND NEURAL NETWORKS
**

ej(t + 1) = dej(t) + (1 d) y0(t) xj(t) ...(15.12) with d the decay rate of the eligibility. The eligibility is a sort of memory; ej is high if the signals from the input state unit j and the output unit are correlated over some time. Using r(t) in expression (15.11) has the disadvantage that learning only finds place when there is an $ external reinforcement signal. Instead of r(t), usually a continuous internal reinforcement signal r (t) given by the ACE, is used. Barto and Anandan (1985) proved convergence for the case of a single binary output unit and a set of linearly independent patterns x p. In control applications, the input vector is the (n-dimensional) state vector s of the system. In order to obtain a linear independent set of patterns x p, often a decoder is used, which divides the range of each of the input variables si in a number of intervals. The aim is to divide the input (state) space in a number of disjunct subspaces or boxes. The input vector can therefore only be in one subspace at a time. The decoder converts the input vector into a binary valued vector x, with only one element equal to one, indicating which subspace is currently visited. It has been shown (Krose and Dam, 1992) that instead of a-priori quantisation of the input space, a self-organizing quantisation, based on methods described in this chapter, results in a better performance.

**15.4.2 Adaptive Critic
**

The Adaptive Critic Element (ACE, or evaluation network) is basically the same as described in section 7.1. An error signal is derived from the temporal difference of two successive predictions (in this case denoted by p!) and is used for training the ACE:

$ r (t) = r(t) + gp(t) p(t 1)

...(15.13) ...(15.14)

p(t) is implemented as a series of weights wCk to the ACE such that p(t) = wCk if the system is in state k at time t, denoted by xk = 1. The function is learned by adjusting the wCk s $ according to a delta-rule with an error signal d given by r (t):

$ DwCk(t) = b r (t)hj(t)

...(15.15) ...(15.16)

b is the learning parameter and hj (t) indicates the trace of neuron xj: hj(t) = lhj(t 1) + (1 l) xj(t 1) This trace is a low-pass filter or momentum, through which the credit assigned to state j increases $ while state j is active and decays exponentially after the activity of j has expired. If r (t) is positive, the $ action u of the system has resulted in a higher evaluation value, whereas a negative r (t) indicates a $ deterioration of the system. r (t) can be considered as an internal reinforcement signal.

15.4.3

The Cart-Pole System

An example of such a system is the cart-pole balancing system (see Fig. 15.3). Here, a dynamics controller must control the cart in such a way that the pole always stands up straight. The controller applies a left or right force F of fixed magnitude to the cart, which may change direction at discrete time intervals. The model has four state variables:

**REINFORCEMENT LEARNING 195
**

q

F

x

Fig. 15.3: The cart-pole system.

x the position of the cart on the track, q the angle of the pole with the vertical,

& x & q

the cart velocity, and the angle velocity of the pole.

Furthermore, a set of parameters specify the pole length and mass, cart mass, coefficients of friction between the cart and the track and at the hinge between the pole and the cart, the control force magnitude, and the force due to gravity. The state space is partitioned on the basis of the following quantisation thresholds: 1. x : ±0.8, ±2.4 m 2. q : 0°, ±1°, ±6°, ±12°

& 3. x : ±0.5, ±¥ m/s

& 4. q : ±50, ±¥ °/s

This yields 3 ´ 6 ´ 3 ´ 3 = 162 regions corresponding to all of the combinations of the intervals. The decoder output is a 162-dimensional vector. A negative reinforcement signal is provided when the state vector gets out of the admissible range: when x > 2.4, x < 2.4, q > 12° or q < 12°. The system has proved to solve the problem in about 75 learning steps.

15.5

REINFORCEMENT LEARNING VERSUS OPTIMAL CONTROL

The objective of optimal control is to generate control actions in order to optimize a predefined performance measure. One technique to find such a sequence of control actions which define an optimal control policy is Dynamic Programming (DP). The method is based on the principle of optimality, formulated by Bellman (1957): Whatever the initial system state, if the first control action is contained

1992).18) from which uk can be derived. The Bellman equations follow directly from the principle of optimality. uk. The temporal difference e(k)between the true and expected performance is again used: $ $ e(k) = [g min J (xk+ 1.. of states and actions.. Solving the equations backwards in time is called dynamic programming. The requirements are a bounded N.17) and (15. In practice. k) (the name Q-learning comes from Watkins notation). k)] u ÎU . then the remaining control actions must constitute an optimal control policy for the problem with as initial system state the state remaining from the first control action. k) = min [Jmin xk+ 1.196 FUZZY LOGIC AND NEURAL NETWORKS in an optimal control policy. the performance measure could be defined as a discounted sum of future costs as expressed by equation 15. Sutton. control actions and disturbances. 1992): Jmin (xk. k)] J (xk . which is assumed to be an exact representation of the system and the environment. Q... RL is therefore often called an heuristic dynamic programming technique (Barto. uk. k) = g Jmin(xk+ 1. The most directly related RL-technique to DP is Q-learning (Watkins & Dayan.20) Watkins has shown that the function converges under some pre-specified conditions to the true optimal Bellmann equation (Watkins & Dayan. The equations for the discrete case are (White & Jordan.(15. uk. & Wilson. k + 1) + r(xk . uk+ 1. Reinforcement learning provides a solution for the problem stated above without the use of a model of the system and environment.6. (2) the learning parameter a must converge to zero. uk . (Werbos. k k $ The estimate of minimum cost J is updated at time step k + 1 according equation 7.. a solution can be derived only for a small N and simple systems. & Watkins. k + 1) + r(xk. uk. 1992): (1) the critic is implemented as a look-up table. uk+ 1. 1990). .(15.17) . Barto. k) with r being the immediate costs.. k) = å J( x . The minimum costs Jmin of cost J can be derived by the Bellman equations of DP. and a model.(Sutton. k) . In order to deal with large or infinity N. 1992). where Q is the minimum discounted sum of future costs Jmin(xk.5. The model has to provide the relation between successive system states resulting from system dynamics. k) u ÎU . the notation with J is continued here: $ J (xk. uk . k + 1) + r(xk . For convenience. The basic idea in Qlearning is to estimate a function.18) Jmin (xN) = r(xN) The strategy for finding the optimal control actions is solving equation (15.(15. This can be achieved backwards. starting at state xN. (3) all actions continue to be tried from all states.(15. u ...2. uk . Assume that a performance measure J(xk. uk+ 1. 1992).19) $ The optimal control rule can be expressed in terms of J by noting that an optimal control action for $ state x is any action u that minimizes J according to equation 7. is i i i=k N to be minimized. uk.

6.W. and M. What are the building blocks of Barto network? Explain them. What are the various approaches of control networks used to find optimal relation between system states and control actions? 3.). and Adaptive Approaches. Machine Learning. A. Neural Networks. White (Eds. 279-292. 3.S. Neural. Neuronlike adaptive elements that can solve difficult learning problems. 1983.S. Watkins. 7. P. Dam. Barto. and P. 4. 11. A menu for designs of reinforcement learning over time. Van Nostrand Reinhold. REFERENCES. White (Eds. Describe dynamic programming to find a sequence of acontrol actions. 3.A. Werbos (Eds. 1983.). D. Sutton. 4. V. R. 13. Vol.G. 1990. Explain reinforcement learning scheme. pp. 834-846. pp. 295-300 1992. 8.T. In D. and D.S.W. In Proceedings of IFAC/IFIP/IMACS International Symposium on Artificial Intelligence in Real-Time Control. Vol. Fuzzy. In D. Man and Cybernetics. R. IEEE Transactions on Systems.REINFORCEMENT LEARNING 197 QUESTION BANK.). Vol. Sofge & D. R. Anderson. White. Luxemburg.M. 8. 9-44. 1992. Anandan. Machine Learning. In W.). and P. Neuron like adaptive elements that can solve difficult learning problems. Van Nostrand Reinhold. New York. Sofge & D. Anderson. Neural Networks for Control. Handbook of Intelligent Control.J. 1988. Jordan. and J. Optimal control: a foundation for intelligent control. 6. Q-learning. pp. Van Nostrand Reinhold. Bellman. 5.S. Vol. pp. D. 1. White (Eds. C.J. 1. 15.G. 1985. Delft: IFAC. Approximate dynamic programming for real-time control and neural modeling. Gullapalli. Vol. & P. 3. 1992.G. Krose. Handbook of Intelligent Control. B.M. 1992. 2. New York. 12. Handbook of Intelligent Control. Sofge. 2. Vol. A stochastic reinforcement learning algorithm for learning real-valued functions. III. Barto. Describe the Barto network of reinforcement learning. 1990. Werbos. 1957. Pattern-recognizing stochastic learning automata. 834-846. Princeton University Press. Fuzzy.H. 360-375. Dynamic Programming. A. pp. 671-692.W. In D. IEEE Transactions on Systems. and C. pp. R. Barto. P. 13. Man and Cybernetics. Sutton. 10. 5. and Adaptive Approaches. pp. Fuzzy. Neural. C. and Adaptive Approaches. Applied learning: optimal control for manufacturing. A. Sofge & D. MIT Press/Bradford. Dayan. 1992. C. R. Explain the cast-pole balancing scheme. IEEE Transactions on Systems. . Neural. Werbos. Learning to avoid collisions: A reinforcement learning paradigm for mobile robot manipulation. 9.J. Learning to predict by the methods of temporal differences. Sutton. Man and Cybernetics. White. New York. Sutton.W.

and C. Sutton. Watkins. and Adaptive Approaches. 19-22. Van Nostrand Reinhold. and P. P. Sofge & D. 8. Advances in Neural Information Processing II. and R. pp.).G. 16. Sutton. Touretsky (Ed.S. 6. R.J. In D. Handbook of Intelligent Control. 1992. Werbos. IEEE Control Systems. 14.S. In D. Q-learning. 1990. C. DUNNO. White (Eds. 1992. A. Vol. Vol. R. . C.198 FUZZY LOGIC AND NEURAL NETWORKS 13. 279-292. Watkins. 15. Sequential decision problems and neural networks. Approximate dynamic programming for real-time control and neural modeling. A. 1992. Wilson. Dayan. Machine Learning. pp. Fuzzy. Neural. Reinforcement learning is direct adaptive optimal control. New York. Barto. Barto.).H.

10. facial recognition. chemical product design analysis. . Credit Card Activity Checking: Neural networks are used to spot unusual credit card activity that might possibly be associated with loss of a credit card. They then replace complex and costly equipment used for this purpose in the past. object discrimination. voice synthesis. warranty activity analysis. 4. Entertainment: Animation. feature extraction and noise suppression. Aerospace: High performance aircraft autopilot. creditline use analysis.+ 0 ) 2 6 . Financial: Real estate appraisal. analysis of grinding operations. chip failure analysis. 6. target tracking. signal/image identification. 9. Industrial: Neural networks are being trained to predict the output gasses of furnaces and other industrial processes. nonlinear modeling. radar and image signal processing including data compression. portfolio trading program. market forecasting.4 Neural Networks Applications 16.1 INTRODUCTION 16 A list of some applications mentioned in the literature follows: 1. product optimization. beer testing. 11. corporate financial analysis. aircraft component simulation. welding quality analysis. dynamic modeling of chemical process system. corporate bond rating. Defense: Weapon steering. 3. new kinds of sensors. flight path simulation. 5. Insurance: Policy application evaluation. visual quality inspection systems. 8. Manufacturing: Manufacturing process control. machine vision. planning and management. special effects. sonar. computer-chip quality analysis. 2. loan advisor. paper quality prediction. aircraft control systems. machine maintenance analysis. process and machine diagnosis. real-time particle identification. integrated circuit chip layout. Electronics: Code sequence prediction. product design and analysis. autopilot enhancements. mortgage screening. process control. 7. currency price prediction. Banking: Check and other document reading. Automotive: Automobile automatic guidance system. project bidding. credit application evaluation. aircraft component fault detection.

. Robotics: Trajectory control. 15. 1989): Forward kinematics Inverse kinematics Dynamics Trajectory generation 16. EEG and ECG analysis. the forward kinematic problem is to compute the position and orientation of the tool frame relative to the base frame (see Fig. manipulator controllers. 13. In robotics. Within this science one studies the position.1). text-to-speech synthesis. Oil and Gas: Exploration. Specifically.2 ROBOT CONTROL An important area of application of neural networks is in the field of robotics. Medical: Breast cancer cell analysis. Speech: Speech recognition. the major task involves making movements dependent on sensor data. vision systems. and of multiple solutions. A very basic problem in the study of mechanical manipulation is that of forward kinematics. Another applications include the steering and path-planning of autonomous robot vehicles. which cause it.1 Forward Kinematics Kinematics is the science of motion. acceleration. automated information services. emergency-room test advisement. Telecommunications: Image and data compression. real-time translation of spoken language. stock trading advisory systems. Securities: Market analysis. these networks are designed to direct a manipulator. speech compression. There are four related problems to be distinguished (Craig. routing systems. Transportation: Truck brake diagnosis systems. vowel classification. Because the kinematic equations are nonlinear. forklift robot. hospital expense reduction. 17.200 FUZZY LOGIC AND NEURAL NETWORKS 12. Usually.2.2 Inverse Kinematics This problem is posed as follows: given the position and orientation of the end-effector of the manipulator.2. optimization of transplant times. vehicle scheduling. customer payment processing systems. to grasp objects. and all higher order derivatives of the position variables. Solving this problem is a least requirement for most robot control systems. 14. based on sensor data. The inverse kinematic problem is not as simple as the forward one. their solution is not always easy or even possible in a closed form. arise. This is the static geometrical problem of computing the position and orientation of the endeffector (hand) of the manipulator. which treats motion without regard to the forces. hospital quality improvement. automatic bond rating. calculate all possible sets of joint angles which could be used to attain this given position and orientation. This is a fundamental problem in the practical use of manipulators. 16. 16. Also. 16. 18. given a set of joint angles. velocity. prosthesis design. 16. the questions of existence of a solution. which is the most important form of the industrial robot.

.2. acceleration). representing the inverse kinematics in combination with sensory transformation).1 An exemplar robot manipulator.2.g. e.5 End-Effector Positioning The final goal in robot manipulator control is often the positioning of the hand or end-effector in order to be able to. pick up an object. a complex set of torque functions must be applied by the joint actuators.g. this task is often relatively simple.2. In the first section of this chapter we will discuss the problems associated with the positioning of the end-effector (in effect. In order to accelerate a manipulator from rest. Typically.. from the image frame determine the position of the object in that frame. 16. this is done with a number of fixed cameras or other sensors which observe the work scene. and perform a pre-determined coordinate transformation. controlled fashion each joint must be moved via a smooth function of time. This is because the weight of the object has to be added to the weight of the arm (thats why robot arms are so heavy. 16. involving the following steps: Determine the target coordinates relative to the base of the robot. previous positions. but also the physical properties of the robot are taken into account. If a robot grabs an object then the dynamics change but the kinematics dont.NEURAL NETWORKS APPLICATIONS 4 3 tool frame 201 2 1 base frame Fig. Its responds to a control signal depends also on its history (e. and finally decelerate to a stop. The robot arm has a memory. In dynamics not only the geometrical properties (kinematics) are used. Exactly how to compute these motion functions is the problem of trajectory generation. 16. which determines the force required to change the motion of the arm. Take for instance the weight (inertia) of the robot arm. 16.3 Dynamics Dynamics is a field of study devoted to studying the forces required to cause motion. glide at a constant end-effector velocity. With the accurate robot arm that are manufactured.4 Trajectory Generation To move a manipulator from here to there in a smooth. speed. The dynamics introduces two extra problems to the kinematic problems. when this position is not always the same. making the relative weight change very small).

5a Involvement of Neural Networks So if these parts are relatively simple to solve with a high accuracy. a neural network uses these samples to represent the whole input space over which the robot is active. less rigid) robot systems. . Constructing the mapping N() from the available learning samples. since in useful applications R() is an unknown function. both on the sensory and motor side. but has the problem that the input space is of a high dimensionality. There are two problems associated with teaching N(): 1. In each of these approaches. and a robot arm. We will discuss three fundamentally different approaches to neural networks for robot end-effector positioning. This controller then generates a joint position q for the robot: q = N(xtarget. need frequent recalibration or parameter determination.e.202 FUZZY LOGIC AND NEURAL NETWORKS With a precise model of the robot (supplied by the manufacturer). When the (usually randomly drawn) learning samples are available. the inverse kinematics). systems. xhand) . yet still with accurate models as starting point) are required and the system must be calibrated..2. Gripper control is not a trivial matter at all. 16.2. This is not trivial.. The target position xtarget together with the visual position of the hand xhand are input to the neural controller N(). 16. and the samples are randomly distributed. calculate the joint angles to reach the target (i.. a form of self-supervised or unsupervised learning is required.2). Finally.e. Instead. xhand) .6 Camera-Robot Coordination in Function Approximation The system we focus on in this section is a work floor observed by fixed cameras.. a solution will be found for both the learning sample generation and the function representation.(16. but we will not focus on that. Also.(16.. Some examples to solve this problem are given below. Move the arm (dynamics control) and close the gripper. why involve neural networks? The reason is the applicability of robots. (8. This is a relatively simple problem. When traditional methods are used to control a robot arm. The visual system must identify the target as well as determine the visual position of the end-effector. which suffer from wear-and-tear. 2. Generating learning samples which are in accordance with eq.. the development of more complex (adaptive!) control methods allows the design and use of more flexible (i.1) We can compare the neurally generated qq with the optimal qq0 generated by a fictitious perfect controller R(): q0 = R(xtarget.2) The task of learning is to make the N generate an output close enough to q0. accurate models of the sensors and manipulators (in some cases with unknown parameters which have to be estimated from the systems behavior. This is evidently a form of interpolation.

3. Sideris and Yamamura (1988). which is constrained to two-dimensional positioning of the robot arm. A Jacobian matrix of a multidimensional function F is a matrix of partial derivatives of F..e. if we have Y = F(x). This target point is fed into the network. i..6a Approach-1: Feed-forward Networks When using a feed-forward system for controlling the manipulator. which generates an angle vector q. x2. Here. a self-supervised learning system must be used. the mapping II). Indirect learning: In indirect learning. xn) y2 = f2(x1. Thus the network can directly minimize |q q¢|.2. The manipulator moves to position q. the network is used in two different places: first in the forward step.. x2. General learning: The method is basically very much like supervised learning.. e.. . Specialized learning: Keep in mind that the goal of the training of the network is to minimize the error at the output of the plant: Î = x x¢..e. . This method requires knowledge of the Jacobian matrix of the plant. In each cycle. the network. This x¢ again is input to the network. Three methods are proposed: 1.g... then for feeding back the error. One such a system has been reported by Psaltis. The network is then trained on the error e1 = q q¢ (see Fig. learns by experimentation. 2. minimization of e1does not guarantee minimization of the overall error e = x x¢.. i.2 Indirect learning system for robotics. xn) . and the cameras determine the new position x¢ of the end-effector in world coordinates. y1 = f1(x1. Correct choice of q may pose a problem.. We can also train the network by backpropagating this error trough the plant (compare this with the backpropagation of the error in Chapter 12).. the network often settles at a solution that maps all xs to a single q (i. x Neural network q x¢ Plant e1 q¢ Neural network Fig. a Cartesian target point x in world coordinates is generated. by a two cameras looking at an object. but here the plant input q must be provided by the user. For example. . For example. However. 16.. resulting in q¢. xn) M ym = fm(x1.e.NEURAL NETWORKS APPLICATIONS 203 16. x2.. the multidimensional form of the derivative. The success of this method depends on the interpolation capabilities of the network. 16.2).

(16.(16.(16. The learning rule applied here regards the plant as an additional and unmodifiable layer in the neural network.3) is also written as dY = J(X) dX . + 2 dxn ¶x1 ¶ x2 ¶x n M dym = or Eq. + m dxn ¶x1 ¶ x2 ¶x n dY = ¶F dX ¶X .(16. in this case we have Jij = ¶fm ¶f ¶f dx1 m dx2 + . When the plant is an unknown function. + 1 dxn ¶x1 ¶ x2 ¶x n ¶f2 ¶f ¶f dx1 + 2 dx2 + .. (16.. so. instead of .6) di = x x¢ where I is used to change the scalar qj into a vector. A somewhat similar approach is taken in (Krose. The total error Î = x x¢is propagated back through the plant by calculating the dj: @j = F(sj) åd i i ¶ P ( q) i ¶q j ...... be approximated by ¶Pi (q) can ¶q j ¶Pi (q) Pi (q + hq j e j ) Pi (q) » ¶q j ¶q j . However..3) LM ¶P OP N ¶q Q i i . 1991).204 FUZZY LOGIC AND NEURAL NETWORKS then dy1 = dy2 = ¶f1 ¶f ¶f dx1 + 1 dx2 + ... Again a two-layer feed-forward network is trained with back-propagation. the Jacobian matrix can be used to calculate the change in the function when its parameters change..5) where Pi(q) the ith element of the plant output for input q. Now...4) where J is the Jacobian matrix of F.. (16. 1990) and (Smagt and Krose. Korst. and Groen...7) This approximate derivative can be measured by slightly changing the input the plant and measuring the changes in the output.

Measure the distance from the current position to the target position in camera domain. Groen. Use this distance. the task is to move the hand such that the object is in the centre of the image and has some predetermined size (in a later article. x¢. Calculate the move made by the manipulator in visual domain. 2. A feed-forward network with one layer of sigmoid units is capable of representing practically any function. x. an accurate representation of the function that governs the learning samples is often not feasible or extremely difficult (Jansen et al. x Ril + 1 x¢. a biologically inspired system is proposed (Smagt. calculating a desired output vector the input vector which should have invoked the current output vector is reconstructed. Smagt. Again measure the distance from the current position to the target position in camera domain. and Groen. 1993). This system has shown to learn correct behavior in only tens of iterations. The reason for this is the global character of the approximation obtained with a feed-forward network with sigmoid units: every weight in the network has a global effect on the final approximation that is obtained. smooth function consisting of a summation of sigmoid functions. & Krose. 1992) in which the visual flow-field is used to account for the monocularity of the system. By using a feed-forward network. 4. 5. 1991. . and to be very adaptive to changes in the sensor or manipulator (Smagt & Krose. such that the dimensions of the object need not to be known anymore to the system). where Ril + 1 is the rotation matrix of the second camera image with respect to the first camera image.NEURAL NETWORKS APPLICATIONS 205 X Neural Network q Plant X¢ e Fig. Krose. Send Dq to the manipulator. and back-propagation is applied to this new input vector and the existing output vector. as input for the neural network. although a reasonable representation can be obtained in a short period of time. The configuration used consists of a monocular manipulator. Dq) to the network. The network then generates a joint displacement vector Dq. But how are the optimal weights determined in finite time to obtain this optimal representation? Experiments have shown that.3 The system used for specialized learning. which has to grasp objects. 16. One step towards the target consists of the following operations: 1. 3.. the available learning samples are approximated by a single. 1994). Teach the learning pair (x Ril + 1 x¢. q. 6. together with the current state q of the robot. Due to the fact that the camera is situated in the hand of the robot.

the observed location of the object x (a four-component vector) is input to the network. 16. The system is observed by two fixed cameras which output their (x. As with the Kohonen network. To correct for the discretization of the working space. In the gross move. this small displacement in Cartesian space is translated to an angle change using the Jacobian Ak: qfinal = qk + Ak(x wk) which is a first-order Taylor expansion of q this fine move are in xf . y) coordinates of the object and the end effector (see Fig. The neurons.206 FUZZY LOGIC AND NEURAL NETWORKS Building local representations is the obvious way out: every part of the network is responsible for a small subspace of the total input space.. During gross move qk is fed to the robot which makes its move. The system described by Ritter et al. an additional move is made which is dependent of the distance between the neuron and the object in space wk x. resulting in retinal coordinates xg of the end-effector. Martinetz. This is typically obtained with a Kohonen network. i. which has to grab objects in 3D-space. We will only describe the kinematics part.. 16. the neuron k with highest activation value is selected as winner. 16.2. Fig.4). Thus accuracy is obtained locally (keep it small and simple).. With each neuron a vector q and Jacobian matrix A are associated.e.4 A Kohonen network merging the output of two cameras. and Schulten (1989) describe the use of a Kohonen-like network for robot control. Each run consists of two movements. The final retinal coordinates of the end-effector after . because its weight vector wk is nearest to x. since it is the most interesting and straightforward. which are arranged in a 3-dimensional lattice. consists of a robot manipulator with three degrees of freedom (orientation of the end-effector is not included).6b Approach 2: Topology Conserving Maps Ritter. correspond in a 11 fashion with subregions of the 3 D workspace of the robot.(16. the neuronal lattice is a discrete representation of the workspace. final .8) .

But the application of neural networks in this field changes these requirements. It appears that after 6. but by carefully choosing the basis functions. jk a distance function is used such that gjk (t) and g¢ (t) are Gaussians depending on the distance between jk neurons j and k with a maximum at j = k. Thus eq.10). A)* is obtained as follows: q* = qk + Ak(x xf) A* = Ak + Ak(x wk xf + xg) . The desired trajectory qd (t). the change in retinal coordinates of the end-effector due to the fine movement. They describe a neural network which generates motor commands from a desired trajectory in joint angles. In fact.1 without wrist joint. the basis functions are thus chosen that the function that is approximated is a linear combination of those basis functions.(16. the system is a feed-forward network. and the robot is not too susceptible to wear-and-tear. is fed into the inverse-dynamics model (Fig.(16. Again.2. this is similar to perceptron learning. Here. and Suzuki (1987).e. A)jold) jk j If gjk(t) = g¢ (t) = djk.000 iterations the system approaches correct behavior. i.7 Robot Arm Dynamics While end-effector positioning via sensor{robot coordination is an important problem to solve. . 16.000 learning steps no noteworthy deviation is present. Furukawa..9). (16.. the related joint angles during fine movement. One of the first neural networks which succeeded in doing dynamic control of a robot arm was presented by Kawato. the network can be restricted to one learning layer such that finding the optimal is a trivial task. 16. 16. In this case. A)jnew = (q..NEURAL NETWORKS APPLICATIONS 207 Learning proceeds as follows: when an improved estimate (q. This error is then added to qk to constitute the improved estimate q* (steepest descent minimization of error). the final error x xf in Cartesian space is translated to an error in joint space via multiplication by Ak. The network is extremely simple. A)* has been found. as with the Kohonen learning rule. the following adaptations are made for all neurons j: wjnew = wjold + g(t) gjk(t) (x wjold) (q. An improved estimate (q. The manipulator used consists of three joints as the manipulator in Fig. A)jold + g¢(t) g¢ (t) ((q. Dx = xf fg. the robot itself will not move without dynamic control of its limbs. i. (16.5).. A)* (q.10) can be recognized as an error-correction rule of the Widrow-Hoff type for Jacobians A.. and that after 30.e..9) = Ak + (Dq Ak D x) ( x f xg )T || x f x g ||2 Dx T || Dx||2 . (16. Dynamics model. The error between qd(t) and q(t) is fed into the neural model. Their system does not include the trajectory generation or the transformation of visual coordinates to body coordinates. which is generated by another subsystem. accurate control with nonadaptive controllers is possible only when accurate models of the robot are available.10) In eq. This requirement has led to the current-day robots that are used in many factories. and Dq = Ak(x wk). In eq.

.2.4)T . 2. 16. 3) . the weights adapt using the delta rule g dwik = xik T1 = xik(Tjk Tik). each one feeding in one joint of the manipulator. (k1.0.3) dt Kvk = 0 unless |qk(t) qdk(objective point) | < e The feedback gains Kp and Kv were computed as (517. 16.7. After 20 minutes of learning the feedback torques are nearly zero such that the system has successfully learned the transformation. training with a repetitive pattern sin (wkt). 16.. 16.6. Next.(16. The desired trajectory qd = (qd1. 8. qd 2(t). 2 : 3 is . qd 3(t)) and fl and gl as in Table 16. qd 2(t). qd 3(t)) xl 2 = xl3 = gl (qd 1(t).2. The feedback torque Tf (t) in Fig.12) A desired move pattern is shown in Fig.. Although the applied patterns are very dedicated. The resulting signals are weighted and summed.1.5 The neural model proposed by Kawato et al.2. 37.. consists of three perceptrons.4)T and (16.(16.2.5 consists of Tfk(t) = Kpk(qdk(t) = qk(t)) + Kvk åw i =1 13 lk xlk (k = 1. qd3) is feed into 13 nonlinear subsystems. such that Tik(t) = with xl1 = f1(qd1(t). which is shown in Fig.208 FUZZY LOGIC AND NEURAL NETWORKS Inverse dynamics model q d( t ) Ti(t) + Tf (t) + K T(t) Manipulator q(t) + – Fig. qd2.11) dq(t ) .2.3) . 191. dt (k = 1. The neural model. 746. with w1: w2: w3 = 1: also successful.

3(t) x13. joint 1 in Figure 16. 1(t) 209 f1 S f13 x13.NEURAL NETWORKS APPLICATIONS x1. one per joint in the robot arm.1: l 1 2 3 4 5 6 7 8 9 10 11 12 13 Nonlinear transformations used in the Kawato model. Table 16. f1(q1. q2.6 The neural network used by Kawato et al. There are three neurons. q2. 2(t) x13. 1(t) x1. 3(t) qd 1(t) qd 2(t) qd 3(t) T1(t) T2(t) Ti1(t) g1 S Ti2(t) Ti3(t) S g13 T3(t) Fig. Each neuron feeds from thirteen nonlinear subsystems. 16. the other two neurons to joints 2 and 3. q3) && q2 && q3 && cos q q2 3 && cos q q3 3 && 2 sin q cos q q1 2 2 & q 2 sin (q2 + q3) cos (q2 + q3) 1 & q 2 sin q2 cos (q2 + q3) 1 & q 2 cos q2 sin (q2 + q3) 1 & q 2 sin q3 2 & q 2 sin q3 3 & q q 2 &&3 sin q3 & q2 & q3 && sin (q + q ) q1 2 3 2 && cos2 (q + q ) q1 2 3 && sin q sin (q + q ) q1 2 2 3 && q sin q cos q q1 & 2 2 2 && q sin (q + q ) cos (q + q ) q1 & 2 2 3 2 3 && q sin q cos (q + q ) q1 & 2 2 2 3 && q cos q sin (q + q ) q1 & 2 2 2 3 && q sin (q + q ) cos (q + q ) q1 & 3 2 3 2 3 && q sin q cos (q + q ) q1 & 3 2 2 3 && q1 . q3) && q1 && sin2 q q1 2 && cos q q1 2 2 g1(q1. 2(t) x1. The upper neuron is connected to the rotary base joint (cf.1).

1998). (a) The training of paradigm is much faster than the back-propagation technique. Joints 2 and 3 have similar time patterns. The milling operations can be monitored with the neural network. ART type unsupervised neural network paradigm was used for detection of tool breakage. 1995). Adaptive Resonance Theory (ART 2) architectures are neural networks that carry out stable self-organization of recognition codes for arbitrary sequence of input patterns. In this section. Neural networks are also classified as supervised and unsupervised according to their learning characteristics. tool breakage can lead to irreparable damage to the workpiece and possibly to the machine tool itself. (b) The back-propagation technique generalizes the given information in order to store it inside the initially selected hidden layers. the neural network classifies the signals by itself.3 DETECTION OF TOOL BREAKAGE IN MILLING OPERATIONS The recent trend in manufacturing is to achieve integrated and self-adjusting machining systems. after training. One of the most important monitoring requirements is a system capable of detecting tool breakages on-line. ART paradigm was used for the following reasons. The back propagation technique cannot give reliable decisions on the sufficiency of previous training. and . The cutting force variation characteristics of normal and broken tools are different. The use of adaptive resonance theory (ART) type neural network was evaluated for detections of tool breakage. are now constructed.210 FUZZY LOGIC AND NEURAL NETWORKS q1 p 0 -p 10 20 30 t/s Fig. Unless recognized in time. ensuring safe and efficient metal removal rate and taking corrective actions in the event of failures and disturbances (Yusuf. Artificial neural networks refer to a group of architectures of the brain (Cheng and Sheng. Neural networks with parallel processing capability and robust performances provide a new approach to adaptive pattern recognition. which are capable of machining varying parts without the supervision of operators. 1993). 16. The absence of human supervision requires on-line monitoring of machining operation. which no longer need a very rigid structure to simplify the controller. 16. Also simulation-based training is proposed to reduce the cost of preparing the systems that monitor the real cutting signals. in this study (Ibrahim and Mclaughlin.7 The desired joint pattern for joint 1. With the normal and broken tool cutting force variation signals is possible to train neural networks. The usefulness of neural algorithms is demonstrated by the fact that novel robot architectures. In unsupervised learning.

Simulated data was used to select the best vigilance of the ART 2 type neural network and to evaluate the performance of paradigm. The STM is divided into two sets of nodes... in addition to the comparison of the bottom-up and top-down signals needed for the reset mechanism.15) .(16. Another important issue is the training of the neural network.. If the input pattern is matched with known pattern in memory.NEURAL NETWORKS APPLICATIONS 211 (c) ART has very important advantage since it can be trained in the field and continuously updates previous experience.18) wi = Si + aui pi = ui + S q( yi) z ji qi = pi e + || p|| wi e + || w|| vi = f(xi) + bf(qi ) xi = . this bottom-up pattern is compared to the top-down. and stored in the LTM (zji).16) ... and the long term memory (LTM) nodes. control.17) . The F1 field in ART 2 includes a combination of normalization and noise suppression.(16. which are permanent and stable. the short term memory (STM) nodes.3.. The STM F1 nodes are used for normalization. The ART 2 neural networks developed by Carpenter and Grossberg (1991) self-organize recognition codes in real time.(16. F1 uses the following equations to calculate the nodes: ui = vi e + ||v|| . it is coded and classified as a new category.(16. If the new pattern cannot be classified in a known category. To overcome this problem. where it is normalized. 16. (1987).. Adaptive resonance occurs when the input to a network and the feed back expectancies match. learned. the weights of the model are changed to update the category.. To accomplish this. It is extremely expensive and time consuming to collect cutting force data at different cutting conditions with normal and broken tools.. 1991).. or already known patterns. F1 and F2.1 Unsupervised Adaptive Resonance Theory (ART) Neural Networks The theory of adaptive resonance networks was first introduces by Carpenter and Grossberg. simulation-based training of neural networks was introduced.. The input pattern ( i ) is received by the STM. When as ART network receives an input pattern.(16.14) . The basic ART 2 architectures consist of two types of nodes. The theoretical background of ART 2 type neural network. The unsupervised ART neural networks can monitor the signal based on previous experience and can update itself automatically while it is monitoring the signals (Carpenter and Grossberg.. the proposed data monitoring system and their performance is presented in the paper. matched. gain and learning procedures. which are temporary and flexible.13) .(16.

. N otherwise . F2 equations select or activate nodes in the LTM...(16. F2 input pattern (bottom-up) is the key property that is used for node selection. 2 . ..(16. The non-linear signal function in equation (5) is used for noise suppression .20) The jth node is selected if equation (16.22) Equation (3) takes the following form: pi = Ru Su + dz T if F2 is inactive if jth node is F2 is active ij . and si is the input...(16..23) The bottom-up and top-down LTM equations are bottom-up (F1 ® F2) : dzij dt topbottom ( F2 ® F1) : = g(yj) [pi zij] . 2. all other nodes in the LTM are inhibited.212 FUZZY LOGIC AND NEURAL NETWORKS Here ||p||.20) is satisfied. then equations (16. Tj = max { Tj: 1. The STM F2 nodes are used for the matching procedure..25) When F2 is active. and e are selected based on the particular application. The constants a.(16.(16..24) dzij dt = g(yj) [pi zij] .. 2004] Tj = åpz i i ji . N} . v and w. q can be set to zero for the case where filtering is not desired.24) and (16. The output function of F2 is given by g(yi) = Rd S0 T i i if Ti : j = 1. f(x) = Rx S0 T if x ³ q if 0 £ x ³ q . The function f filters the noise from the signal.25) are modified from equation (16... Bottom-up inputs are calculated as in ART 2 [Fauselt. When F2 chooses a node. ||v|| and ||w|| denote the norms of the vectors p.(16. b.. the activation function (f ) is given by the equation.. and only one is allowed to interact with the STM. The node that gives the largest sum with the F1.22) to: ...19) where q is an appropriate constant...21) Competition on F2 results in contrast enhancement where a single winning node is chosen each time.(16.

In all the tests. and depth of cut of these different conditions are out lined in Tables 16.016 Feed rate mm/min 50.016 1.26) where d is a constant ( 0 < d < 1).2-16. Vigilance of the neural network was 0.8 101. the neural network inspected the resultant force profile and placed it into a category or initiated a new category if it was found to be different. On the other hand. 1. 3rd (Table16. In each one.07 mm diameter at various cutting conditions. The vigilance of the ART 2 selected either 0. the neural network assigned more nodes to the signal of a good tool with offset. The LTM node weights are recalculated and the pattern is learned by the system. r: ri = ui + cpi e + ||u|| + ||cp|| . It indicates that the broken tool signals are more similar to each other at different cutting conditions compared to the force patterns of normal tools.2-16. a match has been found and the new pattern is learned by the system.2 Results and Discussion The experimental data was collected with a fo1ur flute end mill of 12.016 1.27) If ||r|| < r e. the neural network classified the good and broken tools in different categories. The ART neural network monitored the profile of the resultant force in different tests. Depth of cut (mm) 1.2 Classification of experimental data with the ART.96 or 0.4) and 4th (Table 16. If no match has been found after all nodes have been activated. the neural network generated only one category in the 2nd (Table 16.016 1. 1. then F2 resets another node. In the three tests.2 254 254 Tool condition G B G B G B G B Category 1 2 1 3 1 3 4 5 Spindle speed (rpm) 500 500 500 500 500 500 500 500 .5.NEURAL NETWORKS APPLICATIONS 213 dzij dt = d[pi zij] . 16. 3. if a new pattern can be matched to a known pattern by comparing with a given vigilance parameter. experiments were done at different feed rates with the good and broken tool. 1 and 3 different categories for the good tool.3.3) ..5) tests for the broken tools. As seen in Tables 16. 1 and 3 different categories were selected. For the broken tool 2...016 1.8 50.2 203..016 1. and the new pattern is stored. a new node is created. If ||r|| ³ r e . Table-16. An orienting ART 2 sub system is used to decide. The spindle speed.96.98 in all the tests. feed rate. The ART used four categories to classify all of the data.(16. The ART assigned 2.(16.016 1.6 101.016 1.6 203. The neural network did not have any prior information at the beginning of each test.5. 2.

524 1.2 254 254 Tool condition G B G B G B G B Category 1 2 3 2 1 2 3 2 Spindle speed (rpm) 500 500 500 500 500 500 500 500 Table 16.2 203. The ART 2 used four categories to classify all of the data.6 101.524 1. which requires a minimum number of nodes and has acceptable error rate.8 50.4: Classification of experimental data with the ART 2. After simulation training. Vigilance of the neural network was 0.524 1.98 is used. Depth of cut (mm) 1.8 50.98.2 254 254 Tool condition G B G B G B G B Category 1 2 1 2 1 2 1 2 Spindle speed (rpm) 700 700 700 700 700 700 700 700 The ART gained first experience on the simulation data and later. When the vigilance of 0.016 Feed rate mm/min 50.016 1.524 1.6 203.016 1. the neural network inspected the incoming signals and continued to assign new categories when different types of signals were encountered.6 203.524 1.016 1.214 FUZZY LOGIC AND NEURAL NETWORKS Table16. the neural network started to monitor the experimental data collected at different conditions.6 101.2 203. The studies focused on selection of the best vigilance. Vigilance of the neural network was 0. Vigilance of the neural network was 0.8 101.96.524 1.6 101.6 203.96.8 101. the network classified the perfect too input data into seven different categories and classified the broken tool input data into four different categories.524 1.524 1.524 1.016 1.8 50.524 1.3: Classification of experimental data with the ART.524 1. Depth of cut (mm) 1.524 1.524 Feed rate mm/min 50. The ART used three categories to classify all of the data.2 254 254 Tool condition G B G B G B G B Category 1 2 3 2 4 2 4 2 Spindle speed (rpm) 700 700 700 700 700 700 700 700 Table 16. Depth of cut (mm) 1.524 1.524 1. . The ART 2 used two categories to classify all of the data.8 101.2 203.016 1.5: Classification of experimental data with the ART 2.524 Feed rate mm/min 50.016 1.016 1.

Vol.T. Enumerate various applications of neural networks.M. Nested networks for robot control. A.J. Murray (Ed. 12. Introduction to Robotics. Explain the application self-supervised learning to control manipulator.A. Int.J. Kaynak (Ed. Kawato. A Fast learning ART 1 model without searching. Krose. and F. 28. Rep. 2. Tech. 2. B. pp. Korst. R. Addison-Wesley Publishing Company. University of Amsterdam. 1995. Proceedings of the 1991 International Conference on Artificial Neural Networks. A. T.). Robot Hand-Eye Coordination Using Neural Networks. 10. M. Int. Vol. J. M. No.). Kohonen. 351-356. 2. 8..A.NEURAL NETWORKS APPLICATIONS 215 QUESTION BANK. K. North Holland/Elsevier Science Publishers. No. Makisara. Neural Network Applications. Explain the application of Kohonon network for robot control. J. Schulten. Smagt. 545-558. Explain the application of neural network for robot arm dynamics. Vol. In T.J. K. pp. 8. Adaptive hamming net. & J. Kangas (Eds.H. pp.A. Nos.C. 1994. and F. 1. 11. 4. 5.J. 2. 17-21. pp. A. 2. B. 33. The neural network approach. 4. 1992. J. In T.). A hierarchical neural-network model for control and learning of voluntary movement. 169-185. P.J. and B. Mach. Smagt.P.C. Kangas (Eds. A multilayer neural network controller. In O. No. 1988.Tools Manufact. Krose.A. North-Holland/Elsevier Science Publishers. Kohonen. and A. and K. Groen. Groen. 3.A. 1998. pp. 199-203. IEEE. Psaltis. Krose. O.P. 3. 57.Cheng and F. P. 7. Groen.1991. Martinetz. A real-time learning neural robot controller. Simula. Using time-to-contact to guide a robot manipulator. IEEE International Workshop on Intelligent Motor Control. 9. 1989. Yusuf In-processs detection of tool breakages using time series monitoring of cutting forces. akisara.P. 605-618. 5. Ibrahim and C. N. 351-356. 1989.A. P.A. 157-172. Krose. Groen. pp. B.A. Mclaughlin. Furukawa. Mach. A. Krose. J. 159-168. F. of Neural Networks. & J. 1. Proceedings of the 1991 International Conference on Artificial Neural Networks. 6. O. Neural Networks. Yamamura.. & Suzuki.J. 1993. 4. Korst. 4. In Proceedings of the 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems. CS-93-10.A. 1993. In A. 1991. H. No. K. Vol. 8.F. Smagt. Sideris. Detection of tool breakage in milling II. 13. Int. Ritter. Craig. A real-time learning neural robot controller. IEEE Control Systems Magazine.Tools Manufact. pp.C. Kluwer Academic Publishers. and B.A. and F. Krose.C.M. pp. Sheng. M. and B.J. Department of Computer Systems. Jansen. Topology-conserving maps for learning visuomotor-coordination.J. Learning strategies for a vision based neural controller for a robot arm. (1987).J. Vol. Simula. D.).A.J. . Explain the application of ART for machining applications. 1990. REFERENCES. Biological Cybernetics.J.

17. Carpenter and Grossberg. Graphics. Grossberg. 493-504. Pearson Education. Fundamentals of Neural Networks. and S. G.A.A. 1987. 2004. Fauselt. ART 2-A: Adaptive resonance Algorithm for rapid category learning and recognition.216 FUZZY LOGIC AND NEURAL NETWORKS 14. 2nd Edition. 4919-4930. Grossberg. ART 2: Self-organization of stable category recognition codes for analog input patterns. 26(23). L. A massively parallel architecture for a self-organizing neural pattern recognition machine. of Neural Network. Architectures. pp. Applied Optics. Carpenter. 54115. and Applications. G. 1987. Vol. Computer Vision. Carpenter. and S. Vol. 1991. pp. Algorithms. 15. J. and Image Processing. Int. . 16. 37. 4.

in reasoning task)...2. 17.e. which act on two distinct subproblems. when talking about the neurofuzzy systems. In that case. Normally. The output of one technology becomes the input to another technology and so on. 17. This is one of the weakest forms of hybridization because an integrated combination of the technologies is not present.g. .1. i. on the hardware whereas fuzzy logic systems concentrate on software.4 Hybrid Fuzzy Neural Networks 17 17. Sometimes neurofuzzy is associated to hybrid systems.+ 0 ) 2 6 . neural network is utilized in the first subproblem (e. A simple sequential hybrid system is shown in Fig. The hybrid systems are classified as: Sequential hybrids Auxiliary hybrids Embedded hybrids 17.1 Sequential Hybrid Systems Sequential hybrid systems make use of technologies in a pipeline-like structure.g. In the following light is tried to shed on the most common different interpretations. in signal processing) and fuzzy logic is utilized in the second subproblem (e. Hybrid systems are those for which more than one technology is employed to solve the problem.2 HYBRID SYSTEMS Designation neurofuzzy has several different meanings. Neural networks concentrate on the structure of human brain..1 INTRODUCTION Neural networks and fuzzy systems try to emulate the operation of human brain. Combining neural networks and fuzzy systems in one unified framework has become popular in the last few years. the link between these two soft computing methods is understood to be stronger.

17. Inputs Technology A Technology B Output Fig. The auxiliary hybrid system is better than the sequential hybrid system.1 A sequential hybrid system. the technologies participating are integrated in such a manner that they appear intertwined.2. .3 illustrates an embedded hybrid system. Fig. one technology calls the other as a subroutine to process or manipulate information by it. 17. 17. 17.2. An auxiliary hybrid system is shown in Fig. The embedded hybrid system is better than sequential and auxiliary hybrid systems.2 An auxiliary hybrid system.218 FUZZY LOGIC AND NEURAL NETWORKS Inputs Technology A Technology B Outputs Fig. 17.2.3 Embedded Hybrid Systems In embedded hybrid systems. 17. The fusion is so complete that it would appear that no technology could be used without the others for solving the problem.2 Auxiliary Hybrid Systems In this. The second technology processes the information provided by the first and hands it over for further use.

ZE. . 17. The purpose is to achieve a faster rate of convergence by controlling the learning rate parameter with fuzzy rules. C o G o E is change of G o E (approximation to second order gradient). Rules are of the type: Rule 1: Rule 13: Rule 25: IF(G o E is NB) AND (C o G o E is NB) THEN (C o LP is NS) IF(G o E is ZB) AND (C o G o E is ZE) THEN (C o LP is PS) IF(G o E is PB) AND (C o G o E is PB) THEN (C o LP is NS) where C o LP is change of learning parameter. NB. zero equal. a fuzzy control of back-propagation is illustrated in Fig. For example. negative small. 17. G o E is the gradient of the error surface.3 FUZZY LOGIC IN LEARNING ALGORITHMS A common approach is to use fuzzy logic in neural networks to improve the learning ability. PS and PB are fuzzy sets negative big. (They also incorporated in rules information about the sign change of gradient and information about the momentum constant. MLP is multiplayer perceptron. 17.4 Learning rate control by fuzzy logic.HYBRID FUZZY NEURAL NETWORKS 219 Inputs Technology A Technology B Outputs Fig. NS.) Learning parameter performance Desired performance Error FLC MLP Output Actual performance Fig.3 An embedded hybrid system. FLC is fuzzy logic controller.3. positive small and positive big. 17.

These combinations are again combined (i. Another way to implement fuzzy neuron is to extend weights and/or inputs and/or outputs (or targets) to fuzzy numbers. wd)) .. Input and weight are combined (i. In addition there exists a type of network where the weights and targets are crisp and the inputs are fuzzy. crisp weights. We use the extension principle to calculate wixi Output fuzzy set Y is computed by the Extension principle Y(y) = R(w x + w x ) g S0 T 1 1 2 2 1( y ) if 0 £ y £ 1 otherwise . 17. . w1).(17. which means that the fuzzy neural nets based on the extension principle are universal approximators only for monotonic functions. where the inputs and weights are fuzzy numbers. .1) .e. we get an AND fuzzy neuron (17. w1). fuzzy input..(17. If we choose linear activation function. crisp input.1: Consider a simple network y = g(w1x1 + w2x2). The networks of this type are used in classification problems to map fuzzy input vectors to crisp classes. t-norm (min) for addition and t-conorm (max) for product. we get an OR fuzzy neuron (17. product is replaced) using t-norm y = T(S(x1. Example 17. S (xd.220 FUZZY LOGIC AND NEURAL NETWORKS Simulation results show that the convergence of fuzzy back propagation is faster than standard back propagation. RFNN) A regular fuzzy neural network is a network with fuzzy signals and/or fuzzy weights. and if they are chosen on the contrary.. or some other continuous operation.. (usually membership degrees) and real valued weights wi Î[0..1).2: (regular fuzzy neural network.3) where g1(y) = ln y ln (1 y) is simply the inverse function of logistic sigmoid g(z) = 1/(1 + e z). Output of (17. addition is replaced) using t-norm.. fuzzy output 3... fuzzy weights. t-conorm S(xi.. 1]. sigmoidal activation function and all the operations are defined by extension principle.e. . fuzzy input.2). wi). The choices are as follows: 1.(17.. fuzzy weights. 2.1) corresponds to min-max composition and (17. T (xd. or some continuous operation. fuzzy output which can be used to implement fuzzy IF-THEN rules. wd)) y = S (T(x1. 1]. The problem of regular fuzzy neural networks is that they are monotonic.. Activation function g can be any continuous function. t-conorm.2) T(xi...2) corresponds to max-min composition known from the fuzzy logic. fuzzy output (or crisp output by defuzzification).. Definition 17. wi)...4 FUZZY NEURONS Definition 17.1: (hybrid neural network) A hybrid neural network is a network with real valued inputs xi Î[0.

17.1: g is an increasing function of its arguments.5) .. and/or fuzzy valued weights wi Î [0.. The use of more than 4 inputs may be impractical. the number of combinations is 16. When the dimension of the problem increases the size of the fuzzy model (and the size of training set needed) grows exponentially.HYBRID FUZZY NEURAL NETWORKS 221 Theorem 17. and projecting the data onto this M-dimensional subspace.. These combinations are again combined using t-norm. there is a strong need for data reduction. Therefore. The most common method to decrease the dimension of input space is the principal component analysis (PCA). if x1 Ì x¢ and x2 Ì x¢ (xi. t-conorm. The algorithm goes as follows: 1. HFNN) A hybrid fuzzy neural network is a network with fuzzy valued inputs xi Î [0. if we have five inputs and each input space is partitioned into seven fuzzy sets. This reduction is achieved via a set of linear transformations. Therefore. The number of combinations of input terms (possibly also the number of rules) is Õm i d i . wi). Activation function g can be any continuous function. calculate covariance matrix and its eigenvectors and eigenvalues . then (w1x1 + w2x2) Ì (w1x¢ + w2 x¢ ) 1 2 .6) where mi is the number of fuzzy sets on axis i..(17. but there is no standard path to follow. Definition 17. which transform input variables to a new set of variables (uncorrelated principal components). This is a serious drawback for the networks of this type.e. For example. 1]. wi are 1 2 i fuzzy numbers).. Input and weight are combined using t-norm T(xi. wi).. compute the mean of inputs in data and subtract it off 2. Many researchers working with fuzzy neurons follow the basic principles described above.(17.5 NEURAL NETWORKS AS PRE-PROCESSORS OR POSTPROCESSORS One of the biggest problems with fuzzy systems is the curse of dimensionality. Selecting M attributes from d is equivalent to selecting M basis vectors which span the new subspace. The smallest number of input variables should be used to explain a maximal amount of information. t-conorm S(xi. 1].807. or some continuous operation. The main goal of identifying principal components is to preserve as much relevant information as possible.(17. or some other continuous operation. identifying principal reduce the dimensionality of a data in which there are large number of correlated variables and at the same time retaining as much as possible of the variation present in the data.4) which means that regular fuzzy neural network is not a universal approximator.3: (hybrid fuzzy neural network. x¢. i. then g(w1x1 + w2w2) Ì g(w1x¢ + w2 x¢ ) 1 2 Proof: Because min and max are increasing functions.

If two additional nonlinear hidden layers are allowed to put into the network. is represented as a network. The original purpose of neurofuzzy systems was to incorporate learning (and classification) capability to fuzzy systems or alternatively to achieve similar transparency (intelligibility) in neural networks as in fuzzy systems. the fuzzy logic system with d inputs one output approximator.222 FUZZY LOGIC AND NEURAL NETWORKS 3. The error function can then be minimized by using gradient based optimization algorithms. Since back-propagation can be applied to any feed forward network. 17. It was named error back-propagation. The principle of the method is that it finds the derivatives of an error function with respect to the weights in the network. Learning is assumed to reduce design costs. increase flexibility. 17. centers and widths of membership functions. some researchers began to represent fuzzy logic systems as feed forward networks. . A two-layer perceptron with linear output units (number of hidden units is M. Multivariate memb. retain eigenvectors corresponding to the M largest eigenvalues 4. a computationally efficient training algorithm for multi-layer neural networks was discovered. with M < d ) which is trained to map input vectors onto themselves by minimization of sum-of-squares error is able to perform a linear principal component analysis. The idea was to use the training algorithm to adjust weights. In Fig.5 Neurofuzzy network for back-propagation. which led to the development of neurofuzzy systems. 17. In the 1980s. the network can be made to perform a non-linear principal component analysis. project input vectors onto the eigenvectors Neural networks may be used to perform the dimensionality reduction.6 NEURAL NETWORKS AS TUNERS OF FUZZY LOGIC SYSTEMS The similarities between neural networks and fuzzy logic systems were noticed. If prior knowledge is unavailable and/or the plant is time-varying then the most sensible (possibly the only) solution is to utilize learning capabilities. improve performance and decrease human intervention. functions Pm X1 * * l WM Xd * * Pm k 1 Pm 1 W1 Norma lize l l + Denominator = Gaussian membership function Fig.5.

8 COMMITTEE OF NETWORKS The method of combining networks to form a committee has been used to improve the generalization ability of the networks. 17. Other parameters (width and position of membership functions) have clear meaning 3. 17. The performance of committee can be better than the performance of isolated networks. different types of neural networks. 17. Only the way to illustrate network differs.Consequent lizer parameters Neurofuzzy classifier x1 A1 A2 B1 B2 Class 1 Class 2 Class 3 Class 4 Class 5 y x2 Linguistic variables Fig..5. The most common way to represent neurofuzzy architecture is shown in Figure 17.HYBRID FUZZY NEURAL NETWORKS 223 Neurofuzzy function approximator x1 A1 A2 B1 B2 S y x2 Linguistic Multivariables plier Norma. 17.g. Although it looks different from the network in Fig. e. it is basically the same network. Initial weights can be chosen appropriately (linguistic rules) The drawback is the curse of dimensionality. It can consist of networks with different architectures. Weights are the centers of THEN part fuzzy sets (clear meaning) 2.6.7 ADVANTAGES AND DRAWBACKS OF NEUROFUZZY SYSTEMS The advantages are as follows: 1.6 And Or Max defuzzifier Neurofuzzy network for function approximation and classification problems. fuzzy logic systems and conventional models. .

He does not restrict the form of fuzzy system to be additive or SAM system. which provides a fuzzy weight of a connection between ith neuron of the input layer.7) The reduction of error arises from the reduced variance due to averaging.7 shows this model fuzzy neural network. and III. and jth neuron of the hidden layer. 17. we propose a learning algorithm from the cost function for adjusting three parameters of each strong L-R type fuzzy weight. 17.. Lastly we examine the ability of proposed fuzzy neural network implementing on fuzzy if-then rules. Input x1 Hidden Output y1 y2 x2 y3 Fig. Fig. Kosko (1991) has proposed the use of weighted average to combine different fuzzy systems that try to predict the same input-output relation. II.(17. Type (III) of fuzzy feed forward neural networks is presented here. the connections between the layers may be illustrated as a matrix of fuzzy weights wji. such that at least one system has nonzero credibility.224 FUZZY LOGIC AND NEURAL NETWORKS The committee prediction (output) is calculated as an average of the outputs of the q networks: yCOM(x) = 1 yi ( x ) q i =1 å q .7) is that Kosko weights fuzzy system outputs yi(x) by credibilities wi Î [0.7 A three layered fuzzy neural network According to the type of inputs and weights we define three different kinds of fuzzy neural networks as follows: I. fuzzy weight and crisp inputs. and show its good properties in interval arithmetic... a-cuts) of fuzzy weights and fuzzy inputs. In the fuzzy neural networks based on BP. 1].9 FNN ARCHITECTURE BASED ON BACK PROPAGATION The input-output relation is numerically calculated by interval arithmetic via level sets (i. While defining a cost function for level sets of fuzzy outputs and fuzzy targets. Next we define the strong L-R type fuzzy number. neurons are organized into a number of layers and the signals flow in one direction. In this model. The total fuzzy input of jth neuron in the second layer is defined as: . 17. There are no interactions and feedback loops among the neurons of same layer. fuzzy weight and fuzzy inputs.e. The only difference with (17. crisp weight and fuzzy inputs.

11) Opk = f (Netpk). . 2..... The cost function for the input-output pair (xp.. Opi = xpi is the ith fuzzy input of that neuron. fuzzy weights and fuzzy biases. Tp) is obtained as ep = åe h ph . 2.. Tp) is the fuzzy input-output pairs. and put forward a FNN algorithm ..(17. fuzzy weights and fuzzy biases. . and Qj is fuzzy bias of the jth neuron. The fuzzy output of the jth neuron is defined with the transfer function Opj = f(Neypj). this type fuzzy number has good property so that it can be easily adept to the interval arithmetic.9) åw j =1 NH O kj . Furthermore. . Next.12) The cost function for the h-level sets of the fuzzy output vector Op and the fuzzy target vector are defined as eph = where L epkh = e pkh + eU pkh 2 åe k =1 NO pkh . NH Furthermore. d[T = h. Tp NO) is NO dimensional fuzzy target vector corresponding to the fuzzy input vector xp.HYBRID FUZZY NEURAL NETWORKS 225 Netpj = å w . a .. j = 1.(17...cut) of fuzzy inputs..e. j = 1.(17.13) L e pkh d[T = h..8) Where..(17. Netpj is the total fuzzy input of the jth neuron of hidden layer. and Tp = (Tp1. NO The fuzzy output is numerically calculated for level sets (i.. pj + Qk . L pk ]h L [Opk ]h i 2 [Opk ]U h 2 eU pkh U pk ]h i 2 Next section we introduce the strong L-R type fuzzy number.. the fuzzy output of the kth neuron of output layer is defined as follows: Netpk = .(17.10) . we need to find out a type of fuzzy number to denote the fuzzy inputs...(17. Tp2. let (xp....O ji i =1 N pj + Qj ..

+ ¥ ]. v) of real numbers. the strong L-R type fuzzy number can be uniquely represented by three parameters. which is a wellestablished area of classical mathematics.) is a special class of the strong L-R type fuzzy number. a and b are called left and right spreads. wg )LR kj kj kj Wji = (wa.15) .9.(17.5: A fuzzy number M is said to be an L-R type fuzzy number if R LFG b x IJ |H a K | mM(x) = S |RFGH x b b IJK | T if x £ b a > 0 if x ³ b b > 0 .6: This kind of fuzzy number M is said to be an strong L-R type fuzzy number if L(1) = R(1) = 0. Definition 17. The a-cuts of every fuzzy number are closed intervals of real numbers 2. wg )LR ji ji ji Qk = (qa. Qk by these parameters as Wkj = (wa . The strong L-R type is an important kind of fuzzy numbers. arithmetic operations on fuzzy numbers can be defined in terms of arithmetic operation on closed intervals of real numbers. Since each fuzzy set is uniquely represented by its a-cut.N. S(x) = S (x). symbolically. These operations are the cornerstone of interval analysis. usually denoted L or R.14) L is the left and R for right reference.4: A function. in other words.1 Strong L. 3. Fuzzy numbers are convex fuzzy sets.. qg )LR k k k .226 FUZZY LOGIC AND NEURAL NETWORKS based BP. v)LR. Definition 17. M is the mean value of M. 3. Let (b x)| a = 1. w b .F. These are closed intervals of real numbers. 2. 17. We can write any strong L-R type fuzzy number symbolically as M = (a. we write M = (mab)LR. is a reference function of fuzzy numbers if 1. We will utilize them in next section to define arithmetic operations on fuzzy numbers. S(0) =1. The triangular fuzzy number (T. S is non-increasing on [0. Those properties are essential for defining meaningful arithmetic operations on fuzzy numbers.(17. This kind of fuzzy number has properties as follows: 1. w b. such that the support of every fuzzy number is the interval (a. wkj and fuzzy biases Qj. same as R(b + b) = R(v) = 0.. We express the strong L-R type fuzzy weight wji... L(b a) = L(a) = 0. qb. Accordingly we can adjust three parameters of each strong L-R type fuzzy weight and fuzzy biases.R Representation of Fuzzy Numbers Definition 17. x = b a º a such that. b.

HYBRID FUZZY NEURAL NETWORKS 227 Qj = (q a... wv ) between the jth kj kj kj hidden unit and the kth output unit... it is h-level and 0-level have relations as follows: [ wkj ]a = h y a wkj + ckj + wkj 1 + ckj y a wkj + ckj + wkj y [ wkj ]h = 1 + ckj LM (w w ) OP L (h) NM 1 + c QP L c (w w ) OP R (h) M NM 1 + c QP y kj a kj 1 .. wij ..22) .21) kj Therefore.. qg)LR j j j Let ckj = g b wkj wkj b a wkj wkj ..18) y ¶wkj ¶e ph = ¶[ wkj ]a h .(17. We discuss how to learn the strong L-R type fuzzy weight wk j = (wa . ck = q gj qb q g qb j k k . w b ..(17.. we can count the quantity of adjustment for each parameter by the cost function L Dwkj(t) = h L ¶wkj ¶e ph L + xDw k j (t 1) . Similar to Rumelhart. q b...(17..(17.20) kj kj y kj a kj 1 .(17.23) ..17) The derivates of above can be written as follows: a ¶wkj ¶e ph = ¶[ wkj ]a h ¶e ph ¶e ph y ¶[wkj ]a ¶e ph ¶[wkj ]h h + y a a ¶wkj ¶[ wkj ]h ¶wkj y ¶[wkj ]a ¶e ph ¶[wkj ]h h + y y y ¶wkj ¶[ wkj ]h ¶wkj ..(17.(17. and cj = b a b a qk qk qj qj Then b wkj = v a wkj + ckj + wkj 1 + ckj b b b b . qk have same form as wkj.. a ¶wkj ¶e ph = ¶e ph ¶[ wkj ]a h ¶e ph ¶[ wkj ]= h y ¶wkj ¶e ph = LM c NM1 + c LM c MN1 + c kj kj kj kj OP QP e L ( h) O PP + ¶[¶w 1+ c Q + 1 kj ¶e ph ckj ckj L1 ( h) + R 1 ( h) y 1 + ckj ¶[ wk j ] h 1 + ckj 1 + ckj ph y kj ]h LM NM LM 1 MN1 + c + kj ckj 1 + ckj R 1 OP QP O (h ) P PQ .19) Since wkj is a strong L-R type fuzzy number.(17.16) U Dw kj(t) = h U ¶wkj ¶e ph + xDw Uj (t 1) k . qk . cji = w g wb ji ji wb w a ji ji .

. . the fuzzy weight is updated by the following rules: a a w kj(t + 1) = w a (t) + Dw kj kj g w kj(t . that is to say. we assume that Api is a symmetrical strong L-R type fuzzy number. h1.(17.. So we note the fuzzy input as Ap = (Ap1..... . and Tp = (Tp1... Apn)... 2.27) We should train this network in order to make eph be minimum.. 4.(17. Ù. 1 |x|2) .26) according to the target output T and the real output O. the learning algorithm of the fuzzy neural network can be defined as follows: 1.228 FUZZY LOGIC AND NEURAL NETWORKS These relations explain how the error signals ¶e ph [ wk j ]a h and ¶e ph [ wk j ]a h for the h-levelset propagate to the 0-level of the strong L-R type fuzzy weight Wkj. In this way. . Forward calculation: Calculate the hlevel set of the fuzzy output vector Op corresponding to the fuzzy input vector xp. Where p = 1. and then. . Tp2. TpNo)whaere No dimensional fuzzy taget vector corresponding to the fuzy input vector xp. for the convenience of computing.. THEN xp = (xp1. Repeat the following procedure for p = 1.Api is lingistic term.. 2. the total number of iterations) is not satisfied... It is easy to know that the error function become the classical error function e = å (t p =1 k p op )2 / 2 in BP algorithm when input vector Ap ... h2.24) . go to 2. Initialize the fuzzy weights and the fuzzy biases. for example: large. Tp)). hn) are used for the learning of the fuzzy neural network.(17....(17.. and the target ouput Tp can be defined as follows: Tp = R1 S0 T if Ap Îclass 1 if Ap Îclass 2 .e. L = R + max (0. 17. small etc. xpn) beling to Gp.. . Back-propagation: Adjust the fuzzy weights and the fuzzy biases using the cost function cph. hn 3. We define the error function: eph = max {(tp op)2/2|Op Î[yp]h} .. We can solve the above problem by using the fuzzy neural network we discussed.9. If a pre-specified stopping condition (etc. h2. Let (xp. . m (m input-output pairs (xp.. Tp) is the fuzzy input-output pairs. and xpn is Apn. 2. Repeat 3 for h = h1.2 Simulation We consider a n-dimension classification problem. k. It can be described by IF-THEN rules as follows: IF xp1 is Ap1 and. .25) + 1) = g w kj(t) + Dw g kj We assume that n values of h (i..

8).10 ADAPTIVE NEURO-FUZZY INFERENCE SYSTEM (ANFIS) A fuzzy system can be considered to be a parameterized nonlinear map.(17.. We train the fuzzy neural network with h-levelsets (h = 0.. 0. we get a satisfied curve after 300 echoes.. Using the proposed learning algorithm.. The membership function m A1 (xi) corresponds to the input x = (x1. 17. (17. and A5-A8 belong to class 2. called f. 0. F m ( x )I JK å GÕ H f (x) = F I å G Õ m ( x )J H K m yl n l =1 i =1 Ail i m n . We assume A1-A4 belong to class1. Lets write here explicitly the expression of f..29) l =1 i =1 Ai1 i where y¢is a place of output singleton if Mamdani reasoning is applied or a constant if Sugeno reasoning is applied..2.. .5 14 A3 A4 8 6 Class 1 A2 A1 2 1 5 Fig.8.HYBRID FUZZY NEURAL NETWORKS 229 and yp are real numbers. The i and connective in the premise is carried out by a product and defuzzification by the center-of-gravity method.4. 17.8 A5 A6 A7 Class 2 A8 10 15 20 The result of fuzzy classification 17. 0. . xn) of the rule l. the error function of the pth pair is: eph = å h max {(t p p o p ) 2 / 2| Op Î[ y p ]h} .6.28) The result of the trained fuzzy neural network is shown in Fig. 20 18 16 0.

(17. b]. b].(17..33) max |F(xi) p(xi)| .6: If FÎC[a..e. then f can approximate F to any desired accuracy. . Remark: The message of Theorem 2 is that polynomials are dense in the space of continuous functions in Hilbert space.F.31) Well-known theorems from Approximation theory for polynomials...32) over all polynomials of degree £ n.. The following theorems are found in R.g. Prentice Hall.. We can also consider the simpler problem of approximating at finitely many points... nonlinear map on a compact set.(17. Curtain and A. and E = (x1. pn which minimizes å | F( x ) p( x )| i i i =1 k 2 . can be extended to fuzzy systems (e. and E = (x1.(17. .30) bj(x) = åÕm l =1 i =1 i =1 m n Õm n Ail ( xi ) Ail ( xi ) If F is a continuous.7: If F is a bounded function on [a.. b]... which minimizes n 0£i£ k . there exists a best approximating polynomial pnof degree £ n such that ||F pn||¥ £ ||F = p||¥ over all polynomials p of degree £ n. F » fFS . . b]..J..(17.5: Let F be a bounded function on [a. Then k there exists the least squares polynomial of degree £ n.34) over all polynomials of degree £ n. Pritchard: Functional Analysis in Modern Applied Mathematics as Corollaries of Orthogonal Projection Theorem. Then there exists a best approximating polynomial p k of degree £ n.. xk) a set of points in [a. Theorem 17. The same can also be said of trigonometric functions. 1997).230 FUZZY LOGIC AND NEURAL NETWORKS This can be further written as f (x) = where wi = y¢ m å w b ( x) i i i =1 . then for any n ³ 0. Wang: A course in fuzzy systems and control. i.. b]. Theorem 17.. xk) a set of points in [a. Theorem 17.

. then f1 = p1x + q1 y + r1 2.38) Then f can be written as f = w1 f1 + w2 f2 x A1 x A2 B1 B2 P w2 N w2 x Fig. i = 1. i = 1. choose product for T-norm (logical and).35) 2. 2 ...(17. 1.HYBRID FUZZY NEURAL NETWORKS 231 17. y) w1 f1 + w2 f2 w1 + w2 wi w1 + w2 .. If x is A2 and y is B2. Evaluating the implication and the rule consequences gives f (x..39) y P w1 N w1 w1f1 S w2f2 y f y . y) f2 ( x. Bi. y) + w2 ( x. y) = w1 ( x.37) This can be separated to phases by first defining wi = ..(17. m Bi .(17. 2.36) Or leaving the arguments out f= . Evaluating the rule premises results in wi = m Ai (x) m Bi (x). then f2 = p2x + q2 y + r2 Let the membership functions of fuzzy sets Ai. y) w1 ( x.(17. In evaluating the rules.10. 17..9 ANFIS structure .. If x is A1 and y is B1. be m Ai . y) f1 ( x.(17.1 ANFIS Structure Consider a Sugeno type of fuzzy system having the rule base 1... y) + w2 ( x.

Describe FNN architecture based on back propagation. Vol. Lee and B. Haykin. Kosko. 3. Explain the use of fuzzy logic in neural networks to improve the learning ability. 1991. NJ. Kartalopoulos. No. pp. Describe adaptive neuro-fuzzy inference system. 2. H. What are the different types of hybrid systems? Explain them schematically. 4. pp. Neural Networks: A Comprehensive Foundation. QUESTION BANK. Explain the role of neural networks as pre-processor or post-processor of data. Carpenter and S. IEEE computer society press. B.M. Grossberg. IEEE press. 3. REFERENCES. 2. Fuzzy BP: a neural network model with fuzzy inference. 1. S. 1583-1588. 1994. NY. Macmillan. Understanding Neural Networks and Fuzzy Logic. Prentice Hall.232 FUZZY LOGIC AND NEURAL NETWORKS All computations can be presented in a diagram form. NY. 21.A. 6. G. Liu. IEEE Computer Magazine. 5. Neural Network and Fuzzy Systems. 7.H. 5. Englewood. S. Define the following: Hybrid neural network Regular fuzzy neural network Hybrid fuzzy neural network 4. Proceedings of International conference on artificial neural networks. 1. 77-88. What is committee of networks? Explain. The art of adaptive pattern recognition by a self organizing neural network. . 3.

global competition in industry has led to exploration of new means of more efficient production. By themselves. flexible manufacturing systems (FMS) have been investigated as a tool for raising manufacturing productivity and product quality while decreasing production costs.+ 0 ) 2 6 . a detecting technology for unexpected tool breakage is needed (Lan and Naerheim.4 Hybrid Fuzzy Neural Networks Applications 18. 1986).2 TOOL BREAKAGE MONITORING SYSTEM FOR END MILLING In recent years. Since CNC machines cannot detect tool conditions.1 INTRODUCTION 18 The hybrid fuzzy neural networks have a tremendous potential to solve engineering problems. manufacturers must confirm that the tool is in good condition in process. monitoring equipment and algorithms for the adaptation of the manufacturing process must be executed accurately (Altintas. It is improper to expect that if the individual technologies are good then hybridization of technologies should turn out to be even better. computer numerical control (CNC) machines are not typically capable of tool breakage detection. The decision-making system analyzes information provided by sensors to make appropriate control actions. Hybridization is performed for the purpose of investing better methods of problem solving. Yellowley. 18. since human operators are absent in these systems. which has received a great deal of attention because it replaces human operators with robotic counterparts in manufacturing and assembly cells. One type of FMS is the Unmanned Flexible Manufacturing System (UFMS). To apply the UFMS effectively. electronic sensors associated with a decision-making system must monitor the process. In particular. However. To reduce costs of materials and prevent damaged tools from negatively affecting production. and Tlusty. automatic and rapid detection of tool breakage is central to successful UFMS operation. Materials costs increase and product quality suffers if a broken tool is used in production. they cannot halt the process if the tool becomes damaged. therefore. In addition to performing the same function as the FMS. . the UFMS reduces direct labour costs and prevents personal oversights. 1988). In order to ensure efficiency within the system.

1) Fri is the resultant force of point I . and Chen (1995) used a telemeter technique. An appropriate threshold was built to analyze information and detect tool conditions. the average and median forces of each tooth were used as input information. The neural networks approach was employed as the decision-making system that judges tool conditions. 18. and milling operations can be of two varieties: peripheral and face milling.1 Methodology: Force Signals in the End Milling Cutting Process Milling is a fundamental machining process in the operation of CNC machines. The application of neural networks and fuzzy logic in detecting tool breakage has also been studied in recent years. was applied by Tansel and McLaughlin (1993) to detect tool breakage in milling. generated from x and y directions..2. analyzing force signals and determining amplitude fluctuations allowed on-line tool breakage detection. Lan and Naerheim (1986) proposed a time series auto regression (AR) model of force signals to detect tool breakage. FR. The principle of cutting force can be further defined as resultant force. Zhang. The fuzzy-nets system was designed to build the rule-bank and solve conflicting rules with a computer. to measure force in milling operations. which used the fast a posterior error sequential technique (FAEST).(18. Cho. Ko.234 FUZZY LOGIC AND NEURAL NETWORKS An in-process tool breakage detection system was developed in an end milling operation with cutting force and machining parameters of spindle speed. A dynamometer sensor is the main device used to measure force signals in different machining operations. The resultant force.. In this study. Variance of adjacent peak force was selected as an input parameter to train the system and build a rule-bank for detecting tool breakage. Tae and Dong (1992) developed a fuzzy pattern recognition technique and a time-series AR model to detect tool wear in turning operations. was used in this experiment expressed as: Fri = Where Fx2i + Fy2 i . and Jung (1994) introduced an unsupervised self-organized neural network combined with an adaptive time-series AR modeling algorithm to monitor tool breakage in milling operations. meaning that each cutting tooth moving in the same direction generates a cyclic cutting force ranging from zero to maximum force. generated in x and y directions. The variation of dynamic cutting force was used to construct the fuzzy dispersion pattern needed to distinguish tool conditions. feed rate. Chen and Black (1997) also introduced a fuzzy-nets system to distinguish tool conditions in milling operations. Milling is an interrupted cutting process. a battery-powered sensing force/torque cutter holder mounted on the spindle head with the transmitter. Jemielniak (1992) proposed that sudden changes in the average level of force signals could be due to catastrophic tool failure (CTF) in turning operations. and depth of cut selected as input factors. The machining parameters and average peak force were used to build the AR model and neural network. The time-series-based tooth period model technique (TPMT). Han. Fri. and back to zero. Measured by sensors. The common method of detecting tool breakage in process involves force signals resultant from tool processes on raw materials. Tarng and Lee (1993) proposed using the average and median force of each tooth in the milling operation. This cyclic force is graphed as a series of peaks.

also called a summation or aggregated function.3) åa W jk where. has been proven effective in dealing with this kind of task... Applying the force principle. Wji and Wkj denote weight from input to hidden neuron. such as spindle speed. a back propagation neural network (BPNN) was chosen as the decision-making system because it is the most representative and commonly used algorithm. an in-process tool breakage detection system was developed in an end milling operation. and uses gradient descent on the error provided by the teacher to train the weights.1. Finally. nodes are grouped in layers). feed rate. as shown in Figure. 1 illustrates the diagram of undamaged and broken tools. Back propagation is intended for training layered (i. i is an input neuron. the tooth that follows a broken tooth generates a higher peak force as it extracts the chip that the broken tool could not. and there are no cycles) nets.2) . Maximum variance force of adjacent peaks should differ between good and broken tools. the neural networks approach was used as a decision-making system using input from sensors to judge tool conditions. feed forward (i. was used to combine or aggregate inputs passing through the connections from other neurons.(18. the arcs joining nodes are unidirectional.2 Neural Networks In this work. maximum peak force in each revolution should differ between good and broken tools. Tool conditions and machining parameters affect the magnitude of resultant force.. usually 1. 2. which requires a teacher that knows the correct output for any input. As a result. therefore.2. . two main differences can be used to detect tool breakage: 1.(18. Comparatively.e. and from hidden to output neuron respectively. In this work. j is a hidden neuron. Figure 18.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 235 Fxi is the force in X direction of point I Fyi is the force in Y direction of point i. It is relatively easy to apply. The propagation rule. the peak measurement of each tooths force should be roughly the same from tooth to tooth during one revolution of the cutting process. and depth of cut. and Wji and Wji are the weight of bias. if the tool condition is good. 18. while a0 represents the bias. were selected as input factors. As the weights of the neural network were obtained. and maximum peak force of a broken tool must be larger than that of a good tool. and maximum variance force of adjacent peaks of broken tools must be larger than in undamaged tools. 18. and has also proven successful in practical applications. This approach involves supervised learning.. if a tooth is broken.. it generates a smaller peak force because it carries a smaller chip load. and k is an output neuron. The cutting force and machining parameters. the prediction function was achieved via the weight information.e.. It can be expressed as Si = Sk = åa W i k ji a0Wjo a0Wjo .

feed = 15 ipm depth = 0.4) 1 + e ay where ay is a function of Sj and Sk respectively. . expressed as: 1 Oy = . and one is called the Sigmoid Function. Many different transfer functions can be used to transfer data. the process is repeated until the error percentage falls into a reasonable range. 18. is used to produce output based on level of activation. Comparing actual output of neural networks to desired output. The transfer function..236 FUZZY LOGIC AND NEURAL NETWORKS N 1200 Fa 1000 800 600 400 200 0 –200 One revolution b a Good Tool N 1600 1400 1200 1000 800 600 Broken Tool Fa a One revolution 400 200 0 – 200 Key: a = force signal b = revolution signal Cutting Parameters: speed = 650 rmp.08 inch b b Fig.. also called the output or squashing function.1 The amplitude of cutting force of a good and broken tool.(18.

and 650 revolution per minute). one side of the tool was in proper working order and the other side was broken. 2. 18. The broken side of the tool possessed varying degrees of breakage (Fig.2. The cutting parameters were set as: 1.3). 18. Five levels of depth of cut (0.3 Experimental Design and System Development Experimental Design This experiment employed a CNC vertical machining center. 550. 0. . and 3. 18.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 237 18. In each cutter. 0. Five level of feed rate (6.06. A dynamometer was mounted on the table to measure cutting force. 18 and 24 inch per minute).07.1 inches).2 The experimental setup. 12. 600. Four ¾-inch doubt-end four-flute high-speed steel cutters were used.2.09 and 0. 0. 500. A proximity sensor was built near the spindle to confirm data in each revolution. Five level of spindle speed (740.08. VM C40 Proximity Sensor Workpiece Dynamometer Amplifier A/D Board DC Power Supply Fig. The experimental setup was shown in Fig.

**238 FUZZY LOGIC AND NEURAL NETWORKS
**

T1 T2 T3 T4

1.5

1.5

3

2.5

1.5 Unit of value: mm.

2

2

3

Fig. 18.3 Diagram of broken tool.

The cutters used to execute the experiment were selected randomly. Cutting force was measured in voltage by the Charge Amplifier and transformed to Newtons (N) via computer.

18.2.4

Neural Network-BP System Development

To develop back propagation of neural networks as a decision-making system, MATLAB software was applied to analyze data. Seven steps were conducted. In step one, prediction factors were determined in order to perform the training process. Step 2 was necessary to analyze differences between scaling data and unscaling data. Step 3 dealt with separating data into training and testing categories. From steps 4 through 6, parameters were developed for the training process, including the hidden layer/hidden neuron, learning rate, and momentum factor. Finally, in step 7, information from the training process was used to predict tool conditions. Step 1. Determine the factors Five input neurons were used for tool breakage prediction data: 1. Spindle speed; 2. Feed rate; 3. Depth of cut; 4. Maximum peak force; and 5. Maximum variance of adjacent peak force. Output neurons were either (1) Good, or (2) Broken. Three hundred data points were used in this work. Good tools collected half of these and broken tools collected the rest, and all data were randomized using MS Excel software. Step 2. Analyze unscaling and scaling data In order to avoid experimental errors resulting from bigger values of some data sets, some preprocessing was needed to obtain good training and prediction results. Since histograms of all data sets

HYBRID FUZZY NEURAL NETWORKS APPLICATIONS

239

were uniform or normal distributions, the Simple Linear Mapping method was employed for scaling. To compare the difference between two sample sizes, some parameters were first set and fixed. The number of hidden neurons was set at 4, the learning rate was set at 1, and the momentum item was 0.5. The number of training cycles was 2000, and the testing period was 5. Table 18.1 shows the comparison of the difference between scaling and unscaling data. As one can see, errors in scaling data are smaller than in unscaling data.

Table 18.1 Difference between scaling and non-scaling

Hidden layer Unscale Scale 1 1 Hidden neuron 4 4 Learn rate 1 1 Momentum factor 0.5 0.5 Train Error 0.505 0.040 Testing Error 0.550 0.160 RMS error of training 0.508 0.197 RMS error of testing 0.515 0.388

Step 3. Impact of the ratio of training and testing data The 300 original 300 data records were randomized and separated into three groups. The first group had 200 training data and 100 (200 ´ 100) testing data, the second had 225 training and 75 (225 ´ 75) testing data, and the third had 250 training and 50 testing (250 ´ 50) data. Table 18.2 shows the Back Propagation Neural Network (BPNN) with different sample sizes of training and testing data. The last four columns of Table 18.2 show training and testing errors. The training, testing, and RMS (root mean square) errors of training of the second group were smaller than in other groups. The RMS errors of testing data of the second group sample were larger than in the first; however, the RMS errors of each sample size were very similar. If samples had similar error percentages, the sample with the largest training sample size was selected because it provided sufficient information to predict testing data. From the experimental design, the ideal ratio between training and testing data was 3:1 for neural networks. The 225 ´ 75 sample size was employed in this analysis.

Table 18.2:

Tra*Tes 200* 100 225* 75 250* 50 Hidden layer 1 1 1 Hidden neuron 4 4 4

**Different sample size of training and testing data
**

Learn rate 1 1 1 Momentum factor 0.5 0.5 0.5 Train Error 0.040 0.036 0.044 Testing Error 0.160 0.093 0.106 RMS error of training 0.197 0.185 0.204 RMS error of testing 0.388 0.298 0.285

Step 4. Impact of the hidden layer and hidden neuron In the beginning, the number of hidden neurons was set at 5, and the hidden layer was set at 1. Different hidden neurons and layers were tested to determine which values would lead to the smallest error percentage. To this end, the hidden neurons were set at 4 and 6, and the hidden layers were set at 1 and 2. Table 18.3 shows the BPN with a different number of hidden neurons and layers. According to this data, the percentage error of the trial with 4 hidden neurons and 1 hidden layer was less than it was in all

**240 FUZZY LOGIC AND NEURAL NETWORKS
**

other trials. Thus, the configuration contained in the 4 hidden neuron/1 hidden layer experiment was chosen because it led to the best results. The formula, (input neurons + output neurons)/2, was useful for determining the number of hidden neurons at the beginning.

Table 18.3: Different number of hidden neurons and layers

Neuron in layer-1 3 4 5 3 4 4 4 Neuron in layer-2 ~ ~ ~ 3 2 3 4 Learn rate 1 1 1 1 1 1 1 Momentum factor 0.5 0.5 0.5 0.5 0.5 0.5 0.5 Train Error 0.080 0.036 0.049 0.267 0.489 0.316 0.164 Testing Error 0.200 0.093 0.093 0.320 0.453 0.333 0.227 RMS error of training 0.256 0.185 0.193 0.338 0.512 0.407 0.362 RMS error of testing 0.410 0.298 0.288 0.375 0.504 0.425 0.414

Step 5. Impact of the learning rate This step was necessary to determine the optimal learning rate. The initial learning rate was 1. Three additional learning rates, 0.5, 2, and 10, were used to compare with the initial. Table 18.4 shows the BPN with different learning rate values. Table 18.4 shows that the error percentage of the learning rates of 0.5 and 1 were the same, in addition to being lower than all other learning rates. To achieve the objective of finding the smallest error percentage, the learning rate of 1 was used, because the software originally recommended that value.

Table 18.4

Hidden layer 1 1 1 1 Hidden neuron 4 4 4 4 Learn rate 0.5 1 2 10

**Different learning rate values
**

Train Error 0.036 0.036 0.116 0.111 Testing Error 0.093 0.093 0.133 0.133 RMS error of training 0.185 0.185 0.306 0.286 RMS error of testing 0.298 0.298 0.319 0.317

Momentum factor 0.5 0.5 0.5 0.5

Step 6. Impact of the momentum factor The final step of data analysis was to change the value of the momentum item to obtain the configuration leading to the lowest error percentage. The initial value of the momentum item was 0.5. Another three values, 0.3, 0.6, and 0.8, were selected to compare with the initial value. Table18.5 shows the BPN with different values for the momentum item. Table 18.5 shows that the percentage of errors of momentum items of 0.3 and 0.5 are the same, and smaller than all others. To achieve the smallest error percentage, the 0.5 momentum item was used, because the software originally recommended that value. Step 7. Prediction After completing analysis and obtaining information about weight and input factors, equations to predict tool conditions were constructed. The variables a1, a2, ., and a5 represent 5 input factors, maximum peak force, spindle speed, feed rate, depth of cut, and maximum variance of adjacent peak force, respectively. By application of equations (18.5), the weighted value of hidden factors ah1, ah2, ah3, ah4 can be expressed as:

**HYBRID FUZZY NEURAL NETWORKS APPLICATIONS
**

Table 18.5

Hidden layer 1 1 1 1 Hidden neuron 4 4 4 4 Learn rate 1 1 1 1

241

**Percent error of momentum items
**

Train Error 0.036 0.036 0.049 0.116 Testing Error 0.093 0.093 0.093 0.133 RMS error of training 0.185 0.185 0.212 0.306 RMS error of testing 0.298 0.298 0.288 0.322

Momentum factor 0.3 0.5 0.6 0.8

ah1 = ah2 = ah3 = ah4 = aout1 = aout2 =

1 + exp 1 + exp

[ a1 ´ ( 4.652 ) + a2 ´ ( 0 .448 ) + a3 ´ ( 0.947 ) + a4 ´ (25.237 ) + a5 ´ ( 0 .853 ) (0.221 )]

1

...(18.5) ...(18.6)

[ a1 ´ ( 40 .457 ) + a2 ´ ( 39 .421 ) + a3 ´ (15.261 ) + a4 ´ ( 7.317 ) + a5 ´ (21.054 ) ( 44 .505 )]

1

1 1 + exp

[ a1 ´ (10.224 ) + a2 ´ (3.444 ) + a3 ´ ( 24 .252 ) + a4 ´ ( 3.449 ) + a5 ´ ( 4 .215) (1.289 )]

...(18.7)

1 + exp 1 + exp 1 + exp

[ a1 ´ (1.321 ) + a2 ´ (24.736 ) + a3 ´ ( 0.202 ) + a4 ´ ( 0.79 ) + a5 ´ ( 0.015 ) ( 0.829 )]

1

...(18.8) ...(18.9) ...(18.10)

[ ah1 ´ (11.697) + ah 2 ´ (16.977 ) + ah 3 ´ (12.295 ) + ah 4 ´ (11.807 ) ( 2.945 )]

1

[ ah1 ´ (11.697) + ah 2 ´ (16.977 ) + ah 3 ´ (12.295 ) + ah 4 ´ (11.807 ) (2.945 )]

1

Finally, the output information was used to judge the tool conditions: If aout1 > aout2, then the tool condition is used If aout1 < aout2, then the tool is broken.

18.2.5

Findings and Conclusions

To operate the UFMS successfully, in-process sensing techniques that relate to rapid-response decisionmaking systems were required. In this research, a neural networks model was developed to judge cutting force for accurate in-process tool breakage detection in milling operations. The neural networks were capable of detecting tool conditions accurately and in process. The accuracy of training data was 96.4%, and the accuracy of testing data was 90.7%. Partial results of training and testing data are shown in Tables 18.6 and Table 18.7.

**242 FUZZY LOGIC AND NEURAL NETWORKS
**

Table 18.6 Partial results of training data

Tool Conditions Good Broken Good Broken Good Broken Good Broken Good Broken a1 904.5 1220.54 634.64 780.14 674.36 847.06 1239.4 1677.92 1413.76 1861.72 Input factors a2 600 600 550 550 650 650 500 500 450 450 a3 12 12 10 10 15 15 18 18 15 15 a4 0.09 0.09 0.06 0.06 0.06 0.06 0.07 0.07 0.08 0.08 Output factors a5 45.8 954.04 274.64 368.64 248.64 537.54 225.2 1,159.64 300.56 1,250.92 aout1 1 0 0.98 0.03 1 .02 1 0 0 0.01 aout2 0 1 0.02 0.97 0 0.98 0 1 1 0.99 Good Broken Good Broken Good Broken Good Broken Broken Broken Prediction

**Table 18.7 Partial results of testing data
**

Tool Conditions Good Broken Good Broken Good Broken Good Broken Good Broken a1 711.94 1296.56 723.32 1215.96 1084.32 1542.92 1024.46 1253.28 1507.18 1876.74 Input factors a2 a3 550 550 550 550 550 550 600 600 450 450 15 15 12 12 18 18 18 18 20 20 a4 0.06 0.06 0.07 0.07 0.07 0.07 0.07 0.07 0.08 0.08 Output factors a5 aout1 177.02 481.66 311.52 1,042.50 192.5 1,303.92 172.22 580.3 550.06 1,062.02 1 0.01 0.35 0 0.99 0 0.98 0.03 0.98 0 Prediction aout2 0 0.99 0.65 1 0.01 1 0.02 0.97 0.02 1 Good Broken Broken Good Broken Good Broken Broken Broken Broken

The weights of hidden factors and output factors were generated from pre-trained neural networks, and a program was written to process these weights in order to respond to the tool conditions. Therefore, the in-process detection system demonstrated a very short response-time to tool conditions. Since tool conditions could be monitored in a real-time situation, the broken tool could be replaced immediately to prevent damage to the machine and mis-machining of the product. However, since the weights were obtained from the pre-trained process, they were fixed when they were put into the detection program. Therefore, the whole system does not have the adaptive ability to feed back information into the system. In this work, depth of cut was employed as one input factor. However, in actual industrial environments, the surface of work materials is often uneven, implying that the depth of cut set in the computer might differ from that used to cut the workpiece. Under the circumstances, the neural networks might generate a wrong decision and misjudge the tool conditions due to fluctuating depths of cut across machining.

18. 4 Ai and Bi can be any appropriate fuzzy sets in parameter form. 18. The properties (heat value. Bio fuels and municipal wastes are very inhomogeneous. the type and the parameter of the fuzzy membership functions and rules are crucial for achieving the desired. and oil). it predicts the flue gas properties. density. The combustion of those fuels or fuel-mixtures has different properties compared to the conventional fuels (coal. including the O2 content. then (f2 = p2x + q2y + r2) One possible ANFIS architecture to implement these two rules is shown in Fig.3. Adaptive Neuro-Fuzzy Inference Systems are fuzzy Sugeno models put in the framework of adaptive systems to facilitate learning and adaptation. To present the ANFIS architecture. i = mAi(x)i = 1. even if steady fuel feed volume is maintained. then (f1 = p1x + q1y + r1) if (x is A2) and (y is B2).1 Adaptive Neuro-Fuzzy Inference System Fuzzy Logic Controllers (FLC) has played an important role in the design and enhancement of a vast number of applications. only their effects on the combustion.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 243 18. gas. leading to increase in the emission level and variation of the generated heat flow. This topic presents an ANFIS system. homogeneity. I is the degree of the membership of the input to the fuzzy membership function (MF) represented by the node: O1. mix ability) may vary in a large range. if bell MF is used then. In the following presentation OLi denotes the output of node i in a layer L. Combined with a stoichiometric model. 2 O1. Note that a circle indicates a fixed node whereas a square indicates an adaptive node (the parameters are changed during training).4. It causes non-steady. which determines the amount of fuel fed to the combustion chamber. The proper selection of the number. there are several difficulties with burning bio fuels and municipal wastes. For example. i = mBi 2(y)i = 3. . agitated combustion conditions. Such framework makes FLC more systematic and less relying on expert knowledge. let us consider two-fuzzy rules based on a first order Sugeno model: Rule 1: Rule 2: if (x is A1) and (y is B1).3 CONTROL OF COMBUSTION Beside the economical and environmental advantages. moisture content. on the steam generation and on the power production can be observed through the O2 content of the flue gas. Layer 1: All the nodes in this layer are adaptive nodes. Those property variations are not predictable or directly measurable.

.244 FUZZY LOGIC AND NEURAL NETWORKS N A1 A A2 N B1 Y B2 Layer 1 Forwards Backwards Fig. The output of each node in this layer is given by: O3.. Y W1 M N W1 Y Y S M W2 Layer 2 N W2 Layer 3 Layer 4 Layer 5 mAi (x) = 1 LF x c I OP 1 + MG MNH a JK PQ 2 i i bi i = 1. Layer 2: The nodes in this layer are fixed (not adaptive).(18.(18. 2 .. w1 + w2 .4 ANFIS. Layer 3: Nodes in this layer are also fixed nodes.11) where ai.(18. i = mAi(x) mBi(y) i = 1. i = w i = . 2 The output of each node is this layer represents the firing strength of the rule..(18.12) wi i = 1.14) where pi ... These are labelled M to indicate that they play the role of a simple multiplier. 2 . bi and ci are the parameters for the MF. i = w i fi = w i ( pi x + qi y + ri ) i = 1.. The outputs of these nodes are given by: O2. 18.. The output of each node is simply the product of the normalized firing strength and a first order polynomial: O4. qi and ri are design parameters (consequent parameter since they deal with the then-part of the fuzzy rule) . These are labelled N to indicate that these perform a normalization of the firing strength from previous layer.13) Layer 4: All the nodes in this layer are adaptive nodes. 2.

5 = f = å i wi fi åw f = åw i i i i i i = 1. 1). bi and ci) pertaining to the input MFs [13].. bi and ci describe the sigma.. 2 . slope and the center of the bell MFs. we can apply least square method to identify the consequent parameters. 15). In this ANFIS architecture. These parameters are called premise parameters. there are two adaptive layers (12. we can plug training data and obtain a matrix equation: AQ = y .2 Learning Method of ANFIS The task of training algorithm for this architecture is tuning all the modifiable parameters to make the ANFIS output match the training data [14]. Note here that ai.. and the solution for È. we can divide the parameter set S into two sets: S = S1 Å S2 S = set of total parameters S1 = set of premise (nonlinear) parameters S2 = set of consequent (linear) parameters Å = direct sum For the forward path (see Fig. which is minimizes 2 A y È . is the least square estimator: .15) The ANFIS architecture is not unique.(18. For this observation.(18.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 245 Layer 5: This layer has only one node labelled S to indicate that is performs the function of a simple summer. 18.. Now for a given set of values of S1.3. If these parameters are fixed. The output of this single node is given by: Oi... respectively..18) where Q contains the unknown parameters in S2.(18. qi and ri) pertaining to the first order polynomial. Layer 4 has also three modifiable parameters (pi. the output of the network becomes: f= w1 w2 f1 + f2 w1 + w2 w2 + w2 .. This is a linear square problem.(18. These parameters are called consequent parameters.17) .16) = w1 f1 + w2 f2 = w1 (p1x + q1y + r1) + w2 (p2x + q2y + r2) = w1 x p1 + w1 y q1 + w1 r1 + w x p2 + w2 y q2 + w 2 r2 2 b g b g b g b g b g b g This is a linear combination of the modifiable parameters.. Some layers can be combined and still produce the same output. Layer 1 has three modifiable parameters (ai.

. from the fuel screw QHz. i being the nodes output and eL.. Q)] i =1 N 2 .2662Pcomb 9.(18.246 FUZZY LOGIC AND NEURAL NETWORKS Q* = (AT A)1 AT y . secondary airflow Fs. The higher the burning rate and smaller the waste heat is the higher efficiency..7207 . 2 . In multi-fuel fired fluidised bed power plants (Fig. from the efficiency point of view.. 18. signal primary airflow Fp.22) 18. through minimizing the overall quadratic cost function J(Q) = 1 2 $ å [ y(k) y(k. The combustion model. The aim of the combustion control.. For the backward path (Fig.3. The error signal form the oxygen content drives the PI controller of the flue screw signal. The update of the parameters in the ith node in layer Lth layer can be written as $ QiL (k) = $ QiL (k 1) + h ¶ Q L ( k ) $ i ¶E ( k ) . The O2 content of the flue gas is directly related to the amount of excess air.(18.(18.23) The reference signals for the fuel screw QHz.3 Model of Combustion The role of the combustion process is to produce the required heat energy for steam generation at the possible highest combustion efficiency.20) in a recursive manner with respect È(S2)..5). the error signals propagate backward. However. 18. .... 18. it is a difficult task due to the inhomogeneous properties of the fuel. is to keep the O2 content around 3-5% [16].. The premise parameters are updated by descent method [15].6) The oxygen and combustion power controller (Fig.19) we can use also recursive least square estimator in case of on-line training.21) where h is the learning rate and the gradient vector $ ¶Z L . The efficiency depends on the completeness of burning and the waste heat taken away in the flue gas by the excess airflow.(18. i ¶E = eL. excess air is required for ensuring complete burning. utilising the ANFIS structure based on [20]. The structure of the PI controller is U(s) = Kpi + Kh 1 S i = 1. i $ $ ¶Q iL ¶Q iL $ ¶Z L.6) consists of two parallel PI controllers. is the back propagated error signal. (see Fig. i. including the oxygen content. 18. primary airflow Fp and secondary airflow Fs signals are calculated by the linearization model as a function of the reference of the combustion power such as: QHz = 0. while combustion power is controlled by the primary airflow. calculates the combustion power (Pcomb) and flue gas components (Cf).(18.4).

0737Pcomb + 10..4 Optimization of the PI-Controllers Using Genetic Algorithms Standard genetic or genetic searching algorithms are used for numerical parameter optimization and are based on the principles of evolutionary genetics and the natural selection process [17].24) 18. A general genetic algorithm contains. depending of the selected parents the growing of the fitness of the population is faster or .5 Fluidized bed power plant.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS Steam Pressure Steam flow to network (2) Steam header Air fan (1) Furnace pressure (8) (9) Heater Induced draft fan X (3) Sec. Selection corresponds to keeping the best members of the population to the next generation to preserve the individual with good performance (elite individuals) in fitness function. Fp = 0. Crossover originates new members for the population. These procedures are responsible for the global search minimization function without testing all the solutions. the next three procedures: selection. usually. air (6) (5) I 247 Boiler drum Econ.(18. Steam Boiler super heater drum Oxygen (4) Furnace waterwall Fuel feed Prim.005 . by a process of mixing genetic information from both parents. air Prim.3. air fan (7) Fig.912 Fs = 0.2662Pcomb 4. 18.. crossover and mutation.

much smaller deviation in the oxygen content can be achieved while satisfying the same demand for combustion power. . Mutation is a process by which a percentage of the genes are selected in a random fashion and changed. the parent selection can be done with the roulette method. by tournament. The individuals are randomly selected with equal opportunity to create the new population. The fitness function is J= å 1 N $ ( ycomb ycomb ) N +k´ å (y 1 N O2 $ yO 2 ) .6 Control system of combustion process. The simulation shows that by applying the new controller structure together with the ANFIS model. The mutation operator is a binary mask generated randomly according to a selected rate that is superposed to the existing binary codification of the population changing some of the bits [19].. Crossover is performed over half of the population. always including the elite.248 FUZZY LOGIC AND NEURAL NETWORKS ANFIS fuel flow model and Stoichiometric combustion and fuel gas model Cf Linearlization model Pcomb QH2 FQ Fs Pcomb Oxygen and combustion power controllers Fig. lower.(18. and random[18]. In our case k = 2 to emphasize the importance of oxygen content which is directly related to the flue gas emissions. the crossover of one site splicing is performed and all the members are subjected to mutation except the elite. The performance of the controller based on the ANFIS model is compared to the performance of the real process. Among many other solutions. The reference signal for the combustion power is taken from the measurement data. 18.25) N where the k is a weighting factor. In the implemented algorithm a small population of 20 individuals. an elitism of 2 individuals was used..

HYBRID FUZZY NEURAL NETWORKS APPLICATIONS 6. .5 6+ + Best Average Poorest 249 5.8 PI combustion power controller optimization with GA. 18.5 + + 3 2.5 + 4 + + + + + 3.5 + 5 + + Fitness + 4.5 1 2 3 4 5 6 7 8 9 10 Generations + + + + + Fig. 18.7 Fitness function by the generation of the GA Optimization of the Pl controller 120 110 100 90 80 70 60 50 40 0 100 200 300 400 500 Time [S] 600 700 800 900 1000 Setpoint Output Combustion power [MW] Fig.

8 3.7 0 100 200 300 400 500 Time [S] 600 700 800 900 1000 Fig.95 3.18.2 4.9 PI Oxygen content controller optimization.75 3.85 3. . 120 110 100 Combustion power [MW] Combustion power measurement signal Setpoint Output 90 80 70 60 50 40 30 0 100 200 300 400 500 600 Time [S] 700 800 900 1000 Fig.9 3.10 Combustion power response: comparison of the achievement in real process and in the simulated control system.250 FUZZY LOGIC AND NEURAL NETWORKS Optimization of the Pl Controller 4. 18.05 Setpoint Output Oxygen content [%] 4 3.15 4.1 4.

K. pp. Altintas. C. Jemielniak. Mizutani.HYBRID FUZZY NEURAL NETWORKS APPLICATIONS Oxygen content measurement signal 9 8 7 Oxygen content [%] 251 Measurement Output Setpoint = 4% 6 5 4 3 2 1 0 0 100 200 300 400 500 600 700 800 900 1000 Time [S] Fig.C. 108. 1992. . Chen. REFERENCES. and E. J. 6. 1988. Lan. Engineering for Industry. 271-277. Sun. Detection of Cutting Edge Breakage in Turning. Explain the application of ANFIS to control the combustion process. Vol. 1997. 2. No. 4. The Detection of Tool Breakage in Milling Operations. and J. QUESTION BANK. 1. pp. and J.T. J. International Journal of Machining. 110. 1986. 1996. International Journal of Advanced Manufacturing Technology.T.C. 5. pp. 1997. 6. 2. Tlusty.S. J. 37. 41. Jang. Y.T. Yellowley. 12. J. 1. Tools manufacturing. 191-197. Vol.11 Oxygen content response: comparison of the achievement in real process and in the simulated control system. Engineering for Industry. pp. 783-800. 1. 153-164. New Jersey: Prentice Hall. Annals of the CIRP. A Fuzzy-Nets In-process (FNIP) System for Tool-Breakage Monitoring in End-Milling Operations. and Y. Chen. Vol. Naerheim. M. I. Neuro-Fuzzy and Soft Computing. 18. 97-100. Vol. 3. Explain the application of hybrid fuzzy neural network for the tool breakage monitoring of end milling. Black. No. In-Process Detection of Tool Breakage in Milling. A Fuzzy-Nets Tool-Breakage Detection System for End-Milling Operations.

J. 143. K. V. 1. Innsbruck.33. Alturki. and D. Lee. 35. and D. Grindelwald. pp. I. 10. Mota. IEE Proc. 2. 14.252 FUZZY LOGIC AND NEURAL NETWORKS 7. Kim.Control Theory Applications. Han. Vol. pp. No. Najim. MA. and K. On-line Monitoring of Tool Breakage in Face Milling: Using a Self-Organized Neural Network. 36. Leppäkoski and J. I. Tansel. K. 1993. Chen. Vol. 11. Japan. and E. Vol. D. D. F. 1995. Nov. 20. MIT Press. Proceedings of IASTED International Conference on Modeling Identification and Control. 1999. Neuro. Switzerland. Mizutani. 17. Tools Manufacturing. Vol. International Journal of Machining. Sun. Jang. 13.S.Yang and M. and H. Tools Manufacturing. 545-588.W and M. 2004. and U. The Neural Network Approach. 19-27.N.346. Kortela. Proc. Ko. Water Gas Heater Nonlinear Physical Model: Oprimazation with Genetic Algorithms Proccedings of IASTED International Conference on Modelling Identification and Control. J. and C. Proceeding of the 1999 IEEE. A Sensor for the Detection of Tool Breakage in NC Milling.1997. Z. pp. Proc. 33. Hímer. 80-90. 14. Journal of Manufacturing Systems. pp. T. Y. C. Switzerland 2004. 341. 4. 8.S. Hybrid model of oxygen content in flue gas. S. Prentice Hall. Hawai. 19. 12. 4. E. USA. tools Manufacturing. and B. McLanghlin. R. 259-272. 259-269. and A. pp. July 1993. Detection of Tool Breakage in Milling Operation-I. West.fuzzy model of flue gas oxygen content. pp. Y. 1992. Ikonen. J.Y. No. No. NJ. Journal of Materials Processing Technology. Neuro-fuzzy and Soft Computation. Grindelwald. 15. IEEE/RSJ Conference on intelligent robots and systems Yokohama. 1993. The Time Series Analysis Approach. Adaptation in Natural and Artificial System. pp. Cambridge. Abdennour. 950957. Tansel. 1993.H. 18. Sim. USA. Cho. A Case Study of PID Contoller tunning by Genetic Algorithm Proccedings of IASTED International Conference on Modelling and Control. Fuzzy neural networks and application to the FBC process. 1990. and C. Wertz. Kim. Kovacs.A. No. pp. Tarng. .J. Kovacs.B.H. Neuro-fuzzy control of a steam boiler turbine unit.N. Holland. Vol. Jung. International Conference on Control Applications. and A. 2001. May 1996. 9.B. On-line Detection of Tool Breakage Using Telemetering of Cutting Forces in Milling.Y.T Jeon. IASTED International Conference on Applied Modelling and Simulation. On Developing an adaptive neural-fuzzy control system. International Journal of Machining. Zhang. Detection of Tool Breakage in Milling Operation-II. 531-544. McLanghlin. pp. 2002. Y. International Journal of Machining. 16. J. Vol. 1050-1055.H. Vieria.L.W.

139. 133 Adaptive Critic 194 Adaptive Critic Element 192 Adaptive linear element 133 Adaptive Resonance Theory 182 Andlike 62 Andness is 62 ANFIS 229 ANFIS Structure 231 Anti-Reflexivity 20 Anti-Symmetricity 20 Applications of fuzzy logic 95 Approximate reasoning 41 Arithmetic mean 59 Arithmetical mean 58 ART 1 169. 142 Barto network 192 Bartos approach 192 Basic property 45 Bellman equations 196 Bias 127 Binary fuzzy relation 21 Biological neural network 121.Index A A back propagation neural network 235 =-Cut 11 Activation function 124 Adaline 125. 185 ART 2 211 Artificial network 122 Artificial neural 3 Artificial neural network 3. 9 Characteristic of fuzzy systems 9 Classical modus ponens 44 Classical modus tollens 45 . 183. 55 Asymmetric divergence 166 Auto-associator network 161 Auxiliary Hybrid Systems 218 Auxiliary hybrids 217 Average error 151 Averaging operators 58 Axon 3. 122 Boltzmann machines 165 Boolean logic 3 C Calssical modus ponens 45 Cart-Pole system 194 Cartesian product 24 Cells 122 Center of-gravity method 74 Center-of-area 87 Centroid defuzzification 90. 121 B Back-propagation 3. 123 ASE-ACE combination 192 Associative learning 125 Associative Memory 163 Associative Search 193 Associative Search Element 192 Associativity 54. 129.

21 Fuzzy rule-base system 71 Fuzzy set 9 Fuzzy singleton 63 Fuzzy systems 2 G Gaussian membership functions 89 Generalised delta rule 140 Generalized Modus Ponens 44 Geometric mean 59 Godel implication 32 Graded response 164 . 86 Delta 131 Delta rule 134 Dendrites 3. 121 Dimensionality reduction 169 Direction set minimization method 148 Discrete membership function 10 Disjunction Rule 43 Dot product 170 Dubois and prade 55 Dynamic Programming 195 Dynamics 201 E Eigenvectors 181 Elman Network 159 Embedded Hybrid Systems 218 Embedded hybrids 217 Empty Fuzzy Set 15 End-effector positioning 201 End milling 233 End milling cutting process 234 Entailment Rule 43 Entropy 63 Equivalence 20 Error back-propagation 222 Error function 173 Euclidean distance 171 Evaluation network 194 Exclusive-or 135 Expressive power 151 Extremal conditions 58 F Feature extraction 169 Feed-forward network 139. 6. 103 Defuzzifier 68. 170 Committee 223 Committee of networks 223 Commutative 61 Commutativity 58 Compensatory 58 Competitive learning 170 Complement 17 Component extractor 181 Compositional rule 44 Conjugate gradient minimization 148 Conjunction rule 43 Contrast enhancement 187 Control of combustion 243 Control room temperature 100 Controller network 191 Convergence theorem 131 Convex fuzzy set 11 Cost function 172 Counter propagation 174 Critic 190 D Defuzzification 67. 150 Feed-forward networks 125 Feedback control system 81 First-of-Maxima 87 Flexible manufacturing systems 233 FNN architecture 224 Follow-the-leader clustering algorithm 185 Forward Kinematics 200 Frank 55 Fuzzification 8 Fuzzy Approach 116 Fuzzy control 8 Fuzzy implication operator 32 Fuzzy implications 30 Fuzzy logic 1. 7 Fuzzy logic controller 82.#" FUZZY LOGIC AND NEURAL NETWORKS Classical N-array relation 19 Clustering 169. 1. 84 Fuzzy Logic Controllers 243 Fuzzy logic is 1. 1 Fuzzy neuron 220 Fuzzy Number 12 Fuzzy Point 15 Fuzzy relations 19.

INDEX Grading of apples 104 Gravity 87 H Hamacher 55. 164 Non-fuzzy approach 112 Normal fuzzy set 11 Normalization 186 Number of layers 127 O Offset 127 One identy 54 Ordered weighted averaging 60 Original model 185 Orlike operator 62 Orness 62 Output units 123 P Paradigms of learning 125 Partial order 20 Perceptron 2. 58. 121 Neuro-fuzzy systems 8 Neuro-fuzzy-genetic systems 8 Neurofuzzy network 222 Neurons 3. 190 Least mean square 133. 56 Harmonic mean 59 Hebbian learning 126 Hebbian Rule 180 Height defuzzification 88 Hidden Units 123. 135 Linear discriminant function 130 Linear threshold neural network 130 Linguistic variable 33. 121 Hybrid fuzzy neural network 221 Hybrid neural network 220 Hybrid systems 217 I Idempotency 58. 55. 129 Perceptron learning rule 131 ## . 34 Linguistic variable truth 35 LMS 131 LMS rule 131 Local Minima 148 Long-term memory 183 M Mamdani inference Mechanism 73 Mamdani system 66 Mamdanis implication operator 32. 122. 21 Inverse Kinematics 200 J Jordan Network 158 K Kleene-Dienes implication 32 Kohonen network 177 Kullback information 166 L Larsen inference Mechanism 77 Larsen system 66 Law of the excluded middle 2 Laws of Thought 2 Learning 127 Learning Rate 144 Learning Samples 152. 153 Hopfield network 161 Human brain 3. 63 Material implication 29 Mathematical neuron network 122 Max-Criterion 88 Maximum 56 Measure of dispersion 63 Median 59 Membership function 10 Middle-of-Maxima 87 Milling 210 Minimum 54 Modifiers 33 Momentum 144 Monotonicity 54. 61 Multi-layer network 140 Multi-layer perceptrons 137 Multi-input-multi-output 82 N Negation rule 44 Network paralysis 148 Neural networks 2. 61 Inference mechanisms 72 Input units 123 Interpolation 42 Intersection 16. 125.

54 Symmetry 55 T T 54. 130. 22 Universal approximation theorem 142 Universal approximators 91 Universal fuzzy set 15 Unmanned flexible manufacturing system 233 Unsupervised learning 125 . 157 Reflexivity 20 Regular fuzzy neural network 220 Reinforcement learning 192 Reinforcement learning scheme 190 Representation 127 Road accidents 96 Robot arm dynamics 207 Robot control 200 S Self-organization 125 Self-organizing networks 169 Semi-linear 124 Sequential hybrids 217 Sgn function 124 Shadow of fuzzy relation 24 Short-term memory 183 Sigmod 124 Significance 7 Simplified fuzzy Reasoning 77 Single layer feed-forward network 129 Single layer network 129. 56 t-conorm-based union 57 t-norm-based intersection 57 Taylor series 148 Test error rate 151 The linguistic variable truth 35 Threshold 127 Threshold (sgn) 130 Tool breakage 233 Total error 135 Total indeterminance 46 Total order 20 Traffic accidents and traffic safety 96 Trajectory generation 201 Transitivity 20 Translation rules 43 Trapezoidal fuzzy number 14 Triangular conorm 55 Triangular Fuzzy Number 13 Triangular norm 54 Tsukamoto inference mechanism 73 two layer feed-forward network 139 Two-input-single-output 82 U Union 16.#$ FUZZY LOGIC AND NEURAL NETWORKS Perceptron learning rule 131 Perceptrons 2 Precisiated natural language 8 Precision 7 Principal component analysis 221 Principle of incompatibility 101 Principle of optimality 195 Probabilistic 56 Processing Units 123 Product 55 Product fuzzy conjunction 89 Product fuzzy implication 89 Projection 23 Projection Rule 44 Q Q-learning 196 Quasi fuzzy number 12 R Recurrent networks 125. 134 Singleton fuzzifier 89 Soft computing 8 Standard Strict 32 Stochastic function 193 Strong 56 Subset 46 Subsethood 14 Sugeno Inference Mechanism 75 Summed squared error 135 Sup-Min Composition 26 Superset 47 Supervised learning 125 Support 11 Symmetricity 20.

56 Z Zero identity 55 #% . 174 W Weak 55 Winner Selection 170 Y Yager 55.INDEX V Vector quantisation 169.

- Fuzzy Logic With Engineering Applications
- Fuzzy Sets and Fuzzy Logic Theory and Applications
- Fuzzy Neural
- Fuzzy logic, neural network & genetic algorithms
- A Course in Fuzzy Systems and Control
- Neural Networks and Fuzzy Systems, A Dynamical Systems Approach to Machine Intelligence
- Artificial Neural Networks in Real-Life Applications [Idea, 2006]
- Fuzzy Logic - Controls Concepts Theories and Applications
- Artificial Neural Networks Architecture Applications
- Neural Networks and Computing Learning Algorithms and Applications
- Neural Networks
- Fuzzy Sets and Fuzzy Logic Theory and Applications - George j. Klir , Bo Yuan
- Fuzzy logic
- Zurada - Introduction to Artificial Neural Systems (WPC, 1992)
- Neuro-Fuzzy and Soft Computing (Jang Sun Mizutani)
- Principles of Artificial Neural Networks 9812706240
- Fuzzy Logic
- Support Vector Machines, Neural Networks, And Fuzzy Logic Models (2001)
- Introduction to MATLAB and Simulink a Project Approach
- Intro-to-fuzzy-logic-with-matlab
- Fuzzy Book
- C.Neural.Networks.And.Fuzzy.Logic.pdf
- A First Course in Fuzzy and Neural Control . 2003 . Hung T. Nguyen Et Al
- Neural Networks
- Neural Network in MATLAB
- Neural Network complete
- Neural Networks and Learning Machines (3rd Edition)
- NJ-ebooks
- Multidimensional Neural Networks Unified Theory Rama Murthy_NEW AGE_2007
- Continued Fractions - From Analytic Number Theory to Constructive AMS

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue listening from where you left off, or restart the preview.

scribd