This action might not be possible to undo. Are you sure you want to continue?
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 10, NO. 2, APRIL 2002
DENFIS: Dynamic Evolving Neural-Fuzzy Inference System and Its Application for Time-Series Prediction
Nikola K. Kasabov, Senior Member, IEEE, and Qun Song
Abstract—This paper introduces a new type of fuzzy inference systems, denoted as dynamic evolving neural-fuzzy inference system (DENFIS), for adaptive online and offline learning, and their application for dynamic time series prediction. DENFIS evolve through incremental, hybrid (supervised/unsupervised), learning, and accommodate new input data, including new features, new classes, etc., through local element tuning. New fuzzy rules are created and updated during the operation of the system. At each time moment, the output of DENFIS is calculated through a fuzzy inference system based on -most activated fuzzy rules which are dynamically chosen from a fuzzy rule set. Two approaches are proposed: 1) dynamic creation of a first-order Takagi–Sugeno-type fuzzy rule set for a DENFIS online model; and 2) creation of a first-order Takagi–Sugeno-type fuzzy rule set, or an expanded high-order one, for a DENFIS offline model. A set of fuzzy rules can be inserted into DENFIS before or during its learning process. Fuzzy rules can also be extracted during or after the learning process. An evolving clustering method (ECM), which is employed in both online and offline DENFIS models, is also introduced. It is demonstrated that DENFIS can effectively learn complex temporal sequences in an adaptive way and outperform some well-known, existing models. Index Terms—Dynamic evolving neural-fuzzy inference system (DENFIS), hybrid systems, online adaptive learning, online clustering, time series prediction.
I. INTRODUCTION HE complexity and dynamics of real-world problems, especially in engineering and manufacturing, require sophisticated methods and tools for building online, adaptive intelligent systems (ISs). Such systems should be able to grow as they operate, to update their knowledge and refine the model through interaction with the environment , , . This is especially crucial when solving artificial intelligence (AI) problems such as adaptive speech and image recognition, multimodal information processing, adaptive prediction, adaptive online control, and intelligent agents on the world-wide web , . Seven major requirements of the present ISs (that are addressed in the ECOS framework presented in  and ) are discussed in , , , and . They are concerned with fast learning, online incremental adaptive learning, open structure organization, memorising information, active interaction,
Manuscript received June 13, 2001; revised August 6, 2001. This work was supported by a research program funded by the New Zealand Foundation for Research Science and Technology under Contract UOOX0016. The authors are with the Department of Information Science, University of Otago, Dunedin 09015, New Zealand (e-mail: firstname.lastname@example.org; email@example.com). Publisher Item Identifier S 1063-6706(02)02965-X.
knowledge acquisition and self-improvement, and spatial and temporal learning. Online learning is concerned with learning data as the system operates (usually in real time) and the data might exist only for a short time. Several investigations , , , – proved that the most popular neural network models and algorithms that include multilayer perceptrons (MLPs) trained with the back propagation (BP) algorithm, radial basis function (RBF) networks, and self-organizing maps (SOMs) are not suitable for adaptive, online learning. At the same time, several models that have elements of adaptive, online learning or structure and knowledge adaptation, have been developed that include connectionist models –, –, , , , , , , , , , , –, , , , , , fuzzy logic models , , , , , , models based on genetic algorithms , , hybrid models , –, , , , , , , evolving fuzzyneural networks , , , , and evolving SOMs . The evolving connectionist systems (ECOSs) framework  assumes that a system evolves its structure and functionality from a continuous input data stream in an adaptive, life-long, modular way. The system creates connectionist-based modules and connects them, if that is required according to the input data distribution and the system’s performance at a certain time moment. ECOSs employ local learning (see, for example, , ). The evolving fuzzy neural network (EFuNN) model was introduced in  as one way for creating connectionist modules in an ECOS architecture. In  and , the EFuNN model is further developed mainly in respect of dynamic parameter self-optimization. EFuNNs are fuzzy logic systems that have five-layer structures (Fig. 1). Nodes and connections are created/connected as data examples are presented. An optional short-term memory layer can be used through a feedback connection from the rule (also called case) node layer. The layer of feedback connections could be used if temporal relationships of input data are to be memorised structurally. The input layer represents input variables. The second layer of nodes (fuzzy input neurons, or fuzzy inputs) represents fuzzy quantization of each input variable space. For example, two fuzzy input neurons can be used to represent “small” and “large” fuzzy values. Different membership functions (MFs) can be attached to these neurons (triangular, Gaussian, etc.). The number and the type of MF can be dynamically modified. The task of the fuzzy input nodes is to transfer the input values into membership degrees to which they belong to the MF.
1063-6706/02$17.00 © 2002 IEEE
KASABOV AND SONG: DENFIS: DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM 145 Fig. The radius of the input hyper-sphere is defined as . The structure of EFuNN. Here. ‘ ’ denotes division. and is the activation of the rule node . represents an association between a hyper-sphere from the fuzzy input space and a hyper-sphere connection weights from the fuzzy output space. if which is the maximum than value for the local normalized fuzzy difference. and in the fuzzy output space depending on the output error trough the Widrow-Hoff LMS algorithm (delta algorithm). similar to the input fuzzy neuron representation. the centers of this node hyper-spheres adjust in the fuzzy input space depending on the distance between the new input vector and the rule node through a learning rate . In EFuNNs. A linear activation function is used for the neurons of this layer. The fifth layer represents the real values of the output variables. The third layer contains rule (case) nodes that evolve through supervised and/or unsupervised learning. The fourth layer of neurons represents fuzzy quantization of the output variables. . a weighted sum input function and a saturated linear activation function is used for the neurons to calculate the membership degrees to which the output vector associated with the presented input vector belongs to each of the output MFs. For case of and . The latter represents the error tolerance of the system. that include: online. The pair of fuzzy input–output data vectors will be allocated to the rule node if falls into the input receptive field (hyper-sphere). Through the process of associating (learning) of new data points to a rule node . In initial algorithms for different types of learning in EFuNN. and the the fuzzy output space.g. the local normalized fuzzy distance is used to measure the distance between a new input data vector and a rule node in the local vicinity of the rule node.. This adjustment can be represented mathematically by the change in the connection weights of the rule node from and to and respectively according to the following vector operations: (2) (3) is the activation vector of the fuzzy output neurons where in the EFuNN structure when is presented. Each rule node is defined by two vectors of connecand . and falls in the output reactive field hyper-sphere. the representing the coordinates of the center of the sphere in —the coordinates in the fuzzy input space. e. and the local is smaller normalized fuzzy difference between and than the error threshold . is calculated as (1) denotes the sum of all the absolute values of a where: vector that is obtained after vector subtraction (or summation in ) of two vectors and . The rule nodes represent prototypes (exemplars. Definition: A local normalized fuzzy difference (distance) and that reprebetween two fuzzy membership vectors sent the membership degrees to which two real-value data vectors and belong to predefined MFs. where is of a rule node the sensitivity threshold parameter defining the minimum to a new input vector from a activation of the rule node in order to be considered for association new example with this rule node. example. that a local normalized fuzzy difference between and is smaller than the radius . and the former being adjusted through unsupervised learning based on similarity measure within a local area of the problem space. . clusters) of input–output data associations that can be graphically represented as associations of hyper-spheres from the fuzzy input and the fuzzy output spaces. Here a linear activation function is used to calculate the defuzzified values for the output variables (similar to FuNN ). structures are presented in . the latter being adjusted tion weights— through supervised learning based on the output error. This is ensured through two conditions. Each rule node. 1.
The offline ECM with constrained minimization (ECMc) is an extension of the online mode. the data examples come from a data stream and this process starts with an empty set of clus. is also shown in this section on the same data set (Gas-furnace ). MaxDist. . such as adaptive speech recognition. With defined and its cluster radius. More sophisticated algorithms are included in  where different applications of EFuNN are also developed. and for finding their current centers in the input data space. the distance. The ECM algorithm is described as follows. and setting a value for its cluster first cluster center [Fig. called ECM. the results are compared with the results obtained with the use of resource-allocation network (RAN) . . can perform well as online. it means that the current example belongs with the minimum distance to a cluster subject to the constraint In this case. EFuNN . the algorithm returns to Step 1. denotes a general Euclidean distance defined as follows: (4) . such as EFuNN. and subtractive clustering method . the maximum distance. Here. Conclusions and directions for further research are presented in Section VII. or less than. Dthr. VOL. When a new cluster is created. Section III introduces the DENFIS online model. and in Section IV. a fuzzy inference system for calculating the output fuzzy rules that had been is formed dynamically bases on created during the past learning process. II. is proposed to implement a scatter partitioning of the input space for the purpose of creating fuzzy inference rules. we propose a model called dynamic evolving neural fuzzy inference system (DENFIS). It is a distance-based connectionist clustering method. between an example point and the cluster center. adaptive control. EVOLVING CLUSTERING METHOD: ECM Here.  prediction. or offline. 2 shows a brief ECM clustering process in a 2-D space. neither a new cluster is created. An optimization is then applied. learning applications. cluster centers are represented by evolved nodes (rule nodes in the EFuNN terminology). The DENFIS model is developed with the idea that. the current input example. the example does not belong to any existing clusters. equal to. DENFIS online model is applied to Mackay–Glass (MG) time series . that makes a predefined objective function based on a distance measure to reach a minimum value subject to given constraints. A comparison between ECM. (with center and cluster ra• Step 3: Find cluster ) from all existing cluster centers through caldius . some created clusters will be updated through changing their centers’ positions and increasing their cluster radiuses. -means . . ECMc and some other clustering methods. are calculated. A new cluster is . evolving. is ters. This method has two modes: the first one is usually applied to online learning systems. and culating the values with the minimum then choosing the cluster center : value • Step 4: If is greater than Dthr. by simply taking the • Step 0: Create the first cluster position of the first example from the input stream as the . between vectors and . the algorithm is finished. In Section V. DENFIS is similar to EFuNN in some degree. Which cluster will be updated and how much it will be changed. at least one of the radii. DENFIS offline models are applied to MG time series and Gas-furnace time series prediction. more examples presented one after another. the cluster center. and the multilayer perceptrons trained with the back propagation algorithm (MLP-BP). gene expression data analysis and profiling. and it inherits and develops EFuNNs dynamic features which make DENFIS suitable for online adaptive systems. and evolving self-organizing maps (ESOM) . two DENFIS offline models are introduced. 2. both of which are used in the DENFIS model for partitioning the input space. where In the clustering process. fuzzy -means . online. Section II gives a description of an evolving clustering method (ECM) and its extension—evolving clustering method with constrained minimization (ECMc). passive—sleep learning. and the second one is more suitable for offline learning systems. A cluster will not be . 2). the ECMs. 2(a)]. Online ECM Without any optimization. and especially online. and in Section VI. The online ECM is used in the DENFIS online model. that has been set as a clustering parameter and would affect the number of clusters to be estimated. etc. is taken and the distances between this example and all already created cluster centers . APRIL 2002 offline. It takes the result from the online mode as initial values. With this method. radius • Step 1: If all examples of the data stream have been processed. Dthr. NO. nor any existing cluster is updated (the cases of and in Fig. A. self-organized generic clustering models. is less than a threshold value. This paper is organized as follows. • Step 2: If there is any distance value. Fig. the online ECM is a fast. active. depending on the position of the input vector in the input space. In addition to this. reaches the value updated any more when its cluster radius. ECM and ECMc. The results are compared with the results obtained with the use of adaptive neural-fuzzy inference system (ANFIS) . Else. that is equal to a threshold value. . The comparative analysis clearly indicates the advantages of DENFIS when used for both offline. maximum distance-based clustering method. In any cluster. one-pass algorithm for a dynamic estimation of the number of clusters in a set of data.146 IEEE TRANSACTIONS ON FUZZY SYSTEMS. depends on the position of the current example in the input space. Else—go to the next step. 10. In this paper. is initially set to zero.
the method determines the cluster and the membership matrix iteratively using the centers following steps. the cluster • Step 5: If is updated by moving its center. the maximum distance from any cluster center to the examples that belong to this cluster is not greater than the threshold value. . The updated radius is set value of its radius. A brief clustering process using ECM with samples x to x in a 2-D space. (c) x : update cluster C C . (a) The example x causes the ECM to create a new cluster update cluster C C . the minimizing for (5) and (6) cluster centers is derived as follows: if else for each (7) For a batch-mode operation. is not greater than Dthr. x : create a new cluster C .x . to minimize an objective function based on a distance measure subject to given constraints. called ECMc. and increasing the . In this way. and 0 otherwise. where the element belongs to cluster . Constrained Optimization and Offline ECMc The offline ECM. applies an optimization procedure to the resulted cluster centers after the application of ECM. and the algorithm returns to of Step 1. (d) x : update cluster C C . and finds a cluster center in each cluster. and at the point on the line connecting the to the point is the distance from the new center (the cases of and in Fig. ! ! ! ! C . The ECMc partitions a data set including vector . equal to The algorithm returns to Step 1. B. 2). in cluster and the corresponding cluster center. 2. . . Taking the general Euclidean distance as the measure between an example vector. . 2). and the new center is located to be equal to and . x : update cluster C C a new cluster C . the objective function is defined by the following equation: (5) where cluster is the objective function within and the constraints are defined by the next equation Dthr (6) binary The partitioned clusters are typically defined by a is 1 if the -th membership matrix . (b) x : : create created in the same way as described in Step 0 (the cases and in Fig. x : do nothing. Once the data point are fixed. into clusters . Dthr though the algorithm does not keep any information of passed examples.KASABOV AND SONG: DENFIS: DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM 147 Fig. x : do nothing.
one pass) . the following six clustering methods are implemented and applied to the gas-furnace data set: a) b) c) d) e) f) ECM (online. we can say that if all six clustering methods ob. both ECM and ECMc. VOL. APRIL 2002 Fig. 2. the ECM and the ECMc tained the same values for could achieve less number of partitions than the others. corresponding cluster center and the value of objective function . defined by (2). 3. the clustering simulations are implemented in the input space. ECMc (offline). KMC (offline). one-pass). EFuNN (online. We can see from Table I. or the iteration number of minimization operations is over a certain value. NO. • Step 3: Employ the constrained minimization method  with (5) and (6) to get new cluster centers. From another point of view. which means that these methods partition the value of data set more uniformly than the other methods. or its improvement over previous iteration is below a certain threshold. It consists of 296 consecutive data . Each of them partitions the data into NoC clusters. the algorithm returns to Step 2. one pass). In this case. are taken as criteria for comparison. can obtain the minimum . The results are shown in Table I and Fig. 3. 10. between an example point and the maximum distance. Else. For a comparative analysis. Results of clustering the gas-furnace data set by several clustering methods. • Step 4: Calculate the objective function according to (6). . and the carbon pairs of methane at a time moment dioxide (CO ) produced in a furnace at a time moment as input variables. Application: Clustering of the Gas-Furnace Data Set The gas-furnace time series is a well-known bench-mark data set and has been frequently used by many researches in the area of neural networks and fuzzy system for control. that come from the result of ECM clustering process. C. that ECMs. SC (offline. Stop if the result is below a certain tolerance value. with the produced CO at the moment as an output variable. . • Step 2: Determine the membership matrix by (7). FCMC (offline).148 IEEE TRANSACTIONS ON FUZZY SYSTEMS. • Step 1: Initialize the cluster center . prediction and adaptive learning . The .
KASABOV AND SONG: DENFIS: DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM 149 TABLE I RESULTS OBTAINED BY USING DIFFERENT CLUSTERING METHODS FOR CLUSTERING THE GAS-FURNACE DATA SET INTO 15 CLUSTERS For an input vector . . Dthr. . We can rewrite the (9) sponding cluster center. . . and polynomial . . . is used and the least-square estimator (LSE) of are calculated as the cofficients . . General Principle The DENFIS. . If the consequent functions are crisp constants. the threshold value. . . use Takagi–Sugeno type fuzzy inference engine . and . . . and (10) as follows: (11) (12) if if if is is is and and and is is is and and and and and and is is is then then then is is is . are antecedent variables defined over universes of discourse . . The system is called a first-order Takagi–Sugeno type fuzzy . Such fuzzy inference engine used in DENFIS is composed of rules indicated as shown in the equation at the bottom of . the page. each rule’s output indicated as follows: where. . all fuzzy membership functions are triangular type functions which depend on three parameters as given by the following equation: For obtaining these functions a learning data set. . DENFIS: A DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM A. . functions In both DENFIS online and offline models. . and is the distance between th example and the corre. . .. it is called high-order Takagi–Sugeno fuzzy inference system. we use a weighted least-square estimation method . the result of infer(the output of the system) is the weighted average of ence. by applying the following formula: (9) where . is a consequent variable. i. Dthr and Dthr – . B. Learning Processes in DENFIS Online Model In the DENFIS online model. are employed. which data pairs is composed of . both online and offline models. . and . . are fuzzy sets defined by their fuzzy membership functions . .  (10) (8) where where: is the value of the cluster center on the dimension. are inference system if linear functions. where “ is fuzzy propositions as antecedents form are fuzzy rules respectively.  with learning data. In the consequent parts.” . .e. . In the DENFIS models. we call such system a zero-order Takagi–Sugeno type fuzzy inference system. is a clustering parameter. the first-order Takagi–Sugeno type fuzzy rules are employed and the linear functions in the consequences can be created and updated by linear least-square estimator (LSE) . . . . Each of the linear functions can be expressed as follows: III. If these functions are nonlinear functions.
In the online DENFIS model. find data points whose 3) For every cluster center positions in the input space are closest to the center. In the DENFIS online model. in each of the two groups of rules. the DENFIS model chooses fuzzy rules from the whole fuzzy rule set for forming a current inference system. 4 where two different groups of fuzzy inference rules are formed depending on two input vecand . the inference system may have the same fuzzy rule inference group. C. 10. and the value of should be greater than the number of input elements.150 IEEE TRANSACTIONS ON FUZZY SYSTEMS. . VOL.8–1. the region has a linguistic meaning ‘large’. A new fuzzy rule is created if a new cluster center is found by the ECM. their corresponding inference systems may be different. This operation depends on the position of the current input vector in the input space. Second. however. for each input vector. NO. . for instance. 4(a) and (b). respectively. respectively. data to 2) Implement clustering using ECM with these obtaining cluster centers. direction for Fig. where is the weight defined in (10) and is a forgetting factor which typical value is between 0. a clustering parameter). First. Using (12) on data pairs calculate the values of and of the consequent function. depending on the position of the current input vector in the input space. the initial values of directly from (12) with the use of the first data pairs from the learning data set. In the DENFIS online model. In addition to this. Equation (13) is the formula of recursive LSE . create the antecedents of the fuzzy rule using the position of the cluster center and (8). Here. The antecedent of the new fuzzy rule is formed by using (8) with the position of the cluster center (as a rule node). The region is defined by different the same direction of membership functions. then can be calculated iteratively as shown in (13) at the bottom of and can be calculated the page. several existing fuzzy rules are updated by using (14) if their rule nodes have distances to the data point in the input space that are not greater than Dthr (the threshold value. An existing fuzzy rule is found which rule node is the closest to the new rule node. the antecedents of the fuzzy rules chosen to form an inference system for this input vector may vary. but for the group of in the rules from Fig. An example is illustrated in the Fig. APRIL 2002 Let the -th row vector of matrix defined in (9) be and the -th element of be . . even if two input vectors are exactly the same. DENFIS online learning model. 2. one of these rules may also be updated through changing its antecedent so that. We can see from this example that. respectively. the consequence function of this rule is taken as the consequence function for the new fuzzy rule. if its rule node position is changed by the ECM. the fuzzy rule will have a new antecedent calculated through (8). For every data pair. As new data pairs are presented to the system. The distances between these rule nodes and the data point in the input space are taken as the weights in (14). and are the parameters of the In the above steps. new fuzzy rules may be created and some existing rules are updated. In addition to dynamically creating and updating fuzzy rules the DENFIS online model has some other major differences from the other inference systems. Takagi-Sugeno Fuzzy Inference in DENFIS The Takagi–Sugeno fuzzy inference system utilized in DENFIS is a dynamic inference. If no rule insertion is applied. The distances between data points and the cluster center are taken as the weights in (12). 1) Take the first learning data pairs from the learning data set. (13) (14) . 4(a) group. in a 2-D input space as shown in tors Fig. especially in the DENFIS offline model. 4) For obtaining a fuzzy rule corresponding to a cluster center. we use a weighted recursive LSE with forgetting factor defined as (14) shown at the bottom of the page. the fuzzy following steps are used for the creation of the first rules and for the calculation of the initial values and of the functions. the rules are created and updated at the same time with the input space partitioning using online ECM and (8) and (14). It is due to the reason that these two input vectors are presented to the system at different time moments and the fuzzy rules used for the first input vector might have been updated before the second input vector has arrived. In the case of two input vectors that are very close to each other. 4(b) it denotes a linguistic meaning of ‘small’ in .
batch training mode. similar to the DENFIS online model. Here. Two DENFIS models for offline learning are developed and presented here: (1) a linear model. 5(a). the DENFIS online model will be applied to modeling and predicting the future values of a chaotic time series: the MG data set . TIME SERIES MODELLING AND PREDICTION WITH THE DENFIS ON-LINE MODEL In this section. For each of the aforementioned online models. X X IV. which has been used as a benchmark example in the areas of neural networks.KASABOV AND SONG: DENFIS: DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM 151 TABLE II PREDICTION RESULTS OF ON-LINE LEARNING MODELS MACKEY-GLASS TEST DATA ON THE (a) (b) Fig. The following experiment was conducted: 3000 data points. 4. Then. are used as testing data. V. and EFuNN . and then the testing data is used with the recalling procedure. Here. 500 data points. These models are Neural gas . (b). we assume that time step and for . but it may not be very efficient when used on comparatively small data sets. using the DENFIS offline model I (it will be introduced in next section). The is 0. all represented in the 2-D space of the first two input variables 1 and 2. units) evolved (used) in each model. For the purpose of batch training the DENFIS online model is extended here to work efficiently in an offline. RAN . are extracted from the time series and used for to as learning (training) data. DENFIS MODEL FOR OFFLINE LEARNING The DENFIS online model presented in the previous section. and (2) a MLPbased model. the properties of rule insertion and rule extraction were utilized where we first obtained a group of fuzzy rules from the first half of training data (1500 samples). we also use some existing online learning models applied on the same task. b) DENFIS online model with 883 fuzzy rules (different parameter values are used from those in the model above). we tested the model on the test data. Two fuzzy rule groups are formed by DENFIS to perform inference for an input vector x (a) and for an input vector x (b) that is entered at a later time moment. c) DENFIS online model with 883 fuzzy rules that is evolved after an initial set of rules was inserted. is employed in model . then we inserted these rules to the DENFIS online model and let it learn continuously from the next half of the learning data (1500 samples). batch mode training. and (c) display the testing errors (from the recall processes on the test data) of DENFIS online model with different number of fuzzy rules a) DENFIS online model with 58 fuzzy rules. ESOM . In another experiment. A first-order Takagi–Sugeno type fuzzy inference engine. the learning data is used for the online learning processes. from input vectors task is to predict the values for any value of the time . fuzzy systems and hybrid systems. model I. we take the nondimensional error index (NDEI)  which is defined as the root mean-square error (RMSE) divided by the standard deviation of the target series. the fourth-order Runge–Kutta method was used to find the numerical solution to the above MG equation. for . Fig. can be used also for offline. This time series is created with the use of the MG time-delay differential equation defined as (15) To obtain values at integer time points.1. model II. For the purpose of a comparative analysis. Table II lists the prediction results (NDEI on test data after online learning) and the number of rules (nodes. to .
estimate the functions to create the consequent part for each fuzzy rule using (10) or (12) with data sets. APRIL 2002 Fig. each consequent function of a fuzzy rule (rule node. two-layer (the hidden layer consists of two or three neurons) multilayer perceptrons to realize the function in the consequent part of each fuzzy rule instead of using a predefined function. • find data sets each of them including one cluster center and learning data pairs that are closest to the center in the input space. • for model II. NO. (e).152 IEEE TRANSACTIONS ON FUZZY SYSTEMS. In this section. one data pair can belong to several clusters. For comparison . 5. VI. the distances between each data point and their corresponding center is represented as a connection weight. TABLE III PREDICTION RESULTS OF OFF-LINE LEARNING MODELS ON MACKEY-GLASS TRAINING AND TEST DATA I. rules) by using the offline ECMc. 10. cluster center) is learned by a corresponding MLP network after training it on the corresponding data set with the use of the MLP-BP. (c) and offline (d). (f) models on test data taken from the MG time series. Prediction error of DENFIS online (a). 2. while an extended high-order Takagi–Sugeno fuzzy inference engine is used in model II. TIME SERIES MODELLING AND PREDICTION WITH THE DENFIS OFFLINE MODEL Dynamic time-series modeling of complex time series is a difficult task. In the general case. • create the antecedent part for each fuzzy rule using (8) and the current position of the cluster center (rule node). we applied the two DENFIS offline models for the same task as in Section IV. (b). especially when the type of the model is not known in advance . The latter employs several small-size. • for model I. VOL. The DENFIS offline learning process is implemented in following way: • cluster (partition) the input space to find cluster centers ( rule nodes.
Rev. and a multilayer perceptron trained with MLP-BP . vol. 524–532. D. Giles. Processing Syst. 1992. pp. D. no. S.. (f) DENFIS offline mode II with 58 fuzzy rules.5).. vol.” Neural Networks. 3. and D. 1975. and (2) application of the DENFIS model for adaptive process control and mobile robot navigation. G. Carpenter and S. and MLP  in the case of offline learning. “A decision making model using a fuzzy neural network. and (f) shows the prediction error of three DENFIS models tested on the same test data as follows: (d) DENFIS offline mode I with 116 fuzzy rules. Kasabov. p. “Circuits of production rule—GenNets—The genetic programming of nervous systems. Albrecht. 1987. REFERENCES  J. 1990. vol.  The Handbook of Brain Theory and Neural Networks. 845..” in Proc. Hassibi and D.” Trans. 7. “The cascade—Correlation learning architecture. and T. Goodman. S. RAN . pp. Here.” Math. Pattern Recogntion with Fuzzy Objective Function Algorithms. Control. Amari and N. San Mateo. so it needs more training data than the models which use global generalization such as ANFIS and MLP. Amari. L.. 1992.  S. “Local learning computation. learning epochs. 59. adaptive neural-fuzzy inference system (ANFIS) . 2. Freeman and D. Amari. Grossberg. Dyna.” Adv. pp. DeGaris. Eds. 1994. Giles. Lawrence. J. During the learning process DENFIS forms an area of partitioned regions. 183–192. French. Bollacker. Intel. vol. 1993. like EFuNN and CMAC neural networks . Hashiyama. 1992. vol. 1995. Fuzzy Logic Neural Networks. Further directions for research include: (1) improvement of the DENFIS model for a better online learning with self-modified parameter values.” in 2nd Int. Forecasting and Control.  S.” in Advances in Neural Information Processing Systems.  H. Iizuka. New York: Plenum Press. DENFIS. Cambridge. Fukuda. Reeves. Smyth.” IEEE Trans. Cambridge. “Evolving self-organizing maps for online learning. Table III lists the offline prediction results of MLP. Gori.. Grossberg. 888–900. Deng and N. Turetzky.. 287–289. 1990. 3. vol. G. Cybenko. DENFIS is likely to produce results with a higher error rate.. . S. CA: Morgan Kaufmann. ACM Conf.  R. no. Touretzky. FL: CRC Press. Albus. adaptive learning systems. vol. In addition to the NDEI. Croder III. 1995. Pittsburgh.. Analysis of Fuzzy Information.” Phys. M. Uchikawa. and with ANFIS .  B. R.” in Proc. “Rule-based neural networks for classification and probability estimation. pp. no. C. “A new approach to manipulator control: The cerebellar model articulation controller (CMAC). 1990 Connectionist Models Summer School. pp. the learning time is the CPU-time (in seconds) consumed by each method during the learning process in the same computing environment (MATLAB. Piuri. 2000. and N.” Science. D. Neural Networks. Lebiere. vol.  B. 2. 1970. Both DENFIS online and offline models are based on the Takagi–Sugeno fuzzy inference system. 2nd Int. no.  L. S.” Neural Networks. 4. Int. Goodwin and K. 227. 781–804. learning time (CPU-time). pp..  G. P. Fig. pp. CA: Holden Day. ANFIS and DENFIS. Sept. like Case 1 in Section IV-B. Lett. M. vol. Zrehen. 3–8. Eds. pp. EFuNN . 303–314. “Online learning in radial basis function networks.” Adv. 4. Ed. Oct. Fuzzy Syst. and these results include the number of fuzzy rules (or rule nodes) in the hidden layer. VI. and L. S. 78. “ART3: Hierarchical search using chemical transmitters in self organizing pattern-recognition architectures.  C.. Miller.” in Proc. C. Sejnowski. Adaptive Filtering Prediction and Control.  K. Autonomous Agents. DENFIS uses a local generalization. 1990. vol. Conf. MA. pp. 698–713. Control. New York.-I. Funihashi. “Predicting chaotic time series.  J. 3. Stork.  G. 1998.  G.” in Artificial Neural Networks and Genetic Algorithms. and V. Processing Syst.  T. J. p. T. In case of examples outside this area. 1987.  G. Jenkins.  J. vol. and T. IEEE. Conf. CONCLUSION AND DIRECTIONS FOR FURTHER RESEARCH This paper presents the principles of a fuzzy inference system. ACKNOWLEDGMENT The authors express thanks to the reviewers whose important comments and suggestions lead to a significant improvement of this paper. in the case of offline learning. vol. Y. 1. New York. The proposed systems demonstrate superiority when compared with Neural gas . New York: Springer Verlag. 164–171. Kasabov. and Y.” Neural Comput. 220. DENFIS would give satisfactory results if the recall examples appear inside of these regions.  S. The best results are achieved in the DENFIS II model. 1994.” From Animals to Animats.. C. 1989. 2. Saad.  J. pp. Bezdek.” J. vol. Reynolds. data analysis and modeling. Arbib. “Mathematical foundations of neuro-computing. Ed. 2. NDEI for training data and NDEI for testing data. 3. Measure. VII. vol. Higgins. Fahlman and C. pp. 1984. 365–377. (e) DENFIS offline mode I with 883 fuzzy rules. 1991.  M. 3. ASME: J. 129–152. PA. (e). are also used for this task under the same conditions. Vapnik. Syst. and P. H. S.” Neural Comput. MIT Press.  D. Rosen. for building both online and offline knowledge-based. Hinton. 1991. Mackey and L. J. Ed. C. B. Box and G. and ESOM  in the case of online learning. “Recurrent neural networks with self-adaptive GAs for biped locomotion robot. New York: Springer Verlag. 1992. Neural Networks. 1990. Sept. M. “FuzzyARTMAP: A neural network architecture for incremental supervised learning of analogue multi-dimensional maps. “Second order derivatives for network pruning: Optimal brain surgeon. C.  G.. Chiu. Sept. Carpenter. Eds. Boca Raton. Gaussier and S. “On the approximate realization of continuous mappings by neural networks. 1997.” Proc. Sidorowitch. 116–123. They use the highly activated fuzzy rules to dynamically compose an inference system for calculating the output vector for a given input vector. San Francisco. IJCNN’2000 Neural Networks Neural Computing: New Challenges Perspectives New Millennium. 1992.  T. Arakawa. vol. Bottu and V. E. Pattern Recognition by Self-Organizing Neural Networks.. Steele. “A topological neural map for online learning: Emergence of obstacle avoidance in a mobile robot. pp.KASABOV AND SONG: DENFIS: DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM 153 purposes two other well-known models. 7. Neural Inform.” Connection Sci.  R. Brain-Like Computing and Intelligent Information Systems. 197. 1989.” in Proc. “A growing neural gas network learns topologies.  K. In the recall process. but these regions may not cover the whole input space.  J. pp. 5(d). 2. NJ: Prentice-Hall. Upper Saddle River. M. vol. Farmer and J. 1057–1060. 1997. MA: MIT Press. pp. Bezdek.  R. New York. “Oscillation and chaos in physiological control systems. “Predicting the Mackey-glass timeseries with cascade—Correlation learning. Japan. Glass. “Fuzzy model identification based on cluster estimation. “Semi-destructive representations and catastrophic forgetting in connectionist networks. “Approximation by super-positions of sigmoidal function. pp. 1977. Signals Syst. the learning time is also measured as another comparative criterion. Fritzke. Sin. Komata. 1981. N. UNIX version 5. vol.” Neural Comput. W. 9. Markuzon. 117–123. Eds. Furuhashi. . Neural Inform. 1997. M. C. 282–290.. 14. The prediction error of DENFIS model II with 58 rule nodes is the lowest one. “CiteSeer: An autonomous Web agent for automatic retrieval and identification of interesting publications. Time Series Analysis.  T.
pp. “Fuzzy identification of systems and its applications to modeling and control.” Neural Networks. T. S. book chapters. Takagi and M. Lapedes and R. vol.” Neural Networks.154 IEEE TRANSACTIONS ON FUZZY SYSTEMS. Fifth Berkeley Symp. 291–299. Jang. no. Mar.. 2–20..  T. adaptive. 665–685. B. Eds. San Mateo. Woldrige and N. 1993. pp.. New Zealand.MMS.  L.” Inform. Miki. 116–132. Kawahara and T. “Evolving fuzzy neural networks for online. “Incremental Class Learning Approach and Its Applications to Hand-Written Digit Recognition. vol. pp. “A technique for trimming the fat from a network via relevance assessment.” Proc. . 4.. 82. 19–24. no.  C. 1989. no. 10. Kappen. Found. 1996.  J. “Evolving fuzzy neural networks—Algorithms. 213–225. edited research books and monographs. Farber. He is currently Professor and Chair. 1301–1321.  S. Syst. 9. 2. 199–233. vol. He has published over 260 works. no. Dr. in 1982.  R. no. 281–294.Sc. 1996. degree in mathematical sciences from the Technical University in Sofia.. “ECOS: A framework for evolving connectionist systems and the  eco learning paradigm.  T.” Proc. 2. San Diego. 1996. Engineering Dept. Oct.” Fuzzy Sets Syst. vol.  N. Tokyo University of Mercantile Marine. University of Otago. “Learning fuzzy rules and approximate reasoning in fuzzy neural  networks and hybrid systems. Past President of the Asia Pacific Neural Network Assembly (APNNA). Kasabov is a Fellow of the Royal Society of New Zealand and the New Zealand Computer Society.  T. MA. As an Automation Engineer. 1999. MA: D..” Neural Comp.. no.  M. member of the TC12 group on Artificial Intelligence of IFIP.E. 1987. and the IEEE Computer Society. pp. Control. patents and authorship certificates in the area of intelligent systems. T. Robins and M. Uchino... 1965. Amsterdam. Probability. Rummery and M. 2000. Miyajima. Watts. D. 1990. 1. Kasabov.” in Math. Eds. 1996.” Neurocomputing. M. G. 23.. Self-Organizing Maps.  J.  T. and A. p. vol. Zadeh. IEEE. Kasabov. Neural Networks. SMC-15. and also a member of INNS. “The self-organizing map. no. Tech.. Kasabov. CA. . pp. Bulgaria. and the Ph. Statistics. 2–4. pp. 19–31. Mathematics. C. 1998. Japan. Dec. Syst. Saito. 1996. Smolensky. Ishikawa. The Math Works.  G. T. pp. and bioinformatics. Yamakawa. APRIL 2002  R.  1997. A. 2. H. “Counter-propagation networks.. He is currently a Research Fellow in the Department of Information Science. 1996.  M. Boston. Mozer and P. Reed. Maeda.” IEEE Trans. Natick. ENNS.” in Proc. S. 10. applications and  biological motivation.  Neural Network Toolbox User’s Guide. 1. pp. 2001. CUED/F-INENG/TR 166. degree from the Department of Automation. CA. Adv. pp. 3. pp. Cambridge.  R.” IEEE Trans.  On-Line Learning in Neural Networks.  N.D. 31. 1999. Singapore: World Scientific. . text books. speech recognition. “On-Line -Learning Using Connectionist Systems.” Inform. N. “A resource allocating network for function interpolation. Kasabov (M’94–SM’01) received the M. Design and Application of Soft Computing. 1993. “Fuzzy sets.” Cellular Neural Networks Appl..K. J. vol. New Zealand. LeCam and J. He is the General Chairman of a series of biannual international conferences on Artificial Neural Networks and Expert Systems (ANNES). 1989.  A. L. Lee.” International Computer Science Institute. S. Heskes and B. 1995. 3. the M. Comput.  M. 6. pp. and Director of the Research Laboratory for Knowledge Engineering and Computational Intelligence in the Department of Information Science. Matsumoto. connectionist and hybrid connectionist systems. Inc. Kim and N.D. “A self organizing neural network with creating and deleting methods. “Pruning algorithms—A survey. Yamakawa and G. 1999. Sugeno. 1017–1020. “Local learning algorithms for sequential learning tasks in neural networks. 101. Touretzky.  M. Dunedin. Neural Networks. among them journal papers. vol. Dunedin. 501–521. Kohonen. Cybern.. “Nonlinear Signal Processing Using Neural Networks: Prediction and System Modeling. pp. 1996. Comput. Sci. Fuzzy Systems and Knowledge  Engineering. Conf. Berkeley. Man. Hsia. N-9. vol. Darken. “HyFIS: Adaptive neuro-fuzzy systems and their application to nonlinear dynamical systems. Processing Syst. 13. 1977. Oct. Hertz.  A. Kitakyushu. 1987.  A. 902–918. 1997. 1967. 1993. Neuro Fuzzy Systems: Prentice Hall. Neural Networks. pp.  Optimization Toolbox User’s Guide. degree from the University of Otago. pp. intelligent expert systems. vol. MacQueen. Cambridge. 338–353. pp. LA-UR-87-2662.  J. 5. 740–747. 95–117.” IEEE Trans. “FuNN/2—A fuzzy neural network architecture for adaptive learning and knowledge acquisition. and S. 155–175. vol.” IEEE Trans. Niranjan.  J. conference papers. M.  M. “A new effective algorithm for neo fuzzy neuron model. 12. CA: Morgan Kaufmann. . “Some methods for classification and analysis of multivariate observations. Lin and C. 1464–1497. pp. “Growing and pruning neural tree networks. S. Frean. 1998. expert systems. VOL. Krogh and J. Japan. Natick. J. Q Nikola K..” Advances Neural Inform. degree from the Division of Transportation Engineering.. vol.” Los Alamos National Laboratory. 2. Gray. “On a novel adaptive self-organizing network. NZRS.” in Proc. pp. Shastri. 42. 397–400. East China Textiles Institute. knowl edge-based learning. 1998. CA.” PCT WO01/78003. The Math Works. NM. Kasabov and B. “Fast learning in networks of locally-tuned processing units.” Cambridge Univ. 1992. TR-98-015. MA: MIT Press. FUZZ-IEEE. “Intelligent agents: Theory and practice. Aug. Cybern.” J. Hech-Nielsen. “Adaptable connectionist production systems. New York: Springer-Verlag. “Adaptive Learning System and Method. Man. 41–46. Man.C. Sankar and R.. 1994. New York: McGraw-Hill. vol. Rev. edited conference proceedings. Heath. vol.” Nonlinear Theory Appl. 2nd ed.. A. System Identification: Least-Squares Methods. Ed. Jennings. Kim. “Structural learning with forgetting. pp.” in Methodologies for the Conception. he was a Foreign Researcher at Kansai University and Tokyo University. and T. pp. . Machine Learning. May 1993. “Rule insertion and rule extraction from evolving fuzzy neural networks: Algorithms and applications for building adaptive. Cybern. 2. 78. Qun Song received the B.  T. pp. 598–605. Platt. A. Syst. 1998. “ANFIS: Adaptive network-based fuzzy inference system.—Appl. Dunedin. 271–274. 1996. vol. 951–957.” Neural Comput. NO. .” in IEEE 1st Int. April 20. 1995. pp. respectively.” in Proc. Intell. 3–4. ICONIP’98. Murashima. The Netherlands: Elsevier. degree in computer science and the Ph. pp. vol. “Online learning processes in artificial neural networks. 1985. pp. Mitchell.” in Advances in Neural Information Processing Systems. 4. Los Alamos. His current research interests include evolving neural-fuzzy systems and robotics. 1222–1235.” IEEE Trans. vol. Woodford. University of Otago. Foundations of Neural Networks. vol. “A simple weight decay can improve generalization. pp.” Knowledge Eng. vol.  N. vol. he worked in the Textile Scientific Research Academy of China from 1982 to 1991. MA. New Zealand. M.  J.  A. Kasabov. Kusanagi. 1991. pp. E. Inc. 1993. 1996. Fifth IFSA World Congress. From 1991 to 1993. fuzzy systems. 87 545. and 2002. Berkeley. vol. 9.. 33–34. Mandziuk and L. Mammone. U. . vol. Moody and C. H. 1997.  N. 8. vol. Rep. 281.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.