You are on page 1of 5

Robust Pareto Design of ANFIS Networks for

Nonlinear Systems with Probabilistic Uncertainties
A. Jamali
Department of Mechanical
engineering, Faculty of Engineering,
The University of Guilan, Rasht, Iran
ali.jamali@guilan.ac.ir
N. Nariman-zadeh, H. Ashraf
Department of Mechanical
engineering, Faculty of Engineering,
The University of Guilan, Rasht, Iran
nnzadeh@guilan.ac.ir
Z. Jamali
Department of Civil engineering,
Faculty of Engineering, The
University of Guilan, Rasht, Iran
zahrajamali_civil86@yahoo.com



Abstract—In this paper, multi-objective evolutionary Pareto
optimal design of Adaptive Neuro-Fuzzy Inference System
(ANFIS) have been used for modeling of nonlinear systems using
input-output data sets with probabilistic uncertainties. In this
way, A Monte Carlo Simulation (MCS) is first performed to
generate input-output data set using some probabilistic
distributions. Multi-objective uniform-diversity genetic
algorithms (MUGA) are then used for Pareto optimization of
ANFIS networks. The important conflicting objectives of ANFIS
networks that are considered in this work are, namely, the mean
and variance of both Training Error (TE) and Prediction Error
(PE) of such ANFIS models. It is shown that a robust ANFIS can
be simply obtained using a criterion based on four values of
means and variances of both TE and PE. The probabilistic
evolved ANFIS model exhibits much more robustness to the
uncertainties involved within the input-output data sets than that
of the deterministic evolved ANFIS model. It is shown that
ANFIS can be successfully applied for input-output data set with
uncertainties so that a robust model can be compromisingly
obtained from some non-dominated optimum ANFIS models
Keywords-component; ANFIS; Pareto; Robust model; Monte
Carlo.
I. INTRODUCTION
System identification techniques are applied in many fields
in order to model and predict the behaviors of unknown and/or
very complex systems based on given input-output data [1]. In
this way, soft-computing methods, which concern computation
in an imprecise environment, have gained significant attention
[2]. Many research efforts have been expended to use of
evolutionary methods as effective tools of soft-computing
methods for system identification [2-3]. Among this method,
fuzzy rule-based systems have been an active field of research
because of their unique ability to build models based on
experimental data. Lee presented the concept of fuzzy sets,
which deal with uncertain or vague information, and paved the
way for applying them to real and complex tasks [4]. Wang
shows that fuzzy systems could be effectively employed as
universal approximators to perform input-output mapping [5].
Moreover, Nariman-Zadeh et al. recently applied SVD in
combination with a genetic algorithm to optimally design a
ANFIS system for modelling purposes that demonstrated its
superior performance in comparison with previous works [6].
However, a fuzzy model consisting of a large number of IF-
THEN rules to map inputs to outputs is not desired due to the
phenomenon of over fitting, which reduces the generalizing
property of the fuzzy model to predict the unforeseen data.
Similarly, the Takagi-Sugeno-Kang (TSK) type fuzzy models
are widely used for control and modelling because of their high
accuracy and relatively small model size. In the TSK models,
which are also known as neuro-fuzzy systems, the consequents
of the fuzzy rules are explicit functions, usually with linear
relationships, of the input variables rather than fuzzy sets. In
other words, as presented by Hoffmann and Nelles the crisp
linear relation portion of the consequents of a TSK fuzzy rule
describes the underlying model in the local multi-dimensional
region specified in the premise part of that fuzzy rule [7].
Therefore, two types of tuning procedures are required for
proper partitioning of the input space and number of fuzzy
rules, known as structural tuning, and for the parameters of the
consequent parts of the fuzzy rules, known as parametric
tuning. In recent years, different approaches have been adopted
for optimal tuning of such models based on either heuristic
search or fuzzy clustering for the premise part and least squares
for linear parameters in the conclusion of the fuzzy rules.
Genetic algorithms have received a great deal of attention for
the optimal selection of the premise part of TSK type fuzzy
rules in Wang and Yen’s literature [8]. In order to identify the
parameters of the consequents, there have been attempts in the
literature to use SVD as a linear optimization technique. Jang
proposed an equivalent approach to the TSK models as an
Adaptive Neuro-Fuzzy Inference System, ANFIS [9]. In this
model a hybrid learning method is used for tuning parameters
in both antecedents and consequents of embodied TSK-type
fuzzy rules. In all of these researches uncertainties are not
addressed and the modelling process is accomplished
deterministically. In fact, it has been shown that designing
without considering uncertainties generally leads to such a
potentially high risk solution [10, 11]. Therefore, it is so
desirable to find such a robust model in which performance
variation in the presence of uncertainties is not high.
With the aid of ever increasing computational power, there
have been a great amount of research activities in the field of
robust modelling and design devoted to the use of Monte Carlo
simulation [12, 13]. MCS is a direct and simple numerical
method but can be computationally expensive. In this method,
random samples are generated assuming pre-defined
probabilistic distributions for uncertain parameters. The system
300
978-1-61284-922-5/11/$26.00 ©2011 IEEE
is then simulated with each of these randomly generated
samples and both mean and variance of the performance
metrics are then evaluated statistically. In Multi-objective
optimization problems (MOPs), there are several objective or
cost functions (a vector of objectives) to be optimized
(minimized or maximized) simultaneously [14, 15]. These
objectives often conflict with each other so that improving one
of them will deteriorate another. Therefore, there is no single
optimal solution as the best with respect to all the objective
functions. Instead, there is a set of optimal solutions, known as
Pareto optimal solutions or Pareto front for multi-objective
optimization problems. Recently, a multi-objective uniform-
diversity genetic algorithm (MUGA) have been presented in
[16, 17] which will also be used in this work.
In this paper, MUGA in conjunction with MCS are used for
Pareto robust optimal design of an ANFIS network for
modeling and prediction of a complex nonlinear system using
input-output data sets with probabilistic uncertainties. In this
way, A Monte Carlo Simulation (MCS) is first performed to
generate input-output data set using some probabilistic
distributions. MUGA are then used for Pareto optimization of
ANFIS networks. The important conflicting objectives of
ANFIS networks that are considered in this work are, namely,
the mean and variance of both Training Error (TE) and
Prediction Error (PE) of such model. It is shown that a robust
model can be simply obtained using a criterion based on four
values of means and variances of both TE and PE. The
probabilistic evolved ANFIS model exhibits much more
robustness to the uncertainties involved within the input-output
data sets than that of the deterministic evolved ANFIS model.
It is shown that ANFIS networks can be successfully applied
for input-output data set with uncertainties so that a robust
model.
II. STOCHASTIC ANALYSIS
In real experimental engineering practice, there exist a
variety of typical sources of uncertainty which have to be
considered through robust modeling approach [10, 11]. Those
uncertainties include plant parameter variations due to
environmental condition, incomplete knowledge of the
parameters, inaccuracies involved with measuring and
experimental apparatus, and etc. In conventional optimum
design of inductive modeling based on input-output data,
uncertainties are not addressed and the optimum model of the
process is often accomplished deterministically. In fact, it has
been shown that optimization without considering uncertainty
generally leads to non-optimal and potentially high risk
solution. Therefore, it is very desirable to find robust model
whose performance variation in the presence of uncertainties
is not high. Generally, there exist two approaches addressing
the stochastic robustness issue, namely, robust design
optimization (RDO) and reliability-based design optimization
(RBDO) [18]. Both approaches represent non deterministic
optimization formulations in which the probabilistic
uncertainty is incorporated into the stochastic optimal design
process. In RBDO approach, some evaluated reliability
metrics subjected to probabilistic constraints are satisfied so
that the violation of design requirements is minimized. In
RBDO approach, some evaluated reliability metrics subjected
to probabilistic constraints are satisfied so that the violation of
design requirements is minimized. In RDO approach, the
stochastic performance is required to be less sensitive to the
random variation induced by uncertain parameters so that the
performance degradation from ideal deterministic behavior is
minimized.
Let X be a random variable, then the prevailing model for
uncertainties in stochastic randomness is the probability
density function (PDF), ( ) x f
X

or equivalently by the
cumulative distribution function (CDF), ( ) x F
X
, where the
subscript X refers to the random variable. This can be given by
( ) ( ) ( )

∞ −
= ≤ =
x
X X
dx x f x X x F Pr (1)
where Pr(.) is the probability that an event (X x) will
occur. Some statistical moments such as the first and the
second moment, generally known as mean value (also referred
to as expected value) denoted by E(X) and variance denoted
by ( ) X
2
σ , respectively, are the most important ones. They can
also be computed by
( ) ( ) ( )
∫ ∫

∞ −

∞ −
= = dx x f x xdF X E
X X
(2)
and
( ) ( ) ( ) ( )


∞ −
− = dx x f X E x X
X
2
σ (3)
In the case of discrete sampling, these equations can be
readily represented as
( )

=

N
i
i
x
N
X E
1
1
(4)
and
( ) ( ) ( )

=



N
i
i
X E x
N
X
1
2 2
1
1
σ (5)
where
i
x is the i
th
sample and N is the total number of
samples.

III. MODELLING USING ANFIS
An ANFIS consisting of a set of TSK-type fuzzy IF-THEN
rules can be used in modeling to map inputs to outputs. The
formal definition of such identification problem is to find a
function,
^
f , so that it can be approximately used instead of
the actual one, f . This is done in order to predict output
^
y
for a given input vector { }
n
x x x X ..., , ,
2 1
= as close as
possible to its actual output y . The problem is now to
determine an ANFIS that minimizes the mean square error
(MSE) between the actual and predicted output, that is

301
M
y y
MSE
M
i
i i ∑
=







=
1
2
^
(6)

In this way, a set of linguistic TSK-type fuzzy IF-THEN rules
is designed to approximate
^
f by f using M observations of
n-input-single-output data pairs ) ,..., 2 , 1 )( , ( M i y x
i i
= . The
fuzzy rules embodied in such ANFIS models can be
conveniently expressed using the following generic form

l
n
i
i
l
i
j
l n
j
l
j
l l
w x w y THEN A is x
AND A is x AND A is x IF rule
n
0
) (
) (
2
) (
1
2 1
:

=
+ =

(7)

in which { } r j
l
,..., 2 , 1 ∈ and { }
l l
n
l l l
w w w w W
0 2 1
, ,... , ∈ are
the parameter sets of the consequent of each rule. The entire
fuzzy set in x
i
space is given as

{ }
) ( ) 2 ( ) 1 ( ) (
,..., ,
r i
A A A A = (8)

These fuzzy sets are assumed to have a Gaussian shape
defined on the domains [ ] ( ) n i
i i
,..., 2 , 1 , = + − β α In this
way, the domains are appropriately selected so that all the
fuzzy sets are complete, that is, for any [ ]
i i i
x β α + − ∈ ,
there exists
) ( j
A in Eq.(7) such that the degree of
membership function is non-zero. Each fuzzy set
{ } ( ) r j A
j
,..., 2 , 1
) (
= represented by Gaussian membership
functions in the form

( )









− =
2
2
2
exp
) (
j
j i
A
c x
j
σ
μ (9)

Where
j j
c σ , which optimally select by GAs are adjustable
centers and variances in antecedents, respectively. More
details about modeling of complex system using ANFIS can
be found in [6]. The method are suggested by some author in
[6] is adopted here for optimal design of ANFIS network for
modeling of uncertain input-output data set. Therefore, in this
paper, a hybridization of genetic algorithms and SVD is
applied for the optimal design of ANFIS.

IV. PARETO ROBUST MODELLING OF NONLINEAR SYTEM
The input–output data used in robust modeling involve 100
data pairs randomly generated from a nonlinear system [19]
with three inputs x
1
, x
2
, x
3,
and a single output y given by
5
3
x ,
2
x ,
1
x 1 )
1.5
3
x
1.0
2
x
0.5
1
x (1 y ≤ ≤

+

+ + = (6)
There are 50 pattern numbers which have been randomly
selected from those data pairs to train such ANFIS model.
However, a testing set which consists of 50 unforeseen input-
output data samples during the training process is merely used
for testing to show the prediction ability of such evolved
ANFIS models during the training process.
There are two main concepts involved within ANFIS
networks design, namely, the parametric and the structural
identification problems. In this way, some of authors present a
hybrid method which using genetic algorithms (GA) and
singular value decomposition (SVD) method to optimally
design the Gaussian membership functions of antecedents and
the vector of linear coefficients of consequents, respectively
[6]. The methodology in these references has been successfully
used in this paper to obtain the robust ANFIS model for above
100-input-1-output data minimizing the mean values and
variances of both training and testing of the input-output data
table simultaneously. In the deterministic approach, the mean
square errors (MSE) of training and prediction errors are
chosen for the bi-objective Pareto optimization approach of
ANFIS models using the methodology explained in detail in
[6]. Figure 1 depicts the Pareto front of both training and
prediction errors of ANFIS models for deterministic input-
output data table. Points A and B stands for ANFIS model
having the least prediction and training errors, respectively.
Design point C, however, depicts the tradeoff in such models
which can be reasonably chosen as an optimal model
compromisingly. The ANFIS’ values of TE and PE of
deterministic optimal design points A, B, and C are given in
Table 1. The good behavior of design point C for modeling and
prediction of input-output data table have been shown in figure
2.


Fig. 1: Pareto front of TE and PE of ANFIS models in
deterministic design
302

Fig2. Comparison between actual output and ANFIS output
of design point C for deterministic data set

Ability of design point C (the best design point for
deterministic data table) for modeling and prediction of
uncertain input-output data set is evaluated by 2500 different
data tables which generated by MCS with 10 percent Gaussian
probabilistic distribution of all three input in the input-output
data set. It is clear from figure 3 for uncertain input-output data
set variances of both TE and PE of design point C is highly
increased. Therefore, modeling of uncertain input-output data
set has to consider through robust modeling approach. In robust
modeling approach, the values of means and variances of both
TE and PE of the ANFIS networks is evaluated based on 2500
different data table which generated by MCS. The multi-
objective Pareto design of ANFIS networks is now used to
obtain some non-dominated ANFIS models using statistical
measures given in equations (4) and (5). Consequently, 119
different non-dominated ANFIS models have been obtained
based on four objective functions of the mean and variance of
both TE and PE. In order to compromise among the solutions
obtained, all the values of these objective functions returned by
the MCS are normalized. The minimum value of the sum of
those normalized values is then simply selected as the robust
ANFIS model. The performance of this design point denoted
by point D is given in Table 1 both for deterministic and
probabilistic approaches using MCS for 2500 data sets. It is
very evident from this table that the stochastic behaviour of the
design point D in terms of mean and variance for both TE and
PE is significantly superior to those obtained in deterministic
approach (design points A, B, and C) and thus exhibits more
robustness in the existence of probabilistic uncertainties. Figure
3 depicts the location of the design point D among other non-
dominated design points in the plane of the variances of both
TE and PE. It is clear that the selection criterion leads to the
design point with almost least variances whose values are
given in Table 1.


Figure 2: Non-dominated design points and the selected one
point D in the plane of variances of errors.

I. CONCLUSION
A multi-objective genetic algorithm was used to optimally
design ANFIS model from a robustness point of view in a
probabilistic approach. The objective functions which often
conflict with each other were appropriately defined using some
probabilistic metrics. The multi-objective optimization of
robust ANFIS models led to the discovering some important
trade-off among those objective functions. The framework of
such hybrid application of multi-objective GAs and Monte
Carlo Simulation of this work for the Pareto optimization of
robust ANFIS model using stochastic objective functions is
very promising and can be generally used in the optimum
design of ANFIS models in real-world complex systems with
probabilistic uncertainties.


Table 1: Training and prediction errors of design points in both deterministic and probabilistic approaches
Design
Point
Deterministic Values of TE
&PE
Probabilistic measures of TE & PE (evaluated by
2500 MCS)
Training
Error
Prediction Error
Mean of
TE
Mean of
PE
Variance
of TE
Variance of
PE
A 0.001003 0.009107 0.01235 0.01894
1.32E-
03
1.98E-03
B 0.002251 0.004359 0.06489 0.04297
1.15E-
04
1.98E-03
C 0.001038 0.005660 0.06248 0.07262 1.98E- 1.03E-04
303
03
D 0.009213 0.005916 0.006818 0.021771
4.41E-
08
8.96E-08

REFERENCES
[1] Astrom, K. J. and Eykhoff, P., "System identification, a survey",
Automatica, 7, 123162, 1971.
[2] Sanchez, E., Shibata T. and Zadeh, L. A., "Genetic Algorithms and
Fuzzy Logic Systems", World Scientific, 1997.
[3] Nariman-zadeh, N., Marzbanrad J. and Jamali, A., "Stiffness modelling
of rubber engine mounts by poly-nomial neural networks and genetic
algorithms" Proc. ESM2004, France, 181187, 2004.
[4] Lee, C. C., "Fuzzy logic in control systems: Fuzzy logic controller",
IEEE Trans. Sys., Man, Cybern 20, 2, 404434, 1990.
[5] Wang, L. X., "Fuzzy systems are universal approximators", Proc. IEEE
Int. Conf. Fuzzy Systems, 11631170, 1992.
[6] N. Nariman-Zadeh, A. Darvizeh, M.H., "DadfarmaiDesign of ANFIS
networks using hybrid genetic and SVD methods for the modelling of
explosive cutting process", Journal of Materials Processing Technology
155–156, 1415 – 1421, 2004.
[7] Hoffmann, F. and Nelles, O. "Genetic Programming for Model Selection
of TSK-Fuzzy Systems", Information Sciences, 136, 728. Elsevier.
Germany, 2001.
[8] Wang, L. and Yen, J., "Extracting fuzzy rules for system modelling
using a hybrid of genetic algorithms and Kalman filter", Fuzzy Sets and
Systems, 101, 353-362, 1999.
[9] Jang, J. S. R., "ANFIS: Adaptive-network-based fuzzy inference
system", IEEE Trans. Sys., Man, Cyb., 23, 665685, 1993.
[10] N. Nariman-zadeh, F. Kalantary, A. Jamali, F. Ebrahimi, " Robust
Pareto Design of GMDH-type Neural Networks for Systems with
Probabilistic Uncertainties", International Conference on Inductive
Modelling ICIM’2008, September 15–19, 2008, Kyiv, Ukraine
[11] Papadrakakis, M., Lagaros, N.D., Plevris, V., Design optimization of
steel structures considering uncertainties, Engineering structure, vol. 27,
1408-1418, 2005
[12] Kalos, M.H., Whitlock, P.A., "Monte Carlo Methods", Wiley, New
York, 1986.
[13] Diwekar, U.M., Kalagnaman, J.R., "Efficient sampling technique for
optimization under uncertainty", American Institute of Chemical
Engineering Journal, Vol. 43, n.2, pp. 440-447, 1997.
[14] Coello Coello, C. A., and Christiansen, A. D., "Multiobjective
optimization of trusses using genetic algorithms. Computers &
Structures", 75, pp. 647-660, 2000.
[15] Coello Coello, C. A., Van Veldhuizen, D. A. and Lamont, G. B.,
“Evolutionary Algorithms for Solving Multi-objective problems”,
Kluwer Academic Publishers, NY, 2002.
[16] Jamali, A., Nariman-Zadeh, N., Atashkari, K., “Multi-objective Uniform
diversity Genetic Algorithm (MUGA)”, In Advanced in Evolutionary
Algorithms, witold kosinski (Ed.), IN-TECH, Vienna, 2008.
[17] Jamali, A., Hajiloo, A., and Nariman-zadeh, N., Reliability-based robust
Pareto design of linear state feedback controllers using a multi-objective
uniform-diversity genetic algorithm (MUGA), Expert systems with
Applications, Vol. 37 (1) , pp. 401-413, 2010.
[18] Smith, B.A., Kenny, S.P., Crespo, L.G., Probabilistic Parameter
Uncertainty Analysis of Single input Single Output Control Systems,
NASA report, TM-2005-213280, March, 2005.
[19] Nariman-zadeh N. And Jamali A., "Pareto genetic design of GMDH-
type neural networks for nonlinear systems", IWIM’07, Prague, Czech
Rep, 2007.

304