DISCUSSIONS AND CLOSURES

Discussion of ‘‘Clustering Technique for

Evaluating and Validating Neural Network
Performance’’ by Jonathan Jingsheng Shi
April 2002, Vol. 16, No. 2, pp. 152–155.
DOI: 10.1061/共ASCE兲0887-3801共2002兲16:2共152兲
Downloaded from ascelibrary.org by Universidad Nacional De Ingenieria on 10/23/18. Copyright ASCE. For personal use only; all rights reserved.

Mohamed Marzouk1 and Osama Moselhi, F.ASCE2

1
Postdoctoral Fellow, Dept. of Building, Civil, and Environmental
Engineering, Concordia Univ., 1455 Blvd. de Maisonneuve West,
Montreal, Qc. H3G 1M8. E-mail: marzouk@cbsengr.concordia.ca
2
Professor and Chair, Dept. of Building, Civil, and Environmental
Engineering, Concordia Univ., 1455 Blvd. de Maisonneuve West,
Montreal, Qc. H3G 1M8. E-mail: moselhi@cbsengr.concordia.ca
The technical note has presented a useful technique for evaluating
the performance of neural networks 共NN兲 using fuzzy clustering.
In this discussion, comments pertaining to the presented tech-
nique are made. Second, the writers present an alternative method
using fuzzy subtractive clustering. The method proposed by the
discussers is direct, instead of forming clusters and then building
NN models. The numerical example considered in the note was
analyzed using the proposed method. The results indicate that in
addition to the simplicity of the proposed method, it outperforms
the technique presented in the original note. The note presented a
subjective model for evaluating and validating the performance of
neural networks using clustering techniques. It suggested the use
Fig. 3. Converted input data
of a two-stage model: fuzzy clustering to classify data points with
similarities into groups and, subsequently, the development of a
NN model for each cluster. The note did not provide a procedure
for specifying the number of clusters for the data sets under study.
For example, classifying data sets that consist of two input vari-
ables (x 1 and x 2 ), as shown here in Fig. 1, appears to have been
made in a subjective manner. In addition, the note suggested,
without explanation, a threshold value for membership of 0.7 for
each data point to belong to a given cluster.
An alternative approach to the one presented in the note is to
use a fuzzy clustering technique as an inference system for predi-
cation. For the same ‘‘cowboy-hat’’ problem (sin冑x 21 ⫹x 22 ) ana-
lyzed in the note, the writers developed a fuzzy subtractive clus-
tering model 共FSC兲. The mathematical computations of that

Fig. 1. Grouping data sets into clusters

Fig. 2. Input data points

182 / JOURNAL OF COMPUTING IN CIVIL ENGINEERING © ASCE / APRIL 2004

J. Comput. Civ. Eng., 2004, 18(2): 183-184

 Table 1. Comparison of Results
Author’s data set Proposed fuzzy subtractive model
Actual Cluster A Cluster B Author’s
c
output %FSC %NN
Point x1 x2 value X a
Xn ␮a Za ␮b Zb b
FSCគPrediction error error
1 ⫺2.9 ⫺2.7 ⫺0.73164 3.96 0.934 1.89E⫺1 3.356 7.45E⫺1 ⫺1.756 ⫺0.72310 1.2 2.0
2 ⫺0.5 ⫺2.5 0.558091 2.55 0.601 9.98E⫺1 2.203 5.37E⫺1 ⫺2.466 0.57059 ⫺2.2 35.1
3 ⫺0.5 ⫺0.5 0.649637 0.71 0.167 4.25E⫺2 0.699 1.69E⫺3 ⫺3.392 0.54157 16.6 9.3
4 2.8 ⫺2.5 ⫺0.57456 3.75 0.885 3.02E⫺1 3.186 8.87E⫺1 ⫺1.861 ⫺0.58050 ⫺1.0 ⫺8.5
5 2 2 0.308072 2.83 0.667 9.52E⫺1 2.431 7.58E⫺1 ⫺2.326 0.32123 ⫺4.3 ⫺157.9
6 0.8 2.5 0.494024 2.62 0.619 9.99E⫺1 2.264 5.97E⫺1 ⫺2.428 0.50861 ⫺3.0 ⫺21.3
Downloaded from ascelibrary.org by Universidad Nacional De Ingenieria on 10/23/18. Copyright ASCE. For personal use only; all rights reserved.

7 1 1 0.987766 1.41 0.333 2.91E⫺1 1.276 3.15E⫺2 ⫺3.037 0.85463 13.5 24.1
8 ⫺2 2 0.308072 2.83 0.667 9.52E⫺1 2.431 7.58E⫺1 ⫺2.326 0.32123 ⫺4.3 ⫺25.6
9 ⫺2.5 0.5 0.558091 2.55 0.601 9.98E⫺1 2.203 5.37E⫺1 ⫺2.466 0.57059 ⫺2.2 15.4
Absolute average error 5.4 33.2
a
X n ⫽normalized values for X(X/X max).
b
FSCគPrediction⫽␮ a* Z a ⫹␮ b* Z b /␮ a ⫹␮ b .
c
%FSC Error⫽共Actual Value⫺FSCគPrediction兲/Actual Value.

procedure can be found elsewhere 共Chiu 1994; Marzouk 2002兲. It References

should be noted that the number of clusters 共fuzzy rules兲 is gen-
erated automatically based on the initial selection of r a 共the radius
that represents the neighborhood for each cluster兲. Because the
sine function has a cyclic nature and a repetitive output in the four
quarters, the following condition is applicable to all integer n
共positive or negative兲:
Chiu, S. L. 共1994兲. ‘‘Fuzzy model identification based on cluster estima-
tion.’’ J. Intell. Fuzzy Syst., 2共3兲, 267–278.
Marzouk, M. 共2002兲. ‘‘Optimizing earthmoving operations using com-
puter simulation.’’ PhD thesis, Concordia Univ., Montreal.

Sin X⫽Sin共 X⫹2n␲ 兲 (1)

Closure to ‘‘Clustering Technique for
Therefore, the problem can be solved considering x 1 and x 2 as Evaluating and Validating Neural Network
passive input parameters, which are used to calculate the active
parameter X as shown in Eq. 共2兲. Performance’’ by Jonathan Jingsheng Shi
April 2002, Vol. 16, No. 2, pp. 152–155.
X⫽ 冑x 21 ⫹x 22 ⫺2n␲ (2) DOI: 10.1061/共ASCE兲0887-3801共2002兲16:2共152兲

where n is a positive integer number that makes X less than 2␲. Jonathan Jingsheng Shi, M.ASCE1
FSC was developed using the 64 data points 共Scenario 2 stud- 1
Associate Professor, Dept. of Civil and Architectural Engineering,
ied by the author兲 as training data sets. Eqs. 共3兲 and 共4兲 were Illinois Institute of Technology, 3201 S. Dearborn St., Chicago,
developed accordingly. IL 60616. E-mail: jonathan.shi@iit.edu

Z a ⫽0.8165X⫹0.12117
Z b ⫽0.5024X⫺3.74707
2 Using the response concept 共Smith et al. 1995兲, a neural net-
␮ a ⫽e ⫺16储 X n ⫺0.611储
2 factors 共e.g., Z⫽ 兵 z 1 ,z 2 ,...,z m 其 ) under a given set of variables
␮ b ⫽e ⫺16储 X n ⫺0.7982储
The data points were generated randomly, in a manner similar
to that used in the note, over three zones 共Fig. 2兲. The active
parameter X for these training data sets was estimated as shown in
Fig. 3. For r a ⫽0.5, two clusters 共a and b兲 were obtained from the
training data points as shown 共Fig. 3兲. The output of these clusters
共Z兲 and their membership values 共␮兲 can be calculated using Eqs.
共3兲 and 共4兲, respectively.
The nine data points specified by the author were also used to
test the performance of the FSC. The results obtained were com-
pared to those generated using the author’s technique 共see Table
1兲. The results clearly indicate much higher accuracy for the pro-
posed FSC over the technique presented by the author 共see,
%FSC Error versus %NN Error兲.
(3a) The writer was very interested in reading the fuzzy subtractive
clustering 共FSC兲 results presented in the discussion. The claimed
(3b)
accuracy seems appealing.
(4a)
work 共NN兲 may be expressed as to estimate a set of interested
(4b)
共e.g., X⫽ 兵 x 1 ,x 2 ,...,x n 其 ), i.e.
Z⫽ f 共 X 兲 (1)
where f⫽mapping relationship between Z and X.
After training, f is obtained. It is generally assumed that the
same f be applicable for the entire space due to the embedded
nonlinear ability in NN. The performance of a system consisting
of modular NN models may improve if different relationships
共e.g., f 兲 between Z and X exist in different regions of the space
共Shi et al. 1998兲. Of course, creating modular NN models would
require extra work.
Clustering seems a useful technique to help a user identify the
number of NN modules needed for a given problem based on the
number of clusters identified. Strong clusters are essential to as-

JOURNAL OF COMPUTING IN CIVIL ENGINEERING © ASCE / APRIL 2004 / 183

J. Comput. Civ. Eng., 2004, 18(2): 183-184

 sure that data points belonging to the same cluster are indeed
similar. The number of clusters vary greatly from problem to
problem. A membership value of 0.5 might be a weak point for a
problem with three clusters, but may be a strong point for a prob-
lem with 10 or more clusters. Like any fuzzy approaches 共e.g., an
initial r a is needed for FSC兲, subjective judgment is essential to
properly determine the number of clusters and threshold values,
although problem-specific heuristic rules may be created to auto-
mate such decisions after comprehensive comparative studies.
If the proposed FSC method works in a similar fashion as any
NN model does, the FSCគPred values in the discusser’s Table 1
should be directly estimated based on given x 1 and x 2 values. It
Downloaded from ascelibrary.org by Universidad Nacional De Ingenieria on 10/23/18. Copyright ASCE. For personal use only; all rights reserved.
was noted that an additional step was used in the discussion 关i.e.,
Eq. 共2兲兴 for computing X first, and FSC estimation was then based
184 / JOURNAL OF COMPUTING IN CIVIL ENGINEERING © ASCE / APRIL 2004
J. Comput. Civ. Eng., 2004, 18(2): 183-184
on X 关Eqs. 共3兲,共4兲兴. Because Z⫽sin(冑x 21 ⫹x 22 ) was the given
mathematical equation, the writer was uncertain how X could be
estimated for a problem while f is unknown based on the descrip-
tion in the discussion.
References
Shi, J., Ortigao, J. A. R., and Bai, J. 共1998兲. ‘‘Modular neural network for
predicting settlements during tunneling.’’ J. Geotech. Geoenviron.
Eng., 124共5兲, 389–395.
Smith, S. D., Obsorne, J. R., and Forde, M. C. 共1995兲. ‘‘Analysis of
earth-moving systems using discrete-event simulation.’’ J. Constr.
Eng. Manage., 121共4兲, 388 –396.