Modeling of Component Failure in Neural Networks for Robustness Evaluation: An Application to Object Extraction ‘Ashish Ghosh, Nikhil R. Pal, Member, IEEE, and Sankar K. Pal, Fellow, IEEE Absract— Au investigation on the robustness (or ruggedness) of neural network (NS) based. Information proceag systems ‘with respect to component fallare (damaglag of podesinks) is done. The danaging/oumpooeat fllure process hat been modeled fa Polsson process To choose the instants of moments of ‘damaging, statistical sampling technique is use. The nodesinks tbe damaged are determined randomly. ‘As an lltstration, the model Is Implemented and tested on “iferent object exracton algortthans employing ‘dative memory mode, Cibbe random elds, and ‘mayer nar newort Te petra of the undertyng network model) tated In terms of percentage of pals correct chased wader 0 0 t<0. ® Thus, when the faire proest is governed by a Poisson disribution, the interfilure time i then described by an ‘exponential istbuion (3) with expected value (mean) B= [soe ® B. Sampling ‘To simulate the failure process, one needs to draw random ‘samples from the exponential distribution (3). Before deserit- ng the exact algorithm, let us first consider the general strategy for sampling from any distribution, Let /(2) be the paf of the random deviate x, and F() be the cuinulative density function (cdf) of =, ie, Fe)= [so It can be easily shown thatthe random variable y = F(z) is ‘uniformly distributed over (0,1), regardless of the distribution © of 2, Hence, if isa random number drawn from uniform (0, 1, then z= F(R) is a random sample from the paf f(2) ‘Therefore, sampling from any distribution can be done using the following simple method having two steps. ‘Sep 1: Generate a random number Ria (0, 1] and assign it to Fle) Step 2: Solve for 2 from R= F(z). ‘The above sampling method is known as method of inversion, ‘Sampling from Exponential Distribution: For an exponen- tial distribution the pf is £0) yer! p>0, 60 0 #0. © o “The last step is possible because if is a random number on {0, 1} then so is (1 — R) and we can replace (1-2) by R 1 has been established before that ifthe failure process is described by a Poisson distribution, then the time between ‘the occurence of failures (intrfilure time) must follow the corresponding exponential distribution. Thus to simulate the ‘component failure process described by the Poisson distibu- tion with mean yt, over a time period (0.7) all oe has to do is to sample the corresponding exponential distribution with mean 1/1 a8 many times as necessary until the sum of the ‘coresponding exponential random samples generated exceeds TT for the frst time, It can further be explained as follows. Suppose Ry isthe ith random sample drawn from uifer (0), then 1 IR @ » is the ith sample from the exponential dstibation (3). There fore ned ° ives the ime stan when the ith (component are occurs. ‘The process is repeated forthe maximam number of times CH, s0p)soch that Ty < 7. Il, APPLICATION OF FAILURE MODEL 70 ROBUSTNESS EVALUATION OF NEURAL NETWORKS In any system components may fail with passage of time. In the case of NN-based systems the components are the rneurons/aodes and links. So in such systems, some of the neurons or links or both may get damaged over time. NN- ‘based information processing systems are normally claimed tobe robust under components failure as the NN architectures involve massive processing elements and connectivity among them (mostly witha few redundant components). The systems are also supposed to be noise insensitive and suitable for il-defined/partil information. ‘The robustness of a neural network system can be investi- ‘gated under three diferent situations: + failure of only nodes, + failure of only inks, and + failure of both nodes and links. ‘We have already seen in Section I-A that if the components sre identical and failures are independent, then the inter failure time follows an exponential distribution. Therefore, ‘while considering failures of only nodes or only links, we ‘can use exponential distribution (3) to draw samples fr inter failure time. Section H-B discusses how to compute the time instant 7; ofthe ith failure, but does not mention about the selection ofthe node (link) to damage at tat instant T;. Since all nodes (or all Hinks) are assumed to be identical, one can select a nade (to be damaged) randomly in such a manner that each node (link) bas an equal chance of being selected ‘The situation becomes a litle more complicated when both nodes and links are allowed to fail. A node failure effectively disables several Hinks connected to it. On the other hand, failure of a set of links may effectively disable a node. Some of the assumptions stated in Section II-A are not valid, if wwe consider both node and link failures together, because the set of components are not identical and failure of one ‘component may cause effective flue of another. We propose to investigate the system under the following two simplifying Case I: Do not differentiate between nodes and links, i. treat either of them as a component. The damaging process is then viewed to follow a single Poisson distribution. Case I: Nodes and links are treated as two different com- ‘ponents. The damaging process will then be governed by 1Wo different Poisson distributions, one fo the nodes and the other for the links, No interdependence between the two distributions is assumed. ‘The components (nodestinks) which are geting damaged ‘can be chosen randomly. When we are damaging only one type of component (ie, either node or link), the easiest way to Select a component is to assign a unique number to each ‘component and use random sumbers in such a way that each node has an equal chance of inclusion. If an already selected component reappears, then one has 10 ignore that drawing. When one is damaging both nodes and inks together the selection process will be as follows depending on the previously mentioned two cases. Suppose there are M nodes and links, Case I: Each component is assigned 2 unique integer num ber inthe range {1, +1], and a component can be selected by drawing randomly an integer in that range. (Case I: Assign each node a unique integer in the range “M] and each link a unigue integer in the range (1, NJ Selection of a node can be done by drawing a random integer in (2... Similarly links are selected, {REE TRANSACTIONS ON NEURAL NETWORKS, VOL. 6 NO. 2. MAY 1985 Note that Case I does not discriminate between nodes sand links and bence interdamage times are drawa from one Aisribution for both the nodes and links. Case I, on the other Ihand, does discriminate between nodes and Tinks and uses two ‘independent distribution (one for links and the other for nodes). ‘The model of damaging process developed in this section is, in general, applicable for any NN-based system (both supervised and unsupervised). In the following sections an ‘pplication ofthe model is demonstrated on some NN-based object extraction algorithms. For a supervised system, this failure can occur in either of the learning and testing phases. FFor an unsupervised system there is no testing phase, and the failure can occur during updation of weights/staus for ‘unsupervised learning, TV. APPLICATION To OBJECT EXTRACTION PROBLEMS Let us consider three object extraction algorithms [10}-{12) for demonstration of the validity of the model. The frst two algorithms use modified version of Hopfield’s network model, ‘whereas the third one deals with a self-organizing multi layer neural network. Let us fist brief the algorithms for the convenience of the readers before describing the present experimental procedure and results of investigation. A. Algorithms Algorithm 1 [10]: Jo tis algorithm the task of image seg- ‘mentation is formulated as an optimization problem and solved by & modified version of Hopfield's model. It involves mini- mization of an objective function “Lowy - lw ‘where Wis the connection strength between the ith and the 4th neurons, (€ (0,1) is the input bias to the sth neuron ‘and Vi (€ (0,1) is the ouput status of the ith neuro. In the present study we have chosen a 3x3 neighborhood. Here, since the number of neighbors is fixed (8), the input value fo a neuron lies in the domain [8,8]. A polynomial funetion io) gt) =-1 ie<-s f(e+8"-1 if -8<2<0 ain ke-2 itosece OD 1 228 defined over the fsite domain {—8, 8] is used as an inpavoutput transfer function ‘The weights and input biases are given in such a way that the network self-organizes to form compact clusters. The dynamics is similar to that of Hopfield's model, Algorithm 2 [11]: A modified version (both architecture wise and neuron characteristic wise) of Hopfild's mode! is ‘sed to determine the maximum aposterior probability (MAP) estimate ofa scene from a noise comrupted realization (modeled by Gibbs random field). Here als, a single neuron is assigned to every pixel, and itis connected only to all ofits nearest neighbors. The energy function of the network was designedto be £=-DYwmy-5 Dy + Dv ay such that its minimum value corresponds tothe MAP estimate of the scene, Here o is the noise level and the other symbols have the same meaning a8 in Algorithm 1. ‘The operations (dynamics) are similar to that in Algorithm Inthe stable state the Set of neurons with ON status constiute the object region. Algorithm 3 [12]: This algorithm is based on the integra- tion of the concept of fuzzy sets and neural network models by incorporating various fuzziness measures in a multi-layer (one input, one hidden, and one output layer in the present investigation) network for object extraction. The architecture is feedforward one with back propagation of error and is used as an unsupervised classifier. Each neuron is connected to the corresponding neuron in the previous layer and to its ‘neighbors. There also exists a feedback path from the output to the input layer. The status of neurons in the outpat layer is described as a fuzzy set. A fuzziness measure [13] of this sot is used as a measure of error ofthe system (instability of| the network) ‘The input value (U,) to the ith neuron in any layer [except the input layer) is 2 u=Dwys ay wher Wi th connection eng betwee he ih ero one ge and jh oto revo layer jn ser Belong ttc neihbrinl ford ofthe revo lye ‘sip tne acon hn ape 0 te pt Stas ofthe neurons inthis ayer Tate capt ae ten ed Sept ote net aj Stating fom he inp jer us ‘ay te lp pater pase ono he ota aye nd Conrspiding out sacs a caeuaed The pa vale fea newt en (01 ‘Aferte wegs have bee usted propery, tbe ouput the eon in te pt layer edb othe Conepodng euros inthe inp lye forte ext ps. The Teen (Gyan of weigh) conints wn he network stn, 12 eer ae (near fae) becomes gig ‘When te ctor taste ut ars fhe nero Ine cur ler becomes eer tr or on, The neuen having ouput auc neo consaetepoup ad he hang caper tle oe cote the abet pop Tete prota iveiguon, he qua inde of fsie (13) ued the enor meanre According, he weight Updating rae tecones fame Hl = VFM for the output layer and AW, = (Sams) rane ifos vj <05 AWs a4) ios