You are on page 1of 5

Proceedings of the 9th International Conference on Neural Information Processing (ICONIPOP) , Vol. 2 Lip0 Wang, Jagath C.

Rajapakse, Kunihiko Fukushima, Soo-Young Lee, and Xin Yao (Editors)

A NEW UNSUPERVISED COMPETIVE LEARNING ALGORITHM FOR VECTOR QUANTIZATION


Tzu-Chao Lin and Pao-Ta Yu* Department of Computer Science-andInformation Engineering National Chung Cheng University Chiayi, Taiwan 62107, R.O.C. Phone: 886-5-272041 1 ext. 33 104, E-mail: csipty@cs.ccu.edu.tw
ABSTRACT
In this paper, a novel unsupervised competitive learning
algorithm, called the centroid neural network adaptive resonance theory (CNN-ART) algorithm, is to be proposed to relieve the dependence on the initial codewords of the codebook in contrast to the conventional algorithms with vector quantization in lossy image compression. The design of the CNN-ART algorithm is mainly based on the adaptive resonance theory (ART) structure, and then a gradient-descent based learning rule is derived so that the CNN-ART algorithm does not require a predetermined schedule for learning rate. The appropriate initial weights obtained kom the CNN-ART algorithm can be applied as an initial codebook of the Linde-Buzo-Gray (LBG) algorithm such that the compression performance can be greatly improved. In this paper, the extensive simulations demonstrate that the CNN-ART algorithm does outperform other algorithms like LBG, SOFM and DCL. algorithm is simple and intuitively appealing, it has some inevitable problems [3]. The first one is that the number k of clusters must be pre-known and fixed. Another problem is that the initial codebook strongly affects the performance of the k -means algorithm. Still another problem is that the algorithm may not converge towards an optimal solution for k > 1. A variation of the k means algorithm, known as the LBG (Linde, BUZO, Gray) algorithm, has been proposed @ [4]. The LBG algorithm still suffers kom these problems as the k -means algorithm. The same thing goes for many of the so-called competitive learning algorithms based on a gradient method associated with a neural network whose weight vectors represent the codewords. The self-organizing feature map (SOFM) proposed by Kohonen has a strong connection with the k -means algorithm [5]. During the training procedure, the SOFM algorithm finds a winner neuron and updates the synaptic weights of both the winner and its neighbors. In order to obtain the best results fiom SOFM, the updated neighborhood, the initial learning rate and the total number of iterations for a given set of data should be chosen carefully. However, it is difficult to determine an appropriate set of initial synaptic weights (initial codebook) and the total number of iterations corresponding to it before the algorithm is run such that the final learning performance can be acceptable. The differential competitive learning (DCL) algorithm, proposed by Kong and Kosko [6], provides concepts of reward and punishment to the competitive learning algorithm like the learning vector quantization (LVQ) proposed by Kohonen [ 5 ] . The significance of DCL is that it allows punishment by using negative learning gain in unsupervised learning. However, it is

1. INTRODUCTION

The design of a codebook is an essential process with vector quantization (VQ) techniques in lossy image compression and usually based on the minimization of an average distortion measure between the training vectors and the codebook vectors. Several variations of vector quantization have been successfully used in image compression [I]. The well-known k -means clustering algorithm is one of the most popular competitive learning vector quantization schemes [2]. Although the k -means

* Corresponding author This work is supported by National Science Council of the Republic of China under Grant NSC 90-2213-E-194-043

944

diEcult to determine an appropriate set of initial synaptic weights, and a schedule for the optimal learning, rate as mentioned in the previous paragraph is still nowhere to be found. Based on the adaptive resonance theory (ART) structure, a new unsupervised competitive learning algorithm, called the centroid neural network .adaptive resonance theory (CNN-ART) algorithm, is proposed in this paper. The CNN-ART algorithm can determirte how a new training pattern should be classified into it cluster through a matching threshold called vigilance. Due to this significant property of the CNN-ART algorithm, the appropriate initial codebook can be easily obtained in contrast to the conventional algorithm. The CNN-ART algorithm derives a gradient-descent based learning rule, so one of the very advantageous features is that CNNART does not require a schedule for the learning rate. As the case with DCL, CNN-ART provides reward and punishment for the winner and the loser respectively in accordance with the learning rate. After iterations, the weights can converge fast towards the centroids of clusters for the expected codebook size. In Section 2, the design of CNN-ART is described in details. In Section 3, the computer simulations, the design of a codebook for image compression, are exhibited Finally, conclusions are discussed in Section 4.

codebook size is reached; that is, the number of neurons is proportional to the size of the codebook. Due to this property, the CNN-ART algorithm does not require the initial values of clusters; that is, it is not necessary to select any appropriate initial codewords in advance when we design a codebook. Each neuron of the h 4 I E T is weighted and interconnected so as to maintain its own value (+1) and attempt to be lateral inhibition ( - E ) for the others, and the output value

07 of the neuron j

at

time t is given in terms of the output values at time t - 1 ,by the relationship

01=J; (q- - &E@-) j,i=O,l,.-.,n;


ij t

t21

where Oy = (Ixi - w j

/I2,

ifp<o A(P>= 0, otherwise


/37

and

n is the current number of output layer neurons, 1 E > - . The process of the MINNET is repeated until n only one output OJIremains to be negative with all the

2. A NEW COMPETITIVE L E A R ( G ALGOlUTElM -CNN-ART


2.1. The CNN-ART Architecture

others being 0, and that is when the minimum input is found. In other words, the competitive output layer is a winner-take-all type of net. We can consider the minimum Euclidean distance between input vector xi and

wp, p = 0,1,

,n ,in the CNN-ART algorithm.

The adaptive resonance theory (ART) algorithm proposed by Grossberg is a special type of neural network that can realize classification without supervision [7]. Based on the ART algorithm, we develop the centroid neural network adaptive resonance theory (CNN-ART) algorithm in order to design a codebook. The graphical representation of the CNN-ART architecture is shown in Figure 1. The CNNART network consists of an input layer and ;m output layer called MINNET that is a simple net for determining the nearest cluster exemplar [7]. Each node in the input layer is connected directly to the neurons in the output layer. A synaptic prototype weight w p,p = 0,1, * .,k, that has the same dimension as the input data X, is associated with each neuron in the output layer as shown in Figure 1. As a result, the set of synaptic weights w P ,p = 0,1, k, can be treated as a codebook. In
. - a ,

2.2. Learning Rules in CNN-ART When the cluster j has hdJ members and x , ( j )

denotes the input data vector X, in the cluster j for one instance, the synaptic weights for a given output neuron should be chosen in a way to minimize the total Euclidean distance in L, norm from the input vectors in its cluster

[SI:
MI

wJ =mi&IIXl(i>-w112w

,=o

By the following Theorem 1, the local minimum can be achieved; that is, the centroid of data in a cluster is the solution.

the MINNET subnet, the number of neurons starts with one and grows by one each time until the desired

945

[Theorem 1 The centroid of data in a cluster is the 1 solution that gives the minimum Euclidean distance in L,

n o m [si, ~91.

In the CNN-ART algorithm, according to Theorem 1, the centroid of every cluster is computed by the following equation:

1 MJ w J.=--Cxi(j). I A
1v1 . ;=I

minimum. However, according to the following adaptive vector quantization (AVQ) convergence theorem [9], AVQ can be viewed as a way to learn prototype vector patterns of real numbers; it can guarantee that average synaptic vectors converge to centroids exponentially quickly, so the CNN-ART algorithm makes the convergence to the centroids of clusters much faster than conventional learning algorithms.
[AVQ Convergence Theorem] Competitive synaptic vectors converge exponentially quickly to pattern-class centroids.

In this paper, the CNN-ART algorithm provides reward and punishment to the competitive learning rules. When an input data vector Xi is applied to the network at time t ,

the CNN-ART algorithm rewards the winner neuron j (updates its weights

w? : the weights of a winner neuron

From Equations (1) and (2), a gradient-descent based learning rule is derived. The learning rate can be summarized as

in the current iteration are updated only when the winner neuron does not win the data xi in the previous iteration). The adaptive equation

if j

is a winner,

wy) can be written as follows:


XI

a ( t ) = (-1)-

w(.? M j x w j = J Mj+l

(1-1)

+-

Mj+l
.j

M j +1

Through Equation (3), CNN-ART provides reward and punishment to the competitive learning algorithm by using positive and negative learning gain respectively. Furthermore, the learning rate is adaptive according to the number of members in the winner or loser cluster, so CNN-ART does not require a predetermined schedule.
2.3. The CNN-ART Algorithm

1 M y -1

if y is a loser,

(3)

0, otherwise.

At the same time, the CNN-ART algorithm will punish the loser neuron JJ (update its weights w: : the weights of the loser neuron in the current iteration are updated only when the loser neuron win the data xi in the previous iteration). The adaptive equation w:) can be written as follows:

In the CNN-ART competitive learning algorithm, at first, an input training vector xo is selected as the centroid of the first cluster, and then the next input training vector is compared to the fist cluster centroid. It is classified to the first cluster if its Euclidean distance is smaller than the value of vigilance. Otherwise, it forms the centroid of a new cluster. This process-is repeated for all the training input vectors. The following steps show a pseudo code of the CNN-ART algorithm.
Step 1: [Initialize] Set CodebookSize k ,Total-input-vectors N ; n = 0; w0 = xo; // n :the number of neurons. Step 2: [Set vigilance] Select vigilance; Step 3: P o loop for designing a codebook] Repeat For (i = 0;i I N;i +) (a) [Calculate Euclidean distances]

= w;-

-~ 1
M y -1

[x; - W y ] .

From Equations (1) and (2), we obtain that CNN-ART adapts its weights according to the centroids to achieve the local minimum Euclidean distance; that is, the mechanism is a greedy approach, so it can minimize the total Euclidean distance for designing a codebook. However, it does not guarantee the convergence towards the global

946

Apply an input vector xi to the network, and calculate Euclidean distance between and existing clusters. (b) pecide the winner neuron j ] Find the smallest Euclidean distance d (c)[Test tolerance] If ( d j> vigilance) AND (n < k) goto ( 4 ; else goto (e); (d)[Form a new cluster] n = n 1; w, = x i ; goto (a);

xi

(e)[Update the weights] Updatewinnerweight w j; Updateloserweight w, ; (f)[Check non-uniformed clusters] If a cluster has not many input vectors to itself, in a snapshot Spare this cluster for later use; Endfor Until convergence criterion is satisfied The CNN-ART clustering process is controlled by the Euclidean metric function and by a threshold called the vigilance at two stages respectively. In the first stage, CNN-ART computes the distance between the input vector and the existing clusters by using the Euclidean metric function:

gather a large number of clusters. On the other hand, if the vigilance is higher, then larger departures are tolerated, and a smaller set of clusters will be formed. The optimal value of vigilance is obtained experimentally which do not require a laborious process performed in the simulations. We set some inspection points to check the size of each current cluster. If a cluster has few input vectors to itself, especially only one, in a snapshot, we eliminate the cluster during the process. Thus, those empty clusters can be used again. Once all the input vectors are trained, they will be classified into their own clusters. The CNN-ART algorithm is reiterated until a stable cluster formation occurs, so it is not necessary to decide the total number of iterations to process in advance. The final set of synaptic weights can be treated as a codebook.

3. THE SIMULATION OF IMAGE COMPRESSION FOR CODEBOOK DESIGN


The proposed CNN-ART algorithm is used to perform the design of a codebook, which is the typical application in image compression based on vector quantization. The experiments are conducted on LBG, SOFM, DCL, and the proposed CNN-ART algorithm to see how they perform on the task of designing a codebook for image compression. Under the condition of different codebook sizes or bits per pixel (bpp) values, the Lena and Peppers images, 256 x 256 pixels of size each, are used as the training images in the experiments. From the comparison of the performances between LBG and CNNART shown in Figure 2, we see that our CNN-ART algorithm always does better than the LBG algorithm, especially when it comes to one iteration only. Due to this significant property, the one iterative set of appropriate synaptic weights can be applied as the initial codebook of LBG, now called the CNN-ART-LBG algorithm, such that the compression performance can be greatly improved Note that the first k (the size of codebook) input training vectors are used as the initial codebook of LBG, DCL and SOFM in the experiments. Table I shows the comparison of the resulting PSNR among the algorithms, where CNN-ART has the best result of all. Here, still another experiment is conducted to demonstrate the robustness of the CNN-ART algorithm. The Peppers image is used in this experiment for training the codebook, and the resulting codebook is tested with the Lena image. From Table II, the CNN-ART has exhibited a satisfactory result.
4. CONCLUSIONS

where n limited to the codebook size k is the current neuron sue of the output layer, q is the dimension of the input training vector, and

xi is

the ith inpul training

vector. In the second stage, the nearest clusterj (winner neuron j ) is found by selecting the minimum d . Then, the following rules are used:
0

E ( d j >vigilance) AND ( n < k ) Then form a new cluster, set n = n + 1.


If d j S vigilance
Then update the weights w j and W, . The total number of clusters grows dynamically with the number of input training vectors and depends on the value of vigilance. If we set the vigilance to be low, then it will

The design of the CNN-ART algorithm is not necessary to select appropriate initial codewords when we use the CNN-ART algorithm to design a codebook. The gradient-

947

descent based learning rule is derived to provide reward and punishment to the neuron in the process so that the CNN-ART algorithm does not require a predetermined schedule for learning rate. Furthermore, CNN-ART adaptively learns the training vectors and converges much faster than conventional learning algorithms, and it does not require the total number of iterations in advance. Due to the significant vigilance property of the CNN-ART algorithm, the appropriate initial weights obtained fiom the CNN-ART algorithm can be applied as an initial codebook to the Linde-Buzo-Gray (LBG) algorithm such that the compression performance can be greatly improved. Moreover, experimental results have also demonstrated that CNN-ART for image compression outperforms other competitive learning algorithms.

Lateral inhibum

Create a new

nemm $ynamically

MINNET

Y
XP

x z'

Y
x:,

Y
x ,2
x;. :.

_''

1' ,

x , = [x;,

, x:]'

Figure 1: The architecture of CNN-ART.


5. REFERENCES

[l] N. M. Nasrabadi and Y. Feng, Image compression using address-vector quantization, IEEE Trans. on Communication,pp. 2166-2173, 1990. [2] A. Gersho and R. M. Gray, Vector Quantization and Signal Compression, MA: Kluwer Academic, Boston, 1992.
[3] Lei Xu, Bayesian Ying-Yang machine, clustering and number of clusters, Pattern Recognition Letters, 18, pp. 1167-1178, 1997.

$$ 2

30.5 30 29.5 29
28.5

&LBG

+CNN-ART
0

28
20
40

60

Number of Iterations

Figure 2: Comparison in PSNR (dB) between CNN-ART and LBG with different numbers of iterations.

[4] Y. Linde, A. Buzo and R. M. Gray, An algorithm for vector quantizatizer design, IEEE Trans. on Communication,28, pp. 84-95, 1980. [5] T. Kohonen, Self-organization and Associative Memory, Germany: Springer-Velag, Berlin, third edition, 1989.
[6] S. Kong and B. Kosko, Differential competitive learning for centroid estimation and phoneme recognition, IEEE Trans. on Neural Networks, pp. 118-124,1991.

h a
43ori"

Peppers 0625 32.14 36.21 0.5 29.28 30.16 0.5625 30.58 32.25 0.625 32.31 35.50

0.5 29.23 29.95

0.5625 30.41 31.92

LBG CNN-AKr-LBG CNN-AKr

29.97

31.97

36.63

30.18

32.33

I 3621

Bits per pixd (b.p.p.)

[7] Y. H. Pao, Adaptive Pattern Recognition and Neural Networks, Addison Wesley Reading, MA, 1989. [8] Dong-Chul Park, Centroid neural network for unsupervised competitive learning, IEEE Trans. on Neural Networks, pp. 520-528, March 2000. [9] B. Kosko, Neural Networh and Fuzzy System: A Llynamic Systems Approach to Machine Intelligence, Prentice-Hall,Englewood Cliffs, NJ, 1991.

Algorithms
LBG

0.5625 27.59 29.98

0.625 27.96 30.39

I I

0.6875 28.28 30.74

DCL

948