sional Gaussian distribution and used for training a neu ron.The weight vector of the neuron is initialized to wo as shown in the figure.As the learning proceeds,the weight vector moves progressively closer to the direction w of maximal variance in the data.In fact,wis the eigenvector of the covariance matrix of the data corresponding to the largest eigenvalue. COMPETITIVE LEARNING RULES.Unlike Hebbian learn- ing (in which multiple output units can be fired simulta- neously),competitive-learning output units compete (b) among themselves for activation.As a result,only one out- put unit is active at any given time.This phenomenon is Figure 6.An example of competitive learning:(a) known as winner-take-all.Competitive learning has been before learning;(b)after learning found to exist in biological neural networks.3 Competitive learning often clusters or categorizes the input data.Similar patterns are grouped by the network The most well-known example of competitive learning and represented by a single unit.This grouping is done is vector quantization for data compression.It has been automatically based on data correlations. widely used in speech and image processing for efficient The simplest competitive learning network consists of a storage,transmission,and modeling.Its goal is to repre- single layer of output units as shown in Figure 4.Each out- sent a set or distribution of input vectors with a relatively put unit i in the network connects to all the input units (x,'s) small number of prototype vectors (weight vectors),or a via weights,wj=1,2,...,n.Each output unit also con- codebook.Once a codebook has been constructed and nects to all other output units via inhibitory weights but has agreed upon by both the transmitter and the receiver,you a self-feedback with an excitatory weight.As a result of com- need only transmit or store the index of the corresponding petition,only the unit i with the largest (or the smallest) prototype to the input vector.Given an input vector,its cor- net input becomes the winner,that is,w'x≥w·x,∀i,or responding prototype can be found by searching for the w-xs w;-x,Vi.When all the weight vectors are nearest prototype in the codebook normalized,these two inequalities are equivalent. A simple competitive learning rule can be stated as SUMMARY.Table 2 summaries various learning algo- rithms and their associated network architectures(this n(x-wi),i=i", is not an exhaustive list).Both supervised and unsuper- (1) vised learning paradigms employ learning rules based 0. i≠i*】 Note that only the weights of the winner unit get updated The effect of this learning rule is to move the stored pat 3-3物之20器透80教% tern in the winner unit (weights)a little bit closer to the 1. Initialize the weights to small random values input pattern.Figure 6 demonstrates a geometric inter- 2. Randomly choose an input pattern x. pretation of competitive learning.In this example,we 3. Propagate the signal forward through the network assume that all input vectors have been normalized to have Compute 8 in the output layer (o=y) unit length.They are depicted as black dots in Figure 6. The weight vectors of the three units are randomly ini- =g(h,9d=y] tialized.Their initial and final positions on the sphere after competitive learning are marked as Xs in Figures 6a and 6b,respectively.In Figure 6,each of the three natural where hrepresents the net input to the ith unit in the /th groups (clusters)of patterns has been discovered by an layer,and g'is the derivative of the activation function g. output unit whose weight vector points to the center of Compute the deltas for the preceding layers by propa- gravity of the discovered group. gating the errors backwards: You can see from the competitive learning rule that the network will not stop learning (updating weights)unless 8=gh∑w6 the learning rate n is 0.A particular input pattern can fire different output units at different iterations during learn- for1=(L1),,1 ing.This brings up the stability issue of a learning system. 6.Update weights using The system is said to be stable if no pattern in the training data changes its category after a finite number of learning Aw =n6 v iterations.One way to achieve stability is to force the learn- ing rate to decrease gradually as the learning process pro- Go to step 2 and repeat for the next pattern until the ceeds towards 0.However,this artificial freezing of learning error in the output layer is below a prespecified thresh causes another problem termed plasticity,which is the abil. old or a maximum number of iterations is reached. ity to adapt to new data.This is known as Grossberg's sta- bility-plasticity dilemma in competitive learning. March 1996 37sional Gaussian distribution and used for training a neu￾ron. The weight vector of the neuron is initialized tow, as shown in the figure. As the learning proceeds, the weight vector moves progressively closer to the direction w of maximal variance in the data. In fact, w is the eigenvector of the covariance matrix of the data corresponding to the largest eigenvalue. COMPETITIVE WING RULES. Unlike Hebbian learn￾ing (in which multiple output units can be fired simulta￾neously), competitive-learning output units compete among themselves for activation. As a result, only one out￾put unit is active at any given time. This phenomenon is known as winner-take-all. Competitive learning has been found to exist in biological neural network^.^ Competitive learning often clusters or categorizes the input data. Similar patterns are grouped by the network and represented by a single unit. This grouping is done automatically based on data correlations. The simplest competitive learning network consists of a single layer of output units as shown in Figure 4. Each out￾put unit i in the network connects to all the input units (+) via weights, w,, j= 1,2, . . . , n. Each output unit also con￾nects to all other output units via inhibitoryweights but has a self-feedbackwith an excitatoryweight. As a result of com￾petition, only the unit i‘ with the largest (or the smallest) net input becomes the winner, that is, w,*. x 2 w,. x, Vi, or 11 w; - x /I < /I w, - x 11, Vi. When all the weight vectors are normalized, these two inequalities are equivalent. A simple competitive learning rule can be stated as 0, i+i”. Figure 6. An example of competitive learning: (a) before learning; (b) after learning. The most well-known example of competitive learning is vector quantization for data compression. It has been widely used in speech and image processing for efficient storage, transmission, and modeling. Its goal is to repre￾sent a set or distribution of input vectors with a relatively small number of prototype vectors (weight vectors), or a codebook. Once a codebook has been constructed and agreed upon by both the transmitter and the receiver, you need only transmit or store the index of the corresponding prototype to the input vector. Given an input vector, its cor￾responding prototype can be found by searching for the nearest prototype in the codebook. SUMMARY. Table 2 summaries various learning algo￾rithms and their associated network architectures (this is not an exhaustive list). Both supervised and unsuper￾vised learning paradigms employ learning rules based Note that only the weights of the winner unit get updated. The effect of this learning rule is to move the stored pat￾tern in the winner unit (weights) a little bit closer to the input pattern. Figure 6 demonstrates a geometric inter￾pretation of competitive learning. In this example, we assume that all input vectors have been normalized to have unit length. They are depicted as black dots in Figure 6. The weight vectors of the three units are randomly ini￾tialized. Their initial and final positions on the sphere after competitive learning are marked as Xs in Figures 6a and 6b, respectively. In Figure 6, each of the three natural groups (clusters) of patterns has been discovered by an output unit whose weight vector points to the center of gravity of the discovered group. You can see from the competitive learning rule that the network will not stop learning (updating weights) unless the learning rate q is 0. A particular input pattern can fire different output units at different iterations during learn￾ing. This brings up the stability issue of a learning system. The system is said to be stable if no pattern in the training data changes its category after a finite number of learning iterations. One way to achieve stabilityis to force the learn￾ing rate to decrease gradually as the learning process pro￾ceeds towards 0. However, this artificial freezing of learning causes another problem termed plasticity, which is the abil￾ity to adapt to new data. This is known as Grossberg’s sta￾bility-plasticity dilemma in competitive learning. March 1996