ceptron canonly separate linearly separable patterns as long e8 tron lears2ng5ri含》W as a monotonic activation function is used. 1.Initialize the weights and threshold to small The back-propagation learning algorithm (see the random numbers. "Back-propagation algorithm sidebar")is also based on 2.Present a pattern vector (xX2....x)and the error-correction principle. evaluate the output of the neuron. 3.Update the weights according to BOLTZMANN LEARNING.Boltzmann machines are sym- metric recurrent networks consisting of binary units (+1 wt+1)=W0+门(d-》 for“on”and-lfor“off).By symmetric,.we mean that the weight on the connection from unitito unit jis equal to the where d is the desired output,t is the iteration weight on the connection from unit j to unit i (w=w).A number,and n (0.0<n<1.0)is the gain (step subset of the neurons,called visible,interact with the envi- size). ronment;the rest,called hidden,do not.Each neuron is a stochastic unit that generates an output (or state)accord- ing to the Boltzmann distribution of statistical mechanics. v=wx-u Boltzmann machines operate in two modes:clamped,in which visible neurons are clamped onto specific states deter- minedby the environment;andfree-running,in which both visible and hidden neurons are allowed to operate freely. The outputy of the perceptron is +1ify>0,and 0 oth- Boltzmann learning is a stochastic learning rule derived erwise.In a two-class classification problem,the percep- from information-theoretic and thermodynamic princi- tron assigns an input pattern to one class ify =1,and to ples.0 The objective of Boltzmann learning is to adiust the the other class ify=0.The linear equation connection weights so that the states of visible units satisfy a particular desired probability distribution.According to the Boltzmann learning rule,the change in the connec- w5-=0 tion weight w is given by △wH=(p4-P), defines the decision boundary (a hyperplane in the n-dimensional input space)that halves the space. where n is the learning rate,and p and p are the corre. Rosenblatt5 developed a learning procedure to deter- lations between the states of units i and i when the net- mine the weights and threshold in a perceptron,given a work operates in the clamped mode and free-running set of training patterns(see the"Perceptron learning algo- mode,respectively.The values of p,and p are usually esti- rithm”sidebar) mated from Monte Carlo experiments,which are Note that learning occurs only when the perceptron extremely slow. makes an error.Rosenblatt proved that when training pat- Boltzmann learning can be viewed as a special case of terns are drawn from two linearly separable classes,the error-correction learning in which error is measured not perceptron learning procedure converges after a finite as the direct difference between desired and actual out- number of iterations.This is the perceptron convergence puts,but as the difference between the correlations among theorem.In practice,you do not know whether the pat- the outputs of two neurons under clamped and free- terns are linearly separable.Many variations of this learn- running operating conditions. ing algorithm have been proposed in the literature.2 Other activation functions that lead to different learning char- HEBBIAN RULE.The oldest learning rule is Hebb's pos- acteristics can also be used.However,a single-layer per- tulate oflearning.13 Hebb based it on the following obser- vation from neurobiological experiments:If neurons on both sides of a synapse are activated synchronously and repeatedly,the synapse's strength is selectively increased. X24 W Mathematically,the Hebbian rule can be described as w,(t+1)=w(t)+ny(t)x() where x and y,are the output values of neurons i and j, 。” respectively,which are connected by the synapse wn,andn is the learning rate.Note thatx,is the input to the synapse. W An important property of this rule is that learning is done locally,that is,the change in synapse weight depends only on the activities of the two neurons connected by it. This significantly simplifies the complexity of the learning circuit in a VLSI implementation. A single neuron trained using the Hebbian rule exhibits Figure 5.Orientation selectivity of a single neuron an orientation selectivity.Figure 5 demonstrates this prop- trained using the Hebbian rule. erty.The points depicted are drawn from a two-dimen- 36 Computerctor (x,,x,, . . . , xJ* and of the neuron. red output, t is the iteration i=l The outputy of the perceptron is + 1 ifv > 0, and 0 oth￾erwise. In a two-class classification problem, the percep￾tron assigns an input pattern to one class ify = 1, and to the other class ify=O. The linear equation j=1 defines the decision boundary (a hyperplane in the n-dimensional input space) that halves the space. Rosenblatt5 developed a learning procedure to deter￾mine the weights and threshold in a perceptron, given a set of training patterns (see the “Perceptron learning algo￾rithm” sidebar). Note that learning occurs only when the perceptron makes an error. Rosenblatt proved that when trainingpat￾terns are drawn from two linearly separable classes, the perceptron learning procedure converges after a finite number of iterations. This is the perceptron convergence theorem. In practice, you do not know whether the pat￾terns are linearly separable. Many variations of this learn￾ing algorithm have been proposed in the literature.2 Other activation functions that lead to different learning char￾acteristics can also be used. However, a single-layerper￾Figure 5. Orientation selectivity of a single neuron trained using the Hebbian rule. ceptron can onlyseparate linearly separable patterns as long as a monotonic activationfunction is used. The back-propagation learning algorithm (see the “Back-propagation algorithm sidebar”) is also based on the error-correction principle. BOLTZMA” LFING. Boltzmann machines are sym￾metric recurrent networks consisting of binary units (+ 1 for “on” and -1 for “off’). By symmetric, we mean that the weight on the connection from unit i to unitj is equal to the weight on the connection from unit j to unit i (wy = wJ. A subset of the neurons, called visible, interact with the envi￾ronment; the rest, called hidden, do not. Each neuron is a stochastic unit that generates an output (or state) accord￾ing to the Boltzmann distribution of statistical mechanics. Boltzmann machines operate in two modes: clamped, in whichvisible neurons are clamped onto specific states deter￾mined by the environment; andfree-running, in which both visible and hidden neurons are allowed to operate freely. Boltzmann learning is a stochastic learning rule derived from information-theoretic and thermodynamic princi￾ples.lOThe objective of Boltzmann learning is to adjust the connection weights so that the states ofvisible units satisfy a particular desired probability distribution. According to the Boltzmann learning rule, the change in the connec￾tion weight wg is given by Awij =q(P,j -pij), where q is the learning rate, and p, and py are the corre￾lations between the states of units z and J when the net￾work operates in the clamped mode and free-running mode, respectively. The values of pli and p, are usuallyesti￾mated from Monte Carlo experiments, which are extremely slow. Boltzmann learning can be viewed as a special case of error-correction learning in which error IS measured not as the direct difference between desired and actual out￾puts, but as the difference between the correlations among the outputs of two neurons under clamped and free￾running operating conditions. HEBBIAN RULE. The oldest learning rule is Hebb’spos￾tulate of 1e~rning.l~ Hebb based it on the following obser￾vation from neurobiological experiments: If neurons on both sides of a synapse are activated synchronously and repeatedly, the synapse’s strength is selectively increased. Mathematically, the Hebbian rule can be described as WJt + 1) = W,W + ?lY]@) xm, where x, andy, are the output values of neurons i and J, respectively, which are connected by the synapse w,, and q is the learning rate. Note thatx, is the input to the synapse. An important property of this rule is that learning is done locally, that is, the change in synapse weight depends only on the activities of the two neurons connected by it. This significantly simplifies the complexity of the learning circuit in a VLSI implementation. A single neuron trained using the Hebbian rule exhibits an orientation selectivity. Figure 5 demonstrates this prop￾erty. The points depicted are drawn from a two-dimen￾Computer