handwritten Digit Recognition with a Back-Propagation Network Y. Le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel AT&T Bell Laboratories, Holmdel, N. J. 07733 ABSTRACT We present an application of back-propagation networks to hand- written digit recognition. Minimal preprocessing of the data was required, but architecture of the network was highly constrained and specifically designed for the task. The input of the network consists of normalized images of isolated digits. The method has 1% error rate and about a 9% reject rate on zipcode digits provided by the U.S. Postal Service. 1 INTRODUCTION The main point of this paper is to show that large back-propagation (BP) net- works can be applied to real image-recognition problems without a large, complex preprocessing stage requiring detailed engineering. Unlike most previous work on the subject (Denker et al., 1989), the learning network is directly fed with images, rather than feature vectors, thus demonstrating the ability of BP networks to deal with large amounts of low level information. Previous work performed on simple digit images (Le Cun, 1989) showed that the architecture of the network strongly influences the network's generalization ability. Good generalization can only be obtained by designing a network architecture that contains a certain amount of a priori knowledge about the problem. The basic de- sign principle is to minimize the number of free parameters that must be determined by the learning algorithm, without overly reducing the computational power of the network. This principle increases the probability of correct generalization because
✂✁☎✄✝✆☎✞✠✟☛✡✌☞✍☞✏✎✑✄ ✒✓✡✕✔✖✡✌☞✠✗✘✎✚✙✜✛✜✔✖✄✢✡✌☞✣✡✌✛✖✄ ✞✠✡✌☞✣✤ ✁ ✥✦✁✧✙✩★✫✪✍✬✭✟✮✛✩✯✰✁✱✔✲✁✧☞✣✡✌✛✖✄ ✳✴✎✵☞✣✞✶✛☎✟☛★ ✷✹✸✻✺✽✼✿✾✜❀✌❁❃❂✌❄❅✸✍❄❇❆❉❈❊✼●❋❍❂✻■✍✸✌❏✏✸✍❑❅✼●❁✌▲▼✼●❋❍❂✻❑✫✸✌◆❖✼●❁✍P◗✼❘❋❙❈❙❆▼❁❃❂ ❚✱✸✌❯☛✸✍◆❖❆❲❱❨❳❩❋❬P✽❂◗❭❪✸✌◆☛❀✌❫✌❫✻❳❲❋❬P✏❂✍❳❲❁✻P✖✺❴✸✻❑❵✸◗■❛❳❉❜❊▲▼✼●❝ ❞❴❡✑❢❅❡ ❄❇✼●❝❣❝✮✺❤❳❲❫✻❆❉❋✐❳❲❥❙❆▼❋❙❦❣✼●❈✕❂✌◆☛❆❉❝❣❧✧P◗✼●❝♠❂❛♥✱✸◗■✍✸✻♦❉♣❉♣▼q❉q rts✈✉❨✇②①tr✈③④✇ ❭✢✼✿⑤✌❋❙✼❘❈❙✼●❁◗❥✫❳❩❁④❳❲⑤✌⑤✣❝⑥❦⑦❜●❳❩❥❙❦❣❆❉❁☎❆❲⑧⑨❫✻❳❉❜❊▲▼⑩❶⑤✌❋❊❆❉⑤✍❳❩❷❛❳❩❥❙❦❣❆❉❁☎❁✌✼●❥♠❱✜❆❉❋❙▲◗❈✩❥❙❆☎❸✍❳❲❁✻P▼⑩ ❱☛❋❙❦❣❥❙❥❊✼●❁✝P◗❦❣❷❉❦❣❥✩❋❙✼❍❜❹❆❉❷❉❁✣❦⑥❥❊❦❣❆❉❁❃✸❻❺✧❦❣❁✌❦❣❧✲❳❩❝❃⑤✌❋❊✼●⑤✌❋❊❆❼❜❹✼●❈❙❈❊❦⑥❁✣❷④❆❩⑧❨❥❊❸✌✼✖P✌❳❲❥✐❳✧❱☛❳❲❈ ❋❙✼❍❽▼❀✌❦❣❋❙✼✕P✽❂❴❫✌❀✣❥✹❳❩❋❬❜❊❸✌❦❣❥❙✼❍❜❹❥❙❀✌❋❊✼✰❆❲⑧✑❥❊❸✌✼✖❁✌✼❘❥♠❱⑨❆▼❋❙▲✢❱☛❳❲❈✿❸✌❦❣❷▼❸✌❝❣❾❿❜✐❆▼❁✌❈❙❥❊❋❬❳❩❦⑥❁✣✼❍P ❳❲❁✍P➀❈❙⑤✻✼❍❜✐❦➂➁➃❜●❳❩❝⑥❝❣❾✢P◗✼●❈❊❦⑥❷▼❁✌✼❍P❿⑧➄❆❉❋❵❥❙❸✌✼☎❥❬❳❩❈❙▲✽✸ ❡❸✌✼✖❦❣❁✌⑤✣❀✌❥✿❆❲⑧✑❥❊❸✌✼✖❁✣✼●❥♠❱⑨❆▼❋❙▲ ❜✐❆❉❁✌❈❊❦❣❈❙❥❙❈❵❆❲⑧✜❁✌❆❉❋❊❧✲❳❲❝❣❦❣➅●✼❍P➆❦⑥❧✧❳❲❷▼✼●❈✑❆❩⑧❨❦❣❈❙❆▼❝⑦❳❲❥❙✼✕P✰P◗❦❣❷❉❦❣❥❙❈✕✸ ❡❸✌✼❻❧✖✼●❥❊❸✌❆✌P✧❸✍❳❩❈ ➇❍➈➉✼❘❋❙❋❙❆▼❋➊❋❬❳❲❥❊✼✑❳❲❁✍P❵❳❲❫✻❆❉❀✣❥✮❳✩➋❉➈➌❋❊✼➄➍✐✼❍❜✐❥❇❋❬❳❲❥❊✼☛❆❉❁✿➅❘❦⑥⑤✽❜✐❆✌P◗✼✑P◗❦❣❷▼❦⑥❥❊❈➊⑤✌❋❙❆❲➎◗❦⑦P◗✼❍P ❫❛❾☎❥❙❸✌✼❵➏✿✸➐❏✽✸✌➑❤❆❉❈❊❥❬❳❲❝❃❏❼✼●❋❙➎◗❦⑦❜✐✼❉✸ ➒ ➓❛➔✇④①✈→✝➣↕↔➌③④✇➓→➔ ❡❸✌✼✰❧✧❳❲❦❣❁✢⑤✻❆❉❦❣❁◗❥✫❆❩⑧➙❥❊❸✌❦❣❈❻⑤✍❳❲⑤✻✼●❋☎❦⑥❈❻❥❊❆✢❈❙❸✌❆❲❱➛❥❙❸✻❳❲❥❻❝⑦❳❲❋❊❷❉✼✧❫✍❳❉❜❊▲▼⑩♠⑤✌❋❙❆▼⑤✍❳❲❷◗❳❲❥❙❦❣❆▼❁↕➜❶❄✜➑❴➝➞❁✌✼●❥♠⑩ ❱✜❆❉❋❙▲◗❈✩❜●❳❲❁✰❫✻✼✫❳❩⑤✌⑤✌❝❣❦❣✼❍P☎❥❙❆✖❋❊✼❍❳❲❝❤❦❣❧✲❳❲❷▼✼✐⑩♠❋❙✼❍❜✐❆▼❷❉❁✌❦❣❥❊❦⑥❆▼❁✖⑤✌❋❊❆❉❫✌❝❣✼●❧☎❈⑨❱☛❦❣❥❊❸✌❆❉❀✌❥✩❳❵❝⑦❳❲❋❙❷▼✼❉❂✍❜❹❆❉❧✖⑤✣❝⑥✼❹➟ ⑤✌❋❊✼●⑤✌❋❙❆✌❜✐✼❘❈❙❈❙❦❣❁✌❷②❈❙❥✐❳❲❷▼✼❻❋❙✼❍❽▼❀✌❦❣❋❙❦❣❁✌❷✧P◗✼●❥✐❳❲❦❣❝❣✼❍P✰✼❘❁✌❷❉❦❣❁✌✼●✼❘❋❙❦❣❁✌❷✍✸❻➏❖❁✌❝❣❦⑥▲▼✼✫❧☎❆❉❈❊❥✑⑤✌❋❊✼●➎◗❦⑥❆▼❀✌❈✩❱⑨❆▼❋❙▲✰❆▼❁ ❥❙❸✣✼✿❈❙❀✌❫❉➍❹✼❍❜✐❥❵➜❶❑❅✼●❁✌▲▼✼●❋✑✼❘❥✑❳❲❝♠✸❣❂✽➇❍➋❉➠▼➋❉➝❘❂◗❥❙❸✌✼✿❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷❻❁✣✼●❥♠❱⑨❆▼❋❙▲☎❦⑥❈✑P❼❦⑥❋❊✼❍❜✐❥❊❝⑥❾☎⑧➄✼✕P✖❱☛❦❣❥❊❸✲❦❣❧✧❳❲❷❉✼❘❈❍❂ ❋❬❳❩❥❙❸✌✼❘❋✑❥❙❸✻❳❲❁☎⑧➄✼❍❳❩❥❙❀✌❋❊✼✿➎❉✼❍❜❹❥❙❆❉❋❊❈❍❂✣❥❙❸◗❀✌❈✵P❼✼●❧✖❆▼❁✌❈❙❥❊❋❬❳❲❥❊❦❣❁✌❷✫❥❊❸✌✼✿❳❲❫✌❦❣❝❣❦❣❥♠❾❻❆❲⑧❴❄⑨➑t❁✣✼●❥♠❱⑨❆▼❋❙▲◗❈☛❥❙❆✖P❼✼❍❳❲❝ ❱☛❦❣❥❙❸✧❝⑦❳❲❋❊❷❉✼✵❳❩❧✖❆▼❀✌❁◗❥❙❈✜❆❲⑧✮❝❣❆❲❱↕❝❣✼●➎▼✼●❝✽❦❣❁◗⑧➄❆▼❋❙❧✲❳❩❥❙❦❣❆❉❁❤✸ ➑➊❋❊✼●➎◗❦⑥❆▼❀✌❈✿❱✜❆❉❋❊▲✰⑤✍✼❘❋♠⑧➄❆▼❋❙❧✖✼✕P✰❆▼❁④❈❊❦⑥❧☎⑤✌❝❣✼✫P◗❦❣❷❉❦❣❥❅❦⑥❧✧❳❲❷▼✼●❈✿➜♠✺✽✼❻✾✜❀✌❁❃❂✜➇❍➋❉➠▼➋❉➝❖❈❙❸✌❆❲❱✜✼❍P②❥❙❸✻❳❲❥✩❥❙❸✌✼ ❳❲❋✐❜❊❸✌❦❣❥❙✼❍❜✐❥❊❀✌❋❙✼✱❆❲⑧✮❥❊❸✌✼✿❁✌✼❘❥♠❱⑨❆▼❋❙▲❻❈❙❥❊❋❙❆▼❁✌❷❉❝❣❾✖❦❣❁◗➡✻❀✌✼●❁✍❜✐✼❘❈✑❥❙❸✣✼✵❁✣✼●❥♠❱⑨❆▼❋❙▲✽➢ ❈☛❷❉✼●❁✣✼●❋❬❳❩❝⑥❦❣➅❍❳❩❥❙❦❣❆❉❁✧❳❲❫✌❦❣❝❣❦❣❥♠❾❛✸ ➤❆◗❆✌P❵❷❉✼●❁✣✼●❋❬❳❩❝⑥❦❣➅❍❳❩❥❙❦❣❆❉❁✧❜●❳❲❁☎❆❉❁✌❝❣❾❵❫✍✼❅❆❉❫✌❥✐❳❲❦❣❁✌✼❍P➞❫◗❾✹P◗✼❘❈❙❦❣❷❉❁✌❦❣❁✌❷❻❳✩❁✌✼❘❥♠❱⑨❆▼❋❙▲✖❳❲❋✐❜❊❸✌❦❣❥❙✼❍❜✐❥❊❀✌❋❙✼✱❥❙❸✍❳❩❥ ❜✐❆▼❁❛❥✐❳❲❦❣❁✌❈☛❳✿❜✐✼●❋❊❥❬❳❩❦⑥❁✧❳❲❧☎❆❉❀✌❁◗❥➊❆❲⑧❨➥❅➦✻➧●➨➄➩❲➧❘➨✖▲◗❁✌❆❲❱☛❝❣✼❍P◗❷▼✼✑❳❲❫✻❆❉❀✌❥✜❥❙❸✣✼✑⑤✌❋❊❆❉❫✌❝❣✼●❧②✸ ❡❸✌✼❅❫✍❳❲❈❊❦⑦❜✑P◗✼✐⑩ ❈❙❦❣❷▼❁✿⑤✌❋❙❦❣❁✍❜❹❦⑥⑤✣❝⑥✼✜❦❣❈❃❥❊❆✑❧✖❦❣❁✌❦❣❧☎❦➫➅●✼➊❥❙❸✣✼⑨❁◗❀✌❧❻❫✻✼●❋❃❆❩⑧✌⑧➄❋❊✼●✼⑨⑤✻❳❲❋❬❳❩❧✖✼●❥❊✼●❋❙❈➊❥❊❸✍❳❲❥❤❧❻❀✌❈❙❥❤❫✍✼☛P◗✼●❥❊✼●❋❙❧☎❦⑥❁✣✼❍P ❫◗❾❻❥❙❸✌✼✩❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷❻❳❲❝❣❷❉❆▼❋❙❦❣❥❙❸✣❧④❂✕❱☛❦❣❥❙❸✌❆▼❀✌❥⑨❆❲➎▼✼●❋❙❝❣❾❵❋❙✼❍P❼❀✍❜✐❦❣❁✌❷❻❥❙❸✣✼➙❜❹❆❉❧✖⑤✣❀✌❥❬❳❩❥❙❦❣❆❉❁✍❳❩❝✌⑤✻❆❲❱⑨✼●❋☛❆❩⑧❃❥❙❸✌✼ ❁✌✼●❥♠❱✜❆❉❋❊▲✽✸ ❡❸✌❦❣❈✑⑤✌❋❊❦❣❁✍❜✐❦❣⑤✌❝❣✼✿❦❣❁✍❜✐❋❊✼❍❳❲❈❊✼●❈✿❥❙❸✣✼✿⑤✌❋❙❆▼❫✍❳❲❫✣❦⑥❝❣❦❣❥♠❾✖❆❩⑧⑨❜✐❆❉❋❊❋❙✼❍❜✐❥✱❷❉✼●❁✌✼❘❋❬❳❲❝❣❦❣➅❍❳❩❥❙❦❣❆❉❁✧❫✻✼❍❜●❳❲❀✣❈❙✼
ovof 76 6419-2o87250八 qoa0ǒ L21白 44151 53 Figure 1:Examples of original zipcodes from the testing set. it results in a specialized network architecture that has a reduced entropy(Denker et al.,1987;Patarnello and Carnevali,1987;Tishby,Levin and Solla,1989;Le Cun, 1989).On the other hand,some effort must be devoted to designing appropriate constraints into the architecture. 2 ZIPCODE RECOGNITION The handwritten digit-recognition application was chosen because it is a relatively simple machine vision task:the input consists of black or white pixels,the digits are usually well-separated from the background,and there are only ten output categories.Yet the problem deals with objects in a real two-dimensional space and the mapping from image space to category space has both considerable regularity and considerable complexity.The problem has added attraction because it is of great practical value. The database used to train and test the network is a superset of the one used in the work reported last year (Denker et al.,1989).We emphasize that the method of solution reported here relies more heavily on automatic learning,and much less on hand-designed preprocessing. The database consists of 9298 segmented numerals digitized from handwritten zip- codes that appeared on real U.S.Mail passing through the Buffalo,N.Y.post office. Examples of such images are shown in figure 1.The digits were written by many different people,using a great variety of sizes,writing styles and instruments,with widely varying levels of care.This was supplemented by a set of 3349 printed dig- its coming from 35 different fonts.The training set consisted of 7291 handwritten digits plus 2549 printed digits.The remaining 2007 handwritten and 700 printed digits were used as the test set.The printed fonts in the test set were different from the printed fonts in the training set.One important feature of this database,which
✂✁☎✄✝✆✟✞✡✠☞☛✍✌✑❯❤➟✍❳❩❧✖⑤✌❝❣✼●❈✜❆❲⑧➊❆❉❋❙❦❣❷▼❦⑥❁✻❳❲❝✻➅●❦❣⑤➃❜✐❆✌P◗✼❘❈☛⑧➄❋❊❆❉❧✠❥❊❸✌✼✿❥❙✼❘❈❙❥❙❦❣❁✌❷❻❈❊✼●❥❍✸ ❦❣❥⑨❋❊✼●❈❙❀✣❝⑥❥❊❈✩❦⑥❁✧❳❵❈❙⑤✻✼❍❜✐❦⑦❳❲❝❣❦❣➅●✼❍P☎❁✌✼❘❥♠❱⑨❆▼❋❙▲✲❳❩❋❬❜❊❸✌❦❣❥❙✼✕❜✐❥❙❀✌❋❊✼✫❥❊❸✍❳❲❥☛❸✻❳❲❈✑❳✱❋❙✼❍P◗❀✻❜✐✼❍P ✼●❁◗❥❙❋❙❆▼⑤◗❾✰➜❶❑❅✼●❁✣▲❉✼●❋ ✼●❥☛❳❲❝♠✸❣❂◗➇❍➋❉➠▼♣✏✎❉➑➊❳❩❥❬❳❲❋❊❁✌✼●❝❣❝❣❆✫❳❲❁✻P❻✾☛❳❲❋❙❁✣✼●➎❉❳❲❝❣❦♠❂◗➇❍➋❉➠▼♣✏✎ ❡❦❣❈❙❸◗❫◗❾❛❂❉✺✏✼●➎◗❦❣❁✹❳❩❁✍P❻❏◗❆❉❝❣❝⑦❳✌❂✣➇❍➋❉➠▼➋✏✎❉✺✏✼✑✾✜❀✌❁❃❂ ➇❍➋▼➠❉➋▼➝●✸✒✑❅❁④❥❊❸✌✼❻❆❉❥❊❸✌✼●❋✿❸✍❳❩❁✍P✽❂❤❈❙❆▼❧✖✼✿✼✔✓✽❆❉❋❊❥✿❧❻❀✌❈❊❥✵❫✻✼✖P◗✼●➎▼❆❉❥❊✼❍P②❥❙❆✰P◗✼●❈❊❦❣❷❉❁✌❦❣❁✌❷✰❳❲⑤✣⑤✌❋❙❆▼⑤✌❋❙❦⑦❳❲❥❊✼ ❜✐❆▼❁✌❈❙❥❊❋❬❳❲❦❣❁◗❥❙❈❅❦❣❁❛❥❊❆✫❥❊❸✌✼✿❳❲❋✐❜❙❸✣❦⑥❥❊✼❍❜✐❥❊❀✌❋❙✼❉✸ ✕ ✖✵➓✘✗ ③✢→✝➣✚✙ ①✛✙ ③✢→✛✜➔➓✇➓→➔ ❡❸✌✼✿❸✻❳❲❁✍P◗❱☛❋❊❦❣❥❙❥❙✼❘❁➆P❼❦⑥❷▼❦❣❥♠⑩♠❋❙✼❍❜✐❆▼❷❉❁✌❦❣❥❊❦⑥❆▼❁✲❳❩⑤✌⑤✌❝❣❦⑦❜●❳❲❥❊❦⑥❆▼❁❻❱❨❳❩❈✑❜❊❸✌❆❉❈❊✼●❁✰❫✻✼❍❜●❳❲❀✣❈❙✼❵❦⑥❥☛❦❣❈✑❳❵❋❊✼●❝⑦❳❲❥❙❦❣➎▼✼●❝❣❾ ❈❙❦❣❧☎⑤✌❝❣✼✵❧✧❳❉❜❊❸✌❦❣❁✌✼✿➎◗❦❣❈❙❦❣❆▼❁✰❥❬❳❩❈❙▲✣✢✩❥❙❸✌✼❵❦❣❁✌⑤✌❀✌❥✿❜✐❆▼❁✌❈❙❦❣❈❊❥❙❈✿❆❩⑧❨❫✌❝⑦❳▼❜❙▲✧❆▼❋✵❱☛❸✣❦⑥❥❊✼✫⑤✣❦➫➟✌✼●❝❣❈✕❂✽❥❙❸✌✼✖P❼❦⑥❷▼❦❣❥❙❈ ❳❲❋❊✼④❀✣❈❙❀✍❳❩❝⑥❝❣❾✈❱⑨✼●❝❣❝➂⑩♠❈❙✼●⑤✍❳❩❋❬❳❩❥❙✼❍P ⑧➄❋❙❆▼❧ ❥❊❸✌✼②❫✍❳❉❜❊▲◗❷❉❋❊❆❉❀✌❁✍P✏❂✑❳❲❁✍P ❥❙❸✌✼●❋❊✼✝❳❲❋❊✼④❆▼❁✌❝❣❾✝❥❙✼❘❁ ❆▼❀✌❥❙⑤✌❀✣❥ ❜●❳❩❥❙✼●❷▼❆❉❋❙❦❣✼●❈✕✸➊✷⑨✼❘❥⑨❥❊❸✌✼✩⑤✌❋❙❆▼❫✌❝❣✼●❧✶P◗✼❍❳❩❝❣❈☛❱☛❦⑥❥❊❸✖❆▼❫❉➍✐✼❍❜✐❥❊❈☛❦⑥❁✧❳✿❋❊✼❍❳❲❝✽❥♠❱✜❆❲⑩❙P◗❦❣❧✖✼❘❁✌❈❙❦❣❆❉❁✻❳❲❝✌❈❊⑤✍❳❉❜✐✼✿❳❩❁✍P ❥❙❸✣✼✫❧✧❳❲⑤✌⑤✌❦❣❁✌❷✱⑧➄❋❊❆❉❧ ❦❣❧✲❳❩❷❉✼✑❈❊⑤✍❳❉❜❹✼❻❥❙❆✲❜❘❳❲❥❙✼❘❷❉❆❉❋❊❾✰❈❙⑤✻❳❉❜✐✼❵❸✍❳❲❈✩❫✻❆❉❥❊❸④❜✐❆❉❁✣❈❙❦⑦P◗✼●❋✐❳❲❫✌❝❣✼❵❋❙✼●❷▼❀✌❝⑦❳❲❋❙❦❣❥♠❾ ❳❲❁✻P✝❜✐❆▼❁✌❈❙❦⑦P◗✼●❋✐❳❲❫✌❝❣✼✧❜✐❆❉❧☎⑤✌❝❣✼✐➟✌❦⑥❥♠❾◗✸ ❡❸✌✼✧⑤✌❋❊❆❉❫✌❝❣✼●❧✦❸✍❳❩❈✹❳▼P✌P◗✼❍Pt❳❲❥❊❥❙❋✐❳❉❜✐❥❊❦⑥❆▼❁✝❫✻✼❍❜●❳❩❀✌❈❙✼✧❦❣❥✫❦❣❈❵❆❲⑧ ❷❉❋❊✼❍❳❲❥☛⑤✣❋❬❳❉❜❹❥❙❦⑦❜●❳❲❝❤➎❉❳❲❝❣❀✌✼❉✸ ❡❸✌✼✖P✌❳❩❥❬❳❩❫✍❳❲❈❊✼✫❀✌❈❊✼❍P✢❥❊❆✧❥❙❋❬❳❩❦❣❁❿❳❩❁✍P✰❥❊✼●❈❙❥❵❥❙❸✣✼✫❁✌✼❘❥♠❱⑨❆▼❋❙▲②❦⑥❈✿❳☎❈❙❀✣⑤✍✼●❋❊❈❙✼●❥❻❆❩⑧❨❥❊❸✌✼❻❆❉❁✣✼✫❀✌❈❊✼❍P②❦❣❁ ❥❙❸✣✼✫❱✜❆❉❋❊▲✖❋❙✼❘⑤✍❆▼❋❙❥❙✼✕P②❝ ❳❩❈❙❥❅❾❉✼❍❳❩❋✿➜❶❑❅✼●❁✣▲❉✼●❋❅✼●❥✿❳❲❝♠✸❣❂✽➇❍➋❉➠▼➋❉➝❘✸☛❭✢✼✿✼●❧☎⑤✌❸✍❳❲❈❊❦❣➅●✼✩❥❙❸✍❳❩❥✑❥❙❸✣✼✫❧☎✼●❥❙❸✌❆✌P ❆❲⑧❴❈❊❆❉❝❣❀✌❥❙❦❣❆▼❁✲❋❊✼●⑤✻❆❉❋❙❥❊✼❍P ❸✌✼●❋❙✼✱❋❙✼●❝❣❦❣✼●❈✑❧☎❆❉❋❊✼✵❸✣✼❍❳✕➎◗❦⑥❝❣❾❻❆▼❁➆❳❩❀✌❥❙❆▼❧✲❳❩❥❙❦⑦❜☛❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷✻❂✍❳❲❁✻P✖❧❻❀✻❜❙❸☎❝❣✼●❈❙❈ ❆❉❁✧❸✍❳❩❁✍P▼⑩❙P◗✼●❈❊❦⑥❷▼❁✌✼❍P☎⑤✌❋❊✼●⑤✌❋❙❆✌❜✐✼❘❈❙❈❙❦❣❁✌❷✻✸ ❡❸✌✼✿P✌❳❩❥❬❳❲❫✻❳❲❈❙✼✿❜✐❆▼❁✌❈❙❦❣❈❊❥❙❈✑❆❩⑧✮➋✥✤▼➋❉➠✱❈❙✼●❷▼❧✖✼●❁◗❥❙✼✕P✖❁◗❀✌❧☎✼●❋❬❳❩❝❣❈⑨P◗❦❣❷❉❦❣❥❙❦❣➅●✼✕P❵⑧➄❋❊❆❉❧✠❸✍❳❩❁✍P◗❱☛❋❊❦⑥❥❊❥❙✼●❁✧➅●❦❣⑤◗⑩ ❜✐❆✌P◗✼●❈✜❥❙❸✻❳❲❥❴❳❲⑤✌⑤✻✼❍❳❩❋❙✼❍P➞❆❉❁❵❋❙✼❍❳❩❝➃➏✱✸ ❏✏✸❉❺✰❳❲❦❣❝◗⑤✍❳❩❈❙❈❙❦❣❁✌❷✩❥❊❸✌❋❙❆▼❀✌❷❉❸❻❥❊❸✌✼✑❄❴❀✦✓✏❳❩❝⑥❆✻❂❉♥✿✸✷❻✸❲⑤✍❆▼❈❙❥❴❆★✧✲❜✐✼❉✸ ❯❤➟✍❳❲❧☎⑤✌❝❣✼●❈☛❆❲⑧✜❈❙❀✍❜❊❸✢❦❣❧✲❳❩❷❉✼●❈✑❳❩❋❙✼❵❈❙❸✌❆❲❱☛❁②❦❣❁✧➁✍❷❉❀✣❋❙✼✧➇❉✸ ❡❸✌✼❻P◗❦❣❷❉❦❣❥❙❈❅❱⑨✼❘❋❙✼❵❱☛❋❙❦❣❥❙❥❊✼●❁④❫◗❾✰❧✧❳❲❁◗❾ P◗❦✩✓✽✼●❋❙✼●❁◗❥❅⑤✍✼●❆▼⑤✌❝❣✼❉❂◗❀✌❈❊❦⑥❁✣❷✹❳❵❷▼❋❙✼❍❳❩❥☛➎❉❳❲❋❙❦❣✼●❥♠❾❻❆❩⑧✮❈❊❦⑥➅❘✼●❈❍❂✣❱☛❋❙❦❣❥❙❦❣❁✌❷❵❈❙❥♠❾◗❝❣✼●❈✑❳❲❁✻P❻❦❣❁✌❈❙❥❊❋❙❀✌❧☎✼●❁◗❥❙❈✕❂◗❱☛❦⑥❥❊❸ ❱☛❦⑦P◗✼●❝❣❾✖➎❉❳❩❋❙❾◗❦❣❁✌❷❵❝⑥✼❘➎❉✼●❝❣❈✑❆❩⑧✚❜●❳❩❋❙✼❉✸ ❡❸✌❦❣❈☛❱❨❳❩❈✑❈❙❀✣⑤✌⑤✌❝❣✼●❧✖✼❘❁❛❥❊✼❍P☎❫◗❾✲❳❵❈❙✼❘❥✵❆❩⑧⑨q❉q✡✪◗➋✿⑤✣❋❙❦❣❁◗❥❙✼❍P✰P❼❦⑥❷❩⑩ ❦❣❥❙❈✩❜✐❆❉❧☎❦⑥❁✣❷✿⑧➄❋❊❆❉❧ q✥✫❻P◗❦✩✓✽✼●❋❙✼❘❁❛❥❅⑧➄❆▼❁◗❥❙❈❍✸ ❡❸✌✼✿❥❙❋✐❳❲❦❣❁✌❦❣❁✌❷❻❈❊✼●❥✵❜✐❆▼❁✌❈❙❦❣❈❊❥❙✼❍P ❆❲⑧❨♣✬✤❉➋✣➇✑❸✍❳❩❁✍P◗❱☛❋❊❦⑥❥❊❥❙✼●❁ P◗❦❣❷❉❦❣❥❊❈✵⑤✣❝⑥❀✣❈✭✤✬✫✡✪◗➋❻⑤✌❋❊❦⑥❁◗❥❊✼❍P④P❼❦⑥❷▼❦❣❥❙❈❍✸ ❡❸✌✼❵❋❙✼❘❧✲❳❲❦❣❁✌❦❣❁✌❷✮✤▼♦❉♦▼♣✖❸✻❳❲❁✍P◗❱☛❋❊❦❣❥❙❥❙✼❘❁❿❳❩❁✍P④♣▼♦❉♦☎⑤✌❋❊❦⑥❁◗❥❊✼❍P P◗❦❣❷❉❦❣❥❊❈✮❱✜✼●❋❊✼✑❀✌❈❊✼❍P✖❳❩❈❴❥❙❸✌✼❅❥❙✼●❈❊❥⑨❈❊✼●❥❍✸ ❡❸✣✼✑⑤✌❋❊❦⑥❁◗❥❊✼❍P✱⑧➄❆❉❁◗❥❊❈❴❦❣❁✫❥❊❸✌✼✑❥❊✼●❈❙❥✜❈❙✼●❥✜❱✜✼●❋❙✼✩P◗❦✩✓➃✼●❋❊✼●❁◗❥❴⑧❋❙❆❉❧ ❥❙❸✣✼✿⑤✌❋❙❦❣❁◗❥❙✼✕P❻⑧❆❉❁◗❥❙❈☛❦❣❁✖❥❊❸✌✼✿❥❊❋❬❳❲❦❣❁✌❦❣❁✌❷❵❈❙✼❘❥❍✸✯✑❅❁✌✼✫❦❣❧☎⑤✍❆▼❋❙❥❬❳❩❁◗❥➊⑧➄✼❍❳❩❥❙❀✌❋❊✼✿❆❲⑧➊❥❙❸✌❦❣❈✑P✌❳❩❥❬❳❲❫✻❳❲❈❙✼▼❂✌❱☛❸✌❦⑦❜❊❸
14t0119t34857680322布*1平1 86L3≤97202992997215100467 0」3a色441S91010615401056 3k△641110304752t20099799 66B41天自眉女7855辛1314379554 60t【牙2501母711又99寸0899707 840非09707597331972015519ò 56【07ss1?S5(gz8143580909 43↓787￥1名55"(@554t0354(0 55】82551065030←75R0439+01 Figure 2:Examples of normalized digits from the testing set. is a common feature to all real-world databases,is that both the training set and the testing set contain numerous examples that are ambiguous,unclassifiable,or even misclassified. 3 PREPROCESSING Acquisition,binarization,location of the zipcode,and preliminary segmentation were performed by Postal Service contractors (Wang and Srihari,1988).Some of these steps constitute very hard tasks in themselves.The segmentation(separating each digit from its neighbors)would be a relatively simple task if we could assume that a character is contiguous and is disconnected from its neighbors,but neither of these assumptions holds in practice.Many ambiguous characters in the database are the result of mis-segmentation (especially broken 5's)as can be seen on figure 2. At this point,the size of a digit varies but is typically around 40 by 60 pixels.Since the input of a back-propagation network is fixed size,it is necessary to normalize the size of the characters.This was performed using a linear transformation to make the characters fit in a 16 by 16 pixel image.This transformation preserves the aspect ratio of the character,and is performed after extraneous marks in the image have been removed.Because of the linear transformation,the resulting image is not binary but has multiple gray levels,since a variable number of pixels in the original image can fall into a given pixel in the target image.The gray levels of each image are scaled and translated to fall within the range-1 to 1. 4 THE NETWORK The remainder of the recognition is entirely performed by a multi-layer network.All of the connections in the network are adaptive,although heavily constrained,and are trained using back-propagation.This is in contrast with earlier work (Denker et al.,1989)where the first few layers of connections were hand-chosen constants The input of the network is a 16 by 16 normalized image and the output is composed
✂✁✄✝✆✞✡✠ ✁ ✌❨❯❤➟✍❳❩❧✖⑤✌❝❣✼●❈✜❆❲⑧➊❁✌❆❉❋❊❧✲❳❲❝❣❦❣➅●✼❍P☎P◗❦❣❷❉❦❣❥❙❈❇⑧➄❋❊❆❉❧✠❥❊❸✌✼✿❥❙✼❘❈❙❥❙❦❣❁✌❷❻❈❊✼●❥❍✸ ❦❣❈✵❳✧❜✐❆▼❧✖❧✖❆▼❁❵⑧➄✼❍❳❩❥❙❀✌❋❊✼❻❥❙❆✲❳❩❝❣❝❃❋❙✼✕❳❲❝➂⑩❶❱✜❆❉❋❊❝⑦P✰P✌❳❲❥✐❳❲❫✍❳❩❈❙✼●❈✕❂✽❦❣❈✩❥❙❸✍❳❩❥✑❫✻❆❉❥❙❸②❥❙❸✣✼✫❥❊❋❬❳❲❦❣❁✌❦❣❁✌❷☎❈❙✼●❥✿❳❩❁✍P ❥❙❸✣✼✖❥❙✼❘❈❙❥❙❦❣❁✌❷②❈❙✼❘❥✹❜✐❆▼❁◗❥❬❳❲❦❣❁✢❁◗❀✌❧☎✼●❋❙❆▼❀✌❈✩✼✐➟✍❳❲❧✖⑤✣❝⑥✼❘❈✿❥❙❸✍❳❩❥✫❳❲❋❊✼✲❳❩❧❻❫✌❦❣❷❉❀✌❆▼❀✌❈❍❂✻❀✌❁✍❜❹❝ ❳❩❈❙❈❊❦➫➁✽❳❲❫✌❝❣✼❉❂➊❆▼❋ ✼●➎▼✼●❁✰❧✖❦❣❈✐❜✐❝⑦❳❲❈❙❈❊❦➂➁✍✼❍P✽✸ ✂ ✗①✛✙✗①✈→ ③✙ ✉❅✉ ➓❛➔✜ ❞❜❘❽❉❀✣❦⑥❈❊❦❣❥❙❦❣❆❉❁❃❂❇❫✌❦❣❁✍❳❲❋❊❦❣➅❍❳❲❥❊❦⑥❆▼❁❃❂❴❝❣❆✌❜●❳❩❥❙❦❣❆❉❁t❆❲⑧✩❥❙❸✌✼ ➅●❦❣⑤➃❜✐❆✌P◗✼▼❂✑❳❲❁✍P✈⑤✌❋❊✼●❝❣❦❣❧✖❦❣❁✍❳❲❋❊❾④❈❊✼●❷❉❧☎✼●❁◗❥❬❳❩❥❙❦❣❆❉❁ ❱✜✼●❋❙✼❻⑤✻✼●❋♠⑧❆❉❋❙❧☎✼❍P ❫❛❾✰➑❤❆❉❈❊❥❬❳❩❝⑨❏◗✼●❋❊➎❛❦⑦❜✐✼✖❜❹❆❉❁◗❥❙❋✐❳❉❜✐❥❊❆❉❋❙❈➞➜➄❭✝❳❲❁✣❷✲❳❲❁✻P④❏◗❋❊❦❣❸✍❳❲❋❊❦❶❂➊➇✕➋❉➠❉➠▼➝●✸❵❏◗❆❉❧☎✼✿❆❲⑧ ❥❙❸✣✼●❈❙✼✩❈❙❥❊✼●⑤✌❈✑❜✐❆▼❁✌❈❙❥❊❦❣❥❙❀✌❥❊✼✵➎▼✼●❋❙❾❻❸✻❳❲❋❬P❵❥✐❳❲❈❊▲❛❈✜❦❣❁❻❥❙❸✌✼❘❧✖❈❙✼❘❝⑥➎▼✼●❈❍✸ ❡❸✣✼✑❈❙✼❘❷❉❧✖✼❘❁❛❥✐❳❲❥❊❦⑥❆▼❁✧➜➄❈❊✼●⑤✍❳❲❋✐❳❲❥❊❦⑥❁✣❷ ✼❍❳▼❜❙❸✰P◗❦❣❷▼❦⑥❥❇⑧➄❋❊❆❉❧✠❦❣❥❙❈☛❁✣✼●❦❣❷❉❸◗❫✍❆▼❋❙❈✐➝✜❱⑨❆▼❀✌❝⑦P❻❫✻✼✫❳✿❋❊✼●❝⑦❳❲❥❊❦⑥➎▼✼●❝❣❾❻❈❙❦❣❧✖⑤✣❝⑥✼❅❥❬❳❩❈❙▲✖❦➂⑧➊❱⑨✼✿❜✐❆▼❀✌❝⑦P✖❳❲❈❊❈❙❀✌❧☎✼ ❥❙❸✻❳❲❥✿❳✖❜❊❸✍❳❩❋❬❳❉❜❹❥❙✼●❋✿❦❣❈✩❜✐❆❉❁◗❥❙❦❣❷▼❀✌❆❉❀✌❈✿❳❩❁✍P✧❦❣❈✿P◗❦❣❈❬❜✐❆▼❁✌❁✌✼❍❜✐❥❊✼❍P ⑧➄❋❊❆❉❧ ❦❣❥❙❈❅❁✌✼●❦❣❷❉❸◗❫✻❆❉❋❙❈✕❂✽❫✌❀✌❥✩❁✌✼❘❦⑥❥❊❸✌✼●❋ ❆❲⑧✽❥❙❸✣✼●❈❙✼✿❳❲❈❊❈❙❀✌❧☎⑤✌❥❙❦❣❆▼❁✌❈➊❸✌❆❉❝⑦P◗❈❴❦❣❁❵⑤✌❋❬❳▼❜✐❥❙❦⑦❜✐✼▼✸❴❺✰❳❲❁◗❾❻❳❲❧❻❫✌❦❣❷❉❀✣❆❉❀✌❈➊❜❙❸✻❳❲❋❬❳▼❜✐❥❙✼❘❋❙❈☛❦❣❁✫❥❊❸✌✼✵P✣❳❲❥❬❳❩❫✍❳❲❈❊✼ ❳❲❋❊✼☛❥❙❸✌✼❅❋❙✼●❈❊❀✌❝❣❥❴❆❲⑧✽❧✖❦❣❈⑩❶❈❊✼●❷❉❧☎✼●❁◗❥❬❳❲❥❊❦❣❆❉❁❵➜➄✼●❈❊⑤✍✼✕❜✐❦⑦❳❲❝❣❝❣❾✫❫✌❋❊❆❉▲▼✼●❁✮✫✌➢ ❈❬➝❇❳❲❈⑨❜●❳❩❁❻❫✍✼☛❈❊✼●✼●❁☎❆❉❁✱➁✍❷❉❀✌❋❊✼ ✤✣✸ ❞❥❴❥❊❸✌❦❣❈❴⑤✍❆▼❦❣❁❛❥✕❂❉❥❊❸✌✼☛❈❙❦❣➅●✼❅❆❲⑧❃❳✿P◗❦❣❷❉❦❣❥➊➎❉❳❲❋❊❦❣✼●❈❴❫✌❀✌❥❇❦⑥❈❇❥♠❾❛⑤✣❦ ❜❘❳❲❝❣❝❣❾✫❳❲❋❊❆❉❀✌❁✍P ✪◗♦❅❫❛❾☎✄❉♦❅⑤✌❦➂➟✌✼●❝❣❈❍✸➊❏◗❦❣❁✍❜✐✼ ❥❙❸✣✼✫❦❣❁✌⑤✌❀✣❥✵❆❩⑧❨❳☎❫✍❳❉❜❊▲▼⑩♠⑤✌❋❙❆▼⑤✍❳❲❷◗❳❲❥❙❦❣❆▼❁✲❁✣✼●❥♠❱⑨❆▼❋❙▲✰❦❣❈❅➁✌➟✌✼❍P ❈❙❦❣➅●✼❉❂✏❦⑥❥✩❦❣❈✑❁✣✼❍❜✐✼●❈❊❈❬❳❲❋❊❾✢❥❙❆✧❁✌❆▼❋❙❧✲❳❩❝❣❦⑥➅❘✼ ❥❙❸✣✼✲❈❊❦❣➅●✼✰❆❲⑧✑❥❊❸✌✼④❜❊❸✍❳❩❋❬❳❉❜❹❥❙✼●❋❊❈❍✸ ❡❸✣❦⑥❈❵❱☛❳❲❈❵⑤✍✼●❋⑧➄❆❉❋❊❧✖✼❍P✈❀✌❈❊❦❣❁✌❷❿❳ ❝❣❦❣❁✌✼❍❳❲❋❵❥❙❋✐❳❲❁✌❈⑧➄❆❉❋❊❧✲❳❩❥❙❦❣❆❉❁✢❥❊❆ ❧✲❳❩▲❉✼✿❥❊❸✌✼✖❜❊❸✍❳❲❋✐❳❉❜✐❥❊✼●❋❙❈✱➁✍❥✿❦❣❁✢❳②➇✆✄☎❫◗❾ ➇✝✄✖⑤✣❦➫➟✌✼●❝❇❦⑥❧✧❳❲❷▼✼❉✸ ❡❸✌❦❣❈✩❥❙❋✐❳❲❁✌❈⑧➄❆❉❋❊❧✲❳❩❥❙❦❣❆❉❁✰⑤✣❋❙✼●❈❊✼●❋❙➎▼✼●❈ ❥❙❸✣✼✖❳❲❈❙⑤✻✼❍❜✐❥✱❋❬❳❲❥❊❦❣❆✲❆❩⑧❨❥❊❸✌✼✖❜❊❸✍❳❲❋✐❳❉❜✐❥❊✼●❋❍❂➊❳❲❁✻P✰❦❣❈✿⑤✻✼●❋♠⑧❆❉❋❙❧☎✼❍P②❳✕⑧➄❥❊✼●❋✩✼✐➟✌❥❙❋❬❳❩❁✌✼●❆▼❀✌❈✿❧✲❳❩❋❙▲◗❈✩❦❣❁✰❥❙❸✌✼ ❦❣❧✲❳❩❷❉✼➊❸✍❳✕➎❉✼✜❫✻✼●✼●❁❻❋❙✼❘❧✖❆❲➎❉✼✕P✽✸❃❄❇✼❍❜●❳❲❀✣❈❙✼☛❆❲⑧✽❥❙❸✣✼⑨❝❣❦❣❁✌✼❍❳❩❋❃❥❙❋✐❳❲❁✌❈⑧➄❆❉❋❊❧✲❳❩❥❙❦❣❆❉❁❃❂✕❥❊❸✌✼⑨❋❊✼●❈❙❀✣❝⑥❥❊❦❣❁✌❷✩❦⑥❧✧❳❲❷▼✼ ❦❣❈✑❁✌❆▼❥☛❫✌❦❣❁✍❳❲❋❊❾✧❫✌❀✌❥✑❸✻❳❲❈☛❧❻❀✌❝❣❥❙❦❣⑤✌❝❣✼❅❷❉❋❬❳✕❾☎❝❣✼●➎❉✼●❝❣❈✕❂✍❈❊❦❣❁✍❜✐✼✫❳➞➎❉❳❲❋❙❦⑦❳❩❫✌❝❣✼✑❁◗❀✌❧❻❫✻✼●❋☛❆❲⑧❴⑤✌❦➂➟✌✼●❝❣❈✑❦❣❁✧❥❙❸✌✼ ❆❉❋❊❦❣❷❉❦❣❁✍❳❲❝➊❦❣❧✲❳❩❷❉✼✫❜❘❳❲❁✢⑧♠❳❲❝❣❝➊❦⑥❁◗❥❊❆④❳✧❷❉❦❣➎▼✼●❁④⑤✣❦➫➟✌✼●❝✜❦❣❁✢❥❙❸✌✼☎❥❬❳❩❋❙❷❉✼❘❥✿❦❣❧✲❳❲❷▼✼❉✸ ❡❸✌✼☎❷❉❋✐❳✕❾✰❝❣✼●➎❉✼●❝❣❈❵❆❲⑧ ✼❍❳▼❜❙❸✧❦❣❧✲❳❩❷❉✼✑❳❩❋❙✼✿❈✐❜●❳❲❝❣✼❍P✧❳❲❁✻P✖❥❊❋❬❳❩❁✌❈❙❝⑦❳❲❥❊✼❍P☎❥❙❆❵⑧❶❳❩❝❣❝✍❱☛❦❣❥❙❸✣❦⑥❁☎❥❙❸✌✼✱❋❬❳❲❁✣❷❉✼✟✞✿➇✩❥❙❆✰➇❉✸ ✠ ✇☛✡✙ ➔✙✧✇✌☞→✝①✎✍ ❡❸✌✼✜❋❙✼●❧✧❳❲❦❣❁✍P◗✼❘❋❃❆❲⑧✻❥❙❸✌✼☛❋❊✼❍❜✐❆❉❷▼❁✌❦❣❥❙❦❣❆❉❁✱❦⑥❈➊✼●❁◗❥❊❦⑥❋❊✼●❝❣❾✿⑤✻✼●❋♠⑧➄❆▼❋❙❧☎✼❍P✩❫◗❾✿❳✑❧❻❀✣❝⑥❥❊❦➂⑩❶❝⑦❳❘❾❉✼●❋✽❁✌✼❘❥♠❱⑨❆▼❋❙▲✽✸ ❞❝❣❝ ❆❲⑧✜❥❙❸✌✼✫❜❹❆❉❁✌❁✌✼✕❜✐❥❙❦❣❆❉❁✣❈✿❦❣❁✲❥❊❸✌✼❵❁✌✼●❥♠❱✜❆❉❋❙▲②❳❲❋❙✼❵❳❉P✌❳❩⑤✌❥❙❦❣➎❉✼▼❂➃❳❩❝❣❥❙❸✌❆▼❀✌❷❉❸✧❸✌✼❍❳✕➎◗❦❣❝❣❾✲❜✐❆▼❁✌❈❙❥❊❋❬❳❩❦⑥❁✣✼❍P✽❂❃❳❩❁✍P ❳❲❋❊✼✫❥❊❋❬❳❩❦⑥❁✣✼❍P✰❀✣❈❙❦❣❁✌❷✖❫✻❳❉❜❊▲▼⑩❶⑤✌❋❊❆❉⑤✍❳❩❷❛❳❩❥❙❦❣❆❉❁❃✸ ❡❸✌❦❣❈✑❦❣❈✩❦❣❁④❜✐❆▼❁❛❥❊❋❬❳❩❈❙❥✩❱☛❦❣❥❙❸✰✼❍❳❩❋❙❝❣❦❣✼●❋✩❱⑨❆▼❋❙▲②➜❶❑❅✼●❁✣▲❉✼●❋ ✼●❥✿❳❲❝♠✸❣❂✽➇❍➋▼➠❉➋▼➝☛❱☛❸✌✼●❋❊✼✫❥❊❸✌✼✱➁✍❋❙❈❊❥✑⑧✼●❱➉❝⑦❳✕❾❉✼●❋❊❈✑❆❲⑧⑨❜✐❆▼❁✌❁✌✼❍❜❹❥❙❦❣❆❉❁✌❈✱❱⑨✼●❋❊✼✫❸✻❳❲❁✍P▼⑩❙❜❊❸✌❆❉❈❊✼●❁✢❜✐❆❉❁✌❈❊❥❬❳❩❁❛❥❊❈❍✸ ❡❸✌✼✜❦❣❁✌⑤✌❀✌❥❤❆❲⑧✍❥❊❸✌✼⑨❁✣✼●❥♠❱⑨❆▼❋❙▲✩❦❣❈✮❳❵➇✆✄✜❫◗❾❻➇✆✄❖❁✌❆❉❋❊❧✲❳❩❝⑥❦❣➅●✼✕P✑❦❣❧✧❳❲❷❉✼➊❳❲❁✍P✱❥❙❸✣✼⑨❆▼❀✌❥❙⑤✌❀✣❥❃❦❣❈✮❜✐❆▼❧✖⑤✻❆❉❈❊✼❍P
of 10 units:one per class.When a pattern belonging to class Fis presented,the desired output is +1 for the Eh output unit,and-1 for the other output units. Figure 3:Input image (left),weight vector (center),and resulting feature map (right).The feature map is obtained by scanning the input image with a single neuron that has a local receptive field,as indicated.White represents -1,black represents +1. A fully connected network with enough discriminative power for the task would have far too many parameters to be able to generalize correctly.Therefore a restricted connection-scheme must be devised,guided by our prior knowledge about shape recognition.There are well-known advantages to performing shape recognition by detecting and combining local features.We have required our network to do this by constraining the connections in the first few layers to be local.In addition,if a feature detector is useful on one part of the image,it is likely to be useful on other parts of the image as well.One reason for this is that the salient features of a distorted character might be displaced slightly from their position in a typical char- acter.One solution to this problem is to scan the input image with a single neuron that has a local receptive field,and store the states of this neuron in corresponding locations in a layer called a eea:ure map (see figure 3).This operation is equivalent to a convolution with a small size kernel,followed by a squashing function.The process can be performed in parallel by implementing the feature map as a plane of neurons whose weight vectors are constrained to be equal.That is,units in a feature map are constrained to perform the same operation on different parts of the image.An interesting side-effect of this weige EPrig technique,already described in (Rumelhart,Hinton and Williams,1986),is to reduce the number of free param- eters by a large amount,since a large number of units share the same weights.In addition,a certain level of shift invariance is present in the system:shifting the input will shift the result on the feature map,but will leave it unchanged otherwise. In practice,it will be necessary to have multiple feature maps,extracting different features from the same image
❆❲⑧✿➇❍♦☎❀✌❁✣❦⑥❥❊❈ ✢✱❆❉❁✌✼❻⑤✻✼●❋✫❜✐❝⑦❳❩❈❙❈❍✸☎❭↕❸✌✼❘❁✝❳✖⑤✻❳❲❥❙❥❊✼●❋❙❁✈❫✍✼❘❝⑥❆▼❁✌❷❉❦❣❁✌❷☎❥❙❆④❜❹❝ ❳❩❈❙❈✁❖❦❣❈✵⑤✣❋❙✼●❈❊✼●❁◗❥❙✼❍P✽❂❇❥❙❸✌✼ P◗✼●❈❊❦❣❋❙✼❍P✧❆▼❀✌❥❙⑤✣❀✌❥☛❦❣❈✄✂✫➇❅⑧❆❉❋✜❥❙❸✌✼✁ ❥❙❸✧❆▼❀✌❥❙⑤✌❀✣❥☛❀✌❁✌❦❣❥❍❂✌❳❩❁✍P✌✞➙➇❅⑧❆❉❋✜❥❙❸✌✼✿❆▼❥❙❸✌✼❘❋☛❆❉❀✌❥❊⑤✌❀✌❥☛❀✌❁✌❦❣❥❊❈❍✸ ✂✁☎✄✍✆✞✡✠✆☎ ✌✞✝♠❁✌⑤✌❀✌❥❵❦❣❧✲❳❲❷▼✼✖➜ ❝❣✼✐⑧➄❥❬➝❘❂➊❱⑨✼❘❦⑥❷▼❸◗❥✫➎▼✼❍❜✐❥❙❆▼❋✖➜♠❜✐✼●❁◗❥❙✼❘❋❬➝●❂☛❳❩❁✍P✢❋❊✼●❈❙❀✌❝❣❥❊❦⑥❁✣❷✰⑧➄✼✕❳❲❥❙❀✣❋❙✼✧❧✲❳❩⑤ ➜➄❋❊❦❣❷❉❸◗❥❬➝❘✸ ❡❸✣✼❻⑧➄✼❍❳❩❥❙❀✌❋❊✼✧❧✲❳❲⑤②❦❣❈✫❆▼❫✌❥❬❳❩❦❣❁✌✼❍P②❫◗❾✢❈❬❜●❳❩❁✌❁✌❦❣❁✌❷②❥❙❸✌✼✧❦❣❁✌⑤✌❀✣❥✿❦❣❧✲❳❲❷▼✼✫❱☛❦❣❥❊❸ ❳✧❈❊❦⑥❁✣❷❉❝❣✼ ❁✌✼●❀✣❋❙❆❉❁t❥❊❸✍❳❲❥❵❸✍❳❲❈✖❳ ❝❣❆✌❜●❳❲❝☛❋❊✼❍❜✐✼●⑤✌❥❊❦❣➎❉✼☎➁✍✼●❝⑦P✽❂☛❳❲❈❵❦❣❁✍P◗❦⑦❜●❳❩❥❙✼❍P✽✸✈❭↕❸✌❦❣❥❊✼✖❋❙✼❘⑤✌❋❙✼●❈❊✼●❁◗❥❙❈☎⑩●➇❉❂➊❫✌❝⑦❳❉❜❊▲ ❋❙✼❘⑤✌❋❙✼●❈❊✼●❁◗❥❙❈✁✂❵➇▼✸ ❞ ⑧❀✌❝❣❝⑥❾✩❜✐❆❉❁✣❁✌✼❍❜✐❥❊✼❍P❵❁✌✼●❥♠❱✜❆❉❋❊▲✵❱☛❦❣❥❊❸✿✼●❁✌❆▼❀✌❷❉❸✫P❼❦⑥❈✐❜✐❋❙❦❣❧☎❦⑥❁✻❳❲❥❙❦❣➎▼✼✮⑤✻❆❲❱✜✼●❋✽⑧➄❆▼❋❃❥❊❸✌✼⑨❥✐❳❲❈❊▲✵❱✜❆❉❀✣❝ P❅❸✍❳✕➎▼✼ ⑧❶❳❩❋✑❥❙❆◗❆☎❧✲❳❲❁◗❾❻⑤✍❳❩❋❬❳❩❧✖✼●❥❊✼●❋❙❈✩❥❙❆☎❫✍✼❻❳❲❫✣❝⑥✼✱❥❙❆✖❷▼✼●❁✌✼●❋✐❳❲❝❣❦❣➅●✼❻❜✐❆❉❋❊❋❙✼❍❜✐❥❊❝❣❾❛✸ ❡❸✌✼❘❋❙✼✐⑧➄❆▼❋❙✼✖❳➞❋❙✼●❈❊❥❙❋❙❦⑦❜✐❥❊✼❍P ❜✐❆▼❁✌❁✌✼❍❜✐❥❊❦❣❆❉❁◗⑩♠❈❬❜❊❸✌✼●❧☎✼✧❧❻❀✌❈❙❥✿❫✻✼✰P◗✼●➎◗❦❣❈❙✼❍P✏❂❴❷❉❀✣❦ P❼✼❍P✢❫◗❾✢❆❉❀✣❋✫⑤✌❋❊❦❣❆❉❋❵▲❛❁✣❆❲❱☛❝⑥✼✕P◗❷❉✼✧❳❲❫✻❆❉❀✌❥❵❈❙❸✍❳❩⑤✍✼ ❋❙✼✕❜✐❆❉❷▼❁✌❦❣❥❙❦❣❆❉❁❃✸ ❡❸✌✼●❋❙✼✖❳❩❋❙✼✿❱✜✼●❝❣❝➂⑩❶▲◗❁✌❆❲❱☛❁✰❳▼P◗➎❉❳❲❁◗❥❬❳❩❷❉✼●❈❅❥❙❆☎⑤✍✼●❋⑧➄❆❉❋❊❧✖❦❣❁✌❷❵❈❙❸✍❳❩⑤✍✼❵❋❙✼✕❜✐❆❉❷▼❁✌❦❣❥❙❦❣❆❉❁✰❫◗❾ P◗✼●❥❊✼❍❜✐❥❊❦⑥❁✣❷④❳❲❁✍P②❜✐❆▼❧❻❫✌❦❣❁✌❦❣❁✌❷❻❝❣❆❼❜❘❳❲❝❤⑧➄✼❍❳❩❥❙❀✌❋❊✼●❈❍✸❻❭✢✼❵❸✍❳✕➎▼✼✫❋❊✼❍❽▼❀✌❦❣❋❙✼❍P②❆▼❀✌❋✿❁✌✼●❥♠❱✜❆❉❋❊▲✰❥❙❆✰P◗❆✧❥❊❸✌❦❣❈ ❫◗❾④❜✐❆▼❁✌❈❙❥❊❋❬❳❲❦❣❁✌❦❣❁✌❷✧❥❊❸✌✼✖❜✐❆❉❁✣❁✌✼❍❜✐❥❊❦⑥❆▼❁✌❈❵❦⑥❁②❥❙❸✣✼❵➁✍❋❙❈❊❥✩⑧➄✼●❱❪❝ ❳✕❾▼✼●❋❙❈✱❥❙❆✧❫✍✼❻❝❣❆✌❜●❳❩❝❶✸✟✝♠❁✢❳❉P✌P❼❦⑥❥❊❦❣❆❉❁❃❂✽❦➂⑧ ❳☎⑧➄✼✕❳❲❥❙❀✣❋❙✼✰P◗✼●❥❊✼❍❜✐❥❙❆▼❋❻❦❣❈✫❀✌❈❊✼✐⑧➄❀✌❝☛❆▼❁✢❆❉❁✌✼☎⑤✍❳❲❋❊❥✫❆❩⑧✑❥❙❸✌✼✧❦❣❧✲❳❩❷❉✼❉❂✏❦⑥❥✱❦⑥❈❵❝❣❦❣▲❉✼●❝❣❾✰❥❊❆④❫✻✼✖❀✌❈❊✼✐⑧➄❀✌❝☛❆▼❁ ❆❉❥❊❸✌✼●❋✜⑤✍❳❲❋❊❥❙❈✜❆❲⑧❃❥❊❸✌✼✑❦❣❧✲❳❩❷❉✼⑨❳❩❈⑨❱✜✼●❝❣❝♠✸ ✑❅❁✌✼✩❋❙✼❍❳❩❈❙❆▼❁❻⑧➄❆▼❋❴❥❙❸✌❦❣❈❴❦❣❈❴❥❊❸✍❳❲❥✜❥❙❸✣✼✑❈❬❳❩❝❣❦⑥✼❘❁❛❥✚⑧➄✼❍❳❲❥❊❀✌❋❙✼❘❈⑨❆❩⑧✮❳ P◗❦❣❈❙❥❊❆❉❋❊❥❙✼❍P☎❜❙❸✻❳❲❋❬❳▼❜✐❥❙✼❘❋⑨❧☎❦⑥❷▼❸◗❥❃❫✻✼✑P◗❦❣❈❙⑤✌❝⑦❳❉❜❹✼❍P❵❈❙❝❣❦❣❷❉❸◗❥❙❝❣❾✩⑧❋❙❆❉❧❪❥❙❸✌✼❘❦⑥❋❇⑤✍❆▼❈❙❦❣❥❙❦❣❆❉❁❻❦❣❁❻❳✩❥♠❾◗⑤✌❦⑦❜●❳❲❝✻❜❙❸✻❳❲❋♠⑩ ❳❉❜❹❥❙✼●❋✕✸ ✑❅❁✌✼✑❈❊❆❉❝❣❀✌❥❊❦⑥❆▼❁❻❥❙❆❵❥❙❸✌❦❣❈✜⑤✌❋❙❆▼❫✌❝❣✼●❧✠❦❣❈❴❥❊❆✫❈✐❜●❳❲❁✧❥❙❸✣✼✑❦❣❁✌⑤✌❀✌❥✜❦❣❧✲❳❩❷❉✼✜❱☛❦⑥❥❊❸✖❳✿❈❊❦❣❁✌❷❉❝❣✼☛❁✌✼●❀✣❋❙❆❉❁ ❥❙❸✻❳❲❥✜❸✍❳❲❈☛❳✿❝❣❆✌❜●❳❩❝✍❋❊✼❍❜✐✼●⑤✌❥❊❦❣➎❉✼❅➁✍✼●❝⑦P✽❂✌❳❩❁✍P❵❈❙❥❊❆❉❋❙✼✩❥❊❸✌✼✩❈❙❥❬❳❩❥❙✼●❈☛❆❩⑧❃❥❙❸✌❦❣❈✜❁✌✼●❀✌❋❊❆❉❁✧❦❣❁✖❜✐❆❉❋❊❋❙✼●❈❊⑤✍❆▼❁✍P◗❦❣❁✌❷ ❝❣❆✌❜●❳❲❥❊❦⑥❆▼❁✌❈✜❦⑥❁✧❳✿❝⑦❳✕❾▼✼●❋⑨❜●❳❩❝⑥❝❣✼❍P✧❳✡✠☞☛✐➥✍✌✏✎◗➧✑☛✡✒✹➥❹➦✧➜➄❈❊✼●✼✩➁✍❷▼❀✌❋❙✼✩q❉➝●✸ ❡❸✣❦⑥❈✜❆❉⑤✻✼●❋✐❳❲❥❙❦❣❆▼❁✖❦❣❈☛✼❍❽▼❀✌❦❣➎❉❳❲❝❣✼●❁◗❥ ❥❙❆②❳✰❜✐❆❉❁◗➎▼❆❉❝❣❀✌❥❙❦❣❆▼❁✰❱☛❦❣❥❙❸✝❳✧❈❊❧✲❳❲❝❣❝➊❈❙❦❣➅●✼❻▲▼✼●❋❙❁✣✼●❝♠❂❤⑧➄❆❉❝❣❝❣❆❲❱⑨✼✕P✰❫◗❾✢❳✖❈✐❽▼❀✍❳❲❈❊❸✌❦❣❁✌❷✧⑧➄❀✌❁✻❜✐❥❙❦❣❆❉❁❤✸ ❡❸✌✼ ⑤✌❋❊❆❼❜❹✼●❈❙❈❻❜●❳❲❁②❫✻✼✫⑤✻✼●❋♠⑧❆❉❋❙❧☎✼❍P ❦⑥❁ ⑤✍❳❲❋✐❳❲❝❣❝❣✼●❝➊❫❛❾✧❦❣❧✖⑤✣❝⑥✼❘❧✖✼●❁◗❥❙❦❣❁✌❷❵❥❙❸✣✼✿⑧➄✼✕❳❲❥❙❀✣❋❙✼❻❧✲❳❩⑤✰❳❲❈✿❳❻⑤✌❝⑦❳❩❁✌✼ ❆❲⑧☛❁✌✼●❀✣❋❙❆❉❁✣❈✫❱☛❸✌❆▼❈❙✼☎❱⑨✼●❦❣❷▼❸❛❥✩➎▼✼❍❜✐❥❙❆▼❋❙❈❻❳❲❋❊✼✖❜✐❆❉❁✌❈❊❥❙❋✐❳❲❦❣❁✌✼❍P➀❥❙❆✰❫✻✼❻✼❍❽▼❀✍❳❲❝♠✸ ❡❸✍❳❲❥✱❦⑥❈✕❂❃❀✣❁✌❦❣❥❙❈✿❦❣❁✢❳ ⑧➄✼❍❳❩❥❙❀✌❋❊✼✑❧✧❳❲⑤✫❳❩❋❙✼✑❜✐❆▼❁✌❈❙❥❊❋❬❳❲❦❣❁✌✼✕P❻❥❙❆✱⑤✍✼●❋⑧➄❆❉❋❊❧ ❥❙❸✌✼❅❈❬❳❲❧☎✼⑨❆▼⑤✍✼❘❋❬❳❲❥❊❦❣❆❉❁❻❆❉❁✖P❼❦✓✽✼●❋❊✼●❁◗❥❴⑤✍❳❲❋❊❥❙❈✜❆❲⑧✽❥❙❸✌✼ ❦❣❧✲❳❩❷❉✼❉✸ ❞❁❻❦❣❁◗❥❙✼❘❋❙✼●❈❊❥❙❦❣❁✌❷❵❈❙❦⑦P◗✼✐⑩♠✼✔✓✽✼❍❜✐❥✜❆❲⑧❃❥❙❸✣❦⑥❈✡✓✔☛●➨✖✕☞✗✘✌✚✙✛✗❼➥❩➧●➨✢✜✣✕✜❥❙✼❍❜❊❸✌❁✌❦⑦❽▼❀✌✼❉❂✌❳❩❝❣❋❙✼❍❳▼P◗❾✫P◗✼●❈✐❜✐❋❙❦❣❫✻✼❍P ❦❣❁✧➜❶❚❖❀✌❧✖✼●❝❣❸✍❳❩❋❙❥✕❂❉◆☛❦❣❁◗❥❙❆▼❁✖❳❲❁✍P❵❭↕❦❣❝❣❝❣❦⑦❳❲❧❻❈❍❂◗➇✕➋❉➠ ✄▼➝●❂▼❦❣❈❴❥❙❆✿❋❊✼❍P◗❀✍❜❹✼✿❥❙❸✌✼❅❁◗❀✌❧❻❫✍✼❘❋✮❆❩⑧✽⑧➄❋❙✼❘✼✵⑤✻❳❲❋❬❳❩❧❻⑩ ✼●❥❊✼●❋❙❈✿❫◗❾✧❳❻❝⑦❳❲❋❊❷❉✼✿❳❲❧☎❆❉❀✌❁◗❥❍❂◗❈❊❦❣❁✍❜✐✼❻❳❻❝⑦❳❲❋❊❷❉✼✩❁◗❀✌❧❻❫✍✼❘❋☛❆❲⑧⑨❀✌❁✣❦⑥❥❊❈✑❈❊❸✍❳❲❋❊✼✿❥❙❸✌✼❵❈❬❳❩❧✖✼✩❱✜✼●❦❣❷❉❸◗❥❙❈✕✸✤✝♠❁ ❳❉P✣P◗❦❣❥❙❦❣❆❉❁❃❂➊❳✰❜✐✼❘❋❙❥❬❳❩❦❣❁ ❝❣✼●➎▼✼●❝✜❆❲⑧✑❈❙❸✣❦➫⑧❥✿❦❣❁❛➎❉❳❩❋❙❦⑦❳❲❁✻❜✐✼❻❦❣❈✫⑤✌❋❊✼●❈❙✼❘❁❛❥❻❦❣❁✢❥❊❸✌✼✖❈❊❾❛❈❊❥❙✼●❧ ✢❻❈❙❸✣❦➫⑧❥❙❦❣❁✌❷✧❥❙❸✌✼ ❦❣❁✌⑤✌❀✌❥❇❱☛❦⑥❝❣❝◗❈❙❸✣❦➫⑧❥✮❥❊❸✌✼✑❋❊✼●❈❙❀✣❝⑥❥❇❆❉❁❵❥❙❸✌✼❖⑧➄✼❍❳❲❥❊❀✌❋❙✼☛❧✧❳❲⑤❃❂✕❫✌❀✣❥❴❱☛❦❣❝⑥❝◗❝❣✼❍❳✕➎▼✼☛❦⑥❥➊❀✌❁✻❜❙❸✻❳❲❁✌❷▼✼❍P❵❆❉❥❊❸✌✼●❋❙❱☛❦❣❈❊✼❉✸ ✝♠❁✧⑤✌❋❬❳▼❜✐❥❙❦⑦❜✐✼▼❂✍❦❣❥☛❱☛❦❣❝❣❝✍❫✻✼✿❁✌✼✕❜✐✼●❈❙❈✐❳❲❋❊❾✰❥❙❆❻❸✍❳✕➎▼✼✩❧❻❀✌❝❣❥❙❦❣⑤✌❝❣✼✜⑧➄✼❍❳❩❥❙❀✌❋❊✼✿❧✲❳❩⑤✌❈❍❂▼✼✐➟✌❥❙❋✐❳❉❜✐❥❊❦⑥❁✣❷✲P◗❦✩✓✽✼●❋❙✼●❁◗❥ ⑧➄✼❍❳❩❥❙❀✌❋❊✼●❈☛⑧❋❙❆❉❧ ❥❙❸✌✼✿❈✐❳❲❧☎✼✑❦❣❧✲❳❩❷❉✼❉✸
1b3 5678910111h X 炎文x及炎 Table 1:Connections between Hhand H3 The idea of local,convolutional feature maps can be applied to subsequent hidden layers as well,to extract features of increasing complexity and abstraction.Inter- estingly,higher level features require less precise coding of their location.Reduced precision is actually advantageous,since a slight distortion or translation of the in- put will have reduced effect on the representation.Thus,each feature extraction in our network is followed by an additional layer which performs a local averaging and a subsampling,reducing the resolution of the feature map.This layer introduces a certain level of invariance to distortions and translations.A functional module of our network consists of a layer of shared-weight feature maps followed by an averaging/subsampling layer.This is reminiscent of the Neocognitron architecture (Fukushima and Miyake,198D,with the notable difference that we use backprop (rather than unsupervised learning)which we feel is more appropriate to this sort of classification problem. The network architecture,represented in figure,is a direct extension of the ones described in(Le Cun,1989Le Cun et al.,1990).The network has four hidden layers respectively named H1,H H3,and H_.Layers H1 and H3 are shared- weights feature extractors,while Hhand H_are averaging/subsampling layers. Although the size of the active part of the input is 16 by 16,the actual input is a by plane to avoid problems when a kernel overlaps a boundary.H1 is composed ofgroups of 576 units arranged as independentbyfeature maps.These four feature maps will be designated by H1.1,H1.B H1.3 and H1._.Each unit in a feature map takes its input from a 5 by 5 neighborhood on the input plane.As described above,corresponding connections on each unit in a given feature map are constrained to have the same weight.In other words,all of the 576 units in H1.1 uses the same set ofweights (including the bias).Of course,units in another map (say H1._)share another set of weights. Layer Hhis the averaging/subsampling layer.It is composed ofplanes of size by 1h Each unit in one of these planes takes inputs on_units on the corresponding plane in H1.Receptive fields do not overlap.All the weights are constrained to be equal,even within a single unit.Therefore,Hperforms a local averaging and a to 1 subsampling of H1 in each direction. Layer H3 is composed of 1h feature maps.Each feature map contains 6_units arranged in a 8 by 8 plane.As before,these feature maps will be designated as The connection scheme between Ihand H3 is quite similar to the one between the input and H1,but slightly more complicated because H3 has multiple D maps.Each unit receptive field is composed of one or two 5 by
➇ ✤ q ✪ ✫ ✄ ♣ ➠ ➋ ➇❍♦ ➇❉➇ ➇ ✤ ➇ ✤ q ✪ ✁✄✂✆☎✞✝☎✠ ☛✍✌✑✾✜❆❉❁✣❁✌✼❍❜✐❥❊❦⑥❆▼❁✌❈☛❫✻✼●❥♠❱⑨✼❘✼●❁❿◆ ✤✫❳❩❁✍P✧◆✑q✣✸ ❡❸✌✼✿❦⑦P◗✼✕❳❵❆❲⑧✮❝❣❆✌❜●❳❩❝❶❂✌❜❹❆❉❁◗➎❉❆▼❝⑥❀✣❥❙❦❣❆❉❁✍❳❩❝✌⑧✼❍❳❲❥❊❀✌❋❙✼✿❧✧❳❲⑤✌❈☛❜●❳❩❁✲❫✻✼✫❳❩⑤✌⑤✌❝❣❦❣✼❍P☎❥❙❆❻❈❙❀✣❫✌❈❙✼❍❽▼❀✌✼❘❁❛❥✩❸✌❦⑦P✌P❼✼●❁ ❝⑦❳✕❾❉✼●❋❊❈✵❳❩❈✵❱✜✼●❝❣❝♠❂✌❥❊❆✲✼❹➟❼❥❊❋❬❳▼❜✐❥❅⑧➄✼❍❳❲❥❊❀✌❋❙✼❘❈✿❆❲⑧❴❦❣❁✍❜✐❋❊✼❍❳❲❈❊❦⑥❁✣❷➆❜❹❆❉❧✖⑤✣❝⑥✼❹➟❼❦❣❥♠❾✖❳❩❁✍P④❳❩❫✌❈❙❥❊❋❬❳▼❜✐❥❙❦❣❆❉❁❤✸✄✝♠❁◗❥❙✼❘❋♠⑩ ✼●❈❊❥❙❦❣❁✌❷❉❝❣❾◗❂◗❸✌❦❣❷❉❸✌✼❘❋☛❝⑥✼❘➎❉✼●❝✻⑧➄✼✕❳❲❥❙❀✣❋❙✼●❈❅❋❙✼❍❽▼❀✌❦❣❋❙✼✱❝⑥✼❘❈❙❈☛⑤✌❋❊✼❍❜✐❦❣❈❙✼❻❜✐❆✌P◗❦❣❁✌❷❵❆❲⑧❴❥❙❸✌✼❘❦⑥❋☛❝❣❆✌❜●❳❩❥❙❦❣❆❉❁❃✸➊❚☛✼✕P◗❀✍❜✐✼❍P ⑤✌❋❊✼❍❜✐❦❣❈❙❦❣❆❉❁✧❦❣❈☛❳❉❜✐❥❊❀✍❳❲❝❣❝❣❾✹❳▼P◗➎❉❳❲❁◗❥✐❳❲❷❉✼❘❆❉❀✌❈✕❂◗❈❙❦❣❁✍❜✐✼✿❳✱❈❙❝❣❦❣❷❉❸◗❥⑨P◗❦❣❈❙❥❊❆❉❋❊❥❙❦❣❆❉❁☎❆❉❋✜❥❙❋❬❳❩❁✌❈❙❝⑦❳❩❥❙❦❣❆❉❁☎❆❲⑧✮❥❊❸✌✼✩❦❣❁◗⑩ ⑤✌❀✌❥✜❱☛❦❣❝❣❝✌❸✍❳✕➎❉✼❅❋❙✼❍P❼❀✍❜✐✼❍P☎✼✔✓✽✼❍❜✐❥❅❆❉❁❻❥❙❸✣✼✵❋❊✼●⑤✌❋❊✼●❈❙✼●❁◗❥✐❳❲❥❙❦❣❆▼❁❃✸ ❡❸◗❀✌❈✕❂◗✼❍❳❉❜❊❸❻⑧✼❍❳❲❥❊❀✌❋❙✼❅✼✐➟✌❥❙❋❬❳▼❜✐❥❙❦❣❆▼❁✖❦❣❁ ❆❉❀✣❋❴❁✌✼●❥♠❱✜❆❉❋❊▲✫❦❣❈➊⑧➄❆▼❝❣❝⑥❆❲❱✜✼❍P✱❫◗❾✹❳❩❁✖❳❉P✌P◗❦❣❥❊❦⑥❆▼❁✍❳❲❝◗❝⑦❳✕❾❉✼❘❋✮❱☛❸✣❦ ❜❊❸❻⑤✻✼●❋♠⑧❆❉❋❙❧☎❈⑨❳✩❝❣❆✌❜●❳❲❝✻❳✕➎❉✼●❋✐❳❲❷❉❦❣❁✌❷✿❳❩❁✍P ❳☎❈❙❀✌❫✣❈❬❳❲❧☎⑤✌❝❣❦❣❁✌❷✍❂✻❋❙✼✕P◗❀✍❜✐❦❣❁✌❷✰❥❊❸✌✼❻❋❙✼❘❈❙❆❉❝❣❀✌❥❊❦❣❆❉❁✢❆❲⑧☛❥❙❸✣✼❵⑧➄✼❍❳❲❥❊❀✌❋❙✼❻❧✧❳❲⑤❃✸ ❡❸✌❦❣❈✿❝⑦❳✕❾❉✼●❋✩❦❣❁◗❥❙❋❊❆❼P❼❀✍❜✐✼●❈ ❳✧❜✐✼●❋❊❥❬❳❲❦❣❁✈❝⑥✼❘➎❉✼●❝❴❆❩⑧❨❦❣❁◗➎❉❳❲❋❊❦⑦❳❲❁✍❜✐✼❵❥❙❆②P◗❦❣❈❙❥❙❆▼❋❙❥❊❦⑥❆▼❁✌❈✿❳❲❁✍P②❥❊❋❬❳❩❁✌❈❙❝⑦❳❲❥❊❦❣❆❉❁✌❈✕✸ ❞ ⑧➄❀✌❁✻❜✐❥❙❦❣❆❉❁✻❳❲❝➊❧✖❆✌P◗❀✌❝❣✼ ❆❲⑧✩❆❉❀✌❋❻❁✣✼●❥♠❱⑨❆▼❋❙▲t❜✐❆❉❁✌❈❊❦❣❈❙❥❙❈☎❆❲⑧✿❳②❝ ❳✕❾▼✼●❋❵❆❲⑧✿❈❙❸✍❳❩❋❙✼❍P❛⑩❶❱✜✼●❦❣❷❉❸◗❥❵⑧➄✼❍❳❩❥❙❀✌❋❊✼✰❧✲❳❩⑤✌❈✩⑧➄❆▼❝❣❝⑥❆❲❱✜✼❍P➀❫❛❾✝❳❩❁ ❳✕➎❉✼❘❋❬❳❲❷▼❦❣❁✌❷✠✟❩❈❙❀✌❫✌❈✐❳❲❧☎⑤✌❝❣❦⑥❁✣❷✵❝⑦❳✕❾▼✼●❋❍✸ ❡❸✌❦❣❈☛❦⑥❈❅❋❙✼●❧☎❦❣❁✌❦❣❈❬❜✐✼●❁◗❥☛❆❩⑧✚❥❊❸✌✼✫♥❖✼●❆✌❜✐❆❉❷▼❁✌❦❣❥❙❋❙❆▼❁✰❳❲❋❬❜❊❸✌❦❣❥❊✼❍❜✐❥❙❀✣❋❙✼ ➜☛✡✣❀✌▲◗❀✌❈❊❸✌❦❣❧✲❳❻❳❲❁✻P④❺✧❦❣❾◗❳❲▲❉✼▼❂✮➇✕➋❉➠✬✤❉➝●❂✻❱☛❦❣❥❙❸ ❥❙❸✌✼❻❁✌❆▼❥❬❳❩❫✌❝❣✼✫P◗❦✩✓➃✼❘❋❙✼●❁✍❜❹✼✖❥❙❸✻❳❲❥✩❱⑨✼❵❀✌❈❊✼❻❫✍❳❉❜❊▲◗⑤✌❋❙❆▼⑤ ➜➄❋✐❳❲❥❊❸✌✼●❋✩❥❙❸✍❳❩❁✰❀✌❁✌❈❊❀✌⑤✻✼●❋❙➎◗❦❣❈❙✼❍P ❝❣✼❍❳❲❋❊❁✌❦❣❁✌❷❛➝❖❱☛❸✌❦⑦❜❊❸✲❱✜✼✩⑧➄✼❘✼●❝➊❦⑥❈❅❧✖❆▼❋❙✼✿❳❲⑤✌⑤✣❋❙❆❉⑤✣❋❙❦⑦❳❲❥❊✼✿❥❙❆☎❥❙❸✌❦❣❈☛❈❙❆▼❋❙❥ ❆❲⑧❴❜✐❝⑦❳❲❈❊❈❙❦➂➁➃❜●❳❩❥❙❦❣❆❉❁✧⑤✌❋❊❆❉❫✌❝❣✼●❧②✸ ❡❸✌✼✿❁✣✼●❥♠❱⑨❆▼❋❙▲✰❳❲❋✐❜❊❸✌❦❣❥❙✼❍❜✐❥❊❀✌❋❙✼▼❂✽❋❙✼●⑤✣❋❙✼●❈❊✼●❁◗❥❙✼❍P②❦❣❁✧➁✍❷▼❀✌❋❙✼ ✪✍❂✌❦❣❈✑❳❻P◗❦❣❋❊✼❍❜✐❥✩✼✐➟✌❥❙✼●❁✣❈❙❦❣❆❉❁✰❆❩⑧❴❥❙❸✌✼✿❆▼❁✌✼●❈ P◗✼●❈✐❜✐❋❙❦❣❫✻✼❍P✈❦❣❁t➜❶✺✽✼✰✾✜❀✌❁❤❂☛➇❍➋❉➠▼➋✏✎➊✺✽✼✧✾✜❀✌❁✝✼●❥❻❳❲❝♠✸❣❂✜➇❍➋❉➋▼♦❉➝❘✸ ❡❸✌✼✖❁✌✼❘❥♠❱⑨❆▼❋❙▲✢❸✍❳❩❈✿⑧➄❆▼❀✌❋✿❸✌❦⑦P✌P❼✼●❁ ❝⑦❳✕❾❉✼●❋❊❈❻❋❙✼●❈❊⑤✍✼✕❜✐❥❙❦❣➎❉✼❘❝⑥❾ ❁✍❳❲❧☎✼❍P✝◆✱➇❉❂☛◆✤✣❂❨◆❅q✌❂☛❳❲❁✻P ◆✪✍✸➌✺❃❳✕❾❉✼❘❋❙❈✖◆✿➇✰❳❩❁✍Pt◆✑q✈❳❲❋❙✼✧❈❊❸✍❳❲❋❊✼❍P▼⑩ ❱✜✼●❦❣❷❉❸◗❥❙❈✜⑧➄✼✕❳❲❥❙❀✣❋❙✼✩✼✐➟✌❥❙❋✐❳❉❜✐❥❊❆❉❋❙❈✕❂✻❱☛❸✌❦❣❝⑥✼✿◆ ✤✫❳❩❁✍P✧◆✪✖❳❲❋❊✼✿❳✕➎❉✼●❋✐❳❲❷❉❦❣❁✌❷☞✟❲❈❊❀✌❫✌❈❬❳❩❧✖⑤✌❝❣❦❣❁✌❷✩❝⑦❳✕❾❉✼●❋❊❈❍✸ ❞❝❣❥❊❸✌❆❉❀✌❷▼❸✫❥❊❸✌✼✑❈❊❦❣➅●✼☛❆❲⑧❃❥❊❸✌✼✑❳❉❜✐❥❊❦❣➎❉✼☛⑤✍❳❩❋❙❥❴❆❩⑧➃❥❊❸✌✼✑❦❣❁✌⑤✌❀✣❥❴❦❣❈❨➇✝✄✩❫◗❾✖➇✆✄✣❂❉❥❊❸✌✼✑❳❉❜❹❥❙❀✍❳❩❝✍❦❣❁✌⑤✌❀✣❥✮❦❣❈⑨❳ ✤▼➠ ❫◗❾✮✤❉➠✱⑤✌❝⑦❳❲❁✌✼❅❥❙❆✖❳✕➎▼❆❉❦⑦P❵⑤✌❋❙❆▼❫✌❝❣✼●❧✖❈❇❱☛❸✌✼●❁✰❳✿▲▼✼●❋❙❁✣✼●❝❃❆❲➎▼✼●❋❙❝⑦❳❲⑤✣❈❨❳✱❫✍❆▼❀✌❁✍P✌❳❩❋❙❾◗✸❴◆✿➇❅❦❣❈❨❜✐❆▼❧✖⑤✻❆❉❈❊✼❍P ❆❲⑧✂✪✖❷❉❋❊❆❉❀✌⑤✣❈✑❆❲⑧ ✫❉♣ ✄❵❀✣❁✌❦❣❥❙❈✿❳❲❋❊❋❬❳❲❁✣❷❉✼❍P✰❳❩❈ ✪✖❦❣❁✍P◗✼●⑤✻✼●❁✍P❼✼●❁◗❥✭✤★✪✖❫◗❾ ✤✡✪➞⑧➄✼❍❳❲❥❊❀✌❋❙✼❵❧✲❳❩⑤✌❈❍✸ ❡❸✌✼●❈❙✼ ⑧➄❆▼❀✌❋☛⑧➄✼✕❳❲❥❙❀✣❋❙✼❻❧✲❳❩⑤✌❈☛❱☛❦❣❝❣❝❃❫✻✼✖P◗✼●❈❙❦❣❷▼❁✍❳❲❥❊✼❍P✰❫◗❾④◆✱➇❉✸❣➇❉❂✻◆✿➇▼✸ ✤✣❂✽◆✿➇❉✸➐q☎❳❲❁✍P✰◆✱➇❉✸ ✪✍✸✿❯❇❳❉❜❊❸④❀✣❁✌❦❣❥✑❦❣❁ ❳❵⑧✼❍❳❲❥❊❀✌❋❙✼❵❧✲❳❩⑤✲❥✐❳❲▲▼✼●❈✑❦❣❥❙❈❅❦❣❁✌⑤✌❀✌❥❖⑧➄❋❙❆▼❧ ❳✮✫❻❫◗❾ ✫❻❁✣✼●❦❣❷❉❸◗❫✍❆▼❋❙❸✌❆◗❆✌P✧❆▼❁✰❥❙❸✌✼❵❦❣❁✌⑤✌❀✌❥❅⑤✌❝⑦❳❲❁✌✼▼✸ ❞❈ P◗✼●❈✐❜✐❋❙❦❣❫✻✼❍P✧❳❲❫✻❆❲➎❉✼❉❂◗❜✐❆▼❋❙❋❙✼❘❈❙⑤✻❆❉❁✍P◗❦❣❁✌❷❻❜✐❆▼❁✌❁✌✼❍❜✐❥❊❦❣❆❉❁✌❈✜❆❉❁❻✼❍❳▼❜❊❸❻❀✌❁✌❦❣❥❴❦❣❁✖❳✩❷❉❦❣➎▼✼●❁✿⑧✼❍❳❲❥❊❀✌❋❙✼❅❧✲❳❲⑤❻❳❲❋❊✼ ❜✐❆▼❁✌❈❙❥❊❋❬❳❲❦❣❁✌✼✕P✰❥❊❆✲❸✻❳✕➎❉✼✿❥❊❸✌✼✫❈✐❳❲❧☎✼✿❱⑨✼❘❦⑥❷▼❸◗❥❍✸ ✝♠❁✰❆▼❥❙❸✌✼●❋✩❱✜❆❉❋✐P◗❈❍❂❃❳❩❝❣❝❃❆❲⑧✜❥❙❸✌✼✒✫❉♣ ✄❻❀✌❁✌❦❣❥❊❈✵❦❣❁④◆✱➇❉✸❣➇ ❀✌❈❊✼●❈✫❥❊❸✌✼❻❈❬❳❩❧✖✼❵❈❙✼●❥❵❆❲⑧ ✤ ✄➆❱⑨✼●❦❣❷▼❸❛❥❊❈❵➜➄❦❣❁✍❜✐❝❣❀✍P◗❦❣❁✌❷✧❥❊❸✌✼✖❫✌❦⑦❳❩❈❬➝●✸ ✑☛⑧✵❜❹❆❉❀✌❋❊❈❙✼❉❂➊❀✌❁✣❦⑥❥❊❈✿❦❣❁✝❳❲❁✌❆▼❥❙❸✌✼❘❋ ❧✲❳❩⑤✧➜➄❈✐❳✕❾✲◆✿➇▼✸ ✪❛➝❇❈❙❸✍❳❩❋❙✼✲➥ ✜✍➩✍✌ ✗ ☛❘➧❵❈❙✼●❥☛❆❩⑧ ✤ ✄✿❱✜✼●❦❣❷❉❸◗❥❊❈❍✸ ✺❃❳✕❾▼✼●❋✑◆✤❵❦❣❈☛❥❊❸✌✼✫❳✕➎❉✼❘❋❬❳❲❷▼❦❣❁✌❷✠✟❩❈❙❀✌❫✌❈✐❳❲❧☎⑤✌❝❣❦⑥❁✣❷✵❝⑦❳✕❾▼✼●❋❍✸ ✝♠❥☛❦⑥❈✑❜❹❆❉❧✖⑤✻❆❉❈❊✼❍P☎❆❲⑧ ✪❵⑤✌❝⑦❳❲❁✌✼●❈❅❆❲⑧❴❈❙❦❣➅●✼❻➇ ✤ ❫◗❾✖➇ ✤✣✸✮❯❇❳❉❜❊❸❻❀✌❁✌❦❣❥➊❦❣❁✫❆▼❁✌✼☛❆❲⑧✽❥❙❸✌✼❘❈❙✼☛⑤✌❝⑦❳❲❁✣✼●❈❴❥❬❳❩▲❉✼●❈❇❦⑥❁✣⑤✌❀✌❥❙❈❇❆❉❁ ✪✵❀✣❁✌❦❣❥❙❈❴❆▼❁✫❥❊❸✌✼✑❜✐❆❉❋❊❋❙✼●❈❊⑤✍❆▼❁✍P◗❦❣❁✌❷ ⑤✌❝⑦❳❲❁✣✼✵❦❣❁✧◆✿➇❉✸❴❚❖✼❍❜✐✼●⑤✣❥❙❦❣➎❉✼✩➁✻✼●❝⑦P◗❈✑P◗❆❵❁✌❆❉❥☛❆❲➎▼✼●❋❙❝⑦❳❲⑤❤✸ ❞❝❣❝✍❥❊❸✌✼✿❱✜✼●❦❣❷❉❸◗❥❙❈✑❳❩❋❙✼✿❜✐❆▼❁✌❈❙❥❊❋❬❳❲❦❣❁✌✼✕P✧❥❙❆❵❫✍✼ ✼❍❽▼❀✍❳❩❝❶❂✣✼●➎❉✼❘❁✰❱☛❦❣❥❙❸✌❦❣❁✲❳❵❈❊❦❣❁✌❷❉❝❣✼✩❀✌❁✌❦❣❥❍✸ ❡❸✌✼❘❋❙✼✐⑧➄❆▼❋❙✼❉❂✽◆ ✤❵⑤✍✼●❋⑧➄❆❉❋❊❧✖❈✑❳❵❝❣❆✌❜●❳❲❝➊❳✕➎❉✼●❋✐❳❲❷▼❦⑥❁✣❷✹❳❩❁✍P✧❳✒✤ ❥❙❆✰➇❅❈❙❀✣❫✌❈❬❳❩❧✖⑤✌❝❣❦❣❁✌❷✿❆❩⑧❇◆✱➇✑❦❣❁✧✼❍❳▼❜❙❸✰P◗❦❣❋❊✼❍❜✐❥❙❦❣❆▼❁❃✸ ✺❃❳✕❾▼✼●❋✖◆✑q②❦❣❈✖❜✐❆▼❧✖⑤✻❆❉❈❊✼❍P✢❆❩⑧➞➇ ✤✧⑧✼❍❳❲❥❊❀✌❋❙✼✰❧✧❳❲⑤✌❈✕✸ ❯❇❳❉❜❊❸✈⑧➄✼❍❳❲❥❊❀✌❋❙✼ ❧✲❳❲⑤✝❜❹❆❉❁◗❥❬❳❩❦⑥❁✣❈ ✄✡✪✢❀✌❁✌❦❣❥❊❈ ❳❲❋❊❋❬❳❩❁✌❷❉✼❍P➀❦❣❁✝❳✰➠✧❫◗❾❿➠➆⑤✌❝⑦❳❲❁✌✼❉✸ ❞❈✫❫✻✼✐⑧➄❆▼❋❙✼❉❂❇❥❙❸✌✼❘❈❙✼☎⑧➄✼❍❳❩❥❙❀✌❋❊✼✖❧✲❳❩⑤✌❈✩❱☛❦❣❝⑥❝❇❫✍✼✧P◗✼●❈❙❦❣❷▼❁✍❳❲❥❊✼❍P✝❳❩❈ ◆✤✣✸❣➇❉❂✍◆ ✤✌✸✯✤✍✌✎✌✏✌✻◆✤✌✸❣➇ ✤✌✸ ❡❸✌✼❻❜✐❆❉❁✌❁✣✼❍❜✐❥❙❦❣❆▼❁④❈✐❜❙❸✣✼●❧✖✼✿❫✻✼●❥♠❱✜✼●✼●❁✢◆✤☎❳❲❁✍P✰◆❅q❻❦❣❈✵❽▼❀✌❦❣❥❊✼✿❈❙❦❣❧✖❦❣❝⑦❳❲❋ ❥❙❆ ❥❙❸✌✼❻❆▼❁✌✼❻❫✍✼❘❥♠❱⑨✼●✼❘❁ ❥❊❸✌✼❻❦❣❁✌⑤✌❀✌❥❵❳❲❁✍P②◆✿➇▼❂❃❫✌❀✣❥✫❈❊❝⑥❦❣❷▼❸❛❥❊❝❣❾✲❧☎❆❉❋❊✼✹❜✐❆▼❧✖⑤✌❝❣❦⑦❜●❳❩❥❙✼❍P ❫✍✼✕❜●❳❲❀✌❈❊✼✲◆❅q ❸✍❳❩❈✵❧❻❀✣❝⑥❥❊❦❣⑤✌❝❣✼✭✤ ⑩❑ ❧✲❳❲⑤✣❈❍✸✿❯❴❳▼❜❙❸✢❀✣❁✌❦❣❥✵❋❊✼❍❜✐✼●⑤✣❥❙❦❣➎❉✼❵➁✍✼❘❝ P②❦❣❈✿❜✐❆▼❧✖⑤✻❆❉❈❙✼✕P✧❆❲⑧☛❆❉❁✌✼❵❆❉❋✱❥♠❱⑨❆ ✫✖❫◗❾
d@12x12 H2 Figure 4: Network Architecture with 5 layers of fully-adaptive connections
28x28 INPUT 4@24x24 H1 4@12x12 H2 12@8x8 H3 12@4x4 H4 10 OUTPUT ✂✁☎✄✍✆✞✡✠✁ ✌❨♥❖✼●❥♠❱✜❆❉❋❙▲ ❞❋❬❜❊❸✌❦❣❥❙✼❍❜❹❥❙❀✌❋❊✼✫❱☛❦❣❥❙❸ ✫❵❝⑦❳✕❾❉✼●❋❊❈☛❆❲⑧❃⑧❀✌❝❣❝⑥❾▼⑩❙❳❉P✌❳❩⑤✌❥❙❦❣➎▼✼✑❜✐❆❉❁✌❁✣✼❍❜✐❥❙❦❣❆▼❁✌❈❍✸
5 neighborhoods centered around units that are at identical positions within each H2 maps.Of course,all units in a given map are constrained to have identical weight vectors.The maps in H2 on which a map in H3 takes its inputs are chosen according to a scheme described on table 1.According to this scheme,the network is composed of two almost independent modules.Layer H4 plays the same role as layer H2,it is composed of 12 groups of 16 units arranged in 4 by 4 planes. The output layer has 10 units and is fully connected to H4.In summary,the network has 4635 units,98442 connections,and 2578 independent parameters.This archi- tecture was derived using the Optimal Brain Damage technique (Le Cun,Denker and Solla,1990)starting from a previous architecture (Le Cun et al.,1990)that had 4 times more free parameters. 5 RESULTS After 30 training passes the error rate on training set(7291 handwritten plus 2549 printed digits)was 1.1%and the MSE was .017.On the whole test set (2007 handwritten plus 700 printed characters)the error rate was 3.4%and the MSE was 0.024.All the classification errors occurred on handwritten characters. In a realistic application,the user is not so much interested in the raw error rate as in the number of rejections necessary to reach a given level of accuracy.In our case,we measured the percentage of test patterns that must be rejected in order to get 1%error rate.Our rejection criterion was based on three conditions:the activity level of the most-active output unit should by larger than a given threshold t1,the activity level of the second most-active unit should be smaller than a given threshold t2,and finally,the difference between the activity levels of these two units should be larger than a given threshold ta.The best percentage of rejections on the complete test set was 5.7%for 1%error.On the handwritten set only,the result was 9%rejections for 1%error.It should be emphasized that the rejection thresholds were obtained using performance measures on the test set.About half the substitution errors in the testing set were due to faulty segmentation,and an additional quarter were due to erroneous assignment of the desired category.Some of the remaining images were ambiguous even to humans,and in a few cases the network misclassified the image for no discernible reason. Even though a second-order version of back-propagation was used,it is interesting to note that the learning takes only 30 passes through the training set.We think this can be attributed to the large amount of redundancy present in real data.A complete training session(30 passes through the training set plus test)takes about 3 days on a SUN SPARCstation 1 using the SN2 connectionist simulator(Bottou and Le Cun,1989). After successful training,the network was implemented on a commercial Digital Signal Processor board containing an AT&T DSP-32C general purpose DSP chip with a peak performance of 12.5 million multiply-add operations per second on 32 bit foating point numbers.The DSP operates as a coprocessor in a PC connected to a video camera.The PC performs the digitization,binarization and segmentation
✫❻❁✣✼●❦❣❷❉❸◗❫✍❆▼❋❙❸✌❆◗❆✌P◗❈✿❜✐✼●❁◗❥❊✼●❋❙✼❍P✈❳❲❋❊❆❉❀✌❁✍P➆❀✌❁✌❦❣❥❙❈✩❥❙❸✻❳❲❥✵❳❩❋❙✼❻❳❲❥✩❦⑦P◗✼●❁◗❥❙❦⑦❜●❳❩❝❃⑤✻❆❉❈❊❦⑥❥❊❦❣❆❉❁✌❈✩❱☛❦❣❥❙❸✌❦❣❁✧✼❍❳▼❜❙❸ ◆✤ ❧✲❳❩⑤✌❈❍✸ ✑☛⑧✫❜✐❆▼❀✌❋❙❈❊✼❉❂☛❳❲❝❣❝✜❀✌❁✌❦❣❥❙❈❻❦❣❁ ❳②❷▼❦❣➎❉✼●❁✈❧✲❳❲⑤✢❳❲❋❊✼✰❜✐❆❉❁✌❈❊❥❙❋✐❳❲❦❣❁✌✼❍Pt❥❊❆④❸✻❳✕➎❉✼✧❦⑦P◗✼●❁◗❥❙❦⑦❜●❳❩❝ ❱✜✼●❦❣❷❉❸◗❥☛➎❉✼❍❜❹❥❙❆❉❋❊❈❍✸ ❡❸✌✼✿❧✧❳❲⑤✌❈✜❦❣❁✲◆ ✤❻❆❉❁✧❱☛❸✌❦⑦❜❊❸✰❳❵❧✲❳❩⑤❻❦⑥❁✰◆❅q❵❥❬❳❲▲▼✼●❈✑❦❣❥❊❈☛❦⑥❁✣⑤✌❀✌❥❙❈✑❳❩❋❙✼✫❜❊❸✌❆▼❈❙✼●❁ ❳❉❜❘❜✐❆❉❋✐P◗❦❣❁✌❷✫❥❊❆✹❳✱❈❬❜❊❸✌✼❘❧✖✼✿P◗✼●❈✐❜✐❋❙❦❣❫✻✼❍P✧❆❉❁❻❥✐❳❲❫✌❝❣✼❻➇❉✸ ❞❜❘❜✐❆❉❋✐P◗❦❣❁✌❷❻❥❙❆❵❥❙❸✌❦❣❈✜❈❬❜❊❸✌✼●❧☎✼❉❂◗❥❙❸✣✼✵❁✣✼●❥♠❱⑨❆▼❋❙▲ ❦❣❈✵❜✐❆▼❧✖⑤✻❆❉❈❊✼❍P☎❆❲⑧❴❥♠❱⑨❆✧❳❲❝❣❧✖❆▼❈❙❥❴❦❣❁✍P❼✼●⑤✍✼❘❁✍P◗✼●❁◗❥✿❧☎❆❼P❼❀✌❝❣✼●❈❍✸✜✺❃❳✕❾❉✼❘❋✵◆✪☎⑤✌❝⑦❳✕❾◗❈☛❥❙❸✌✼✿❈✐❳❲❧☎✼✵❋❊❆❉❝❣✼✫❳❩❈ ❝⑦❳✕❾❉✼●❋☛◆✤✣❂✌❦❣❥✜❦⑥❈✑❜❹❆❉❧✖⑤✻❆❉❈❊✼❍P❻❆❩⑧✑➇ ✤✱❷❉❋❙❆▼❀✌⑤✌❈☛❆❩⑧❨➇✆✄❵❀✣❁✌❦❣❥❙❈✑❳❲❋❊❋❬❳❩❁✌❷❉✼❍P☎❦❣❁✮✪❻❫◗❾ ✪✫⑤✌❝⑦❳❩❁✌✼●❈❍✸ ❡❸✌✼❴❆▼❀✌❥❙⑤✣❀✌❥➊❝ ❳✕❾▼✼●❋✽❸✍❳❩❈⑨➇❍♦❖❀✌❁✌❦❣❥❙❈➊❳❲❁✍P❅❦❣❈✽⑧❀✌❝❣❝⑥❾✩❜✐❆❉❁✣❁✌✼❍❜✐❥❊✼❍P❵❥❙❆✑◆✪✍✸ ✝♠❁✿❈❙❀✌❧☎❧✲❳❩❋❙❾◗❂❙❥❊❸✌✼⑨❁✣✼●❥♠❱⑨❆▼❋❙▲ ❸✍❳❩❈ ✪ ✄❉q✬✫❵❀✌❁✌❦❣❥❙❈✕❂➃➋▼➠✡✪✬✪ ✤✖❜❹❆❉❁✌❁✌✼✕❜✐❥❙❦❣❆❉❁✣❈❍❂❃❳❩❁✍P ✤✥✫▼♣❉➠➞❦⑥❁✻P◗✼●⑤✻✼●❁✍P◗✼●❁◗❥✩⑤✍❳❩❋❬❳❲❧☎✼●❥❙✼❘❋❙❈❍✸ ❡❸✣❦⑥❈✩❳❲❋✐❜❙❸✣❦➫⑩ ❥❙✼✕❜✐❥❙❀✌❋❊✼✖❱☛❳❲❈✿P◗✼●❋❊❦❣➎❉✼❍P ❀✌❈❙❦❣❁✌❷✧❥❊❸✌✼ ✑❅⑤✌❥❊❦❣❧✲❳❲❝➊❄❴❋✐❳❲❦❣❁✢❑✵❳❩❧✲❳❩❷❉✼✩❥❙✼❍❜❊❸✌❁✣❦ ❽▼❀✌✼✧➜♠✺✽✼❻✾✜❀✌❁❃❂❃❑❅✼●❁✣▲❉✼●❋ ❳❲❁✻P✢❏◗❆❉❝❣❝⑦❳✌❂➊➇❍➋▼➋❉♦▼➝✑❈❊❥❬❳❩❋❙❥❙❦❣❁✌❷➆⑧➄❋❙❆▼❧ ❳✧⑤✌❋❊✼●➎◗❦❣❆❉❀✌❈✫❳❩❋❬❜❊❸✌❦❣❥❙✼✕❜✐❥❙❀✌❋❊✼②➜❶✺✽✼✖✾✜❀✣❁✢✼●❥✫❳❲❝♠✸❣❂❴➇✕➋❉➋❉♦▼➝❅❥❙❸✍❳❩❥ ❸✍❳▼P✮✪❵❥❙❦❣❧☎✼●❈⑨❧☎❆❉❋❊✼☛⑧➄❋❊✼●✼✿⑤✍❳❩❋❬❳❲❧☎✼●❥❙✼❘❋❙❈❍✸ ①✙ ✉☛↔✂✁⑨✇➀✉ ❞⑧❥❙✼●❋✑q▼♦✿❥❊❋❬❳❲❦❣❁✌❦❣❁✌❷❵⑤✍❳❩❈❙❈❙✼❘❈✑❥❙❸✣✼✵✼❘❋❙❋❙❆▼❋✑❋✐❳❲❥❙✼✩❆▼❁✖❥❙❋✐❳❲❦❣❁✌❦❣❁✌❷❵❈❙✼●❥✱➜❶♣✬✤❉➋✌➇❖❸✍❳❲❁✍P❼❱☛❋❙❦❣❥❙❥❊✼●❁✧⑤✌❝❣❀✌❈ ✤✥✫★✪❛➋ ⑤✌❋❊❦⑥❁◗❥❊✼❍P P❼❦⑥❷▼❦❣❥❙❈❬➝❵❱☛❳❲❈✢➇☎✄⑥➇✕➈ ❳❩❁✍Pt❥❊❸✌✼❿❺❏✌❯ ❱☛❳❲❈✆✄➐♦✌➇❍♣✣✸ ✑❅❁↕❥❙❸✣✼④❱☛❸✌❆▼❝❣✼✰❥❙✼●❈❊❥✰❈❙✼❘❥④➜ ✤❉♦▼♦❉♣ ❸✍❳❩❁✍P◗❱☛❋❙❦❣❥❊❥❙✼●❁☎⑤✌❝❣❀✌❈⑨♣❉♦▼♦✩⑤✌❋❊❦⑥❁◗❥❊✼❍P✖❜❊❸✍❳❩❋❬❳❉❜❹❥❙✼●❋❊❈❬➝✜❥❙❸✣✼✑✼●❋❊❋❙❆❉❋☛❋✐❳❲❥❊✼✑❱☛❳❲❈⑨q✝✄ ✪◗➈ ❳❩❁✍P❵❥❙❸✌✼✩❺✰❏✌❯➀❱❨❳❩❈ ♦✞✄➐♦✬✤✡✪✍✸ ❞❝❣❝✽❥❊❸✌✼✿❜✐❝⑦❳❲❈❙❈❊❦➂➁➃❜●❳❲❥❊❦❣❆❉❁✧✼●❋❙❋❊❆❉❋❊❈✑❆✌❜●❜✐❀✌❋❊❋❙✼❍P ❆❉❁☎❸✍❳❲❁✻P◗❱☛❋❙❦❣❥❙❥❊✼●❁④❜❊❸✍❳❩❋❬❳❉❜❹❥❙✼●❋❊❈❍✸ ✝♠❁✢❳✖❋❊✼❍❳❲❝❣❦❣❈❙❥❊❦⑦❜✫❳❩⑤✌⑤✌❝❣❦⑦❜●❳❲❥❊❦⑥❆▼❁❃❂✌❥❊❸✌✼❻❀✌❈❊✼●❋✿❦❣❈✩❁✌❆❉❥✩❈❙❆✧❧❻❀✍❜❊❸✰❦❣❁◗❥❙✼❘❋❙✼●❈❊❥❙✼❍P②❦❣❁②❥❙❸✌✼❻❋✐❳✕❱➉✼●❋❙❋❊❆❉❋✿❋✐❳❲❥❙✼ ❳❲❈❅❦❣❁✰❥❙❸✌✼✱❁❛❀✣❧❻❫✍✼●❋❅❆❲⑧❴❋❙✼➍✐✼❍❜✐❥❊❦⑥❆▼❁✌❈✩❁✌✼❍❜✐✼❘❈❙❈❬❳❩❋❙❾②❥❙❆☎❋❙✼❍❳▼❜❙❸✢❳❵❷❉❦❣➎▼✼●❁✰❝❣✼●➎❉✼❘❝✮❆❩⑧⑨❳❉❜●❜✐❀✣❋❬❳❉❜❹❾❛✸✡✝♠❁✰❆❉❀✣❋ ❜●❳❩❈❙✼❉❂➊❱✜✼✫❧☎✼❍❳❲❈❊❀✌❋❙✼✕P②❥❙❸✌✼❻⑤✻✼●❋❬❜❹✼●❁◗❥❬❳❲❷▼✼✖❆❩⑧❨❥❊✼●❈❙❥❵⑤✍❳❲❥❊❥❙✼●❋❊❁✌❈❵❥❙❸✍❳❩❥✵❧❻❀✣❈❙❥✩❫✍✼❻❋❊✼➄➍✐✼❍❜❹❥❙✼❍P➀❦⑥❁②❆❉❋✐P◗✼●❋ ❥❙❆ ❷❉✼●❥✧➇❍➈✭✼●❋❙❋❊❆❉❋❻❋❬❳❩❥❙✼❉✸ ✑❅❀✌❋❵❋❙✼➍✐✼❍❜✐❥❊❦⑥❆▼❁ ❜✐❋❊❦❣❥❙✼●❋❊❦⑥❆▼❁✝❱☛❳❲❈✿❫✻❳❲❈❙✼✕P✢❆❉❁✢❥❊❸✌❋❙✼❘✼➆❜❹❆❉❁✍P◗❦❣❥❊❦⑥❆▼❁✌❈ ✢✱❥❙❸✌✼ ❳❉❜❹❥❙❦❣➎◗❦⑥❥♠❾✱❝⑥✼❘➎❉✼●❝✻❆❲⑧✽❥❙❸✌✼❅❧✖❆▼❈❙❥♠⑩❙❳❉❜❹❥❙❦❣➎❉✼❴❆▼❀✌❥❙⑤✣❀✌❥❴❀✌❁✌❦❣❥❴❈❊❸✌❆❉❀✣❝ P❵❫◗❾✿❝⑦❳❩❋❙❷❉✼❘❋❴❥❙❸✍❳❩❁✖❳✩❷❉❦❣➎▼✼●❁✫❥❊❸✌❋❙✼❘❈❙❸✌❆▼❝ P ✟✡✠ ❂◗❥❙❸✌✼✫❳▼❜✐❥❙❦❣➎◗❦❣❥♠❾✖❝❣✼●➎▼✼●❝✽❆❲⑧❴❥❙❸✣✼✿❈❙✼❍❜❹❆❉❁✍P✧❧☎❆❉❈❊❥♠⑩❙❳❉❜✐❥❊❦⑥➎▼✼✑❀✌❁✣❦⑥❥☛❈❊❸✌❆❉❀✣❝ P☎❫✻✼✿❈❙❧✧❳❲❝❣❝❣✼●❋❴❥❊❸✍❳❲❁④❳✱❷❉❦❣➎▼✼●❁ ❥❙❸✣❋❙✼●❈❊❸✌❆❉❝⑦P ✟☞☛ ❂❉❳❲❁✍P✱➁✻❁✍❳❲❝❣❝❣❾❛❂✕❥❊❸✌✼✑P◗❦✩✓➃✼❘❋❙✼●❁✍❜❹✼✑❫✻✼●❥♠❱⑨✼❘✼●❁✖❥❊❸✌✼✑❳❉❜✐❥❊❦❣➎❛❦❣❥♠❾✿❝❣✼●➎▼✼●❝❣❈❴❆❲⑧✽❥❙❸✣✼●❈❙✼❅❥♠❱⑨❆✩❀✌❁✌❦❣❥❊❈ ❈❙❸✣❆❉❀✌❝⑦P②❫✻✼✖❝⑦❳❲❋❊❷❉✼●❋✿❥❊❸✍❳❲❁✝❳➆❷❉❦❣➎❉✼●❁②❥❙❸✣❋❙✼●❈❊❸✌❆❉❝⑦P ✟☞✌ ✸ ❡❸✌✼❻❫✻✼●❈❙❥❻⑤✻✼●❋✐❜✐✼●❁◗❥❬❳❩❷❉✼✖❆❩⑧✑❋❙✼➄➍❹✼❍❜✐❥❙❦❣❆▼❁✌❈✫❆▼❁ ❥❙❸✣✼➆❜❹❆❉❧✖⑤✣❝⑥✼❘❥❙✼✖❥❊✼●❈❙❥☎❈❙✼●❥☎❱❨❳❩❈ ✫✝✄➐♣❉➈✘⑧➄❆▼❋✲➇✕➈✦✼●❋❊❋❙❆❉❋✕✸✚✑❅❁ ❥❊❸✌✼✧❸✍❳❲❁✻P◗❱☛❋❙❦❣❥❙❥❊✼●❁↕❈❊✼●❥❻❆❉❁✌❝❣❾◗❂➊❥❙❸✌✼ ❋❙✼❘❈❙❀✌❝❣❥✩❱❨❳❩❈✿➋❉➈➛❋❙✼➍✐✼❍❜✐❥❊❦⑥❆▼❁✌❈✩⑧➄❆▼❋✫➇✕➈➛✼●❋❙❋❊❆❉❋❍✸ ✝♠❥✩❈❙❸✌❆▼❀✌❝⑦P✰❫✻✼✫✼❘❧✖⑤✌❸✍❳❩❈❙❦❣➅●✼❍P➆❥❙❸✍❳❩❥✵❥❊❸✌✼❵❋❙✼➄➍✐✼✕❜✐❥❙❦❣❆❉❁ ❥❙❸✣❋❙✼●❈❊❸✌❆❉❝⑦P◗❈✿❱✜✼●❋❊✼❻❆❉❫✌❥✐❳❲❦❣❁✌✼❍P✧❀✣❈❙❦❣❁✌❷✧⑤✍✼❘❋♠⑧➄❆▼❋❙❧✲❳❩❁✍❜✐✼❵❧✖✼❍❳❩❈❙❀✌❋❊✼●❈✩❆❉❁✰❥❊❸✌✼✞✌ ☛✛✙✛✌✤✙☞☛✛✌♠✸ ❞❫✍❆▼❀✌❥✩❸✍❳❲❝➂⑧ ❥❙❸✣✼✫❈❊❀✌❫✌❈❙❥❊❦❣❥❙❀✌❥❊❦⑥❆▼❁④✼❘❋❙❋❙❆▼❋❙❈❵❦❣❁✰❥❙❸✌✼❵❥❙✼❘❈❙❥❙❦❣❁✌❷ ❈❙✼●❥✩❱✜✼●❋❙✼✖P◗❀✣✼✫❥❊❆✖⑧♠❳❲❀✌❝❣❥♠❾✧❈❙✼●❷▼❧✖✼●❁◗❥✐❳❲❥❙❦❣❆▼❁❃❂✽❳❲❁✍P✰❳❩❁ ❳❉P✣P◗❦❣❥❙❦❣❆❉❁✍❳❩❝➃❽▼❀✍❳❩❋❙❥❊✼●❋☛❱⑨✼❘❋❙✼✿P◗❀✌✼✩❥❙❆❵✼●❋❊❋❙❆❉❁✣✼●❆❉❀✌❈✩❳❲❈❊❈❙❦❣❷❉❁✌❧☎✼●❁◗❥❴❆❲⑧➊❥❙❸✌✼✿P◗✼❘❈❙❦❣❋❙✼❍P✧❜●❳❩❥❙✼●❷▼❆❉❋❊❾❛✸❴❏◗❆▼❧✖✼ ❆❲⑧☛❥❙❸✣✼❻❋❙✼●❧✧❳❲❦❣❁✌❦❣❁✌❷✖❦❣❧✧❳❲❷❉✼❘❈☛❱⑨✼●❋❊✼✲❳❩❧❻❫✌❦❣❷❉❀✌❆▼❀✌❈☛✼●➎❉✼❘❁✢❥❙❆✰❸◗❀✌❧✧❳❲❁✌❈✕❂➃❳❩❁✍P②❦❣❁❿❳➞⑧➄✼❘❱ ❜●❳❩❈❙✼●❈❵❥❙❸✌✼ ❁✌✼●❥♠❱✜❆❉❋❊▲✖❧☎❦⑥❈✐❜✐❝⑦❳❲❈❊❈❙❦➂➁✍✼❍P➞❥❙❸✌✼✩❦❣❧✲❳❩❷❉✼✜⑧➄❆▼❋☛❁✌❆✖P◗❦❣❈❬❜✐✼❘❋❙❁✌❦❣❫✌❝❣✼✩❋❙✼❍❳❩❈❙❆▼❁❃✸ ❯➊➎▼✼●❁✰❥❙❸✣❆❉❀✌❷▼❸✲❳❵❈❙✼✕❜✐❆❉❁✍P❛⑩❶❆▼❋❬P◗✼●❋✩➎▼✼●❋❙❈❊❦⑥❆▼❁✧❆❲⑧❴❫✍❳❉❜❊▲▼⑩♠⑤✌❋❙❆▼⑤✍❳❲❷◗❳❲❥❙❦❣❆▼❁❻❱❨❳❩❈☛❀✌❈❙✼❍P✏❂✍❦❣❥✜❦❣❈☛❦⑥❁◗❥❊✼●❋❙✼●❈❊❥❙❦❣❁✌❷ ❥❙❆☎❁✌❆❉❥❊✼✫❥❊❸✍❳❲❥✩❥❊❸✌✼✫❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷❻❥✐❳❲▲▼✼●❈✩❆❉❁✌❝❣❾✰q❉♦➞⑤✍❳❲❈❊❈❙✼●❈✿❥❊❸✌❋❙❆▼❀✌❷❉❸ ❥❙❸✌✼❵❥❙❋✐❳❲❦❣❁✌❦❣❁✌❷✖❈❊✼●❥❍✸✩❭✢✼✱❥❙❸✌❦❣❁✌▲ ❥❙❸✣❦⑥❈✿❜●❳❩❁✰❫✻✼✹❳❩❥❙❥❙❋❊❦❣❫✌❀✌❥❙✼✕P✰❥❊❆✲❥❊❸✌✼❵❝ ❳❩❋❙❷▼✼✫❳❲❧☎❆❉❀✌❁◗❥☛❆❲⑧❴❋❊✼❍P◗❀✌❁✍P✣❳❲❁✍❜✐❾②⑤✌❋❊✼●❈❙✼●❁◗❥✿❦❣❁✰❋❊✼❍❳❲❝❴P✌❳❩❥❬❳✣✸ ❞ ❜✐❆▼❧✖⑤✌❝❣✼●❥❊✼☛❥❙❋❬❳❩❦❣❁✌❦❣❁✌❷✿❈❙✼❘❈❙❈❙❦❣❆▼❁④➜♠q❉♦✩⑤✻❳❲❈❙❈❊✼●❈✑❥❊❸✌❋❙❆▼❀✌❷❉❸☎❥❙❸✌✼❅❥❙❋✐❳❲❦❣❁✌❦❣❁✌❷✿❈❊✼●❥☛⑤✌❝❣❀✌❈✜❥❙✼●❈❊❥❬➝✜❥❬❳❩▲❉✼●❈☛❳❲❫✻❆❉❀✣❥ q✖P✣❳✕❾❛❈✩❆▼❁④❳✧❏✌➏✑♥ ❏✌➑❞❚☛✾✜❈❙❥❬❳❩❥❙❦❣❆❉❁✢➇❵❀✌❈❊❦⑥❁✣❷✖❥❙❸✣✼✹❏✌♥ ✤✧❜✐❆❉❁✌❁✣✼❍❜✐❥❙❦❣❆▼❁✌❦❣❈❙❥✩❈❙❦❣❧❻❀✌❝⑦❳❲❥❊❆❉❋✩➜♠❄❴❆▼❥❙❥❙❆▼❀ ❳❲❁✻P✲✺✏✼✿✾✜❀✌❁❃❂✽➇❍➋▼➠❉➋▼➝●✸ ❞⑧❥❙✼●❋❻❈❊❀✍❜●❜✐✼●❈❊❈♠⑧➄❀✣❝✵❥❊❋❬❳❩❦⑥❁✣❦⑥❁✣❷✍❂➊❥❙❸✣✼✖❁✌✼●❥♠❱✜❆❉❋❊▲✝❱☛❳❲❈❵❦❣❧✖⑤✌❝❣✼●❧☎✼●❁◗❥❙✼❍P ❆❉❁t❳④❜✐❆▼❧✖❧☎✼●❋❬❜✐❦⑦❳❩❝✚❑❅❦❣❷❉❦❣❥✐❳❲❝ ❏◗❦❣❷❉❁✻❳❲❝❴➑➊❋❙❆✌❜✐✼❘❈❙❈❙❆▼❋✫❫✻❆❛❳❩❋❬P②❜✐❆❉❁◗❥✐❳❲❦❣❁✌❦❣❁✌❷✲❳❩❁ ❞❇❡✑❢✑❡ ❑✩❏✌➑❤⑩❙q✥✤▼✾➌❷❉✼●❁✣✼●❋❬❳❩❝⑨⑤✌❀✣❋❙⑤✻❆❉❈❙✼✖❑✩❏✌➑➌❜❊❸✌❦❣⑤ ❱☛❦❣❥❙❸✿❳✜⑤✻✼❍❳❲▲✩⑤✻✼●❋♠⑧➄❆▼❋❙❧✧❳❲❁✍❜✐✼➊❆❩⑧✏➇ ✤✌✸✯✫❇❧✖❦❣❝❣❝⑥❦❣❆❲❁☛❧❻❀✌❝❣❥❊❦⑥⑤✣❝⑥❾▼⑩❙❳❉P✌P❇❆❉⑤✻✼●❋✐❳❲❥❙❦❣❆▼❁✌❈❃⑤✻✼●❋❃❈❊✼❍❜✐❆▼❁✍P✩❆❉❁✿q✬✤✜❫✌❦❣❥ ➡✍❆◗❳❲❥❙❦❣❁✌❷❻⑤✻❆❉❦❣❁◗❥✑❁◗❀✌❧❻❫✻✼●❋❊❈❍✸ ❡❸✌✼❻❑✩❏✌➑t❆▼⑤✍✼●❋✐❳❲❥❊✼●❈✿❳❲❈✵❳☎❜✐❆❉⑤✣❋❙❆✌❜✐✼●❈❊❈❙❆❉❋✱❦⑥❁②❳✖➑❴✾➉❜✐❆▼❁✌❁✌✼❍❜❹❥❙✼❍P②❥❊❆ ❳❻➎◗❦⑦P◗✼●❆✖❜❘❳❲❧✖✼❘❋❬❳✌✸ ❡❸✌✼✫➑❴✾➌⑤✻✼●❋♠⑧❆❉❋❙❧☎❈✑❥❊❸✌✼✫P◗❦❣❷❉❦❣❥❊❦⑥➅✕❳❲❥❙❦❣❆▼❁❃❂◗❫✌❦❣❁✍❳❲❋❊❦❣➅❍❳❲❥❊❦⑥❆▼❁✰❳❲❁✍P✧❈❊✼●❷❉❧☎✼●❁◗❥❬❳❩❥❙❦❣❆❉❁
Figure 5:Atypical data.The network classifies these correctly,even though they are quite unlike any thing in the training set. of the image,while the DSP performs the size-normalization and the classification. The overall throughput of the digit recognizer including image acquisition is 10 to 12 classifications per second and is limited mainly by the normalization step.On normalized digits,the DSP performs more than 30 classifications per second. 6 CONC2 USION Back-propagation learning was successfully applied to a large,real-world task.Our results appear to be at the state of the art in handwritten digit recognition.The network had many connections but relatively few free parameters.The network architecture and the constraints on the weights were designed to incorporate geo- metric knowledge about the task into the system.Because of its architecture,the network could be trained on a low-level representation of data that had minimal preprocessing (as opposed to elaborate feature extraction).Because of the redun- dant nature of the data and because of the constraints imposed on the network,the learning time was relatively short considering the size of the training set.Scaling properties were far better than one would expect just from extrapolating results of back-propagation on smaller,artificial problems.Preliminary results on alphanu- meric characters show that the method can be directly extended to larger tasks. The final network of connections and weights obtained by back-propagation learn- ing was readily implementable on commercial digital signal processing hardware. Throughput rates,from camera to classified image,of more than ten digits per second were obtained. Acknowledgments We thank the US Postal Service and its contractors for providing us with the zip- code database.We thank Henry Baird for useful discussions and for providing the printed-font database. References Bottou,L.-Y.and Le Cun,Y.(1989).SN2:A Simulator for Connectionist Models. Neuristique SA,Paris,France
✂✁☎✄✍✆✞✡✠✁✣✌ ❞❥♠❾◗⑤✌❦⑦❜●❳❩❝❃P✌❳❲❥✐❳✌✸ ❡❸✣✼✿❁✌✼●❥♠❱✜❆❉❋❊▲✲❜✐❝⑦❳❲❈❊❈❙❦➂➁✍✼●❈❅❥❙❸✌✼❘❈❙✼❻❜✐❆❉❋❊❋❙✼❍❜✐❥❊❝❣❾❛❂✣✼●➎❉✼●❁ ❥❙❸✌❆▼❀✌❷❉❸✧❥❙❸✣✼●❾ ❳❲❋❊✼✿❽❉❀✣❦⑥❥❊✼✩❀✌❁✌❝❣❦❣▲❉✼✵❳❩❁◗❾❛❥❊❸✌❦❣❁✌❷❵❦⑥❁✧❥❊❸✌✼✩❥❙❋❬❳❩❦❣❁✌❦❣❁✌❷✫❈❊✼●❥❍✸ ❆❲⑧➊❥❙❸✌✼✩❦❣❧✲❳❩❷❉✼❉❂❩❱☛❸✌❦❣❝⑥✼☛❥❊❸✌✼✿❑✩❏✌➑✈⑤✍✼●❋⑧➄❆❉❋❊❧✖❈✜❥❙❸✌✼✩❈❊❦⑥➅❘✼✐⑩❶❁✣❆❉❋❙❧✧❳❲❝❣❦❣➅❍❳❲❥❊❦❣❆❉❁❻❳❲❁✍P➞❥❙❸✌✼✿❜✐❝⑦❳❲❈❊❈❙❦➂➁➃❜●❳❩❥❙❦❣❆❉❁❤✸ ❡❸✌✼✿❆❲➎▼✼●❋❬❳❩❝❣❝✽❥❙❸✌❋❊❆❉❀✌❷▼❸✌⑤✌❀✌❥❅❆❲⑧❴❥❙❸✌✼❵P◗❦❣❷❉❦❣❥☛❋❙✼❍❜❹❆❉❷❉❁✣❦⑥➅❘✼●❋✩❦⑥❁✻❜✐❝❣❀✍P◗❦❣❁✌❷❵❦⑥❧✧❳❲❷▼✼✵❳❉❜❘❽❉❀✣❦⑥❈❊❦❣❥❙❦❣❆❉❁✧❦❣❈✿➇❍♦❵❥❊❆ ➇ ✤☎❜✐❝⑦❳❲❈❊❈❙❦➂➁➃❜●❳❩❥❙❦❣❆❉❁✌❈✩⑤✻✼●❋✿❈❊✼❍❜✐❆❉❁✻P✢❳❲❁✍P✧❦❣❈✩❝❣❦❣❧✖❦❣❥❙✼❍P☎❧✧❳❲❦❣❁✌❝❣❾❻❫◗❾✲❥❊❸✌✼❻❁✌❆▼❋❙❧✲❳❩❝❣❦⑥➅✕❳❲❥❙❦❣❆▼❁✖❈❙❥❊✼●⑤❃✸✒✑❅❁ ❁✌❆▼❋❙❧✲❳❩❝❣❦⑥➅❘✼❍P❻P◗❦❣❷❉❦❣❥❙❈✕❂◗❥❙❸✌✼❵❑✵❏✣➑✢⑤✻✼●❋♠⑧➄❆▼❋❙❧☎❈☛❧✖❆❉❋❊✼✑❥❊❸✍❳❲❁✰q❉♦❵❜✐❝⑦❳❲❈❊❈❙❦➂➁➃❜●❳❩❥❙❦❣❆❉❁✣❈☛⑤✍✼●❋❅❈❙✼❍❜❹❆❉❁✍P✽✸ ✂ ③✢→➔③✁❵↔➌✉ ➓→➔ ❄✜❳❉❜❊▲▼⑩❶⑤✌❋❊❆❉⑤✍❳❩❷❛❳❩❥❙❦❣❆❉❁❻❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷✿❱☛❳❲❈❇❈❙❀✍❜●❜❹✼●❈❙❈⑧➄❀✌❝❣❝❣❾✲❳❲⑤✣⑤✌❝❣❦⑥✼✕P❻❥❙❆❵❳✿❝⑦❳❲❋❊❷❉✼❉❂▼❋❙✼✕❳❲❝➂⑩❶❱✜❆❉❋❊❝⑦P❵❥❬❳❲❈❊▲✽✸✟✑❅❀✌❋ ❋❙✼❘❈❙❀✌❝❣❥❙❈❵❳❲⑤✌⑤✻✼❍❳❲❋✩❥❊❆✲❫✻✼❻❳❲❥✩❥❙❸✌✼❵❈❙❥✐❳❲❥❊✼❻❆❲⑧⑨❥❊❸✌✼✖❳❲❋❊❥✵❦❣❁✰❸✻❳❲❁✍P◗❱☛❋❊❦❣❥❙❥❙✼❘❁❿P◗❦❣❷▼❦⑥❥❅❋❙✼❍❜❹❆❉❷❉❁✣❦⑥❥❊❦❣❆❉❁❃✸ ❡❸✌✼ ❁✌✼●❥♠❱✜❆❉❋❊▲✢❸✍❳❉P➀❧✲❳❩❁❛❾✢❜✐❆▼❁✌❁✌✼❍❜✐❥❊❦❣❆❉❁✌❈❻❫✌❀✣❥✫❋❊✼●❝⑦❳❲❥❙❦❣➎▼✼●❝❣❾②⑧➄✼●❱ ⑧➄❋❊✼●✼✧⑤✍❳❲❋✐❳❲❧☎✼●❥❙✼●❋❊❈❍✸ ❡❸✣✼✲❁✣✼●❥♠❱⑨❆▼❋❙▲ ❳❲❋✐❜❊❸✌❦❣❥❙✼❍❜✐❥❊❀✌❋❙✼✧❳❲❁✍P➆❥❙❸✌✼❻❜✐❆❉❁✣❈❙❥❙❋✐❳❲❦❣❁◗❥❙❈✿❆▼❁✰❥❙❸✣✼❻❱⑨✼●❦❣❷▼❸❛❥❊❈✑❱✜✼●❋❙✼✖P❼✼●❈❙❦❣❷❉❁✣✼❍P✰❥❊❆✧❦⑥❁✻❜✐❆❉❋❊⑤✍❆▼❋❬❳❲❥❊✼✫❷▼✼●❆❲⑩ ❧✖✼❘❥❙❋❙❦⑦❜✩▲◗❁✌❆❲❱☛❝❣✼❍P❼❷❉✼✿❳❲❫✻❆❉❀✌❥✩❥❊❸✌✼✿❥❬❳❩❈❙▲✰❦❣❁◗❥❙❆❻❥❊❸✌✼❵❈❙❾◗❈❙❥❊✼●❧④✸✩❄❴✼✕❜●❳❲❀✌❈❊✼❻❆❲⑧❴❦❣❥❙❈✩❳❲❋❬❜❊❸✌❦❣❥❊✼❍❜✐❥❙❀✣❋❙✼❉❂❤❥❙❸✌✼ ❁✌✼●❥♠❱✜❆❉❋❊▲❿❜❹❆❉❀✌❝⑦P②❫✻✼✖❥❙❋✐❳❲❦❣❁✌✼❍P②❆▼❁✝❳✧❝❣❆❲❱✜⑩❶❝❣✼●➎▼✼●❝❴❋❊✼●⑤✌❋❙✼❘❈❙✼●❁◗❥❬❳❩❥❙❦❣❆❉❁✈❆❲⑧✵P✌❳❩❥❬❳☎❥❙❸✻❳❲❥✿❸✍❳▼P②❧✖❦❣❁✌❦❣❧✲❳✕❝ ⑤✌❋❊✼●⑤✌❋❙❆✌❜✐✼❘❈❙❈❙❦❣❁✌❷➀➜❶❳❲❈✩❆▼⑤✌⑤✍❆▼❈❙✼❍P ❥❙❆☎✼●❝⑦❳❲❫✻❆❉❋✐❳❲❥❙✼✱⑧➄✼❍❳❩❥❙❀✌❋❊✼✫✼❹➟❼❥❊❋❬❳▼❜✐❥❙❦❣❆❉❁✻➝●✸✿❄❇✼❍❜●❳❲❀✣❈❙✼❻❆❲⑧✜❥❙❸✌✼❵❋❙✼❍P❼❀✌❁◗⑩ P✌❳❩❁❛❥❇❁✍❳❲❥❊❀✌❋❙✼❅❆❲⑧✽❥❙❸✌✼✩P✌❳❲❥✐❳✿❳❲❁✍P❵❫✻✼❍❜●❳❩❀✌❈❙✼❅❆❲⑧❃❥❊❸✌✼✵❜✐❆▼❁✌❈❙❥❊❋❬❳❩❦⑥❁◗❥❊❈⑨❦❣❧✖⑤✻❆❉❈❊✼❍P✱❆❉❁❵❥❙❸✌✼❅❁✌✼●❥♠❱✜❆❉❋❙▲✽❂▼❥❙❸✌✼ ❝❣✼❍❳❲❋❊❁✌❦❣❁✌❷✖❥❊❦❣❧✖✼✩❱❨❳❩❈✩❋❙✼●❝⑦❳❲❥❊❦❣➎❉✼●❝❣❾✧❈❙❸✌❆▼❋❙❥✿❜✐❆▼❁✌❈❙❦⑦P◗✼●❋❊❦❣❁✌❷✧❥❙❸✌✼❻❈❊❦⑥➅❘✼✫❆❩⑧⑨❥❙❸✌✼❻❥❊❋❬❳❩❦⑥❁✣❦⑥❁✣❷❻❈❙✼●❥✕✸✹❏✣❜●❳❲❝❣❦❣❁✌❷ ⑤✌❋❊❆❉⑤✻✼●❋❙❥❊❦⑥✼❘❈✑❱✜✼●❋❙✼❅⑧❶❳❩❋⑨❫✻✼●❥❙❥❊✼●❋✑❥❊❸✍❳❲❁✧❆▼❁✌✼✑❱✜❆❉❀✌❝⑦P➞✼✐➟✌⑤✍✼❍❜❹❥✮➍✐❀✌❈❊❥✜⑧➄❋❙❆▼❧ ✼✐➟✌❥❙❋❬❳❩⑤✍❆▼❝⑦❳❲❥❙❦❣❁✌❷❵❋❙✼❘❈❙❀✌❝❣❥❙❈❅❆❲⑧ ❫✍❳▼❜❙▲▼⑩♠⑤✌❋❙❆▼⑤✍❳❲❷◗❳❲❥❊❦⑥❆▼❁✰❆❉❁②❈❙❧✧❳❲❝❣❝❣✼●❋❍❂✽❳❩❋❙❥❙❦➂➁➃❜❹❦ ❳❩❝❴⑤✌❋❊❆❉❫✌❝❣✼●❧☎❈❍✸❻➑➊❋❙✼❘❝⑥❦❣❧☎❦⑥❁✻❳❲❋❙❾☎❋❙✼●❈❊❀✌❝❣❥❙❈❵❆❉❁✢❳❲❝❣⑤✌❸✍❳❩❁◗❀◗⑩ ❧✖✼❘❋❙❦⑦❜✑❜❊❸✍❳❩❋❬❳❉❜❹❥❙✼●❋❊❈✵❈❊❸✌❆❲❱➌❥❙❸✍❳❩❥☛❥❙❸✌✼✩❧✖✼❘❥❙❸✌❆✌P✖❜❘❳❲❁✧❫✍✼❵P◗❦❣❋❙✼❍❜✐❥❊❝❣❾✖✼✐➟✌❥❙✼❘❁✍P◗✼❍P✧❥❊❆❻❝⑦❳❲❋❙❷▼✼●❋☛❥❬❳❩❈❙▲◗❈❍✸ ❡❸✌✼✩➁✻❁✍❳❲❝❤❁✌✼●❥♠❱✜❆❉❋❙▲☎❆❲⑧⑨❜✐❆▼❁✌❁✌✼❍❜✐❥❊❦❣❆❉❁✌❈✩❳❲❁✍P☎❱✜✼●❦❣❷❉❸◗❥❙❈❅❆❉❫✌❥✐❳❲❦❣❁✌✼❍P☎❫◗❾✖❫✍❳▼❜❊▲❉⑩♠⑤✌❋❊❆❉⑤✍❳❩❷❛❳❲❥❊❦❣❆❉❁✖❝❣✼❍❳❩❋❙❁◗⑩ ❦❣❁✌❷✰❱☛❳❲❈✿❋❊✼❍❳❉P❼❦⑥❝❣❾②❦❣❧✖⑤✌❝❣✼●❧☎✼●❁◗❥❬❳❲❫✣❝⑥✼✱❆❉❁✝❜✐❆▼❧✖❧✖✼❘❋❬❜✐❦⑦❳❲❝❴P◗❦❣❷▼❦⑥❥✐❳❲❝❴❈❊❦❣❷❉❁✍❳❩❝❴⑤✌❋❙❆✌❜✐✼❘❈❙❈❙❦❣❁✌❷✢❸✻❳❲❋❬P❼❱❨❳❩❋❙✼❉✸ ❡❸✌❋❊❆❉❀✌❷▼❸✌⑤✌❀✌❥❻❋✐❳❲❥❙✼❘❈❍❂✜⑧➄❋❊❆❉❧ ❜❘❳❲❧✖✼❘❋❬❳②❥❙❆✝❜❹❝ ❳❩❈❙❈❊❦➫➁✻✼❍P✈❦⑥❧✧❳❲❷▼✼❉❂➊❆❲⑧✿❧✖❆▼❋❙✼☎❥❙❸✍❳❩❁ ❥❊✼●❁ P◗❦❣❷▼❦⑥❥❊❈✫⑤✻✼●❋ ❈❙✼✕❜✐❆❉❁✍P➆❱⑨✼●❋❊✼✿❆❉❫✣❥❬❳❲❦❣❁✌✼✕P✽✸ ✄✆☎✞✝✠✟☛✡✌☞✝☎✠✎✍✄✑✏✠✒✟✔✓✖✕ ❭✢✼✩❥❙❸✍❳❩❁✌▲✧❥❙❸✌✼❵➏✑❏✰➑❤❆❉❈❊❥❬❳❩❝✚❏❼✼●❋❙➎◗❦⑦❜✐✼❻❳❲❁✍P☎❦❣❥❙❈✩❜✐❆❉❁◗❥❙❋✐❳❉❜✐❥❊❆❉❋❊❈✑⑧❆❉❋✑⑤✣❋❙❆❲➎◗❦⑦P◗❦❣❁✌❷✫❀✣❈✑❱☛❦❣❥❙❸✰❥❊❸✌✼✿➅●❦❣⑤◗⑩ ❜✐❆✌P◗✼✫P✣❳❲❥❬❳❩❫✍❳❲❈❊✼❉✸✜❭✢✼✑❥❊❸✍❳❲❁✌▲✰◆❖✼●❁✌❋❙❾✰❄✜❳❲❦❣❋❬P➞⑧➄❆▼❋☛❀✌❈❙✼❹⑧➄❀✌❝✮P❼❦⑥❈✐❜✐❀✌❈❊❈❙❦❣❆❉❁✌❈✩❳❲❁✍P➞⑧➄❆▼❋☛⑤✌❋❙❆❲➎◗❦⑦P◗❦❣❁✌❷❵❥❙❸✌✼ ⑤✌❋❊❦⑥❁◗❥❊✼❍P▼⑩ ⑧➄❆❉❁◗❥✑P✌❳❩❥❬❳❲❫✻❳❲❈❙✼▼✸ ①✘✗✠✙✚✗✔✛✜✗✔✢✤✣✠✗✦✥ ❄❇❆❉❥❙❥❊❆❉❀❃❂✌✺❇✸ ⑩➄✷❻✸❉❳❲❁✍P☎✺✽✼✩✾✜❀✌❁❃❂▼✷✹✸✣➜❬➇✕➋❉➠▼➋❉➝●✸★✧✌✩✫✪✭✬✯✮✰✧✻➨✢✒✁✎✒✱❣➥✍✌❙➩❲➧ ✠❍➩❲➧✳✲⑨➩ ✜ ✜ ☛✵✴ ✌♠➨➄➩ ✜✌➨✖✙✑✌★✶➩✞✷ ☛✸✱✙●✸ ♥❖✼●❀✌❋❙❦❣❈❊❥❙❦⑦❽▼❀✌✼✫❏❞ ❂✌➑✮❳❩❋❙❦❣❈❍❂ ✡✌❋✐❳❲❁✍❜❹✼❉✸
Denker,J.,Schwartz,D.,Wittner,B.,Solla,S.A.,Howard,R.,Jackel,L.,and Hopfield,J.(1987).Large Automatic Learning,Rule Extraction and General- ization.Compler S6:ems,1:877c922. Denker,J.S.,Gardner,W.R.,Graf,H.P.,Henderson,D.,Howard,R.E.,Hubbard, W.,Jackel,L.D.,Baird,H.S.,and Guyon,I.(1989).Neural Network Rec- ognizer for Hand-Written kip Code Digits.In Touretzky,D.,editor,Adnances in Neural oneorma:ion wrocessing S6s:ems,volume 1,pages 323c331,Denver, 1988.Morgan daufmann. Fukushima,d.and Miyake,S.(1982).Neocognitron:A new algorithm for pattern recognition tolerant of deformations and shifts in position.wa::ern Recogni:ion, 15:455c4g9. Le Cun,Y.(1989).Generalization and Network Design Strategies.In Pfeifer,R. Schreter,k.,Fogelman,F.,and Steels,L.,editors,Connec:ionism in werspec- ie,kurich,Switzerland.Elsevier. Le Cun,Y.,Boser,B.,Denker,J.S.,Henderson,D.,Howard,R.E.,Hubbard,W. and Jackel,L.D.(1990).Back-Propagation Applied to Handwritten kipcode Recognition.Neural Compu:a:ion,1(4). Le Cun,Y.,Denker,J.S.,and Solla,S.(1990).Optimal Brain Damage.In Touret- zky,D.,editor,Adnances in Neural oneorma:ion wrocessing 56:ems,volume 2, (Denver,1989).Morgan daufman. Patarnello,S.and Carnevali,P.(1987).Learning Networks of Neurons with Boolean Logic.Europh6sics Le::ers,4(4):503c508. Rumelhart,D.E.,Hinton,G.E.,and Williams,R.J.(198g).Learning internal representations by error propagation.In warallel dis:riSu:ed processing:Explo- ra:ions in he micros:ruc:ure ce cogni:ion,volume I,pages 318c3g2.Bradford Books,Cambridge,MA. Tishby,N.,Levin,E.,and Solla,S.A.(1989).Consistent Inference of Probabilities in Layered Networks:Predictions and Generalization.In wroceedings oe:he on:erna:ional Join:Coneerence on Neural Ne:fforks,Washington DC. Wang,C.H.and Srihari,S.N.(1988).A Framework for Object Recognition in a Visually Complex Environment and its Application to Locating Address Blocks on Mail Pieces.on:erna:ional Journal oe Compu:er Vision,2:125
❑❅✼●❁✌▲▼✼●❋❍❂❇■✍✸❣❂⑨❏✌❜❊❸◗❱☛❳❲❋❙❥❊➅❉❂⑨❑❵✸❣❂➊❭↕❦❣❥❙❥❊❁✌✼●❋✕❂⑨❄❅✸⑥❂✜❏◗❆❉❝❣❝⑦❳✌❂➊❏✽✸ ❞ ✸⑥❂❴◆❖❆❲❱❨❳❩❋❬P✽❂❴❚✱✸❣❂➊■❛❳❉❜❊▲▼✼●❝♠❂⑨✺❴✸❣❂❴❳❩❁✍P ◆❖❆❉⑤◗➁✍✼❘❝ P✏❂✌■✍✸✻➜✐➇❍➋▼➠❉♣❉➝❘✸❴✺❃❳❩❋❙❷❉✼ ❞❀✌❥❙❆▼❧✲❳❲❥❊❦⑦❜☛✺✽✼❍❳❩❋❙❁✌❦❣❁✌❷✻❂✍❚☛❀✣❝⑥✼✩❯❤➟✌❥❙❋✐❳❉❜✐❥❊❦⑥❆▼❁✰❳❲❁✍P ➤✼●❁✣✼●❋❬❳❩❝➫⑩ ❦❣➅❍❳❩❥❙❦❣❆❉❁❃✸ ✲⑨➩✒❅➦ ✱ ☛ ✁ ✧✄✂✍✙✛✌ ☛✒✙●❂❃➇✥✢➐➠▼♣❉♣✆☎✌➋✥✤✬✤✌✸ ❑❅✼●❁✌▲▼✼●❋❍❂❲■✻✸❉❏✽✸❣❂ ➤❳❲❋✐P◗❁✌✼●❋✕❂❲❭✸❲❚✿✸❣❂ ➤❋❬❳⑧❬❂❲◆✿✸❉➑➊✸❣❂❲◆☛✼●❁✻P◗✼●❋❙❈❊❆❉❁❃❂◗❑❵✸❣❂ ◆❖❆❲❱❨❳❩❋❬P✽❂❲❚✱✸ ❯❖✸⑥❂❲◆❖❀✌❫✌❫✍❳❩❋❬P✽❂ ❭✸❣❂✽■◗❳❉❜❊▲❉✼❘❝❶❂➊✺❴✸❤❑✫✸❣❂❃❄✜❳❲❦❣❋✐P✽❂✮◆✱✸❃❏✽✸❣❂❃❳❩❁✍P ➤❀✌❾▼❆❉❁❃❂ ✝●✸✚➜❬➇❍➋▼➠❉➋▼➝●✸❻♥☛✼●❀✣❋❬❳❲❝☛♥☛✼❘❥♠❱⑨❆▼❋❙▲✢❚☛✼❍❜❊⑩ ❆▼❷❉❁✌❦❣➅●✼●❋✜⑧❆❉❋✑◆✑❳❩❁✍P▼⑩♠❭✢❋❙❦❣❥❙❥❊✼●❁✞✝✻❦❣⑤✰✾✜❆✌P◗✼✿❑❅❦⑥❷▼❦❣❥❙❈❍✸ ✝♠❁ ❡❆❉❀✣❋❙✼●❥❊➅●▲◗❾❛❂✌❑❵✸❣❂◗✼❍P◗❦❣❥❙❆▼❋❍❂✌✮✫✷✠✟❲➥ ✜ ✴✛☛ ✙ ➨✢✜ ✩☛ ✎◗➧❬➥ ✱☛✡✜✠❍➩❩➧✒✹➥ ✌❶➨ ➩ ✜✌☞⑨➧✐➩ ✴✛☛ ✙✑✙●➨✢✜ ✕ ✧✄✂✍✙✛✌ ☛✒✙●❂✽➎▼❆❉❝❣❀✌❧☎✼✫➇❉❂✻⑤✍❳❩❷❉✼●❈✿q✬✤❉q✆☎✌q❉q✣➇❉❂✍❑❅✼●❁◗➎▼✼●❋❍❂ ➇✕➋❉➠❉➠✣✸❛❺➆❆❉❋❙❷◗❳❲❁✎✍✩❳❲❀◗⑧❧✲❳❲❁✣❁❃✸ ✡✣❀✌▲◗❀✌❈❙❸✌❦❣❧✧❳✌❂✏✍❵✸✍❳❩❁✍P✧❺✧❦❣❾❛❳❩▲❉✼❉❂◗❏✽✸✽➜✐➇❍➋▼➠✥✤❉➝❘✸✮♥❖✼●❆✌❜✐❆❉❷▼❁✌❦❣❥❙❋❊❆❉❁ ✢ ❞ ❁✌✼❘❱ ❳❩❝⑥❷▼❆❉❋❊❦⑥❥❊❸✌❧ ⑧➄❆▼❋⑨⑤✻❳❲❥❙❥❊✼●❋❙❁ ❋❊✼❍❜✐❆❉❷▼❁✌❦❣❥❙❦❣❆❉❁✱❥❙❆❉❝❣✼●❋✐❳❲❁◗❥✽❆❲⑧✽P◗✼✐⑧➄❆▼❋❙❧✲❳❩❥❙❦❣❆❉❁✣❈➃❳❩❁✍P✩❈❙❸✣❦➫⑧❥❙❈❃❦❣❁✩⑤✻❆❉❈❙❦❣❥❊❦⑥❆▼❁❃✸✑☞❨➥ ✌ ✌ ☛❘➧✜✓✒✤☛✵✴✐➩✕ ✜✣➨ ✌❶➨ ➩ ✜✌❂ ➇ ✫✏✢ ✪ ✫✬✫✆☎✪ ✄❉➋✣✸ ✺✽✼❻✾✜❀✌❁❃❂✣✷❻✸❃➜✐➇❍➋▼➠❉➋❉➝❘✸ ➤✼●❁✣✼●❋❬❳❩❝⑥❦❣➅❍❳❩❥❙❦❣❆❉❁②❳❲❁✍P✰♥❖✼●❥♠❱✜❆❉❋❙▲✰❑❅✼●❈❊❦⑥❷▼❁❿❏❼❥❙❋❬❳❩❥❙✼●❷▼❦❣✼●❈❍✸ ✝♠❁④➑✮⑧➄✼●❦➂⑧➄✼●❋✕❂➃❚✱✸❣❂ ❏✌❜❊❸✌❋❊✼●❥❙✼❘❋❍❂✑✝➊✸⑥❂ ✡✌❆▼❷❉✼●❝❣❧✧❳❲❁❃❂☞✡✜✸⑥❂✻❳❲❁✍P ❏◗❥❙✼●✼❘❝⑥❈✕❂✽✺❴✸❣❂✌✼❍P❼❦⑥❥❊❆❉❋❊❈❍❂ ✲⑨➩ ✜ ✜ ☛✵✴ ✌♠➨➄➩ ✜✌➨✖✙✒ ➨ ✜✎☞☛❘➧✛✙♠➦ ☛✵✴✕✔ ✌♠➨✖✟ ☛●❂✗✝✻❀✌❋❊❦⑦❜❙❸❤❂✍❏◗❱☛❦❣❥❙➅❘✼●❋❙❝⑦❳❲❁✻P✽✸✍❯➊❝❣❈❊✼●➎◗❦⑥✼❘❋❍✸ ✺✽✼✿✾✜❀✌❁❤❂▼✷✹✸❣❂✌❄❴❆▼❈❙✼●❋✕❂✍❄❅✸❣❂✌❑✑✼❘❁✌▲❉✼●❋✕❂✌■✻✸✍❏✽✸❣❂◗◆☛✼●❁✍P❼✼●❋❙❈❊❆❉❁❃❂✽❑❵✸❣❂❛◆❖❆❲❱❨❳❩❋❬P✽❂✌❚✱✸✌❯☛✸❣❂✌◆☛❀✌❫✌❫✻❳❲❋❬P✏❂◗❭✸❣❂ ❳❩❁✍P✧■❛❳▼❜❊▲❉✼●❝♠❂✽✺❴✸✽❑❵✸✽➜✐➇❍➋❉➋▼♦❉➝❘✸✵❄✜❳❉❜❊▲▼⑩ ➑✚❋❙❆❉⑤✻❳❲❷❛❳❩❥❙❦❣❆❉❁ ❞⑤✌⑤✌❝❣❦❣✼❍P✧❥❙❆✧◆✑❳❩❁✍P◗❱☛❋❙❦❣❥❊❥❙✼●❁✘✝✻❦❣⑤➃❜✐❆✌P◗✼ ❚❖✼❍❜✐❆❉❷▼❁✌❦❣❥❙❦❣❆❉❁❤✸ ✩☛ ✎◗➧❬➥ ✱☛✲⑨➩✒❅➦ ✎✘✌❙➥✍✌♠➨➄➩ ✜✌❂❤➇❲➜☎✪❛➝●✸ ✺✽✼✩✾✜❀✌❁❃❂❩✷✹✸❣❂◗❑✑✼❘❁✌▲❉✼●❋✕❂❉■✻✸✌❏✽✸❣❂❉❳❲❁✻P✖❏◗❆▼❝⑥❝⑦❳✣❂ ❏✏✸✌➜✐➇❍➋❉➋▼♦❉➝❘✸✟✑❅⑤✌❥❊❦⑥❧✧❳❲❝✌❄❇❋❬❳❲❦❣❁✖❑✩❳❲❧✧❳❲❷▼✼❉✸ ✝♠❁ ❡❆❉❀✌❋❊✼●❥♠⑩ ➅●▲◗❾◗❂❛❑❵✸❣❂❲✼❍P❼❦⑥❥❊❆❉❋✕❂✌✮ ✷✙✟ ➥ ✜ ✴✛☛✛✙✩➨✢✜ ✩☛ ✎◗➧❬➥ ✱✚✡✜✠❍➩❩➧✒✹➥ ✌❶➨ ➩ ✜✛☞⑨➧❬➩✞✴✛☛✛✙✛✙●➨✢✜✣✕✫✧✄✂✍✙✛✌ ☛✒✙●❂❉➎▼❆❉❝❣❀✌❧☎✼ ✤✣❂ ➜♠❑✑✼❘❁❛➎▼✼●❋❍❂❤➇❍➋▼➠❉➋❉➝❘✸✌❺✧❆❉❋❊❷❛❳❩❁✞✍✩❳❲❀◗⑧❧✲❳❲❁❤✸ ➑➊❳❲❥✐❳❲❋❙❁✣✼●❝❣❝⑥❆✻❂❍❏✽✸❲❳❩❁✍P✿✾☛❳❲❋❊❁✌✼●➎❉❳❩❝⑥❦♠❂✕➑➊✸❲➜❬➇✕➋❉➠▼♣❉➝●✸✽✺✏✼❍❳❲❋❊❁✌❦❣❁✌❷✵♥❖✼●❥♠❱⑨❆▼❋❙▲◗❈➊❆❲⑧✍♥☛✼❘❀✌❋❙❆▼❁✌❈➊❱☛❦⑥❥❊❸✿❄❴❆◗❆❉❝❣✼❍❳❩❁ ✺✏❆❉❷❉❦⑦❜❲✸✢✜✔✎◗➧❬➩❹➦ ✗✣✂✍✙❹➨ ✴✛✙✓✤✚☛ ✌ ✌ ☛●➧✙❹❂✬✪✌➜☎✪❛➝ ✢ ✫▼♦❉q✥☎✦✫❉♦▼➠✌✸ ❚☛❀✣❧✖✼●❝❣❸✍❳❩❋❙❥❍❂❤❑✫✸➊❯☛✸❣❂❴◆☛❦❣❁◗❥❙❆▼❁❃❂ ➤ ✸❃❯❖✸⑥❂❴❳❩❁✍P✢❭❦⑥❝❣❝❣❦⑦❳❲❧❻❈✕❂➃❚✱✸❃■✻✸✜➜❬➇✕➋❉➠ ✄▼➝●✸✰✺✏✼❍❳❲❋❊❁✌❦❣❁✌❷✰❦❣❁◗❥❙✼●❋❊❁✍❳❲❝ ❋❊✼●⑤✌❋❙✼❘❈❙✼●❁◗❥❬❳❩❥❙❦❣❆❉❁✣❈☛❫❛❾✱✼●❋❙❋❊❆❉❋✜⑤✌❋❙❆▼⑤✍❳❲❷◗❳❲❥❙❦❣❆▼❁❃✸ ✝♠❁✦☞❨➥❩➧❬➥✖✱ ✱☛✸✱ ✷❲➨ ✙✛✌♠➧●➨★✧ ✎✘✌ ☛ ✷✜➦✻➧❬➩✞✴✑☛ ✙✛✙●➨✢✜✣✕✖✬✩✜✁➦ ✱❣➩✆✔ ➧✐➥✍✌♠➨➄➩ ✜✣✙❵➨✢✜ ✌ ✗ ☛✟✒❵➨ ✴❘➧❬➩ ✙✑✌♠➧✎✌✴ ✌ ✎◗➧✑☛❵➩✏✠ ✴✐➩✛✕ ✜✌➨ ✌♠➨➄➩ ✜✌❂✽➎❉❆▼❝❣❀✌❧✖✼✡✝●❂✍⑤✻❳❲❷❉✼❘❈➙q✣➇❍➠✥☎❼q ✄✬✤✌✸✌❄❇❋❬❳▼P▼⑧➄❆❉❋✐P ❄❇❆◗❆❉▲◗❈❍❂✻✾☛❳❲❧❻❫✌❋❊❦ P❼❷❉✼❉❂❉❺❞ ✸ ❡❦❣❈❙❸◗❫◗❾❛❂◗♥✿✸❣❂◗✺✽✼●➎◗❦❣❁❃❂✌❯☛✸❣❂✌❳❩❁✍P✖❏◗❆▼❝❣❝ ❳✣❂❉❏✽✸ ❞ ✸✌➜✐➇❍➋❉➠▼➋❉➝❘✸✮✾✜❆▼❁✌❈❙❦❣❈❙❥❊✼●❁◗❥✤✝♠❁◗⑧➄✼●❋❊✼●❁✍❜✐✼✱❆❲⑧✚➑✚❋❙❆❉❫✻❳❲❫✌❦❣❝❣❦❣❥❙❦❣✼●❈ ❦❣❁✝✺❃❳✕❾▼✼●❋❙✼✕P✝♥☛✼❘❥♠❱⑨❆▼❋❙▲◗❈ ✢✧➑➊❋❙✼✕P◗❦⑦❜✐❥❙❦❣❆❉❁✣❈✹❳❩❁✍P ➤✼❘❁✌✼●❋❬❳❩❝❣❦⑥➅✕❳❲❥❙❦❣❆▼❁❃✸ ✝♠❁✪☞⑨➧✐➩ ✴✛☛✛☛ ✷❲➨✢✜ ✕☞✙✰➩✏✠ ✌ ✗ ☛ ✡✜✣✌ ☛❘➧✜✍➥ ✌❶➨ ➩ ✜✍➥ ✱✬✫◗➩❲➨✢✜ ✌ ✲⑨➩ ✜✠☞☛❘➧✑☛ ✜ ✴✛☛✫➩ ✜ ✩☛ ✎◗➧❬➥✖✱✑✩☛✛✌✏✓⑨➩❩➧✮✭☞✙●❂◗❭✝❳❩❈❙❸✌❦❣❁✌❷▼❥❙❆❉❁✖❑✩✾✩✸ ❭✝❳❲❁✣❷✍❂✍✾✩✸✻◆✿✸✽❳❲❁✻P✰❏◗❋❙❦❣❸✍❳❩❋❙❦♠❂✽❏✽✸✍♥✱✸❤➜❬➇❍➋▼➠❉➠▼➝●✸ ❞ ✡✌❋✐❳❲❧✖✼❘❱⑨❆▼❋❙▲❻⑧❆❉❋ ✑❅❫❉➍✐✼✕❜✐❥✿❚☛✼❍❜✐❆▼❷❉❁✌❦❣❥❊❦⑥❆▼❁✰❦❣❁✰❳ ✯❅❦⑥❈❊❀✍❳❲❝❣❝❣❾✵✾✜❆▼❧✖⑤✌❝❣✼✐➟✿❯➊❁◗➎◗❦⑥❋❊❆❉❁✌❧☎✼●❁◗❥❃❳❲❁✍P➙❦⑥❥❊❈ ❞⑤✌⑤✌❝❣❦⑦❜●❳❩❥❙❦❣❆❉❁✩❥❙❆✿✺✏❆❼❜❘❳❲❥❙❦❣❁✌❷ ❞P✌P◗❋❊✼●❈❙❈☛❄❴❝❣❆✌❜❊▲◗❈ ❆▼❁➆❺❳❲❦❣❝➃➑✚❦⑥✼✕❜✐✼●❈❍✸✩✡✜✣✌ ☛❘➧✜✍➥ ✌❶➨ ➩ ✜✍➥ ✱✬✫◗➩ ✎◗➧✜✍➥ ✱✮➩✠ ✲⑨➩ ✒✑➦ ✎✘✌ ☛❘➧✎✰❃➨✖✙❹➨ ➩ ✜✌❂✏✤ ✢⑥➇ ✤✥✫✣✸