ovof 76 6419-2o87250八 qoa0ǒ L21白 44151 53 Figure 1:Examples of original zipcodes from the testing set. it results in a specialized network architecture that has a reduced entropy(Denker et al.,1987;Patarnello and Carnevali,1987;Tishby,Levin and Solla,1989;Le Cun, 1989).On the other hand,some effort must be devoted to designing appropriate constraints into the architecture. 2 ZIPCODE RECOGNITION The handwritten digit-recognition application was chosen because it is a relatively simple machine vision task:the input consists of black or white pixels,the digits are usually well-separated from the background,and there are only ten output categories.Yet the problem deals with objects in a real two-dimensional space and the mapping from image space to category space has both considerable regularity and considerable complexity.The problem has added attraction because it is of great practical value. The database used to train and test the network is a superset of the one used in the work reported last year (Denker et al.,1989).We emphasize that the method of solution reported here relies more heavily on automatic learning,and much less on hand-designed preprocessing. The database consists of 9298 segmented numerals digitized from handwritten zip- codes that appeared on real U.S.Mail passing through the Buffalo,N.Y.post office. Examples of such images are shown in figure 1.The digits were written by many different people,using a great variety of sizes,writing styles and instruments,with widely varying levels of care.This was supplemented by a set of 3349 printed dig- its coming from 35 different fonts.The training set consisted of 7291 handwritten digits plus 2549 printed digits.The remaining 2007 handwritten and 700 printed digits were used as the test set.The printed fonts in the test set were different from the printed fonts in the training set.One important feature of this database,which￾✂✁☎✄✝✆✟✞✡✠☞☛✍✌✑❯❤➟✍❳❩❧✖⑤✌❝❣✼●❈✜❆❲⑧➊❆❉❋❙❦❣❷▼❦⑥❁✻❳❲❝✻➅●❦❣⑤➃❜✐❆✌P◗✼❘❈☛⑧➄❋❊❆❉❧✠❥❊❸✌✼✿❥❙✼❘❈❙❥❙❦❣❁✌❷❻❈❊✼●❥❍✸ ❦❣❥⑨❋❊✼●❈❙❀✣❝⑥❥❊❈✩❦⑥❁✧❳❵❈❙⑤✻✼❍❜✐❦⑦❳❲❝❣❦❣➅●✼❍P☎❁✌✼❘❥♠❱⑨❆▼❋❙▲✲❳❩❋❬❜❊❸✌❦❣❥❙✼✕❜✐❥❙❀✌❋❊✼✫❥❊❸✍❳❲❥☛❸✻❳❲❈✑❳✱❋❙✼❍P◗❀✻❜✐✼❍P ✼●❁◗❥❙❋❙❆▼⑤◗❾✰➜❶❑❅✼●❁✣▲❉✼●❋ ✼●❥☛❳❲❝♠✸❣❂◗➇❍➋❉➠▼♣✏✎❉➑➊❳❩❥❬❳❲❋❊❁✌✼●❝❣❝❣❆✫❳❲❁✻P❻✾☛❳❲❋❙❁✣✼●➎❉❳❲❝❣❦♠❂◗➇❍➋❉➠▼♣✏✎ ❡❦❣❈❙❸◗❫◗❾❛❂❉✺✏✼●➎◗❦❣❁✹❳❩❁✍P❻❏◗❆❉❝❣❝⑦❳✌❂✣➇❍➋❉➠▼➋✏✎❉✺✏✼✑✾✜❀✌❁❃❂ ➇❍➋▼➠❉➋▼➝●✸✒✑❅❁④❥❊❸✌✼❻❆❉❥❊❸✌✼●❋✿❸✍❳❩❁✍P✽❂❤❈❙❆▼❧✖✼✿✼✔✓✽❆❉❋❊❥✿❧❻❀✌❈❊❥✵❫✻✼✖P◗✼●➎▼❆❉❥❊✼❍P②❥❙❆✰P◗✼●❈❊❦❣❷❉❁✌❦❣❁✌❷✰❳❲⑤✣⑤✌❋❙❆▼⑤✌❋❙❦⑦❳❲❥❊✼ ❜✐❆▼❁✌❈❙❥❊❋❬❳❲❦❣❁◗❥❙❈❅❦❣❁❛❥❊❆✫❥❊❸✌✼✿❳❲❋✐❜❙❸✣❦⑥❥❊✼❍❜✐❥❊❀✌❋❙✼❉✸ ✕ ✖✵➓✘✗ ③✢→✝➣✚✙ ①✛✙ ③✢→✛✜➔➓✇➓→➔ ❡❸✌✼✿❸✻❳❲❁✍P◗❱☛❋❊❦❣❥❙❥❙✼❘❁➆P❼❦⑥❷▼❦❣❥♠⑩♠❋❙✼❍❜✐❆▼❷❉❁✌❦❣❥❊❦⑥❆▼❁✲❳❩⑤✌⑤✌❝❣❦⑦❜●❳❲❥❊❦⑥❆▼❁❻❱❨❳❩❈✑❜❊❸✌❆❉❈❊✼●❁✰❫✻✼❍❜●❳❲❀✣❈❙✼❵❦⑥❥☛❦❣❈✑❳❵❋❊✼●❝⑦❳❲❥❙❦❣➎▼✼●❝❣❾ ❈❙❦❣❧☎⑤✌❝❣✼✵❧✧❳❉❜❊❸✌❦❣❁✌✼✿➎◗❦❣❈❙❦❣❆▼❁✰❥❬❳❩❈❙▲✣✢✩❥❙❸✌✼❵❦❣❁✌⑤✌❀✌❥✿❜✐❆▼❁✌❈❙❦❣❈❊❥❙❈✿❆❩⑧❨❫✌❝⑦❳▼❜❙▲✧❆▼❋✵❱☛❸✣❦⑥❥❊✼✫⑤✣❦➫➟✌✼●❝❣❈✕❂✽❥❙❸✌✼✖P❼❦⑥❷▼❦❣❥❙❈ ❳❲❋❊✼④❀✣❈❙❀✍❳❩❝⑥❝❣❾✈❱⑨✼●❝❣❝➂⑩♠❈❙✼●⑤✍❳❩❋❬❳❩❥❙✼❍P ⑧➄❋❙❆▼❧ ❥❊❸✌✼②❫✍❳❉❜❊▲◗❷❉❋❊❆❉❀✌❁✍P✏❂✑❳❲❁✍P ❥❙❸✌✼●❋❊✼✝❳❲❋❊✼④❆▼❁✌❝❣❾✝❥❙✼❘❁ ❆▼❀✌❥❙⑤✌❀✣❥ ❜●❳❩❥❙✼●❷▼❆❉❋❙❦❣✼●❈✕✸➊✷⑨✼❘❥⑨❥❊❸✌✼✩⑤✌❋❙❆▼❫✌❝❣✼●❧✶P◗✼❍❳❩❝❣❈☛❱☛❦⑥❥❊❸✖❆▼❫❉➍✐✼❍❜✐❥❊❈☛❦⑥❁✧❳✿❋❊✼❍❳❲❝✽❥♠❱✜❆❲⑩❙P◗❦❣❧✖✼❘❁✌❈❙❦❣❆❉❁✻❳❲❝✌❈❊⑤✍❳❉❜✐✼✿❳❩❁✍P ❥❙❸✣✼✫❧✧❳❲⑤✌⑤✌❦❣❁✌❷✱⑧➄❋❊❆❉❧ ❦❣❧✲❳❩❷❉✼✑❈❊⑤✍❳❉❜❹✼❻❥❙❆✲❜❘❳❲❥❙✼❘❷❉❆❉❋❊❾✰❈❙⑤✻❳❉❜✐✼❵❸✍❳❲❈✩❫✻❆❉❥❊❸④❜✐❆❉❁✣❈❙❦⑦P◗✼●❋✐❳❲❫✌❝❣✼❵❋❙✼●❷▼❀✌❝⑦❳❲❋❙❦❣❥♠❾ ❳❲❁✻P✝❜✐❆▼❁✌❈❙❦⑦P◗✼●❋✐❳❲❫✌❝❣✼✧❜✐❆❉❧☎⑤✌❝❣✼✐➟✌❦⑥❥♠❾◗✸ ❡❸✌✼✧⑤✌❋❊❆❉❫✌❝❣✼●❧✦❸✍❳❩❈✹❳▼P✌P◗✼❍Pt❳❲❥❊❥❙❋✐❳❉❜✐❥❊❦⑥❆▼❁✝❫✻✼❍❜●❳❩❀✌❈❙✼✧❦❣❥✫❦❣❈❵❆❲⑧ ❷❉❋❊✼❍❳❲❥☛⑤✣❋❬❳❉❜❹❥❙❦⑦❜●❳❲❝❤➎❉❳❲❝❣❀✌✼❉✸ ❡❸✌✼✖P✌❳❩❥❬❳❩❫✍❳❲❈❊✼✫❀✌❈❊✼❍P✢❥❊❆✧❥❙❋❬❳❩❦❣❁❿❳❩❁✍P✰❥❊✼●❈❙❥❵❥❙❸✣✼✫❁✌✼❘❥♠❱⑨❆▼❋❙▲②❦⑥❈✿❳☎❈❙❀✣⑤✍✼●❋❊❈❙✼●❥❻❆❩⑧❨❥❊❸✌✼❻❆❉❁✣✼✫❀✌❈❊✼❍P②❦❣❁ ❥❙❸✣✼✫❱✜❆❉❋❊▲✖❋❙✼❘⑤✍❆▼❋❙❥❙✼✕P②❝ ❳❩❈❙❥❅❾❉✼❍❳❩❋✿➜❶❑❅✼●❁✣▲❉✼●❋❅✼●❥✿❳❲❝♠✸❣❂✽➇❍➋❉➠▼➋❉➝❘✸☛❭✢✼✿✼●❧☎⑤✌❸✍❳❲❈❊❦❣➅●✼✩❥❙❸✍❳❩❥✑❥❙❸✣✼✫❧☎✼●❥❙❸✌❆✌P ❆❲⑧❴❈❊❆❉❝❣❀✌❥❙❦❣❆▼❁✲❋❊✼●⑤✻❆❉❋❙❥❊✼❍P ❸✌✼●❋❙✼✱❋❙✼●❝❣❦❣✼●❈✑❧☎❆❉❋❊✼✵❸✣✼❍❳✕➎◗❦⑥❝❣❾❻❆▼❁➆❳❩❀✌❥❙❆▼❧✲❳❩❥❙❦⑦❜☛❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷✻❂✍❳❲❁✻P✖❧❻❀✻❜❙❸☎❝❣✼●❈❙❈ ❆❉❁✧❸✍❳❩❁✍P▼⑩❙P◗✼●❈❊❦⑥❷▼❁✌✼❍P☎⑤✌❋❊✼●⑤✌❋❙❆✌❜✐✼❘❈❙❈❙❦❣❁✌❷✻✸ ❡❸✌✼✿P✌❳❩❥❬❳❲❫✻❳❲❈❙✼✿❜✐❆▼❁✌❈❙❦❣❈❊❥❙❈✑❆❩⑧✮➋✥✤▼➋❉➠✱❈❙✼●❷▼❧✖✼●❁◗❥❙✼✕P✖❁◗❀✌❧☎✼●❋❬❳❩❝❣❈⑨P◗❦❣❷❉❦❣❥❙❦❣➅●✼✕P❵⑧➄❋❊❆❉❧✠❸✍❳❩❁✍P◗❱☛❋❊❦⑥❥❊❥❙✼●❁✧➅●❦❣⑤◗⑩ ❜✐❆✌P◗✼●❈✜❥❙❸✻❳❲❥❴❳❲⑤✌⑤✻✼❍❳❩❋❙✼❍P➞❆❉❁❵❋❙✼❍❳❩❝➃➏✱✸ ❏✏✸❉❺✰❳❲❦❣❝◗⑤✍❳❩❈❙❈❙❦❣❁✌❷✩❥❊❸✌❋❙❆▼❀✌❷❉❸❻❥❊❸✌✼✑❄❴❀✦✓✏❳❩❝⑥❆✻❂❉♥✿✸✷❻✸❲⑤✍❆▼❈❙❥❴❆★✧✲❜✐✼❉✸ ❯❤➟✍❳❲❧☎⑤✌❝❣✼●❈☛❆❲⑧✜❈❙❀✍❜❊❸✢❦❣❧✲❳❩❷❉✼●❈✑❳❩❋❙✼❵❈❙❸✌❆❲❱☛❁②❦❣❁✧➁✍❷❉❀✣❋❙✼✧➇❉✸ ❡❸✌✼❻P◗❦❣❷❉❦❣❥❙❈❅❱⑨✼❘❋❙✼❵❱☛❋❙❦❣❥❙❥❊✼●❁④❫◗❾✰❧✧❳❲❁◗❾ P◗❦✩✓✽✼●❋❙✼●❁◗❥❅⑤✍✼●❆▼⑤✌❝❣✼❉❂◗❀✌❈❊❦⑥❁✣❷✹❳❵❷▼❋❙✼❍❳❩❥☛➎❉❳❲❋❙❦❣✼●❥♠❾❻❆❩⑧✮❈❊❦⑥➅❘✼●❈❍❂✣❱☛❋❙❦❣❥❙❦❣❁✌❷❵❈❙❥♠❾◗❝❣✼●❈✑❳❲❁✻P❻❦❣❁✌❈❙❥❊❋❙❀✌❧☎✼●❁◗❥❙❈✕❂◗❱☛❦⑥❥❊❸ ❱☛❦⑦P◗✼●❝❣❾✖➎❉❳❩❋❙❾◗❦❣❁✌❷❵❝⑥✼❘➎❉✼●❝❣❈✑❆❩⑧✚❜●❳❩❋❙✼❉✸ ❡❸✌❦❣❈☛❱❨❳❩❈✑❈❙❀✣⑤✌⑤✌❝❣✼●❧✖✼❘❁❛❥❊✼❍P☎❫◗❾✲❳❵❈❙✼❘❥✵❆❩⑧⑨q❉q✡✪◗➋✿⑤✣❋❙❦❣❁◗❥❙✼❍P✰P❼❦⑥❷❩⑩ ❦❣❥❙❈✩❜✐❆❉❧☎❦⑥❁✣❷✿⑧➄❋❊❆❉❧ q✥✫❻P◗❦✩✓✽✼●❋❙✼❘❁❛❥❅⑧➄❆▼❁◗❥❙❈❍✸ ❡❸✌✼✿❥❙❋✐❳❲❦❣❁✌❦❣❁✌❷❻❈❊✼●❥✵❜✐❆▼❁✌❈❙❦❣❈❊❥❙✼❍P ❆❲⑧❨♣✬✤❉➋✣➇✑❸✍❳❩❁✍P◗❱☛❋❊❦⑥❥❊❥❙✼●❁ P◗❦❣❷❉❦❣❥❊❈✵⑤✣❝⑥❀✣❈✭✤✬✫✡✪◗➋❻⑤✌❋❊❦⑥❁◗❥❊✼❍P④P❼❦⑥❷▼❦❣❥❙❈❍✸ ❡❸✌✼❵❋❙✼❘❧✲❳❲❦❣❁✌❦❣❁✌❷✮✤▼♦❉♦▼♣✖❸✻❳❲❁✍P◗❱☛❋❊❦❣❥❙❥❙✼❘❁❿❳❩❁✍P④♣▼♦❉♦☎⑤✌❋❊❦⑥❁◗❥❊✼❍P P◗❦❣❷❉❦❣❥❊❈✮❱✜✼●❋❊✼✑❀✌❈❊✼❍P✖❳❩❈❴❥❙❸✌✼❅❥❙✼●❈❊❥⑨❈❊✼●❥❍✸ ❡❸✣✼✑⑤✌❋❊❦⑥❁◗❥❊✼❍P✱⑧➄❆❉❁◗❥❊❈❴❦❣❁✫❥❊❸✌✼✑❥❊✼●❈❙❥✜❈❙✼●❥✜❱✜✼●❋❙✼✩P◗❦✩✓➃✼●❋❊✼●❁◗❥❴⑧❋❙❆❉❧ ❥❙❸✣✼✿⑤✌❋❙❦❣❁◗❥❙✼✕P❻⑧❆❉❁◗❥❙❈☛❦❣❁✖❥❊❸✌✼✿❥❊❋❬❳❲❦❣❁✌❦❣❁✌❷❵❈❙✼❘❥❍✸✯✑❅❁✌✼✫❦❣❧☎⑤✍❆▼❋❙❥❬❳❩❁◗❥➊⑧➄✼❍❳❩❥❙❀✌❋❊✼✿❆❲⑧➊❥❙❸✌❦❣❈✑P✌❳❩❥❬❳❲❫✻❳❲❈❙✼▼❂✌❱☛❸✌❦⑦❜❊❸