handwritten Digit Recognition with a Back-Propagation Network Y. Le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel AT&T Bell Laboratories, Holmdel, N. J. 07733 ABSTRACT We present an application of back-propagation networks to hand- written digit recognition. Minimal preprocessing of the data was required, but architecture of the network was highly constrained and specifically designed for the task. The input of the network consists of normalized images of isolated digits. The method has 1% error rate and about a 9% reject rate on zipcode digits provided by the U.S. Postal Service. 1 INTRODUCTION The main point of this paper is to show that large back-propagation (BP) net- works can be applied to real image-recognition problems without a large, complex preprocessing stage requiring detailed engineering. Unlike most previous work on the subject (Denker et al., 1989), the learning network is directly fed with images, rather than feature vectors, thus demonstrating the ability of BP networks to deal with large amounts of low level information. Previous work performed on simple digit images (Le Cun, 1989) showed that the architecture of the network strongly influences the network's generalization ability. Good generalization can only be obtained by designing a network architecture that contains a certain amount of a priori knowledge about the problem. The basic de- sign principle is to minimize the number of free parameters that must be determined by the learning algorithm, without overly reducing the computational power of the network. This principle increases the probability of correct generalization because￾✂✁☎✄✝✆☎✞✠✟☛✡✌☞✍☞✏✎✑✄ ✒✓✡✕✔✖✡✌☞✠✗✘✎✚✙✜✛✜✔✖✄✢✡✌☞✣✡✌✛✖✄ ✞✠✡✌☞✣✤ ✁ ✥✦✁✧✙✩★✫✪✍✬✭✟✮✛✩✯✰✁✱✔✲✁✧☞✣✡✌✛✖✄ ✳✴✎✵☞✣✞✶✛☎✟☛★ ✷✹✸✻✺✽✼✿✾✜❀✌❁❃❂✌❄❅✸✍❄❇❆❉❈❊✼●❋❍❂✻■✍✸✌❏✏✸✍❑❅✼●❁✌▲▼✼●❋❍❂✻❑✫✸✌◆❖✼●❁✍P◗✼❘❋❙❈❙❆▼❁❃❂ ❚✱✸✌❯☛✸✍◆❖❆❲❱❨❳❩❋❬P✽❂◗❭❪✸✌◆☛❀✌❫✌❫✻❳❲❋❬P✏❂✍❳❲❁✻P✖✺❴✸✻❑❵✸◗■❛❳❉❜❊▲▼✼●❝ ❞❴❡✑❢❅❡ ❄❇✼●❝❣❝✮✺❤❳❲❫✻❆❉❋✐❳❲❥❙❆▼❋❙❦❣✼●❈✕❂✌◆☛❆❉❝❣❧✧P◗✼●❝♠❂❛♥✱✸◗■✍✸✻♦❉♣❉♣▼q❉q rts✈✉❨✇②①tr✈③④✇ ❭✢✼✿⑤✌❋❙✼❘❈❙✼●❁◗❥✫❳❩❁④❳❲⑤✌⑤✣❝⑥❦⑦❜●❳❩❥❙❦❣❆❉❁☎❆❲⑧⑨❫✻❳❉❜❊▲▼⑩❶⑤✌❋❊❆❉⑤✍❳❩❷❛❳❩❥❙❦❣❆❉❁☎❁✌✼●❥♠❱✜❆❉❋❙▲◗❈✩❥❙❆☎❸✍❳❲❁✻P▼⑩ ❱☛❋❙❦❣❥❙❥❊✼●❁✝P◗❦❣❷❉❦❣❥✩❋❙✼❍❜❹❆❉❷❉❁✣❦⑥❥❊❦❣❆❉❁❃✸❻❺✧❦❣❁✌❦❣❧✲❳❩❝❃⑤✌❋❊✼●⑤✌❋❊❆❼❜❹✼●❈❙❈❊❦⑥❁✣❷④❆❩⑧❨❥❊❸✌✼✖P✌❳❲❥✐❳✧❱☛❳❲❈ ❋❙✼❍❽▼❀✌❦❣❋❙✼✕P✽❂❴❫✌❀✣❥✹❳❩❋❬❜❊❸✌❦❣❥❙✼❍❜❹❥❙❀✌❋❊✼✰❆❲⑧✑❥❊❸✌✼✖❁✌✼❘❥♠❱⑨❆▼❋❙▲✢❱☛❳❲❈✿❸✌❦❣❷▼❸✌❝❣❾❿❜✐❆▼❁✌❈❙❥❊❋❬❳❩❦⑥❁✣✼❍P ❳❲❁✍P➀❈❙⑤✻✼❍❜✐❦➂➁➃❜●❳❩❝⑥❝❣❾✢P◗✼●❈❊❦⑥❷▼❁✌✼❍P❿⑧➄❆❉❋❵❥❙❸✌✼☎❥❬❳❩❈❙▲✽✸ ❡❸✌✼✖❦❣❁✌⑤✣❀✌❥✿❆❲⑧✑❥❊❸✌✼✖❁✣✼●❥♠❱⑨❆▼❋❙▲ ❜✐❆❉❁✌❈❊❦❣❈❙❥❙❈❵❆❲⑧✜❁✌❆❉❋❊❧✲❳❲❝❣❦❣➅●✼❍P➆❦⑥❧✧❳❲❷▼✼●❈✑❆❩⑧❨❦❣❈❙❆▼❝⑦❳❲❥❙✼✕P✰P◗❦❣❷❉❦❣❥❙❈✕✸ ❡❸✌✼❻❧✖✼●❥❊❸✌❆✌P✧❸✍❳❩❈ ➇❍➈➉✼❘❋❙❋❙❆▼❋➊❋❬❳❲❥❊✼✑❳❲❁✍P❵❳❲❫✻❆❉❀✣❥✮❳✩➋❉➈➌❋❊✼➄➍✐✼❍❜✐❥❇❋❬❳❲❥❊✼☛❆❉❁✿➅❘❦⑥⑤✽❜✐❆✌P◗✼✑P◗❦❣❷▼❦⑥❥❊❈➊⑤✌❋❙❆❲➎◗❦⑦P◗✼❍P ❫❛❾☎❥❙❸✌✼❵➏✿✸➐❏✽✸✌➑❤❆❉❈❊❥❬❳❲❝❃❏❼✼●❋❙➎◗❦⑦❜✐✼❉✸ ➒ ➓❛➔✇④①✈→✝➣↕↔➌③④✇➓→➔ ❡❸✌✼✰❧✧❳❲❦❣❁✢⑤✻❆❉❦❣❁◗❥✫❆❩⑧➙❥❊❸✌❦❣❈❻⑤✍❳❲⑤✻✼●❋☎❦⑥❈❻❥❊❆✢❈❙❸✌❆❲❱➛❥❙❸✻❳❲❥❻❝⑦❳❲❋❊❷❉✼✧❫✍❳❉❜❊▲▼⑩♠⑤✌❋❙❆▼⑤✍❳❲❷◗❳❲❥❙❦❣❆▼❁↕➜❶❄✜➑❴➝➞❁✌✼●❥♠⑩ ❱✜❆❉❋❙▲◗❈✩❜●❳❲❁✰❫✻✼✫❳❩⑤✌⑤✌❝❣❦❣✼❍P☎❥❙❆✖❋❊✼❍❳❲❝❤❦❣❧✲❳❲❷▼✼✐⑩♠❋❙✼❍❜✐❆▼❷❉❁✌❦❣❥❊❦⑥❆▼❁✖⑤✌❋❊❆❉❫✌❝❣✼●❧☎❈⑨❱☛❦❣❥❊❸✌❆❉❀✌❥✩❳❵❝⑦❳❲❋❙❷▼✼❉❂✍❜❹❆❉❧✖⑤✣❝⑥✼❹➟ ⑤✌❋❊✼●⑤✌❋❙❆✌❜✐✼❘❈❙❈❙❦❣❁✌❷②❈❙❥✐❳❲❷▼✼❻❋❙✼❍❽▼❀✌❦❣❋❙❦❣❁✌❷✧P◗✼●❥✐❳❲❦❣❝❣✼❍P✰✼❘❁✌❷❉❦❣❁✌✼●✼❘❋❙❦❣❁✌❷✍✸❻➏❖❁✌❝❣❦⑥▲▼✼✫❧☎❆❉❈❊❥✑⑤✌❋❊✼●➎◗❦⑥❆▼❀✌❈✩❱⑨❆▼❋❙▲✰❆▼❁ ❥❙❸✣✼✿❈❙❀✌❫❉➍❹✼❍❜✐❥❵➜❶❑❅✼●❁✌▲▼✼●❋✑✼❘❥✑❳❲❝♠✸❣❂✽➇❍➋❉➠▼➋❉➝❘❂◗❥❙❸✌✼✿❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷❻❁✣✼●❥♠❱⑨❆▼❋❙▲☎❦⑥❈✑P❼❦⑥❋❊✼❍❜✐❥❊❝⑥❾☎⑧➄✼✕P✖❱☛❦❣❥❊❸✲❦❣❧✧❳❲❷❉✼❘❈❍❂ ❋❬❳❩❥❙❸✌✼❘❋✑❥❙❸✻❳❲❁☎⑧➄✼❍❳❩❥❙❀✌❋❊✼✿➎❉✼❍❜❹❥❙❆❉❋❊❈❍❂✣❥❙❸◗❀✌❈✵P❼✼●❧✖❆▼❁✌❈❙❥❊❋❬❳❲❥❊❦❣❁✌❷✫❥❊❸✌✼✿❳❲❫✌❦❣❝❣❦❣❥♠❾❻❆❲⑧❴❄⑨➑t❁✣✼●❥♠❱⑨❆▼❋❙▲◗❈☛❥❙❆✖P❼✼❍❳❲❝ ❱☛❦❣❥❙❸✧❝⑦❳❲❋❊❷❉✼✵❳❩❧✖❆▼❀✌❁◗❥❙❈✜❆❲⑧✮❝❣❆❲❱↕❝❣✼●➎▼✼●❝✽❦❣❁◗⑧➄❆▼❋❙❧✲❳❩❥❙❦❣❆❉❁❤✸ ➑➊❋❊✼●➎◗❦⑥❆▼❀✌❈✿❱✜❆❉❋❊▲✰⑤✍✼❘❋♠⑧➄❆▼❋❙❧✖✼✕P✰❆▼❁④❈❊❦⑥❧☎⑤✌❝❣✼✫P◗❦❣❷❉❦❣❥❅❦⑥❧✧❳❲❷▼✼●❈✿➜♠✺✽✼❻✾✜❀✌❁❃❂✜➇❍➋❉➠▼➋❉➝❖❈❙❸✌❆❲❱✜✼❍P②❥❙❸✻❳❲❥✩❥❙❸✌✼ ❳❲❋✐❜❊❸✌❦❣❥❙✼❍❜✐❥❊❀✌❋❙✼✱❆❲⑧✮❥❊❸✌✼✿❁✌✼❘❥♠❱⑨❆▼❋❙▲❻❈❙❥❊❋❙❆▼❁✌❷❉❝❣❾✖❦❣❁◗➡✻❀✌✼●❁✍❜✐✼❘❈✑❥❙❸✣✼✵❁✣✼●❥♠❱⑨❆▼❋❙▲✽➢ ❈☛❷❉✼●❁✣✼●❋❬❳❩❝⑥❦❣➅❍❳❩❥❙❦❣❆❉❁✧❳❲❫✌❦❣❝❣❦❣❥♠❾❛✸ ➤❆◗❆✌P❵❷❉✼●❁✣✼●❋❬❳❩❝⑥❦❣➅❍❳❩❥❙❦❣❆❉❁✧❜●❳❲❁☎❆❉❁✌❝❣❾❵❫✍✼❅❆❉❫✌❥✐❳❲❦❣❁✌✼❍P➞❫◗❾✹P◗✼❘❈❙❦❣❷❉❁✌❦❣❁✌❷❻❳✩❁✌✼❘❥♠❱⑨❆▼❋❙▲✖❳❲❋✐❜❊❸✌❦❣❥❙✼❍❜✐❥❊❀✌❋❙✼✱❥❙❸✍❳❩❥ ❜✐❆▼❁❛❥✐❳❲❦❣❁✌❈☛❳✿❜✐✼●❋❊❥❬❳❩❦⑥❁✧❳❲❧☎❆❉❀✌❁◗❥➊❆❲⑧❨➥❅➦✻➧●➨➄➩❲➧❘➨✖▲◗❁✌❆❲❱☛❝❣✼❍P◗❷▼✼✑❳❲❫✻❆❉❀✌❥✜❥❙❸✣✼✑⑤✌❋❊❆❉❫✌❝❣✼●❧②✸ ❡❸✌✼❅❫✍❳❲❈❊❦⑦❜✑P◗✼✐⑩ ❈❙❦❣❷▼❁✿⑤✌❋❙❦❣❁✍❜❹❦⑥⑤✣❝⑥✼✜❦❣❈❃❥❊❆✑❧✖❦❣❁✌❦❣❧☎❦➫➅●✼➊❥❙❸✣✼⑨❁◗❀✌❧❻❫✻✼●❋❃❆❩⑧✌⑧➄❋❊✼●✼⑨⑤✻❳❲❋❬❳❩❧✖✼●❥❊✼●❋❙❈➊❥❊❸✍❳❲❥❤❧❻❀✌❈❙❥❤❫✍✼☛P◗✼●❥❊✼●❋❙❧☎❦⑥❁✣✼❍P ❫◗❾❻❥❙❸✌✼✩❝❣✼❍❳❩❋❙❁✌❦❣❁✌❷❻❳❲❝❣❷❉❆▼❋❙❦❣❥❙❸✣❧④❂✕❱☛❦❣❥❙❸✌❆▼❀✌❥⑨❆❲➎▼✼●❋❙❝❣❾❵❋❙✼❍P❼❀✍❜✐❦❣❁✌❷❻❥❙❸✣✼➙❜❹❆❉❧✖⑤✣❀✌❥❬❳❩❥❙❦❣❆❉❁✍❳❩❝✌⑤✻❆❲❱⑨✼●❋☛❆❩⑧❃❥❙❸✌✼ ❁✌✼●❥♠❱✜❆❉❋❊▲✽✸ ❡❸✌❦❣❈✑⑤✌❋❊❦❣❁✍❜✐❦❣⑤✌❝❣✼✿❦❣❁✍❜✐❋❊✼❍❳❲❈❊✼●❈✿❥❙❸✣✼✿⑤✌❋❙❆▼❫✍❳❲❫✣❦⑥❝❣❦❣❥♠❾✖❆❩⑧⑨❜✐❆❉❋❊❋❙✼❍❜✐❥✱❷❉✼●❁✌✼❘❋❬❳❲❝❣❦❣➅❍❳❩❥❙❦❣❆❉❁✧❫✻✼❍❜●❳❲❀✣❈❙✼