《机器学习 Machine Learning》课程教学资源（书籍文献）Gradient-Based Learning Applied to Document Recognition

PROC. OF THE IEEE, NOVEMBER 1998 Gradient-Based Learning Applied to Document Recognition Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner Abstract I. INTRODUCTION Multilayer Neural Networks trained with the backpropa- gation algorithm constitute the best example of a successful Over the last several years, machine learning techniques, Gradient-Based Learning technique. Given an appropriate particularly when applied to neural networks, have played network architecture, Gradient-Based Learning algorithms can be used to synthesize a complex decision surface that can an increasingly important role in the design of pattern classify high-dimensional patterns such as handwritten char- recognition systems. In fact, it could be argued that the acters, with minimal preprocessing. This paper reviews var availability of learning techniques has been a crucial fac- ious methods applied to handwritten character recognition and compares them on a standard handwritten digit recog- tor in the recent success of pattern recognition applica- nition task. Convolutional Neural Networks, that are specif- tions such as continuous speech recognition and handwrit- ically designed to deal with the variability of 2D shapes, are ing recognition. shown to outperform all other techniques. The main message of this paper is that better pattern Real-life document recognition systems are composed of multiple modules including field extraction, segmenta recognition systems can be built by relying more on auto- tion, recognition, and language modeling. A new learning matic learning, and less on hand-designed heuristics. This paradigm, called Graph Transformer Networks (GTN), al- is made possible by recent progress in machine learning lows such multi-module systems to be trained globally using Gradient-Based methods so as to minimize an overall per and computer technology. Using character recognition as formance meas ure. a case study, we show that hand-crafted feature extrac- Two systems for on-line handwriting recognition are de- tion can be advantageously replaced by carefully designed scribed. Experiments demonstrate the advantage of global learning machines that operate directly on pixel images. training, and the flexibility of Graph Transformer Networks A Graph Transformer Network for reading bank check is Using document understanding as a case study, we show also described. It uses Convolutional Neural Network char- that the traditional way of building recognition systems by acter recognizers combined with global training techniques manually integrating individually designed modules can be to provides record accuracy on business and personal checks. It is deployed commercially and reads several million checks replaced by a unified and well-principled design paradigm, per day. called Graph Transformer Networks, that allows training Keywords- Neural Networks, OCR, Document Recogni all the modules to optimize a global performance criterion. tion, Machine Learning, Gradient-Based Learning, Convo- Since the early days of pattern recognition it has been lutional Neural Networks, Graph Transformer Networks, Fi- known that the variability and richness of natural data, nite State Transducers be it speech, glyphs, or other types of patterns, make it almost impossible to build an accurate recognition system NOMENCLATURE entirely by hand. Consequently, most pattern recognition . GT Graph transformer. systems are built using a combination of automatic learn- . GTN Graph transformer network. ing techniques and hand-crafted algorithms. The usual . HMM Hidden Markov model. method of recognizing individual patterns consists in divid- . HOS Heuristic oversegmentation ing the system into two main modules shown in figure 1. The first module, called the feature extractor, transforms . K-NN K-nearest neighbor. .NN Neural network. the input patterns so that they can be represented by low- . OCR Optical character recognition. dimensional vectors or short strings of symbols that(a)can . PCA Principal component analysis. be easily matched or compared, and (b) are relatively in- . RBF Radial basis function. variant with respect to transformations and distortions of . RS-SVM Reduced-set support vector method. the input patterns that do not change their nature. The . SDNN Space displacement neural network. feature extractor contains most of the prior knowledge and . SVM Support vector method. is rather specific to the task. It is also the focus of most of . TDNN Time delay neural network. the design effort, because it is often entirely hand-crafted. . V-SVM Virtual support vector method. The classifier, on the other hand, is often general-purpose and trainable. One of the main problems with this ap- proach is that the recognition accuracy is largely deter The authors are with the Speech and Image Pro- cessing Services Research Laboratory, AT&T Labs- mined by the ability of the designer to come up with an Research, 100 Schulz Drive Red Bank, NJ 07701. E-mail: appropriate set of features. This turns out to be a daunt- {yann,leonb,yoshua,haffner}@research.att.com. Yoshua Bengio ing task which, unfortunately, must be redone for each new is also with the Departement d'Informatique et de Recherche Operationelle, Universite de Montreal, C.P. 6128 Succ. Centre-Ville, problem. A large amount of the pattern recognition liter- 2920 Chemin de la Tour, Montreal, Quebec, Canada H3C 3J7. ature is devoted to describing and comparing the relative

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✜ ★✩✫✪✭✬✯✮✟✰✲✱✴✳✶✵✶✷✪✹✸✺✰✻✬ ✼✽✰✾✪✭✩✿✱✯✮❀✱❂❁ ❃❄❅❄✯❆❇✮✟✰✻✬ ✳✫❈ ❉❈❋❊❍●❅■✰✲✱✴✳ ❏✰❍❊❑❈▲❁▼✱❅✮◆✳❖✮☛❈P✱ ◗❙❘◆❚❯❚❲❱❨❳✂❩❭❬❯❚✛❪❫❱✺❴❳❛❵✟❚✾❜❭❵◆❝❞❝❞❵✟❬✛❪❇◗✒❵◆❡❞❢✝❬❣❘❤❜❭❳✐❚❦❥◆❧♠❵❦❪❣❘✎❚❦♥❲♦♣❘✎❝rq✖❧♠s✠t❍✉✢❘✎✈✇❚❦❳✐q ①❣②④③⑥⑤✓⑦⑨⑧❶⑩✧⑤✓❷ ❸❺❹✂❻❽❼❿❾➀❻❽➁❞➂➄➃✖➅✛➆♣➃❞❹➇➅✥➁➈❻◆➆➉➃➊❼④➋❫➌➈➅❿➍✐➎➏❼❿➅✥➁➈❾➑➐✐➃❞➒➓➋➔❾→❼↔➣❭❼❿➣✐➃✌↕➇➁➄➙❿➍❛➛✂➅✥➌➈➛➇➁r➜ ➝➁r❼↔❾❽➌❛➐➞➁➈❻➝➌➈➅↔❾→❼↔➣✂➟✾➙❶➌❛➐➇➎➠❼↔❾→❼↔❹❛❼✧➃✛❼❿➣✐➃♣↕✙➃✖➎④❼❫➃➊➡✐➁➈➟❭➛✙❻❽➃❨➌➈➢✎➁➤➎✧❹➇➙❶➙❶➃✖➎✤➎✤➢⑨❹✂❻ ➥➅✥➁➄➒✙❾❽➃r➐✠❼✧➜♠➦✇➁➄➎④➃✖➒✫➧◆➃✖➁➄➅❿➐✂❾➀➐➝ ❼✥➃✖➙↔➣✂➐✂❾➩➨➇❹✐➃➄➫ ➥❾➩➭➄➃r➐✿➁➈➐✺➁➄➛✂➛✂➅✥➌➈➛✂➅↔❾❽➁r❼✥➃ ➐✐➃❶❼④➋✇➌➈➅❿➍✿➁➄➅✥➙↔➣✂❾❽❼✧➃❞➙❿❼❿❹➇➅✥➃➄➯ ➥➅✥➁➄➒✙❾➩➃❞➐➄❼✤➜⑥➦❫➁➄➎✤➃✖➒➲➧◆➃➊➁➄➅↔➐✂❾➀➐➝ ➁➈❻➝➌➈➅↔❾❽❼❿➣✂➟❭➎ ➙↔➁➈➐♣↕✙➃✇❹➇➎✤➃✖➒➳❼✥➌➵➎✤➂➇➐➄❼❿➣✐➃❞➎✧❾➩➸❶➃➏➁✛➙↔➌❛➟❭➛✙❻❽➃✖➡➔➒➇➃✖➙➊❾➑➎✧❾➩➌❛➐♣➎✧❹➇➅✥➢➺➁➄➙↔➃❦❼↔➣✐➁r❼☛➙❶➁➈➐ ➙➊❻➩➁➄➎✤➎✥❾❽➢➺➂➤➣✂❾➝➣❛➜⑥➒✙❾➑➟❙➃r➐➇➎✧❾➩➌❛➐✐➁➈❻r➛➇➁r❼✤❼✥➃✖➅↔➐➇➎✝➎✧❹➇➙↔➣➤➁➄➎❇➣✐➁➈➐➇➒➇➋✛➅↔❾❽❼✤❼✥➃❞➐➳➙↔➣✐➁➄➅✤➜ ➁➄➙❿❼✥➃✖➅❿➎↔➯➄➋➔❾❽❼❿➣✌➟➻❾➑➐✂❾➀➟❙➁➈❻✠➛✂➅✥➃✖➛✂➅✥➌➇➙↔➃✖➎✧➎✧❾➀➐➝ ➫❯➼➵➣✂❾➩➎❯➛➇➁➄➛✙➃❞➅❣➅✧➃✖➭➇❾➩➃➊➋❨➎❇➭✖➁➄➅✧➜ ❾➩➌❛❹➇➎➉➟❙➃❶❼↔➣✐➌✐➒✂➎❨➁➄➛✂➛✙❻➀❾➩➃✖➒✢❼✧➌✶➣✐➁➈➐➇➒➇➋❨➅↔❾→❼✧❼✧➃r➐✢➙↔➣✐➁➄➅✥➁➄➙❿❼✥➃✖➅➉➅✥➃❞➙↔➌➝➐✂❾❽❼❿❾➩➌❛➐ ➁➈➐➇➒✶➙❶➌❛➟❭➛➇➁➄➅✧➃❞➎➏❼↔➣✐➃❞➟✭➌❛➐➽➁❙➎④❼✧➁➈➐➇➒➇➁➄➅❿➒✶➣✐➁➈➐➇➒➇➋❨➅↔❾→❼✧❼✧➃❞➐➽➒✙❾➝❾❽❼❨➅✧➃❞➙↔➌➝➜ ➐✂❾❽❼❿❾➩➌❛➐➉❼✧➁➄➎✤➍☎➫❯➾✇➌❛➐➈➭➄➌❛❻➀❹❛❼❿❾➩➌❛➐✐➁➈❻r➆♣➃❞❹➇➅✥➁➈❻➈➆♣➃❶❼④➋✇➌➈➅❿➍❛➎❶➯✖❼↔➣✐➁r❼❀➁➄➅✥➃✇➎✧➛✙➃✖➙✖❾❽➢✓➜ ❾➑➙↔➁➈❻➑❻➩➂➞➒➇➃✖➎✥❾➝➐✐➃✖➒❙❼✧➌✌➒➇➃➊➁➈❻✙➋➔❾→❼↔➣✌❼↔➣✐➃❨➭✖➁➄➅↔❾➩➁➄↕✙❾➑❻➀❾→❼④➂➤➌➄➢☛➚✠➪✻➎✥➣✐➁➄➛✙➃❞➎↔➯➇➁➄➅✧➃ ➎✧➣✐➌r➋➔➐➻❼✥➌✒➌❛❹❛❼❿➛✙➃❞➅✧➢➺➌❛➅❿➟➶➁➈❻➀❻✎➌✠❼❿➣✐➃❞➅✛❼✧➃❞➙❿➣✂➐✂❾➑➨➇❹✐➃✖➎↔➫ ➹❨➃➊➁➈❻❽➜⑥❻➀❾❽➢➺➃❲➒➇➌✐➙✖❹✂➟✒➃r➐✠❼➘➅✥➃✖➙↔➌➝➐✂❾→❼↔❾❽➌❛➐▲➎✤➂✐➎④❼✧➃r➟❙➎❤➁➄➅✥➃❲➙↔➌✐➟❙➛☎➌➈➎④➃❞➒ ➌➄➢✒➟❙❹✂❻→❼↔❾➩➛✙❻➩➃❖➟❙➌✐➒✙❹✂❻➩➃✖➎✿❾➑➐➇➙➊❻➀❹➇➒✙❾➀➐➝➷➴➃❞❻➑➒❍➃➊➡❛❼✥➅✥➁➄➙❿❼↔❾❽➌❛➐✎➯➉➎✤➃➝➟❙➃r➐✠❼✥➁r➜ ❼❿❾➩➌❛➐✎➯❇➅✥➃❞➙↔➌➝➐✂❾❽❼❿❾➩➌❛➐✎➯❀➁➈➐➇➒➷❻❽➁➈➐➝❹✐➁➝➃➻➟❙➌➇➒➇➃❞❻➀❾➑➐➝ ➫✶➬▲➐✐➃✖➋✭❻➩➃➊➁➄➅↔➐✂❾➀➐➝ ➛➇➁➄➅✥➁➄➒✙❾➝➟➽➯☛➙↔➁➈❻➀❻❽➃❞➒ ➥➅✥➁➄➛✙➣➲➼☛➅✥➁➈➐➇➎④➢➺➌➈➅↔➟❙➃✖➅♣➆➉➃➊❼④➋❫➌➈➅❿➍✐➎❙➮➥➼❫➆➤➱❿➯☛➁➈❻→➜ ❻➩➌❞➋❨➎❯➎✧❹➇➙↔➣❭➟✒❹✂❻❽❼❿❾❽➜⑥➟❙➌✐➒✙❹✂❻➩➃❫➎✤➂✐➎④❼✧➃❞➟❭➎❀❼✧➌➤↕✙➃✛❼❿➅✧➁➈❾➀➐✐➃✖➒ ➝❻❽➌❛↕➇➁➈❻➑❻➩➂➤❹➇➎✥❾➑➐➝ ➥➅✥➁➄➒✙❾❽➃r➐✠❼✧➜♠➦✇➁➄➎④➃✖➒➲➟❙➃➊❼❿➣✐➌➇➒✂➎♣➎✤➌✶➁➄➎➉❼✧➌✿➟➻❾➀➐✂❾➑➟➻❾➩➸❶➃✌➁➈➐✫➌r➭➄➃❞➅✥➁➈❻➑❻❀➛✙➃✖➅✧➜ ➢➺➌➈➅↔➟❙➁➈➐➇➙↔➃➳➟❙➃✖➁➄➎✧❹➇➅✥➃➄➫ ➼➏➋❫➌➽➎✤➂✐➎④❼✧➃r➟❙➎➉➢➺➌➈➅➳➌❛➐❛➜⑥❻➀❾➀➐✐➃❭➣✐➁➈➐➇➒➇➋❨➅↔❾→❼↔❾➑➐➝ ➅✥➃✖➙❶➌➝➐✂❾❽❼❿❾➩➌❛➐✢➁➄➅✧➃❭➒➇➃❶➜ ➎✤➙➊➅❿❾➑↕✙➃❞➒◆➫✒✃❀➡➇➛✙➃❞➅↔❾➑➟❙➃❞➐➄❼✥➎➉➒➇➃❞➟❙➌❛➐➇➎➠❼❿➅✥➁r❼✧➃✌❼↔➣✐➃✒➁➄➒➇➭❞➁➈➐✠❼✥➁➝➃✒➌➄➢ ➝❻❽➌➈↕➇➁➈❻ ❼✥➅✥➁➈❾➀➐✂❾➑➐➝ ➯✖➁➈➐➇➒➳❼❿➣✐➃✇❐✙➃➊➡✂❾➑↕✙❾➑❻➀❾→❼④➂➉➌➄➢ ➥➅✧➁➄➛✙➣➳➼☛➅✥➁➈➐➇➎④➢➺➌❛➅❿➟❙➃❞➅❀➆♣➃❶❼④➋❫➌❛➅✥➍✐➎↔➫ ➬ ➥➅✥➁➄➛✙➣➽➼☛➅✥➁➈➐➇➎④➢➺➌❛➅❿➟❙➃❞➅✇➆♣➃❶❼④➋✇➌➈➅❿➍✒➢➺➌➈➅➉➅✧➃✖➁➄➒✙❾➑➐➝ ↕➇➁➈➐➇➍➻➙❿➣✐➃❞➙❿➍➽❾➩➎ ➁➈❻➑➎④➌✒➒➇➃❞➎✤➙❶➅↔❾➑↕✙➃✖➒◆➫➏❒♠❼➔❹➇➎④➃❞➎✛➾✇➌❛➐➈➭➄➌❛❻➀❹❛❼❿❾➩➌❛➐✐➁➈❻➇➆♣➃❞❹➇➅✥➁➈❻◆➆➉➃➊❼④➋❫➌➈➅❿➍❭➙↔➣✐➁➄➅✤➜ ➁➄➙❿❼✥➃✖➅➳➅✧➃❞➙↔➌➝➐✂❾➩➸❶➃❞➅✥➎➉➙↔➌❛➟➞↕✙❾➀➐✐➃✖➒✢➋➔❾→❼↔➣ ➝❻❽➌❛↕➇➁➈❻◆❼✥➅✥➁➈❾➀➐✂❾➑➐➝ ❼✧➃❞➙↔➣✂➐✂❾➩➨➇❹✐➃✖➎ ❼✧➌➔➛✂➅✥➌r➭✂❾➩➒➇➃❞➎☛➅✧➃❞➙↔➌➈➅❿➒➳➁➄➙❶➙➊❹➇➅✥➁➄➙↔➂➔➌✐➐➤↕✙❹➇➎✧❾➀➐✐➃✖➎✤➎✝➁➈➐➇➒✌➛✙➃✖➅❿➎✤➌❛➐✐➁➈❻✠➙↔➣✐➃✖➙❿➍✐➎↔➫ ❒♠❼✇❾➑➎❣➒➇➃✖➛✙❻➩➌❞➂➈➃✖➒❭➙↔➌❛➟➻➟❙➃✖➅❿➙➊❾➩➁➈❻➑❻➩➂➉➁➈➐➇➒❭➅✥➃➊➁➄➒✂➎❣➎④➃✖➭➄➃✖➅✥➁➈❻✂➟➻❾➀❻➑❻➀❾❽➌✐➐➤➙❿➣✐➃❞➙❿➍❛➎ ➛✙➃❞➅➉➒➇➁❞➂✠➫ ❮✇❰✤Ï↔Ð◆Ñ ⑦⑨Ò↔③♠❷ ➆♣➃❞❹➇➅✥➁➈❻✇➆➉➃➊❼④➋❫➌➈➅❿➍✐➎↔➯➏Ó➔➾➵➹➤➯❯➪➉➌✐➙➊❹✂➟❙➃r➐✠❼➻➹❨➃✖➙↔➌➝➐✂❾→➜ ❼❿❾➩➌❛➐✎➯❦❸➲➁➄➙↔➣✂❾➑➐✐➃➽➧◆➃✖➁➄➅❿➐✂❾➀➐➝ ➯ ➥➅✥➁➄➒✙❾➩➃❞➐➄❼✤➜⑥➦❫➁➄➎✤➃✖➒❖➧◆➃✖➁➄➅❿➐✂❾➀➐➝ ➯❯➾✇➌❛➐➈➭➄➌✠➜ ❻➀❹❛❼❿❾➩➌❛➐✐➁➈❻r➆➉➃r❹➇➅✥➁➈❻➄➆♣➃❶❼④➋❫➌❛➅✥➍✐➎↔➯ ➥➅✧➁➄➛✙➣➳➼☛➅✥➁➈➐➇➎④➢➺➌❛➅❿➟❙➃❞➅☛➆➉➃➊❼④➋❫➌➈➅❿➍✐➎↔➯❞Ô❇❾→➜ ➐✂❾❽❼✧➃➞Õ❛❼✧➁r❼✥➃➳➼☛➅✥➁➈➐➇➎✤➒✙❹➇➙❶➃✖➅❿➎↔➫ Ö➻×❀Ø♣Ù✙Ú❀Û◆Ü➈Ý✂Þ☛ß✝à☛Ù á➲â➤ã❲â➳ä✥å➈æ✙ç✿è✧ä❿å➄é☎ê✤ëíì➈ä✥î➻ï➊ä❞ð á➲â➤ã❨ñòâ➳ä❿å➄æ✙ç✢è✧ä❿å➄é✎ê④ëíì❛ä✧î➻ï➊ä✛é✙ï➊è④ó✇ì❛ä✧ô✟ð á✫õ♣ö➲ö÷õ➔ø✓ù✙ù✂ï➊é✫ö➲å➈ä✧ô❛ì✠ú➻î❭ì✂ù✂ï✖ûüð á✫õ➤ý➤þ✶õ➉ï➊ÿ✙ä✥ø➀ê✤è✧ø✁￾➤ì✠ú➈ï✖ä✥ê✧ï✄✂❛î➻ï➊é✐è✥å➄è✧ø➀ì➈é✝ð á✆☎✞✝➠ñ➉ñ✟☎✞✝⑥é✙ï✖å➈ä✧ï❞ê④è✛é✙ï➊ø✠✂➈ç☛✡◆ì➈ä❞ð á✫ñ♣ñ ñ➔ï➊ÿ☎ä✥å➈û☛é✙ï❶è④ó➵ì➈ä✥ô✟ð á❖ý✌☞✎✍ ý♣æ✂è✧ø✁￾➊å➈û✏￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä➵ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é✝ð á✆✓✔☞✎✕✖✓❫ä✧ø➀é✗￾➊ø➑æ☎å➈û✏￾❶ì❛î❭æ◆ì➈é☎ï➊é✐è➔å➄é☎å➈û✙✘✂ê✧ø➀ê✖ð á✆✍✛✚✔✜✢✍➉å➈ù✂ø✓å➄û✏✡✎å➈ê✧ø➀ê✇ëíÿ✙é✗￾❶è✧ø➀ì➈é✝ð á✆✍➉þ☛✝➠þ☛✣➳ö✤✍❨ï✖ù✂ÿ✥￾❶ï✖ù✦✝⑥ê✧ï❶è➔ê✧ÿ✙æ✙æ◆ì➈ä✧è➔ú➈ï✒￾❶è✧ì❛ä➵î➻ï➊è✧ç✙ì✂ù☛ð á➲þ✦✧➳ñ➉ñòþ✂æ☎å✑￾➊ï➞ù✂ø➀ê✧æ✙û✓å✑￾➊ï➊î➻ï➊é✐è❨é✙ï➊ÿ☎ä✥å➈û☛é✙ï❶è④ó➵ì➈ä✥ô✟ð á➲þ☛✣➳ö þ➇ÿ✙æ✙æ◆ì➈ä✧è➔ú➈ï★￾↔è✧ì❛ä✛î❭ï➊è✧ç✙ì✂ù☛ð á✫ã✩✧➳ñ➉ñ ã✛ø➀î➻ï➞ù✂ï➊û✓å✪✘➽é✙ï✖ÿ✙ä✥å➈û☛é✙ï❶è④ó➵ì➈ä✥ô✟ð á✫✣✬✝➠þ☛✣➳ö✭✣➉ø➀ä✤è✥ÿ☎å➄û✝ê✧ÿ✙æ✙æ◆ì➈ä✧è➔ú➈ï★￾↔è✧ì❛ä✛î❭ï➊è✧ç✙ì✂ù☛ð ✮✏✯✪✰✲✱✴✳✒✵✶✯✪✷✹✸✻✺✼✱✴✸✶✰✾✽❀✿ ✵✶✯ ✵✶✯✪✰❂❁✒❃❄✰❅✰❇❆✶✯ ✱✴❈★❉ ❊●❋✛✱✴❍✹✰❏■✥✸✶✷✴❑ ❆❅✰❅✺✶✺✶✿▲❈★❍ ❁✒✰❅✸✻▼✒✿▲❆❇✰❅✺ ◆❀✰❅✺✶✰❇✱✴✸✻❆❖✯ P✦✱✴◗❄✷✹✸❖✱❘✵✻✷✹✸✻❙❯❚ ❱✗✮❳❲❨✮ P☛✱✴◗✪✺✻❑ ◆✏✰❇✺✻✰❇✱✴✸✶❆❖✯❩❚❭❬❇❪✹❪❫❁✒❆❖✯✄✳✪❴▲❵✖❛❜✸✶✿▼❯✰❝◆✏✰❇❉❡❞❢✱✴❈✪❣❄❚✢❤❳✐❫❪❯❥✹❥✴❪★❬✹❦ ❧✗❑●❋✛✱✴✿▲❴✁♠ ♥ ❙✄✱✴❈✪❈❩❚ ❴▲✰❅✷✹❈✄◗❩❚ ❙❯✷✹✺✶✯✄✳♦✱★❚ ✯♦✱❘♣✑❈✪✰❅✸rq✴s❜✸✶✰❅✺✶✰❇✱✴✸✶❆✶✯❩❦ ✱❘✵✻✵❇❦ ❆❅✷✹❋t❦ ✉✈✷✹✺✻✯✒✳♦✱✭❞✇✰❇❈★❍✹✿▲✷ ✿▲✺①✱✴❴▲✺✻✷❝✽❀✿ ✵✻✯②✵✻✯✪✰❭❛④③✰❇❃✪✱✴✸✻✵✻✰❇❋✩✰❅❈❯✵✢❉❩⑤ ❊●❈★⑥✙✷✹✸✶❋✛✱❘✵✻✿▲⑦✒✳★✰⑧✰r✵✢❉✪✰⑨◆❀✰❅❆❖✯✪✰❅✸✻❆❖✯✪✰ ⑩❃✈③✰❅✸❖✱❘✵✻✿▲✷✹❈✪✰❅❴▲❴▲✰✹❚❄❶❳❈★✿▼❯✰❇✸✻✺✶✿ ✵✒③✰❷❉✪✰❹❸t✷✹❈❯✵✶✸✒③✰❺✱✴❴✁❚❄❻❨❦ ■❼❦♦❽★❬❇❾✹❿✛❁✄✳✪❆❅❆❯❦❢❻✇✰❇❈❯✵✻✸✶✰r❑✁➀❨✿▲❴▲❴▲✰✹❚ ❾✹➁✹❾✹❪✛❻✇✯✪✰❅❋✩✿▲❈✛❉✪✰❹❴➂✱➃✮✦✷✹✳✪✸❇❚♦❸t✷✹❈❯✵✶✸✒③✰❺✱✴❴✁❚❄➄❜✳✇③✰❅◗✑✰❅❆✹❚✑❻❢✱✴❈♦✱✴❉♦✱✩➅❳➆❯❻✫➆✹✐✄❥✒❦ ➇✪➈➉➇❿Ú☛Þ☛à✟×❀➊❇ß❀Û✎Þ❢➋í×❀Ú ý♣ú➈ï➊ä❦è✥ç✙ï➔û✓å➈ê✤è❫ê✤ï✖ú➈ï✖ä✥å➈û✦✘➈ï✖å➈ä✥ê✒➌➄î➓å✑￾❿ç☎ø➑é✙ï❨û➀ï✖å➈ä✧é☎ø➑é❼✂➳è✥ï✒￾❿ç✙é✙ø✁➍✐ÿ✙ï✖ê✒➌ æ☎å➈ä✤è✥ø✠￾➊ÿ✙û➀å➈ä✧û✠✘➽ó❨ç✙ï✖é✫å➄æ✙æ✙û➀ø➀ï✖ù✢è✧ì➓é✙ï✖ÿ✙ä❿å➄û✝é✙ï➊è④ó✇ì❛ä✧ô✂ê✒➌➇ç☎årú➈ï➤æ✙û✓å✪✘➈ï❞ù å➄é ø➀é✗￾➊ä✧ï❞å➈ê✧ø➑é❼✂❛û✙✘✲ø➑î➻æ◆ì➈ä✧è✥å➄é✐è❖ä✥ì➈û➀ï ø➀é è✧ç✙ï❑ù✂ï✖ê✧ø✙✂❛é ì➄ë➓æ☎å➄è✤è✧ï✖ä✧é ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é➷ê❺✘➇ê✤è✧ï✖î➓ê➊ð➐➏➠é➷ë⑨å✑￾❶è✒➌❯ø➩è➑￾➊ì➈ÿ✙û✓ù➒✡✎ï✢å➄ä❘✂➈ÿ✙ï❞ù❖è✥ç☎å✠è✒è✧ç✙ï årú✠å➄ø➀û➀å✑✡✙ø➀û➑ø➑è❅✘❖ì➈ë➔û➑ï❞å➄ä✥é✙ø➑é✗✂✫è✧ï★￾❿ç✙é✙ø✁➍❛ÿ☎ï✖ê➞ç✎å➈ê➓✡◆ï➊ï✖é❤å➔￾➊ä✧ÿ✥￾❶ø✓å➄û➏ë⑨å➎￾✹✝ è✧ì❛ä✢ø➀é✻è✥ç✙ï ä✧ï★￾❶ï➊é✐è✺ê✤ÿ✗￾✒￾❶ï❞ê✧ê✢ì➄ë➞æ☎å➄è✤è✥ï➊ä✥é✲ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✻å➈æ✙æ✙û➀ø✠￾✖å♦✝ è✧ø➀ì➈é✎ê❨ê✤ÿ✗￾❿ç✿å➈ê✩￾➊ì➈é✐è✧ø➀é➇ÿ✙ì➈ÿ☎ê✛ê✧æ✎ï✖ï✒￾❿ç✢ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✢å➈é☎ù✢ç☎å➄é☎ù✙ó❨ä✧ø➑è❇✝ ø➀é❼✂➻ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✝ð ã✛ç✙ï✶î➓å➄ø➀é î➻ï✖ê✥ê✥å❄✂➈ï➓ì➈ë❨è✧ç✙ø✓ê❙æ☎å➄æ◆ï➊ä❙ø➀ê✒è✧ç☎å➄è➑✡◆ï❶è✧è✧ï➊ä❭æ☎å➄è✤è✧ï✖ä✧é ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✺ê❺✘➇ê✤è✧ï✖î➓ê✬￾➊å➈é✫✡✎ï➓✡☎ÿ✙ø➑û➑è✬✡☛✘✿ä✧ï✖û✙✘➇ø➀é❼✂➓î➻ì➈ä✥ï➳ì❛é➲å➄ÿ✂è✥ì❄✝ î➓å✠è✥ø✠￾➤û➀ï✖å➈ä✧é✙ø➀é❼✂✥➌✙å➄é☎ù✢û➑ï❞ê✧ê➔ì➈é✿ç☎å➄é☎ù✦✝⑥ù✂ï❞ê✤ø✠✂➈é☎ï✖ù✢ç✙ï➊ÿ✙ä✥ø✓ê④è✥ø✠￾✖ê➊ð➏ã✛ç✙ø✓ê ø✓ê✶î➓å➈ù✙ï➲æ✎ì✐ê✧ê✧ø✠✡✙û➑ï➔✡❩✘➘ä✥ï✒￾➊ï➊é✐è✶æ✙ä✥ì✑✂❛ä✧ï❞ê✧ê➻ø➀é✻î➓å✑￾❿ç✙ø➀é✙ï➲û➀ï✖å➈ä✧é☎ø➑é❼✂ å➄é✎ù➒￾➊ì➈î➻æ✙ÿ✂è✥ï➊ä➤è✥ï✒￾❿ç✙é✙ì❛û➑ì➎✂✑✘❛ð➣→➉ê✧ø➑é❼✂✆￾❿ç☎å➈ä✥å➎￾↔è✥ï➊ä➤ä✧ï★￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é➷å➈ê å✢￾✖å➈ê✧ï❖ê④è✥ÿ☎ù✦✘➎➌❨ó✇ï❖ê✧ç✙ì✠ó▼è✧ç✎å✠è✢ç✎å➄é☎ù☛✝r￾❶ä❿å✠ë➺è✥ï✖ù❑ëíï✖å➄è✧ÿ✙ä✥ï➲ï✄↔✐è✥ä✥å➎￾✹✝ è✧ø➀ì➈é↕￾✖å➄é✫✡◆ï✒å➈ù✂ú✠å➈é❛è❿å❄✂❛ï➊ì➈ÿ✎ê✤û✠✘➽ä✥ï➊æ✙û✓å✑￾➊ï✖ù➐✡☛✘➙￾➊å➈ä✧ï➊ëíÿ✙û➀û✙✘✺ù✂ï✖ê✧ø✠✂➈é✙ï❞ù û➀ï✖å➄ä✥é✙ø➀é❼✂➷î➓å✑￾❿ç✙ø➀é✙ï✖ê❭è✥ç☎å✠è➽ì❛æ✎ï✖ä✥å➄è✧ï➲ù✂ø➀ä✥ï✒￾↔è✥û✙✘❤ì➈é➘æ✙ø✙↔➇ï✖û➔ø➀î➻å✑✂➈ï❞ê➊ð →➉ê✧ø➀é❼✂❺ù✂ì✦￾➊ÿ✙î➻ï➊é✐è❙ÿ✙é✎ù✂ï➊ä❿ê④è❿å➄é☎ù✙ø➑é❼✂❺å➈ê✒å➛￾➊å❛ê✤ï✶ê✤è✧ÿ✎ù✦✘✑➌❦ó➵ï✢ê✧ç✙ì✠ó è✧ç✎å✠è➏è✥ç✙ï❨è✧ä❿å➈ù✙ø➩è✥ø➑ì❛é☎å➄û✙ó✛å✪✘➞ì➈ë✇✡☎ÿ✙ø➑û✓ù✂ø➀é❼✂➞ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é❭ê❺✘✂ê④è✥ï➊î➓ê❷✡☛✘ î➓å➄é➇ÿ☎å➈û➑û✠✘➞ø➀é❛è✥ï✄✂❛ä✥å➄è✧ø➀é❼✂➤ø➑é☎ù✙ø➑ú➇ø✓ù✂ÿ☎å➄û➀û✠✘❭ù✂ï❞ê✤ø✠✂➈é☎ï✖ù❙î❭ì✂ù✂ÿ☎û➑ï❞ê➜￾➊å➈é➝✡✎ï ä✥ï➊æ✙û✓å✑￾➊ï✖ù➣✡☛✘➽å❙ÿ✙é✙ø✙➞☎ï✖ù✿å➄é✎ù➽ó➵ï➊û➀û➟✝⑥æ✙ä✥ø➑é✗￾➊ø➑æ☎û➑ï❞ù✶ù✂ï❞ê✤ø✠✂➈é✢æ☎å➄ä❿å➈ù✂ø✠✂➈î✫➌ ￾➊å➈û➑û➀ï✖ù②➠✎➡❘➢❘➤❼➥➧➦✦➡❺➢❄➨❼➩➭➫❯➯♦➡✹➲➵➳✄➡➝➸➑➳❯➺✶➻➜➯♦➡❺➼♦➩✹➌❣è✧ç☎å➄è➻å➈û➑û➀ì✠ó➔ê✌è✥ä✥å➈ø➑é☎ø➑é❼✂ å➄û➀û✙è✥ç✙ï➉î➻ì✂ù✂ÿ✙û➀ï✖ê➏è✧ì➞ì❛æ✂è✧ø➀î➻ø✙➽✖ï♣å✌✂❛û➑ì➎✡☎å➄û☎æ✎ï✖ä✤ëíì❛ä✧î➓å➄é✥￾❶ï✛￾❶ä✥ø➑è✧ï➊ä✥ø➀ì➈é✝ð þ➇ø➑é✥￾❶ï➓è✥ç✙ï➽ï✖å➈ä✧û✠✘❺ù☎å✪✘➇ê➞ì➄ë❨æ☎å✠è✧è✧ï✖ä✧é ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é❺ø➑è✒ç☎å❛ê➓✡◆ï➊ï➊é ô➇é✙ì✠ó❨é è✧ç☎å➄è➽è✧ç☎ï➲ú✠å➄ä✥ø➀å✑✡✙ø➑û➀ø➑è❅✘❑å➄é✎ù❑ä✥ø✁￾❿ç✙é✙ï✖ê✥ê➓ì➈ë➤é✎å✠è✧ÿ☎ä✥å➈û➔ù✙å✠è❿å❼➌ ✡◆ï✢ø➑è➽ê✤æ◆ï➊ï★￾❿ç✏➌❹✂➈û✠✘➇æ✙ç☎ê✒➌❣ì➈ä❭ì➄è✥ç✙ï➊ä❭è❅✘➇æ✎ï❞ê❭ì➄ë➉æ☎å➄è✤è✥ï➊ä✥é☎ê✄➌❣î➓å➈ô➈ï✿ø➩è å➄û➀î➻ì❛ê✤è❨ø➑î➻æ◆ì❛ê✥ê✤ø✠✡✙û➀ï➤è✧ì➵✡✙ÿ✙ø➀û✓ù✿å➈é✫å✑￾✄￾➊ÿ✙ä❿å✠è✧ï➤ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✢ê❺✘➇ê✤è✧ï✖î ï➊é✐è✥ø➑ä✥ï➊û✠✘✫✡❩✘✫ç✎å➄é☎ù☛ð➉☞✇ì❛é☎ê✤ï★➍✐ÿ✙ï➊é✐è✧û✠✘✑➌✟î➻ì❛ê✤è➤æ☎å✠è✧è✧ï➊ä✥é➲ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é ê❺✘➇ê✤è✧ï✖î➓ê➉å➄ä✥ï✌✡✙ÿ✙ø➀û➑è♣ÿ☎ê✧ø➑é✗✂✶å➣￾❶ì❛î➉✡✙ø➀é☎å➄è✧ø➀ì➈é✫ì➄ë➏å➈ÿ✂è✧ì❛î➻å➄è✧ø✁￾✌û➀ï✖å➄ä✥é✦✝ ø➀é❼✂ è✧ï✒￾❿ç☎é✙ø✠➍✐ÿ✙ï❞ê✫å➄é☎ù✾ç☎å➈é☎ù☛✝r￾❶ä❿å✠ë➺è✧ï❞ù✲å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î➓ê➊ð▼ã✛ç✙ï➷ÿ☎ê✤ÿ✎å➄û î➻ï❶è✥ç✙ì✂ù➞ì➄ë✎ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ø➀é❼✂➔ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û➇æ☎å✠è✧è✧ï➊ä✥é☎ê❜￾❶ì❛é☎ê✤ø✓ê✤è✥ê❇ø➑é❭ù✂ø➑ú➇ø✓ù☛✝ ø➀é❼✂✺è✧ç✙ï➽ê❺✘✂ê④è✥ï➊î❂ø➑é✐è✧ì✺è④ó➵ì✺î➓å➄ø➀é î➻ì✂ù✂ÿ✙û➀ï✖ê✒ê✤ç✙ì✠ó❨é ø➀é➛➞✥✂➈ÿ✙ä✥ï✫➾➈ð ã✛ç✙ï➚➞☎ä❿ê④è➤î➻ì✂ù✂ÿ✙û➀ï✑➌❀￾✖å➄û➀û➑ï❞ù✫è✧ç☎ï❙ëíï❞å✠è✥ÿ✙ä✧ï❭ï❯↔➇è✥ä✥å➎￾↔è✧ì❛ä✒➌◆è✥ä✥å➈é☎ê④ëíì❛ä✧î➓ê è✧ç☎ï♣ø➀é✙æ✙ÿ✂è✛æ☎å➄è✤è✧ï✖ä✧é✎ê➵ê✧ì✌è✥ç☎å✠è➵è✧ç✙ï✒✘➣￾➊å➈é➣✡✎ï➳ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù➵✡☛✘➻û➑ì✠ó✔✝ ù✂ø➀î➻ï➊é☎ê✧ø➑ì❛é☎å➄û➇ú❛ï✒￾↔è✥ì➈ä❿ê✝ì❛ä❯ê✧ç✙ì➈ä✧è❯ê✤è✧ä✥ø➀é❼✂❛ê❯ì➄ë◆ê❇✘➇î➉✡◆ì➈û✓ê❀è✧ç☎å➄è✔➪üå➎➶❳￾➊å➄é ✡◆ï❭ï❞å➈ê✧ø➑û✠✘✫î➓å✠è✴￾❿ç✙ï✖ù❖ì➈ät￾❶ì➈î➻æ☎å➈ä✧ï❞ù❢➌✝å➄é✎ù➹➪➭✡✈➶♣å➈ä✧ï❭ä✥ï➊û✓å✠è✧ø➀ú➈ï✖û✙✘✺ø➀é✦✝ ú✠å➄ä✥ø➀å➈é✐è➉ó❨ø➩è✥ç❖ä✥ï✖ê✧æ✎ï★￾↔è♣è✧ì➽è✥ä✥å➈é☎ê✤ëíì➈ä✥î➻å➄è✧ø➀ì➈é☎ê➉å➄é☎ù➲ù✙ø➀ê✤è✧ì❛ä✤è✥ø➑ì❛é☎ê➔ì➈ë è✧ç☎ï➽ø➑é☎æ✙ÿ✂è✒æ☎å➄è✤è✥ï➊ä✥é☎ê✌è✥ç☎å✠è❙ù✙ì✫é✙ì➄è➉￾❿ç☎å➈é❼✂➈ï➻è✥ç✙ï➊ø➀ä✒é✎å✠è✧ÿ☎ä✧ï❛ð✢ã✛ç✙ï ëíï✖å➄è✧ÿ✙ä✥ï➔ï✄↔✐è✥ä✥å➎￾↔è✥ì➈ä❹￾❶ì❛é✐è✥å➄ø➀é☎ê❣î➻ì✐ê④è❫ì➈ë✟è✧ç✙ï➔æ☎ä✧ø➀ì➈ä➏ô➇é✙ì✠ó❨û➀ï✖ù❼✂➈ï❨å➄é☎ù ø✓ê✇ä❿å✠è✥ç✙ï➊ä➵ê✤æ◆ï✒￾➊ø➟➞✈￾➔è✥ì➞è✧ç☎ï♣è❿å➈ê✧ô✟ð❳➏⑥è✛ø➀ê✛å➈û➀ê✧ì✌è✧ç☎ï➉ëíì✦￾❶ÿ✎ê✇ì➈ë❀î➻ì❛ê✤è✇ì➈ë è✧ç☎ï✒ù✂ï✖ê✧ø✠✂➈é✺ï❯➘✟ì➈ä✧è✒➌✥✡✎ï★￾➊å➄ÿ✎ê✤ï✌ø➑è➉ø✓ê❨ì➄ë➺è✥ï➊é✫ï➊é✐è✥ø➑ä✥ï➊û✠✘✶ç☎å➈é☎ù☛✝r￾❶ä❿å✠ë➺è✥ï✖ù☛ð ã✛ç✙ï➚￾❶û✓å➈ê✥ê✤ø✙➞☎ï➊ä★➌✙ì❛é✺è✧ç✙ï❙ì➄è✧ç☎ï➊ä➉ç✎å➄é☎ù❢➌◆ø✓ê➉ì➈ë➺è✧ï➊é✆✂❛ï➊é✙ï✖ä✥å➈û➟✝⑥æ✙ÿ✙ä✥æ◆ì❛ê✧ï å➄é✎ù è✧ä❿å➄ø➀é☎å❄✡✙û➀ï➈ðòý♣é✙ï➲ì➈ë➤è✧ç✙ï➲î➓å➈ø➑é❍æ✙ä✥ì✑✡☎û➑ï✖î➻ê✶ó❨ø➩è✥ç➘è✧ç✙ø✓ê✿å➄æ✦✝ æ✙ä✥ì❛å➎￾❿ç ø✓ê✶è✧ç☎å➄è✶è✧ç☎ï❖ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✻å➎￾✄￾➊ÿ✙ä✥å➎￾❯✘❑ø➀ê✶û➀å➈ä❺✂❛ï➊û✠✘ ù✂ï➊è✧ï✖ä❇✝ î➻ø➑é☎ï✖ù①✡☛✘❺è✧ç✙ï✿å❄✡✙ø➀û➑ø➑è❅✘ ì➄ë❨è✧ç☎ï✢ù✂ï❞ê✤ø✠✂➈é✙ï✖ä✌è✧ì➛￾❶ì➈î➻ï✢ÿ✙æ ó❨ø➩è✥ç❤å➄é å➄æ☎æ✙ä✧ì❛æ✙ä✥ø➀å➄è✧ï➞ê✧ï❶è➉ì➈ë❯ëíï❞å✠è✥ÿ✙ä✧ï❞ê➊ð➔ã✛ç☎ø➀ê➔è✧ÿ✙ä✥é☎ê➔ì❛ÿ✂è➉è✥ì➣✡◆ï❙å➓ù✙å➈ÿ✙é✐è❇✝ ø➀é❼✂➳è✥å➈ê✧ô➞ó❨ç✙ø✁￾❿ç✏➌➄ÿ☎é✂ëíì➈ä✧è✧ÿ✙é✎å✠è✧ï✖û✙✘➎➌➈î❙ÿ☎ê④è❷✡◆ï❨ä✧ï❞ù✂ì➈é☎ï✇ëíì❛ä❦ï✖å✑￾❿ç❭é✙ï✖ó æ✙ä✥ì✑✡✙û➀ï➊î✺ð✞✕òû✓å➄ä❘✂➈ï✒å➈î❭ì❛ÿ✙é✐è➉ì➄ë❣è✥ç✙ï❭æ☎å✠è✧è✧ï➊ä✥é➲ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✿û➀ø➑è✧ï➊ä❺✝ å✠è✥ÿ✙ä✥ï❙ø✓ê➤ù✂ï➊ú❛ì➄è✧ï❞ù✫è✥ì✿ù✂ï❞ê❺￾➊ä✧ø✠✡✙ø➀é❼✂✺å➄é☎ù➛￾❶ì➈î➻æ☎å➈ä✧ø➀é❼✂✶è✧ç✙ï➻ä✥ï➊û✓å✠è✧ø➀ú➈ï

PROC.OF THE IEEE,NOVEMBER 1998 Class scores manipulate directed graphs.This leads to the concept of trainable Graph Transformer Network (GTN)also intro- TRAINABLE CLASSIFIER MODULE duced in Section IV.Section V describes the now clas- sical method of heuristic over-segmentation for recogniz- Feature vector ing words or other character strings.Discriminative and non-discriminative gradient-based techniques for training a recognizer at the word level without requiring manual FEATURE EXTRACTION MODULE segmentation and labeling are presented in Section VI.Sec- tion VII presents the promising Space-Displacement Neu- Raw input ral Network approach that eliminates the need for seg- mentation heuristics by scanning a recognizer at all pos Fig.1.Traditional pattern recognition is performed with two mod- ules:a fixed feature extractor,and a trainable classifier. sible locations on the input.In section VIIL,it is shown that trainable Graph Transformer Networks can be for- mulated as multiple generalized transductions,based on a merits of different feature sets for particular tasks general graph composition algorithm.The connections be- Historically,the need for appropriate feature extractors tween GTNs and Hidden Markov Models,commonly used was due to the fact that the learning techniques used by in speech recognition is also treated.Section IX describes the classifiers were limited to low-dimensional spaces with a globally trained GTN system for recognizing handwrit- easily separable classes [1].A combination of three factors ing entered in a pen computer.This problem is known as have changed this vision over the last decade.First,the "on-line"handwriting recognition,since the machine must availability of low-cost machines with fast arithmetic units produce immediate feedback as the user writes.The core of allows to rely more on brute-force "numerical"methods the system is a Convolutional Neural Network.The results than on algorithmic refinements.Second,the availability clearly demonstrate the advantages of training a recognizer of large databases for problems with a large market and at the word level,rather than training it on pre-segmented, wide interest,such as handwriting recognition,has enabled hand-labeled,isolated characters.Section X describes a designers to rely more on real data and less on hand-crafted complete GTN-based system for reading handwritten and feature extraction to build recognition systems.The third machine-printed bank checks.The core of the system is the and very important factor is the availability of powerful ma- Convolutional Neural Network called LeNet-5 described in chine learning techniques that can handle high-dimensional Section II.This system is in commercial use in the NCR. inputs and can generate intricate decision functions when Corporation line of check recognition systems for the bank- fed with these large data sets.It can be argued that the ing industry.It is reading millions of checks per month in recent progress in the accuracy of speech and handwriting several banks across the United States. recognition systems can be attributed in large part to an increased reliance on learning techniques and large training A.Learning from Data data sets.As evidence to this fact,a large proportion of There are several approaches to automatic machine modern commercial OCR.systems use some form of multi- learning,but one of the most successful approaches,pop- layer Neural Network trained with back-propagation. ularized in recent years by the neural network community, In this study,we consider the tasks of handwritten char- can be called "numerical"or gradient-based learning.The acter recognition (Sections I and II)and compare the per- learning machine computes a function Yp F(ZP,W) formance of several learning techniques on a benchmark where ZP is the p-th input pattern,and W represents the data set for handwritten digit recognition (Section III). collection of adjustable parameters in the system.In a While more automatic learning is beneficial,no learning pattern recognition setting,the output Yp may be inter- technique can succeed without a minimal amount of prior preted as the recognized class label of pattern ZP,or as knowledge about the task.In the case of multi-layer neu- scores or probabilities associated with each class.A loss ral networks,a good way to incorporate knowledge is to function EP =D(DP,F(W,ZP)),measures the discrep- tailor its architecture to the task.Convolutional Neu- ancy between DP,the "correct"or desired output for pat- ral Networks [2]introduced in Section II are an exam-tern ZP,and the output produced by the system.The ple of specialized neural network architectures which in- average loss function Etrain(W)is the average of the er- corporate knowledge about the invariances of 2D shapes rors EP over a set of labeled examples called the training by using local connection patterns,and by imposing con- set {D),....(,DP)}.In the simplest setting,the straints on the weights.A comparison of several methods learning problem consists in finding the value of W that for isolated handwritten digit recognition is presented in minimizes Etrain(W).In practice,the performance of the section III.To go from the recognition of individual char- system on a training set is of little interest.The more rel- acters to the recognition of words and sentences in docu-evant measure is the error rate of the system in the field, ments,the idea of combining multiple modules trained to where it would be used in practice.This performance is reduce the overall error is introduced in Section IV.Rec- estimated by measuring the accuracy on a set of samples ognizing variable-length objects such as handwritten words disjoint from the training set,called the test set.Much using multi-module systems is best done if the modules theoretical and experimental work [3],[4],[5]has shown

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ￾ TRAINABLE CLASSIFIER MODULE FEATURE EXTRACTION MODULE Class scores Feature vector Raw input ✁❼✿▲❍✪❦✦❬✹❦↕✮✦✸❖✱✴❉✪✿ ✵✻✿▲✷✹❈♦✱✴❴✦❃♦✱❘✵ ✵✻✰❇✸✻❈✌✸✶✰❅❆❅✷✹❍✹❈✪✿ ✵✻✿▲✷✹❈✌✿▲✺❳❃❄✰❅✸✻⑥✙✷✹✸✶❋✩✰❅❉④✽❀✿ ✵✻✯✞✵●✽❢✷✩❋✩✷✄❉★❑ ✳✪❴▲✰❅✺❺♠✥✱✄✂✆☎✒✰❅❉④⑥✙✰❺✱❘✵✻✳✪✸✻✰➜✰✝☎✄✵✶✸✶✱✴❆r✵✶✷✹✸❇❚❄✱✴❈✪❉t✱➜✵✶✸✶✱✴✿▲❈♦✱✴◗✪❴▲✰❷❆❅❴➂✱✴✺✶✺✻✿✂♦✰❅✸❺❦ î➻ï➊ä✥ø➩è❿ê❨ì➄ë❦ù✂ø✙➘◆ï✖ä✧ï✖é❛è❨ëíï❞å✠è✥ÿ✙ä✧ï✌ê✧ï❶è❿ê➵ëíì➈ä❨æ☎å➈ä✤è✥ø✠￾➊ÿ✙û➀å➈ä✛è✥å❛ê✤ô✂ê✖ð õ➔ø✓ê④è✥ì➈ä✥ø✠￾✖å➄û➀û✙✘➎➌☎è✥ç✙ï❙é☎ï➊ï✖ù✫ëíì➈ä➤å➄æ✙æ✙ä✥ì➈æ☎ä✧ø✓å✠è✥ï➤ëíï✖å✠è✥ÿ✙ä✥ï✒ï❯↔➇è✥ä✥å➎￾↔è✧ì❛ä✥ê ó✛å➈ê➞ù✂ÿ☎ï➓è✧ì✫è✥ç✙ï➓ë⑨å➎￾↔è✒è✥ç☎å✠è✒è✧ç✙ï➽û➀ï✖å➈ä✧é☎ø➑é❼✂✺è✥ï✒￾❿ç✙é☎ø✠➍✐ÿ✙ï❞ê➞ÿ☎ê✧ï✖ù ✡☛✘ è✧ç☎ï➉￾❶û✓å➈ê✥ê✤ø✙➞☎ï✖ä✥ê✛ó➵ï➊ä✥ï✌û➑ø➀î➻ø➩è✥ï✖ù✿è✥ì➓û➑ì✠ó✔✝➠ù✂ø➀î❭ï✖é☎ê✧ø➑ì❛é☎å➄û✝ê✧æ☎å✑￾➊ï✖ê❨ó❨ø➑è✧ç ï✖å❛ê✤ø➀û✠✘➽ê✤ï✖æ☎å➄ä❿å❄✡✙û➀ït￾❶û✓å➈ê✥ê✤ï❞ê✟✞✙➾✡✠⑥ð❹✕✟￾❶ì➈î➑✡✙ø➀é☎å✠è✥ø➑ì❛é✶ì➄ë❇è✧ç✙ä✥ï➊ï♣ë⑨å✑￾❶è✧ì❛ä✥ê ç☎årú❛ï➣￾❿ç☎å➄é❼✂❛ï✖ù è✥ç✙ø➀ê❙ú➇ø➀ê✧ø➀ì➈é ì✠ú➈ï✖ä➤è✧ç✙ï✢û➀å❛ê④è❭ù✂ï✒￾✖å➈ù✂ï❛ð➔✜❯ø➑ä❿ê④è★➌❯è✧ç✙ï årú✠å➄ø➀û➀å✑✡✙ø➀û➑ø➑è❅✘❙ì➈ë❀û➀ì✠ó✔✝❖￾➊ì❛ê✤è➏î➻å➎￾❿ç✙ø➀é✙ï✖ê✇ó❨ø➩è✥ç➽ë⑨å❛ê④è✛å➄ä✥ø➩è✥ç✙î➻ï❶è✧ø✁￾♣ÿ✙é✙ø➑è✥ê å➄û➀û➀ì✠ó➔ê➻è✧ì❑ä✧ï✖û✙✘ î➻ì➈ä✥ï✫ì➈é ✡✙ä✥ÿ✂è✥ï❯✝♠ëíì➈ä✴￾❶ï☞☛✤é➇ÿ✙î➻ï✖ä✧ø✁￾➊å➈û✍✌➷î➻ï❶è✥ç✙ì✂ù✙ê è✧ç✎å➄é ì➈é å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î➻ø✠￾❭ä✥ï❯➞☎é✙ï✖î➻ï➊é✐è✥ê✖ð✿þ➇ï★￾❶ì❛é☎ù❢➌❇è✧ç✙ï➽årú✠å➄ø➀û✓å❄✡✙ø➀û➑ø➑è❅✘ ì➄ë♣û➀å➈ä❺✂❛ï✢ù✙å➄è✥å❄✡✎å➈ê✧ï✖ê✌ëíì❛ä➻æ✙ä✧ì➎✡✙û➀ï➊î➓ê✒ó❨ø➑è✧ç➘å❺û➀å➈ä❺✂❛ï➽î➓å➄ä✥ô➈ï❶è➻å➄é☎ù ó❨ø✓ù✂ï❨ø➑é✐è✥ï➊ä✥ï✖ê✤è✒➌➈ê✧ÿ✗￾❿ç➻å➈ê❦ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é❼✂➳ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❀➌➄ç☎å➈ê❯ï✖é☎å❄✡✙û➀ï✖ù ù✂ï❞ê✤ø✠✂➈é✙ï✖ä✥ê✝è✧ì➳ä✧ï✖û✙✘➳î➻ì➈ä✥ï❫ì❛é➞ä✧ï❞å➄û➇ù✙å✠è❿å➉å➈é☎ù✌û➀ï✖ê✥ê❀ì❛é➞ç☎å➄é✎ù☛✝❖￾➊ä✥å➄ë➺è✧ï❞ù ëíï✖å➄è✧ÿ✙ä✥ï✌ï❯↔➇è✥ä✥å➎￾↔è✧ø➀ì➈é✢è✧ì➵✡☎ÿ✙ø➑û✓ù✺ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✺ê❺✘➇ê✤è✧ï✖î➓ê➊ð❫ã✛ç☎ï➤è✧ç✙ø➀ä✥ù å➄é✎ù➤ú➈ï➊ä❘✘➔ø➀î➻æ✎ì❛ä✤è❿å➄é✐è☛ë⑨å✑￾❶è✧ì➈ä❀ø➀ê✟è✥ç✙ï❫årú✠å➄ø➀û✓å❄✡✙ø➀û➑ø➑è❅✘➔ì➈ë➇æ◆ì✠ó➵ï➊ä✧ëíÿ✙û➄î➓å♦✝ ￾❿ç✙ø➀é✙ï➵û➑ï❞å➄ä✥é✙ø➑é✗✂➔è✧ï★￾❿ç✙é✙ø✁➍✐ÿ✙ï✖ê❇è✧ç✎å✠è❷￾➊å➄é✒ç☎å➄é☎ù✙û➑ï➵ç✙ø✠✂➈ç✦✝➠ù✂ø➀î❭ï✖é☎ê✧ø➑ì❛é☎å➄û ø➀é✙æ✙ÿ✂è❿ê➳å➈é☎ù➔￾✖å➄é↕✂❛ï➊é✙ï✖ä✥å➄è✧ï❙ø➑é✐è✧ä✥ø✁￾➊å✠è✥ï❙ù✂ï★￾❶ø✓ê✤ø➀ì➈é➲ëíÿ✙é✗￾↔è✥ø➑ì❛é☎ê➳ó❨ç✙ï➊é ëíï✖ù ó❨ø➑è✧ç è✧ç☎ï✖ê✧ï➓û➀å➈ä❺✂❛ï❭ù✙å➄è✥å✺ê✧ï❶è✥ê✖ð➵➏⑥è➉￾➊å➈é➛✡◆ï✶å➈ä❺✂❛ÿ✙ï✖ù➲è✧ç☎å➄è✌è✧ç✙ï ä✥ï✒￾❶ï✖é✐è✛æ☎ä✧ì➎✂➈ä✥ï✖ê✥ê❫ø➀é✿è✧ç✙ï➞å➎￾✄￾❶ÿ☎ä✥å➎￾❯✘➓ì➄ë❣ê✧æ✎ï✖ï✒￾❿ç✺å➄é☎ù✿ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é❼✂ ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é ê❇✘✂ê✤è✧ï✖î➻ê✌￾✖å➄é➒✡✎ï➽å➄è✤è✥ä✧ø✠✡✙ÿ✂è✥ï✖ù ø➑é➷û✓å➄ä❘✂➈ï❭æ☎å➄ä✧è➤è✧ì➲å➈é ø➀é✗￾❶ä✥ï✖å❛ê✤ï❞ù✌ä✧ï✖û➑ø✓å➄é✥￾❶ï➵ì➈é✒û➀ï✖å➈ä✧é☎ø➑é❼✂➉è✧ï★￾❿ç✙é✙ø✁➍❛ÿ☎ï✖ê❣å➄é✎ù➞û✓å➄ä❘✂➈ï➏è✥ä✥å➈ø➑é☎ø➑é❼✂ ù✙å➄è✥å✫ê✧ï❶è✥ê✖ð➙✕♣ê✌ï➊ú➇ø✓ù✂ï➊é✥￾❶ï➻è✧ì✺è✥ç✙ø✓ê✌ë⑨å✑￾❶è✒➌❦å✿û➀å➈ä❺✂❛ï❭æ☎ä✧ì❛æ✎ì❛ä✤è✥ø➑ì❛é❺ì➈ë î➻ì✂ù✂ï➊ä✥é➙￾➊ì➈î➻î➻ï➊ä✴￾❶ø✓å➄û✝ý✌☞✎✍✲ê❺✘➇ê✤è✧ï✖î➓ê✇ÿ✎ê✤ï➤ê✤ì❛î➻ï➉ëíì❛ä✧î ì➈ë❀î✒ÿ✙û➑è✧ø✙✝ û✓å✪✘➈ï➊ä❨ñ➉ï➊ÿ✙ä❿å➄û✝ñ➉ï❶è④ó➵ì➈ä✥ô➻è✧ä❿å➄ø➀é✙ï✖ù✿ó❨ø➑è✧ç✫✡☎å✑￾❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é✝ð ➏➠é➓è✧ç✙ø✓ê➵ê④è✥ÿ☎ù✦✘✑➌✐ó➵ï✬￾❶ì❛é☎ê✤ø✓ù✂ï✖ä❣è✧ç✙ï➉è✥å❛ê✤ô✂ê➏ì➈ë☛ç☎å➄é☎ù✙ó❨ä✧ø➑è✤è✥ï➊é ￾❿ç☎å➈ä❇✝ å✑￾❶è✧ï✖ä❨ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é➔➪♠þ➇ï✒￾❶è✧ø➀ì➈é☎ê✩➏➵å➄é☎ù➙➏❇➏❇➶✛å➈é☎ù➐￾❶ì➈î➻æ☎å➈ä✧ï♣è✧ç☎ï✌æ✎ï✖ä❇✝ ëíì➈ä✥î➓å➄é✗￾➊ï✫ì➄ë➞ê✧ï➊ú➈ï✖ä✥å➈û❨û➑ï❞å➄ä✥é✙ø➀é❼✂➷è✧ï★￾❿ç✙é✙ø✁➍❛ÿ☎ï✖ê➽ì❛é❍å①✡✎ï✖é✗￾❿ç✙î➓å➄ä✥ô ù✙å➄è✥å➘ê✤ï➊è✺ëíì➈ä✺ç☎å➄é✎ù✂ó❨ä✧ø➑è✤è✥ï➊é ù✂ø✠✂➈ø➑è✺ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é❫➪♠þ➇ï✒￾❶è✧ø➀ì➈é✖➏❇➏❇➏❇➶❶ð ✎ç✙ø➀û➑ï✫î➻ì❛ä✧ï✫å➄ÿ✙è✧ì➈î➓å➄è✧ø✁￾✢û➀ï✖å➈ä✧é☎ø➑é❼✂➷ø✓ê➵✡◆ï➊é✙ï✄➞✥￾❶ø✓å➄û✶➌✇é☎ì û➀ï✖å➈ä✧é☎ø➑é❼✂ è✧ï★￾❿ç✙é✙ø✁➍✐ÿ✙ï➉￾➊å➈é➲ê✤ÿ✥￾✄￾❶ï✖ï✖ù✺ó❨ø➑è✧ç✙ì❛ÿ✂è➳å➽î❭ø➀é✙ø➀î➓å➄û❇å➈î➻ì➈ÿ✙é✐è➉ì➈ë❦æ☎ä✧ø➀ì➈ä ô➇é✙ì✠ó❨û➀ï✖ù✦✂❛ï✒å❄✡◆ì➈ÿ✂è♣è✧ç✙ï✒è✥å❛ê✤ô✟ð✬➏➠é➲è✧ç✙ï➝￾➊å❛ê✤ï➞ì➈ë➏î✒ÿ✙û➑è✧ø✙✝⑥û➀å✪✘❛ï➊ä➔é✙ï✖ÿ✦✝ ä❿å➄û➵é✙ï❶è④ó➵ì➈ä✥ô✂ê✄➌❣å↕✂❛ì➇ì➇ù ó➵å✪✘ è✧ì❺ø➑é✥￾❶ì➈ä✥æ◆ì➈ä❿å✠è✧ï✶ô✐é☎ì✠ó❨û➑ï❞ù✦✂➈ï✢ø➀ê✒è✧ì è✥å➈ø➑û➀ì➈ä❺ø➩è❿ê❺å➈ä❘￾❿ç☎ø➩è✥ï✒￾↔è✥ÿ✙ä✥ï è✧ì✲è✧ç✙ï❤è✥å➈ê✧ô✟ð ☞✇ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✒ñ➔ï✖ÿ✦✝ ä❿å➄û✒ñ➉ï❶è④ó➵ì➈ä✥ô➇ê✏✞✑✆✠❙ø➑é✐è✧ä✥ì✂ù✂ÿ✗￾➊ï✖ù❲ø➑éòþ➇ï★￾↔è✥ø➑ì❛é ➏❇➏✫å➄ä✥ï å➄é ï❯↔✙å➄î➚✝ æ✙û➀ï➲ì➄ë✌ê✧æ✎ï★￾❶ø✓å➄û➀ø✙➽✖ï✖ù é✙ï➊ÿ☎ä✥å➈û❨é✙ï❶è④ó➵ì➈ä✥ô❤å➄ä✴￾❿ç✙ø➑è✧ï★￾↔è✧ÿ☎ä✧ï❞ê➻ó❨ç✙ø✠￾❿ç➘ø➀é✦✝ ￾❶ì❛ä✧æ◆ì➈ä❿å✠è✥ï✢ô➇é✙ì✠ó❨û➀ï✖ù❼✂➈ï✺å❄✡◆ì➈ÿ✂è➻è✧ç☎ï✫ø➑é➇ú✠å➄ä✥ø➀å➈é✗￾❶ï❞ê❭ì➄ë✒✑✑✧ ê✧ç☎å➄æ◆ï✖ê ✡☛✘➲ÿ☎ê✧ø➑é✗✂✢û➀ì✦￾➊å➄û❹￾➊ì➈é✙é☎ï✒￾↔è✥ø➑ì❛é❖æ☎å➄è✤è✥ï➊ä✥é☎ê✄➌❇å➄é☎ù➛✡☛✘✫ø➑î➻æ◆ì❛ê✧ø➑é❼✂✆￾➊ì➈é✦✝ ê✤è✧ä❿å➄ø➀é❛è❿ê➔ì❛é✫è✧ç✙ï❭ó➵ï➊ø✠✂➈ç✐è✥ê✖ð✬✕➧￾➊ì➈î➻æ☎å➈ä✧ø✓ê✤ì❛é✫ì➄ë❫ê✧ï➊ú❛ï➊ä❿å➄û✝î➻ï❶è✥ç✙ì✂ù✙ê ëíì➈ä➻ø✓ê✤ì❛û➀å➄è✧ï❞ù ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï➊é➘ù✂ø✠✂➈ø➑è❭ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é❤ø✓ê❭æ✙ä✧ï❞ê✤ï✖é✐è✧ï✖ù❤ø➑é ê✧ï✒￾↔è✥ø➑ì❛é✆➏❇➏❺➏↔ð✛ã❇ì➵✂➈ì➻ëíä✥ì➈î✴è✧ç✙ï❙ä✧ï★￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✿ì➄ë❣ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û❳￾❿ç☎å➈ä❇✝ å✑￾❶è✧ï✖ä✥ê➳è✥ì✢è✥ç✙ï➓ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❺ì➄ë➵ó✇ì❛ä✥ù✙ê✌å➈é☎ù ê✤ï✖é❛è✥ï➊é✗￾➊ï✖ê➤ø➀é ù✂ì✦￾❶ÿ✦✝ î➻ï➊é✐è✥ê✒➌✟è✧ç☎ï❙ø✓ù✂ï✖å✶ì➄ë➜￾➊ì➈î➉✡☎ø➑é✙ø➀é❼✂✢î✒ÿ✙û➑è✧ø➀æ✙û➀ï❭î➻ì✂ù✂ÿ✙û➀ï✖ê➉è✧ä❿å➄ø➀é✙ï❞ù✺è✧ì ä✥ï✖ù✂ÿ✗￾➊ï✒è✥ç✙ï❭ì✠ú➈ï➊ä❿å➄û➀û❀ï✖ä✧ä✥ì➈ä♣ø➀ê➳ø➀é✐è✧ä✥ì➇ù✙ÿ✗￾❶ï❞ù✫ø➑é➷þ➇ï✒￾❶è✧ø➀ì➈é➛➏r✣✒ð➉✍❨ï★￾✹✝ ì✑✂❛é✙ø✠➽➊ø➀é❼✂♣ú✠å➈ä✧ø✓å❄✡✙û➀ï❯✝⑥û➀ï➊é❼✂➈è✧ç➞ì➎✡✔✓④ï✒￾❶è✥ê❣ê✧ÿ✗￾❿ç❙å❛ê❦ç☎å➄é☎ù✙ó❨ä✧ø➑è✤è✥ï➊é❭ó✇ì❛ä✥ù✙ê ÿ☎ê✧ø➑é✗✂❤î✒ÿ☎û➩è✥ø➟✝⑥î➻ì➇ù✙ÿ✙û➑ï ê❇✘✂ê✤è✧ï✖î➻ê✢ø✓ê➣✡◆ï✖ê✤è✺ù✂ì❛é✙ï➲ø➑ë➞è✧ç☎ï❖î➻ì✂ù✂ÿ✙û➀ï✖ê î➓å➄é✙ø➀æ✙ÿ✙û✓å✠è✥ï➓ù✂ø➑ä✥ï✒￾❶è✧ï❞ù➔✂❛ä✥å➈æ✙ç☎ê✖ð❭ã✛ç✙ø✓ê➤û➀ï✖å➈ù☎ê➳è✧ì✿è✧ç✙ï➣￾➊ì➈é✗￾➊ï➊æ✂è✌ì➈ë è✧ä❿å➄ø➀é☎å✑✡✙û➑ï⑨➠✔➡❺➢❘➤✗➥➧➦❼➡❺➢♦➨✗➩➭➫❯➯♦➡✹➲➵➳❯➡ ➸➑➳❯➺✶➻➜➯♦➡❺➼ ➪♠â➤ã❨ñ④➶➻å➈û➀ê✧ì❺ø➀é✐è✧ä✥ì❄✝ ù✂ÿ✗￾➊ï✖ù➘ø➀é✲þ➇ï✒￾❶è✧ø➀ì➈é ➏r✣✒ðòþ➇ï★￾↔è✥ø➑ì❛é⑨✣❂ù✂ï✖ê❘￾❶ä✥ø✙✡◆ï✖ê➓è✧ç✙ï➲é☎ì✠ó ￾➊û➀å❛ê❅✝ ê✧ø✠￾✖å➄û❨î➻ï❶è✥ç✙ì✂ù❑ì➈ë➳ç☎ï➊ÿ✙ä✥ø➀ê✤è✧ø✁￾✿ì✠ú➈ï✖ä❇✝➠ê✤ï✒✂➈î➻ï➊é✐è❿å✠è✧ø➀ì➈é ëíì➈ä➓ä✥ï✒￾❶ì➎✂➈é✙ø✠➽❯✝ ø➀é❼✂❖ó➵ì➈ä❿ù✙ê➞ì❛ä✒ì➈è✧ç✙ï✖ä➑￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✒ê✤è✧ä✥ø➑é❼✂✐ê➊ð➒✧➉ø✓ê❘￾❶ä✥ø➑î➻ø➀é☎å✠è✥ø➑ú❛ï✶å➄é☎ù é✙ì❛é✦✝⑥ù✙ø➀ê❘￾❶ä✥ø➑î➻ø➀é☎å✠è✥ø➑ú❛ï➙✂❛ä✥å❛ù✂ø➀ï➊é✐è❇✝❖✡☎å➈ê✧ï✖ù è✧ï★￾❿ç✙é✙ø✁➍❛ÿ☎ï✖ê❭ëíì➈ä➻è✥ä✥å➈ø➑é☎ø➑é❼✂ å➷ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï✖ä➓å✠è➓è✧ç☎ï➲ó✇ì❛ä✥ù❑û➀ï➊ú➈ï✖û❨ó❨ø➩è✥ç✙ì➈ÿ✙è✶ä✧ï★➍✐ÿ✙ø➑ä✥ø➀é❼✂➷î➻å➈é➇ÿ☎å➄û ê✧ï✄✂➈î➻ï✖é❛è❿å✠è✥ø➑ì❛é➤å➈é☎ù➤û✓å❄✡◆ï➊û➀ø➑é❼✂➔å➈ä✧ï❣æ☎ä✧ï❞ê✤ï✖é❛è✥ï✖ù➤ø➀é✒þ➇ï✒￾❶è✧ø➀ì➈é✌✣④➏↔ð❯þ➇ï★￾✹✝ è✧ø➀ì➈é➛✣✬➏❺➏➉æ✙ä✥ï✖ê✧ï➊é✐è✥ê➉è✧ç☎ï❙æ✙ä✥ì➈î➻ø✓ê✤ø➀é❼✂✺þ➇æ☎å✑￾➊ï❯✝r✧➉ø✓ê✧æ✙û➀å➎￾❶ï✖î❭ï✖é✐è➳ñ➔ï✖ÿ✦✝ ä❿å➄û➞ñ➔ï➊è④ó✇ì❛ä✧ô✾å➄æ✙æ✙ä✥ì❛å➎￾❿ç✻è✥ç☎å✠è➲ï✖û➑ø➀î➻ø➑é☎å➄è✧ï❞ê✺è✧ç✙ï❤é✙ï➊ï❞ù✲ëíì❛ä➲ê✤ï✒✂❄✝ î➻ï➊é✐è✥å➄è✧ø➀ì➈é ç☎ï➊ÿ✙ä✥ø➀ê✤è✧ø✁￾➊ê➑✡☛✘ ê❺￾✖å➄é✙é✙ø➀é❼✂ å➲ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï✖ä✒å✠è➻å➄û➀û✛æ✎ì✐ê❅✝ ê✧ø✙✡✙û➀ï➽û➀ì✦￾➊å✠è✥ø➑ì❛é☎ê➞ì❛é➷è✧ç✙ï✢ø➑é☎æ✙ÿ✂è✖ð➔➏➠é❤ê✤ï★￾↔è✥ø➑ì❛é①✣✬➏❇➏❺➏✹➌❣ø➩è❭ø➀ê❙ê✤ç☎ì✠ó❨é è✧ç✎å✠è✿è✧ä❿å➄ø➀é☎å❄✡✙û➀ï â➳ä✥å➈æ✙ç✲ã❇ä✥å➈é☎ê④ëíì❛ä✧î➻ï✖ä✢ñ➔ï➊è④ó✇ì❛ä✧ô✂ê➐￾➊å➈é ✡✎ï ëíì❛ä❇✝ î✒ÿ☎û➀å➄è✧ï✖ù✫å❛ê❨î✒ÿ☎û➩è✥ø➑æ✙û➀ï➓✂❛ï➊é✙ï✖ä✥å➈û➑ø✠➽➊ï❞ù➽è✥ä✥å➈é☎ê✧ù✙ÿ✗￾↔è✥ø➑ì❛é☎ê✄➌✗✡☎å➈ê✧ï✖ù✿ì➈é✫å ✂➈ï✖é✙ï➊ä❿å➄û✗✂➈ä❿å➄æ✙ç➣￾➊ì➈î➻æ✎ì✐ê✤ø➑è✧ø➀ì➈é➽å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î✺ð❯ã✛ç☎ï④￾❶ì❛é✙é✙ï★￾↔è✧ø➀ì➈é✎ê❹✡◆ï❯✝ è④ó➵ï➊ï➊é➲â➤ã❨ñ♣ê➉å➄é✎ù✫õ➔ø✓ù✙ù✂ï➊é➲ö❖å➄ä✥ô➈ì✠ú➽ö✫ì✂ù✂ï✖û➀ê✒➌✥￾➊ì➈î➻î➻ì➈é✙û✠✘✶ÿ☎ê✧ï✖ù ø➀é❖ê✤æ◆ï➊ï★￾❿ç✫ä✧ï★￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✺ø✓ê➉å➄û✓ê✧ì➓è✧ä✥ï✖å➄è✧ï✖ù✝ð➉þ➇ï✒￾❶è✧ø➀ì➈é✆➏✝✕➶ù✙ï✖ê❘￾❶ä✥ø✙✡◆ï✖ê å✫✂➈û➀ì✑✡✎å➄û➀û✙✘✫è✥ä✥å➈ø➑é☎ï✖ù â➤ã❨ñ ê❇✘✂ê✤è✧ï✖î ëíì➈ä✒ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ø➀é❼✂✺ç☎å➄é☎ù✙ó❨ä✧ø➑è❇✝ ø➀é❼✂➽ï➊é✐è✥ï➊ä✥ï✖ù✺ø➀é❖å➓æ◆ï➊é➔￾➊ì➈î➻æ✙ÿ✂è✥ï➊ä❞ð➔ã✛ç✙ø➀ê➉æ✙ä✥ì✑✡✙û➀ï➊î▼ø➀ê➉ô➇é✙ì✠ó❨é➲å➈ê ☛✤ì❛é✦✝⑥û➑ø➀é✙ï✖✌➞ç☎å➄é☎ù✙ó❨ä✧ø➑è✧ø➀é❼✂❭ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✏➌➇ê✤ø➀é✗￾➊ï➔è✧ç☎ï➳î➓å➎￾❿ç✙ø➑é☎ï➉î✒ÿ✎ê④è æ✙ä✥ì✂ù✂ÿ✗￾❶ï✇ø➑î➻î➻ï✖ù✙ø➀å➄è✧ï✇ëíï➊ï✖ù❼✡☎å✑￾❿ô✌å➈ê✝è✧ç✙ï➵ÿ☎ê✧ï➊ä❯ó❨ä✧ø➑è✧ï❞ê➊ð❯ã✛ç✙ï✔￾❶ì➈ä✥ï➏ì➈ë è✧ç☎ï➉ê❇✘✂ê✤è✧ï✖îòø➀ê✇å➓☞✇ì➈é➇ú➈ì❛û➑ÿ✂è✥ø➑ì❛é☎å➄û✙ñ➉ï➊ÿ✙ä❿å➄û✙ñ➉ï❶è④ó➵ì➈ä✥ô✟ð❇ã✛ç✙ï➔ä✥ï✖ê✧ÿ✙û➑è✥ê ￾❶û➀ï✖å➈ä✧û✠✘➞ù✂ï✖î❭ì❛é☎ê✤è✧ä❿å✠è✧ï✇è✧ç☎ï➔å➈ù✂ú✠å➄é✐è❿å❄✂➈ï❞ê❀ì➈ë☎è✥ä✥å➈ø➑é✙ø➀é❼✂➤å➳ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ï➊ä å✠è❦è✥ç✙ï❨ó✇ì❛ä✥ù✒û➑ï✖ú➈ï✖û✻➌➄ä❿å✠è✥ç✙ï➊ä❯è✥ç☎å➄é❙è✧ä❿å➄ø➀é✙ø➀é❼✂➤ø➩è❣ì➈é❭æ✙ä✥ï❯✝➠ê✤ï✒✂➈î➻ï➊é✐è✥ï✖ù❢➌ ç☎å➈é☎ù☛✝⑥û➀å✑✡✎ï✖û➑ï❞ù❢➌✛ø➀ê✧ì➈û✓å✠è✥ï✖ù ￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä✥ê✖ð✲þ➇ï✒￾❶è✧ø➀ì➈é☞✕❋ù✙ï✖ê❘￾❶ä✥ø✙✡◆ï✖ê➓å ￾❶ì❛î➻æ✙û➑ï➊è✧ï➻â➤ã❨ñ✩✝❖✡☎å➈ê✧ï✖ù✫ê❺✘✂ê④è✥ï➊î▼ëíì➈ä♣ä✧ï❞å➈ù✂ø➀é❼✂➽ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï✖é❖å➄é☎ù î➓å✑￾❿ç✙ø➀é✙ï✄✝♠æ✙ä✥ø➀é❛è✥ï✖ù✌✡✎å➄é✙ô➓￾❿ç✙ï★￾❿ô✂ê➊ð❯ã✛ç✙ï✎￾➊ì➈ä✥ï✇ì➈ë✙è✧ç☎ï➵ê❺✘✂ê④è✥ï➊îòø✓ê❇è✧ç✙ï ☞✇ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✎ñ➉ï➊ÿ✙ä❿å➄û✟ñ➔ï➊è④ó✇ì❛ä✧ô➵￾➊å➈û➑û➀ï✖ù✘✗✝ï✖ñ➉ï❶è❺✝✝✙✒ù✂ï✖ê❘￾❶ä✥ø✙✡◆ï✖ù✶ø➑é þ➇ï★￾↔è✧ø➀ì➈é➒➏❇➏↔ð❭ã✛ç✙ø✓ê✌ê❇✘✂ê✤è✧ï➊î❋ø➀ê➳ø➀é➒￾➊ì➈î➻î➻ï➊ä✴￾❶ø✓å➄û❦ÿ☎ê✧ï❙ø➀é è✧ç✙ï➓ñ✞☞✎✍ ☞✇ì➈ä✥æ◆ì➈ä❿å✠è✧ø➀ì➈é✒û➑ø➀é✙ï❨ì➄ë✇￾❿ç✙ï✒￾❿ô✌ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é❙ê❺✘✂ê④è✥ï➊î➓ê❯ëíì➈ä❯è✧ç☎ï✩✡☎å➄é✙ô❩✝ ø➀é❼✂➽ø➀é☎ù✂ÿ☎ê✤è✧ä❘✘➈ð✎➏⑥è➉ø✓ê➔ä✥ï✖å➈ù✙ø➑é❼✂➻î➻ø➀û➑û➀ø➑ì❛é☎ê➔ì➈ë❹￾❿ç✙ï✒￾❿ô✂ê✛æ◆ï➊ä➉î➻ì❛é❛è✥ç✫ø➑é ê✧ï➊ú➈ï✖ä✥å➈û✇✡☎å➄é✙ô✂ê❨å➎￾❶ä✥ì❛ê✥ê➏è✥ç✙ï➓→➔é✙ø➑è✧ï❞ù➲þ✐è✥å➄è✧ï❞ê➊ð ✚✜✛✒✢➳❘➢❄➡✹➨✤✣●➨✦✥✬➫✴➡❺➯❄➲★✧➚➢♦➺r➢ ã✛ç✙ï➊ä✥ï✹å➄ä✥ïòê✧ï➊ú➈ï✖ä✥å➈û❖å➄æ☎æ✙ä✧ì✐å✑￾❿ç✙ï❞ê è✧ì✴å➈ÿ✂è✧ì❛î➓å✠è✧ø✁￾ î➓å✑￾❿ç✙ø➀é✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂✗➌❢✡☎ÿ✂è✌ì➈é✙ï❭ì➄ë✇è✧ç✙ï➻î➻ì❛ê✤è➤ê✧ÿ✗￾✒￾❶ï✖ê✥ê✤ëíÿ✙û➏å➄æ✙æ☎ä✧ì✐å✑￾❿ç✙ï❞ê✄➌◆æ✎ì❛æ✦✝ ÿ✙û✓å➄ä✥ø✙➽✖ï✖ù✶ø➑é✿ä✧ï★￾❶ï✖é❛è✩✘❛ï✖å➄ä❿ê➃✡❩✘➻è✥ç✙ï✌é✙ï✖ÿ✙ä✥å➈û✟é✙ï❶è④ó➵ì➈ä✥ô➵￾❶ì➈î➻î❙ÿ✙é✙ø➑è❅✘✑➌ ￾➊å➈é✆✡✎ï➚￾➊å➈û➑û➀ï✖ù✩☛✧é✐ÿ☎î❭ï✖ä✧ø✁￾➊å➈û✍✌➻ì❛ä✟✥✑➡❺➢✫✪✬✣ ➳❯➨✥➺✮✭✰✯✴➢✪➩✄➳✱✪✳✲✙➳✴➢♦➡✹➨✴✣●➨✵✥➈ð➉ã✛ç✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂➘î➓å✑￾❿ç✙ø➀é✙ï✢￾❶ì❛î➻æ✙ÿ✂è✧ï❞ê❖å ëíÿ✙é✥￾↔è✧ø➀ì➈é✷✶✹✸✻✺✽✼➵➪✿✾❀✸✔❁❃❂➶ ó❨ç✙ï✖ä✧ï❄✾❅✸❭ø✓ê❨è✥ç✙ï❇❆✗✝üè✥ç✫ø➑é☎æ✙ÿ✂è♣æ✎å✠è✤è✥ï➊ä✥é✏➌✟å➄é☎ù❈❂ ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✥ê❨è✧ç✙ï ￾❶ì❛û➑û➀ï✒￾❶è✧ø➀ì➈é✻ì➈ë✒å➈ù✆✓④ÿ✎ê④è❿å❄✡✙û➀ï❖æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê✶ø➑é✻è✧ç✙ï➷ê❇✘✂ê✤è✧ï✖î✿ð ➏➠é✲å æ☎å➄è✤è✧ï✖ä✧é ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é ê✤ï➊è✤è✧ø➀é❼✂✥➌❇è✥ç✙ï✶ì❛ÿ✂è✧æ☎ÿ✂è❄✶✹✸✿î➓å✪✘➛✡◆ï✶ø➀é✐è✧ï✖ä❇✝ æ✙ä✥ï❶è✥ï✖ù❤å❛ê❙è✧ç✙ï✺ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï❞ù✢￾❶û✓å➈ê✥ê❙û➀å✑✡✎ï✖û❨ì➄ë➉æ☎å➄è✤è✥ï➊ä✥é✩✾✸ ➌✇ì➈ä➓å➈ê ê❘￾❶ì➈ä✥ï✖ê✒ì➈ä❙æ✙ä✧ì➎✡☎å❄✡☎ø➑û➀ø➩è✥ø➑ï❞ê✒å❛ê✧ê✧ì✦￾❶ø✓å✠è✧ï❞ù➷ó❨ø➩è✥ç❤ï❞å✑￾❿ç✢￾❶û✓å➈ê✥ê✖ð➛✕Pû➑ì✐ê✧ê ëíÿ✙é✗￾❶è✧ø➀ì➈é✷❉❊✸❋✺❍●➙➪❏■❑✸✔❁❃✼➵➪✮❂▲❁▼✾❀✸✑➶❇➶❯➌➞î➻ï✖å❛ê✤ÿ✙ä✥ï✖ê✿è✥ç✙ï❤ù✂ø✓ê❘￾❶ä✥ï➊æ✦✝ å➄é✥￾❯✘➣✡◆ï❶è④ó➵ï➊ï✖é◆■❑✸✦➌✙è✧ç✙ï❖☛❘￾❶ì➈ä✥ä✥ï✒￾↔è✱✌➞ì❛ä➔ù✂ï✖ê✧ø➑ä✥ï✖ù✿ì❛ÿ✂è✧æ✙ÿ✙è❨ëíì➈ä❨æ☎å➄è❇✝ è✧ï✖ä✧éP✾❀✸✦➌➉å➄é✎ù➘è✧ç✙ï❺ì➈ÿ✂è✥æ✙ÿ✂è✿æ✙ä✧ì✂ù✂ÿ✥￾❶ï✖ù ✡❩✘❑è✧ç☎ï ê❺✘✂ê④è✥ï➊î✺ð✹ã✛ç✙ï årú➈ï✖ä✥å✑✂➈ï➻û➀ì❛ê✥ê➞ëíÿ✙é✥￾↔è✧ø➀ì➈é◗❉❙❘❯❚❃❱✡❲❨❳✏➪❩❂➶➞ø✓ê✒è✧ç✙ï✿årú➈ï➊ä❿å❄✂❛ï➻ì➄ë❨è✧ç☎ï✢ï✖ä❇✝ ä✥ì➈ä❿ê✟❉❊✸➽ì✠ú❛ï➊ä➤å✢ê✧ï❶è➤ì➄ë❫û✓å❄✡◆ï➊û➀ï✖ù❺ï❯↔✙å➈î❭æ☎û➑ï❞ê④￾✖å➄û➀û➑ï❞ù✫è✧ç☎ï❙è✥ä✥å➈ø➑é☎ø➑é❼✂ ê✧ï❶è◆❬❩➪✿✾ ✜ ❁❃■✜ ➶✡❁❪❭✍❭❨❭✍❭✁➪❩✾❇❫❙❁❴■❵❫❹➶▼❛❛ð⑧➏➠é❤è✥ç✙ï✫ê✤ø➀î➻æ✙û➑ï❞ê④è➽ê✧ï❶è✧è✧ø➀é❼✂✗➌❫è✧ç✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂➲æ✙ä✥ì✑✡☎û➑ï✖î✤￾➊ì➈é☎ê✧ø✓ê④è❿ê✒ø➀é①➞☎é☎ù✙ø➑é❼✂❺è✧ç☎ï✢ú✠å➄û➀ÿ✙ï✢ì➄ë✟❂ è✥ç☎å✠è î➻ø➑é☎ø➑î➻ø✠➽➊ï✖ê❊❉❙❘❯❚✱❱▼❲✍❳✏➪✮❂➶❶ð✛➏➠é✫æ✙ä❿å✑￾↔è✥ø✠￾➊ï✑➌✎è✧ç✙ï❙æ✎ï✖ä✤ëíì❛ä✧î➓å➈é✗￾❶ï✌ì➈ë❣è✧ç✙ï ê❺✘➇ê✤è✧ï✖î✴ì❛é✿å❭è✥ä✥å➈ø➑é✙ø➀é❼✂➓ê✧ï❶è➔ø✓ê❨ì➄ë❦û➑ø➑è✤è✥û➑ï✌ø➀é✐è✧ï✖ä✧ï❞ê④è❞ð❫ã✛ç✙ï✌î➻ì➈ä✥ï➤ä✧ï✖û➟✝ ï➊ú✠å➈é❛è➤î➻ï✖å❛ê✤ÿ✙ä✥ï✒ø✓ê♣è✧ç✙ï➻ï✖ä✧ä✥ì➈ä♣ä✥å➄è✧ï✒ì➈ë➏è✧ç☎ï➻ê❺✘✂ê④è✥ï➊î ø➀é➲è✥ç✙ï➚➞☎ï➊û✓ù❢➌ ó❨ç✙ï✖ä✧ï➽ø➑è➻ó✇ì❛ÿ✙û➀ù ✡◆ï✿ÿ☎ê✤ï❞ù ø➀é æ✙ä❿å✑￾↔è✥ø✠￾➊ï➈ð❖ã✛ç☎ø➀ê❙æ✎ï✖ä✤ëíì❛ä✧î➓å➈é✗￾❶ï➽ø✓ê ï✖ê✤è✧ø➀î➓å✠è✥ï✖ù➒✡☛✘➲î➻ï✖å➈ê✧ÿ✙ä✥ø➑é✗✂✶è✥ç✙ï➽å✑￾✒￾❶ÿ✙ä❿å✑￾✄✘✫ì➈é➷å✿ê✧ï❶è✌ì➈ë✛ê✥å➄î➻æ✙û➀ï✖ê ù✂ø✓ê❯✓④ì❛ø➑é✐è➻ëíä✥ì➈î è✥ç✙ï✿è✧ä❿å➄ø➀é✙ø➑é✗✂➷ê✧ï❶è★➌➃￾✖å➄û➀û➑ï❞ù è✥ç✙ï✿è✧ï✖ê✤è✶ê✧ï❶è✖ð➘ö✫ÿ✗￾❿ç è✧ç☎ï➊ì➈ä✥ï❶è✥ø✠￾✖å➄û❨å➄é✎ù ï❯↔✂æ◆ï➊ä✥ø➑î➻ï✖é❛è❿å➄û❨ó➵ì➈ä✥ô☞✞❜✬✠✶➌❊✞❝✫✠✶➌❇✞✙✆✠➔ç✎å➈ê➻ê✤ç☎ì✠ó❨é

PROC.OF THE IEEE,NOVEMBER 1998 that the gap between the expected error rate on the test Hessian matrix as in Newton or Quasi-Newton methods. set d trt and the error rate on the training set dt,Ano de- The ConRigate Gradient method [can also be used. creases with the number of training samples approximately However,Appendix B shows that despite many claims as to the contrary in the literature,the usefulness of these dtt-d,A[(]“”)W (1) second-order methods to large learning machines is very limited. where"is the number of training samples,is a measure of A popular minimization procedure is the stochastic gra- “effective capacity'”or complexity of the machine[j],[☑，a dient algorithm,also called the on-line update.It consists is a number between 0.5 and 1.0,and is a constant.This in updating the parameter vector using a noisy,or approx- gap always decreases when the number of training samples increases.Furthermore,as the capacity increases,dt,Ao imated,version of the average gradient.In the most com- mon instance of it,W is updated on the basis of a single decreases.Therefore,when increasing the capacity],there sample: is a trade-off between the decrease of dt,As and the in- crease of the gap,with an optimal value of the capacity W视sW-1-e arRW) ∂W (3) that achieves the lowest generalization error dtt.Most learning algorithms attempt to minimize dt,Ao as well as With this procedure the parameter vector Zuctuates some estimate of the gap.A formal version of this is called around an average tralectory,but usually converges consid- structural risk minimization [j],[7],and is based on defin- erably faster than regular gradient descent and second or- ing a sequence of learning machines of increasing capacity, der methods on large training sets with redundant samples corresponding to a sequence of subsets of the parameter (such as those encountered in speech or character recogni- space such that each subset is a superset of the previous tion).The reasons for this are explained in Appendix B subset.In practical terms,Structural Risk Minimization The properties of such algorithms applied to learning have is implemented by minimizing dt,AsoigD(W),where the been studied theoretically since the 19j0,s [9],[10],[11], function D(W)is called a regularization function,and g is but practical successes for non-trivial tasks did not occur a constant.D(W)is chosen such that it takes large val- until the mid eighties. ues on parameters W that belong to high-capacity subsets C.Grad ent Back-Propagat on of the parameter space.Minimizing D(W)in effect lim- its the capacity of the accessible subset of the parameter Gradient-Based Learning procedures have been used space,thereby controlling the tradeoff between minimiz- since the late 1950,s,but they were mostly limited to lin- ing the training error and minimizing the expected gap ear systems [1].The surprising usefulness of such sim- between the training error and test error. ple gradient descent techniques for complex machine learn- ing tasks was not widely realized until the following three B.Gradent-Based Learn Vig events occurred.The first event was the realization that, The general problem of minimizing a function with re- despite early warnings to the contrary [12],the presence of local minima in the loss function does not seem to spect to a set of parameters is at the root of many issues in computer science.Gradient-Based Learning draws on the be a mapor problem in practice.This became apparent fact that it is generally much easier to minimize a reason- when it was noticed that local minima did not seem to be a mabor impediment to the success of early non-linear ably smooth,continuous function than a discrete (combi- natorial)function.The loss function can be minimized by gradient-based Learning techniques such as Boltzmann ma- chines [13],[14].The second event was the popularization estimating the impact of small variations of the parame- ter values on the loss function.This is measured by the by Rumelhart,Hinton and Williams [15]and others of a gradient of the loss function with respect to the param- simple and eb cient procedure,the back-propagation al- gorithm,to compute the gradient in a non-linear system eters.-b cient learning algorithms can be devised when the gradient vector can be computed analytically (as op- composed of several layers of processing.The third event posed to numerically through perturbations).This is the was the demonstration that the back-propagation proce- basis of numerous gradient-based learning algorithms with dure applied to multi-layer neural networks with sigmoidal continuous-valued parameters.In the procedures described units can solve complicated learning tasks.The basic idea in this article,the set of parameters W is a real-valued vec- of back-propagation is that gradients can be computed eb- tor,with respect to which d(W)is continuous,as well as ciently by propagation from the output to the input.This idea was described in the control theory literature of the differentiable almost everywhere.The simplest minimiza- tion procedure in such a setting is the gradient descent early sixties [1j],but its application to machine learning algorithm where W is iteratively adRuisted as follows: was not generally realized then.Interestingly,the early derivations of back-propagation in the context of neural a(W) network learning did not use gradients,but "virtual tar- WIs WI-1-e- aw (2) gets”for units in intermediate layers[lT],[l月，or minimal disturbance arguments [19].The Lagrange formalism used In the simplest case,e is a scalar constant.More sophisti- in the control theory literature provides perhaps the best cated procedures use variable e,or substitute it for a diag- rigorous method for deriving back-propagation [20],and for onal matrix,or substitute it for an estimate of the inverse deriving generalizations of back-propagation to recurrent

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ￾ è✧ç✎å✠è➞è✥ç✙ï➵✂❛å➈æ➒✡◆ï❶è④ó➵ï➊ï✖é è✧ç✙ï➽ï✄↔✂æ✎ï★￾↔è✧ï❞ù ï➊ä✥ä✧ì❛ä✌ä✥å➄è✧ï➻ì➈é è✧ç☎ï➓è✧ï❞ê④è ê✧ï❶è✹❉❘✂✁☎✄✿❘ å➈é☎ù✫è✥ç✙ï❭ï➊ä✥ä✧ì❛ä➉ä❿å✠è✧ï❙ì➈é✫è✥ç✙ï❙è✧ä❿å➄ø➀é✙ø➑é✗✂✢ê✧ï❶è✹❉❘❯❚❃❱✡❲❨❳ ù✙ï❯✝ ￾❶ä✥ï✖å❛ê✤ï❞ê❯ó❨ø➩è✥ç❙è✥ç✙ï✛é➇ÿ✙î➉✡◆ï➊ä❣ì➈ë✎è✥ä✥å➈ø➑é✙ø➀é❼✂➤ê✧å➈î➻æ✙û➑ï❞ê❣å➄æ✙æ☎ä✧ì✪↔✂ø➀î➻å➄è✧ï✖û✙✘ å➈ê ❉❘✂✁☎✄✿❘✝✆ ❉❘❯❚❃❱✡❲❨❳ ✺✟✞❢➪✡✠☞☛✍✌➓➶✏✎ ➪❇➾★➶ ó❨ç✙ï✖ä✧ï✑✌❑ø➀ê✝è✧ç✙ï❣é➇ÿ✙î➑✡✎ï✖ä❀ì➈ë❛è✥ä✥å➈ø➑é✙ø➀é❼✂➔ê✥å➄î➻æ✙û➀ï✖ê✒➌✒✠➞ø➀ê✝å✛î➻ï❞å➈ê✧ÿ✙ä✧ï❦ì➈ë ☛✤ï✄➘◆ï★￾↔è✥ø➑ú❛ï✛￾➊å➄æ✎å✑￾❶ø➑è❅✘✔✌➳ì➈ä➜￾➊ì➈î➻æ✙û➀ï❯↔✂ø➩è❅✘❙ì➄ë◆è✧ç✙ï➉î➻å➎￾❿ç✙ø➀é✙ï✜✞✓✬✠✶➌ ✞✕✔✠❖➌✗✖ ø✓ê✇å✌é➇ÿ✙î➑✡✎ï✖ä➃✡◆ï❶è④ó➵ï➊ï✖é✙✘ ❭ ✙➤å➄é✎ù✆➾✫❭ ✘❼➌➇å➄é☎ù✚✞➓ø➀ê➵å➓￾❶ì❛é☎ê✤è✥å➄é✐è❞ð❯ã✛ç✙ø✓ê ✂❛å➈æ➽å➄û➀ó➵å✪✘✂ê➏ù✙ï✒￾❶ä✥ï✖å❛ê✤ï❞ê❣ó❨ç✙ï✖é➓è✧ç✙ï♣é➇ÿ✙î➉✡◆ï➊ä✇ì➄ë☛è✧ä❿å➄ø➀é✙ø➀é❼✂✒ê✥å➄î➻æ✙û➀ï✖ê ø➀é✗￾❶ä✥ï✖å❛ê✤ï❞ê➊ð❨✜✙ÿ✙ä✧è✧ç☎ï➊ä✥î❭ì❛ä✧ï➎➌✐å❛ê❣è✥ç✙ï✞￾✖å➄æ☎å➎￾❶ø➑è❅✘✚✠➽ø➀é✗￾❶ä✥ï✖å❛ê✤ï❞ê✄➌✵❉❙❘❯❚✱❱▼❲❨❳ ù✂ï★￾❶ä✥ï✖å➈ê✧ï✖ê✖ð❀ã✛ç✙ï✖ä✧ï➊ëíì➈ä✥ï✑➌ró❨ç✙ï✖é❙ø➀é✗￾❶ä✥ï✖å❛ê✤ø➀é❼✂❨è✥ç✙ï✩￾➊å➈æ☎å✑￾➊ø➩è❅✘✛✠❀➌rè✧ç✙ï✖ä✧ï ø✓ê❭å❖è✧ä❿å➈ù✙ï❯✝⑥ì❄➘ ✡✎ï➊è④ó✇ï✖ï➊é❑è✧ç✙ï✺ù✂ï✒￾➊ä✧ï❞å➈ê✧ï✶ì➈ë✟❉❘❯❚✱❱▼❲❨❳ å➄é✎ù è✧ç☎ï✢ø➀é✦✝ ￾❶ä✥ï✖å❛ê✤ï✌ì➈ë❇è✥ç✙ï➉✂✐å➄æ✏➌☎ó❨ø➩è✥ç➲å➄é✫ì➈æ✙è✧ø➀î➻å➈û❀ú✠å➄û➀ÿ✙ï✌ì➈ë❯è✥ç✙ï➑￾✖å➄æ☎å➎￾❶ø➑è❅✘✜✠ è✧ç✎å✠è❭å➎￾❿ç✙ø➀ï➊ú➈ï❞ê➤è✧ç☎ï✶û➀ì✠ó✇ï❞ê④è➉✂❛ï➊é✙ï✖ä✥å➈û➑ø✠➽✖å➄è✧ø➀ì➈é➷ï➊ä✥ä✥ì➈ä ❉❙❘✂✁✢✄✿❘↔ð❺ö✫ì✐ê④è û➀ï✖å➄ä✥é✙ø➀é❼✂➽å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê➉å✠è✤è✥ï➊î➻æ✂è♣è✧ì➽î➻ø➀é✙ø➑î➻ø✠➽➊ï❄❉❙❘❯❚❃❱✡❲❨❳✺å➈ê➉ó✇ï✖û➑û❯å➈ê ê✧ì➈î➻ï➔ï✖ê✤è✧ø➀î➓å✠è✥ï➔ì➄ë✟è✧ç☎ï✬✂❛å➈æ✝ð❨✕➘ëíì➈ä✥î➓å➄û☎ú❛ï➊ä❿ê✤ø➀ì➈é➻ì➈ë◆è✥ç✙ø✓ê➏ø✓ê➃￾✖å➄û➀û➑ï❞ù ê✤è✧ä✥ÿ✗￾↔è✥ÿ✙ä✥å➈û✟ä✧ø✓ê✤ô✶î❭ø➀é✙ø➀î➻ø✙➽❞å✠è✧ø➀ì➈é▲✞✓✠❖➌ ✞✔✆✠✶➌✙å➈é☎ù✶ø✓ê✔✡☎å❛ê✤ï❞ù➽ì➈é✺ù✂ï✄➞☎é✦✝ ø➀é❼✂❭å➻ê✧ï✒➍✐ÿ✙ï✖é✗￾❶ï➳ì➄ë❀û➀ï✖å➈ä✧é☎ø➑é❼✂❭î➓å✑￾❿ç✙ø➀é✙ï❞ê✇ì➈ë❀ø➀é✗￾❶ä✥ï✖å❛ê✤ø➀é❼✂➑￾✖å➄æ☎å➎￾❶ø➑è❅✘✑➌ ￾❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✂ø➀é❼✂❖è✧ì å➷ê✤ï★➍❛ÿ☎ï➊é✗￾➊ï✢ì➈ë➳ê✧ÿ❼✡☎ê✧ï❶è✥ê➻ì➈ë➉è✥ç✙ï✫æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä ê✧æ☎å✑￾➊ï➽ê✤ÿ✗￾❿ç➷è✥ç☎å✠è❙ï✖å✑￾❿ç ê✧ÿ❼✡☎ê✧ï❶è✒ø✓ê✒å❖ê✤ÿ✙æ◆ï➊ä❿ê✧ï❶è➞ì➄ë✛è✥ç✙ï➽æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê ê✧ÿ❼✡☎ê✧ï❶è✖ð➔➏➠é æ✙ä❿å✑￾↔è✥ø✠￾✖å➄û❫è✧ï➊ä✥î➓ê✄➌➏þ✐è✥ä✧ÿ✥￾↔è✧ÿ☎ä✥å➈û✎✍❨ø✓ê✧ô ö✫ø➀é✙ø➀î➻ø✙➽❞å✠è✧ø➀ì➈é ø✓ê✇ø➀î➻æ✙û➀ï➊î➻ï➊é✐è✧ï❞ù➣✡☛✘➻î➻ø➑é✙ø➀î➻ø✙➽✖ø➑é✗✂❄❉❙❘❯❚✱❱▼❲✍❳✤✣✦✥★✧✢➪❩❂➶✹➌✂ó❨ç✙ï✖ä✧ï➉è✧ç✙ï ëíÿ✙é✗￾❶è✧ø➀ì➈é✙✧➹➪✮❂➶❣ø✓ê➜￾✖å➄û➀û➑ï❞ù➓å✌ä✥ï✄✂➈ÿ☎û➀å➈ä✧ø✠➽✖å➄è✧ø➀ì➈é❙ëíÿ✙é✗￾↔è✥ø➑ì❛é✏➌➇å➄é☎ù✩✥✫ø✓ê å✆￾❶ì❛é☎ê✤è✥å➄é✐è❞ð✪✧✢➪❩❂➶➳ø➀ê➑￾❿ç✙ì❛ê✧ï➊é ê✧ÿ✗￾❿ç è✧ç☎å➄è✒ø➑è➞è✥å➈ô➈ï✖ê➞û➀å➈ä❺✂❛ï❭ú✠å➈û➟✝ ÿ✙ï❞ê✇ì❛é✶æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê❀❂ è✥ç☎å✠è✔✡✎ï✖û➑ì❛é❼✂❙è✧ì❭ç✙ø✙✂❛ç✦✝r￾➊å➄æ✎å✑￾❶ø➑è❅✘➻ê✤ÿ❼✡✎ê✤ï➊è✥ê ì➄ë❨è✥ç✙ï✶æ✎å➄ä❿å➄î➻ï❶è✥ï➊ä✒ê✧æ☎å✑￾➊ï➈ð❖ö➲ø➑é✙ø➀î➻ø✙➽✖ø➑é✗✂✫✧✢➪❩❂➶✌ø➀é ï❯➘✟ï✒￾❶è❙û➀ø➑î➚✝ ø➑è✥ê✌è✥ç✙ï ￾✖å➄æ☎å➎￾❶ø➑è❅✘➲ì➄ë✛è✧ç☎ï➽å✑￾✄￾➊ï✖ê✥ê✤ø✠✡✙û➀ï➽ê✤ÿ✗✡☎ê✤ï➊è➞ì➄ë✛è✧ç☎ï➽æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä ê✧æ☎å✑￾➊ï✑➌✇è✧ç✙ï✖ä✧ï✒✡❩✘⑨￾❶ì❛é❛è✥ä✧ì❛û➑û➀ø➀é❼✂ è✧ç✙ï✺è✥ä✥å❛ù✂ï➊ì✑➘❝✡✎ï➊è④ó✇ï✖ï➊é➘î➻ø➑é☎ø➑î➻ø✠➽❯✝ ø➀é❼✂❤è✥ç✙ï❺è✧ä❿å➄ø➀é✙ø➑é✗✂❤ï➊ä✥ä✥ì➈ä✢å➈é☎ù✻î➻ø➀é✙ø➀î❭ø✠➽➊ø➀é❼✂❑è✧ç✙ï ï❯↔✂æ◆ï✒￾❶è✧ï✖ù❝✂❛å➈æ ✡◆ï❶è④ó➵ï➊ï➊é✿è✥ç✙ï➤è✧ä❿å➄ø➀é✙ø➀é❼✂➻ï➊ä✥ä✧ì❛ä➵å➈é☎ù✢è✧ï✖ê✤è➔ï➊ä✥ä✥ì➈ä❞ð ✬✛ ➠✔➡❺➢ ✪✣ ➳✄➨✗➺❩✭✬ ➢♦➩❯➳✱✪ ✢ ➳❘➢❄➡✹➨✤✣●➨✦✥ ã✛ç✙ï➣✂➈ï✖é✙ï➊ä❿å➄û❣æ☎ä✧ì➎✡✙û➑ï✖î ì➄ë❨î➻ø➀é✙ø➑î➻ø✠➽➊ø➀é❼✂➲å✿ëíÿ✙é✗￾❶è✧ø➀ì➈é ó❨ø➑è✧ç ä✥ï❯✝ ê✧æ✎ï★￾↔è➏è✥ì➞å✌ê✤ï➊è❫ì➄ë✟æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê❦ø✓ê➏å✠è❣è✥ç✙ï➔ä✥ì✐ì➈è➏ì➄ë✟î➓å➄é☛✘✒ø✓ê✧ê✧ÿ✙ï❞ê❣ø➑é ￾❶ì❛î➻æ✙ÿ✂è✧ï✖ä✌ê❺￾➊ø➑ï✖é✗￾❶ï❛ð✌â➳ä❿å➈ù✙ø➑ï✖é❛è❺✝❖✚✛å❛ê✤ï❞ù✳✗❀ï✖å➈ä✧é✙ø➀é❼✂✿ù✂ä✥åró➔ê➉ì➈é❖è✧ç✙ï ë⑨å✑￾❶è➔è✧ç✎å✠è➉ø➑è➉ø✓ê✛✂➈ï✖é✙ï➊ä❿å➄û➀û✠✘✶î✒ÿ✥￾❿ç✿ï❞å➈ê✧ø➑ï✖ä❨è✧ì➓î➻ø➀é✙ø➀î❭ø✠➽➊ï❭å➻ä✧ï❞å➈ê✧ì➈é✦✝ å❄✡☎û✙✘❖ê✧î➻ì➇ì➄è✧ç❀➌✏￾❶ì➈é✐è✥ø➑é➇ÿ✙ì❛ÿ☎ê♣ëíÿ✙é✗￾↔è✥ø➑ì❛é è✧ç☎å➈é å✿ù✂ø✓ê❺￾➊ä✧ï➊è✧ï✆➪ ￾➊ì➈î➉✡☎ø➟✝ é☎å➄è✧ì➈ä✥ø✓å➄û●➶➏ëíÿ☎é✗￾↔è✥ø➑ì❛é✝ð➵ã✛ç☎ï✌û➑ì✐ê✧ê➵ëíÿ✙é✥￾↔è✧ø➀ì➈é↕￾✖å➄é✫✡◆ï➞î➻ø➑é✙ø➀î➻ø✙➽✖ï✖ù✫✡☛✘ ï✖ê✤è✧ø➀î➓å✠è✥ø➑é❼✂❖è✧ç✙ï✿ø➀î❭æ✎å✑￾↔è❭ì➄ë➉ê✧î➓å➄û➀û✇ú✠å➄ä✥ø✓å✠è✧ø➀ì➈é✎ê➞ì➄ë❨è✥ç✙ï✿æ☎å➄ä❿å➄î➻ï❯✝ è✧ï✖ä❙ú✠å➈û➑ÿ✙ï❞ê✒ì❛é➷è✧ç✙ï✿û➀ì❛ê✥ê✌ëíÿ✙é✗￾❶è✧ø➀ì➈é✝ð➷ã✛ç✙ø✓ê✒ø✓ê❙î❭ï❞å➈ê✧ÿ✙ä✥ï✖ù ✡☛✘ è✧ç✙ï ✂➈ä❿å➈ù✙ø➑ï✖é❛è➓ì➈ë➉è✧ç☎ï✫û➑ì✐ê✧ê❭ëíÿ✙é✥￾↔è✧ø➀ì➈é➘ó❨ø➑è✧ç❍ä✥ï✖ê✧æ✎ï★￾↔è➻è✧ì➷è✥ç✙ï✫æ☎å➈ä✥å➈î➑✝ ï❶è✥ï➊ä❿ê➊ð✮✭✑✯➣￾❶ø➀ï➊é✐è❭û➀ï✖å➈ä✧é☎ø➑é❼✂➷å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê➑￾➊å➈é➹✡◆ï✺ù✙ï➊ú➇ø➀ê✧ï✖ù❤ó❨ç✙ï➊é è✧ç☎ï➣✂➈ä❿å➈ù✂ø➀ï➊é✐è➞ú❛ï✒￾↔è✥ì➈ä➉￾✖å➄é ✡◆ï➙￾❶ì❛î➻æ✙ÿ✂è✧ï❞ù å➄é☎å➈û✙✘✐è✥ø✠￾✖å➄û➀û✙✘➹➪üå➈ê✒ì➈æ✦✝ æ◆ì❛ê✧ï✖ù❖è✧ì✫é➇ÿ✙î➻ï➊ä✥ø✠￾✖å➄û➀û✙✘✺è✥ç✙ä✥ì➈ÿ❼✂❛ç❺æ◆ï➊ä✧è✧ÿ✙ä❘✡☎å➄è✧ø➀ì➈é☎ê✴➶↔ð➓ã✛ç✙ø✓ê➤ø➀ê✌è✧ç✙ï ✡☎å❛ê✤ø✓ê❫ì➈ë❀é➇ÿ✙î➻ï➊ä✥ì➈ÿ☎ê➃✂➈ä❿å➈ù✙ø➑ï✖é❛è❺✝✶✡✎å➈ê✧ï✖ù➻û➀ï✖å➄ä✥é✙ø➀é❼✂❙å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î➓ê❫ó❨ø➑è✧ç ￾❶ì❛é✐è✧ø➀é✐ÿ☎ì➈ÿ☎ê❇✝♠ú✠å➈û➑ÿ✙ï❞ù➤æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê✖ð✏➏➠é➞è✥ç✙ï✇æ☎ä✧ì✦￾❶ï❞ù✂ÿ✙ä✥ï✖ê❇ù✂ï✖ê❘￾❶ä✥ø✙✡◆ï✖ù ø➀é❙è✥ç✙ø✓ê➏å➄ä✧è✧ø✁￾❶û➀ï✑➌✠è✥ç✙ï❨ê✤ï➊è➏ì➄ë◆æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê ❂ ø✓ê❣å➳ä✧ï❞å➄û✙✝♠ú✠å➄û➀ÿ✙ï❞ù➞ú➈ï★￾✹✝ è✧ì❛ä✒➌✟ó❨ø➑è✧ç➲ä✥ï✖ê✧æ◆ï✒￾↔è➳è✧ì✢ó❨ç✙ø✁￾❿ç ❉➣➪✮❂➶➉ø✓ê④￾➊ì➈é✐è✧ø➀é➇ÿ✙ì➈ÿ☎ê✒➌✟å➈ê♣ó✇ï✖û➑û❦å➈ê ù✂ø✙➘◆ï✖ä✧ï✖é✐è✧ø✓å❄✡✙û➀ï➓å➄û➀î❭ì✐ê④è➤ï➊ú❛ï➊ä❘✘➇ó❨ç✙ï➊ä✥ï➈ð➞ã✛ç✙ï➓ê✧ø➑î➻æ✙û➀ï✖ê✤è➤î❭ø➀é✙ø➀î➻ø✙➽❞å♦✝ è✧ø➀ì➈é✲æ✙ä✧ì✦￾➊ï✖ù✂ÿ✙ä✥ï❖ø➀é✲ê✧ÿ✗￾❿ç✲å❑ê✤ï➊è✤è✧ø➀é❼✂❑ø➀ê✶è✧ç✙ï➛✂➈ä❿å➈ù✂ø➀ï➊é✐è✢ù✙ï✖ê❘￾❶ï➊é✐è å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î ó❨ç✙ï➊ä✥ï✜❂ ø✓ê❨ø➑è✧ï➊ä❿å✠è✥ø➑ú❛ï➊û✠✘➽å➈ù✆✓④ÿ✎ê④è✥ï✖ù✺å➈ê➵ëíì❛û➑û➀ì✠ó➔ê✱✰ ❂✳✲✹✺ ❂✳✲✒✴ ✜ ✆✶✵✸✷❉➣➪❩❂➶ ✷❂ ❭ ➪❩✑✑➶ ➏➠é✿è✧ç✙ï❙ê✤ø➀î➻æ✙û➑ï❞ê④è✬￾✖å➈ê✧ï✑➌ ✵ ø➀ê➉å❭ê❘￾➊å➈û➀å➈ä✛￾❶ì❛é☎ê④è❿å➄é✐è✖ð✇ö✫ì➈ä✥ï✌ê✤ì❛æ✙ç✙ø✓ê④è✥ø➟✝ ￾➊å➄è✧ï❞ù➽æ✙ä✥ì✦￾❶ï✖ù✙ÿ✙ä✧ï❞ê❫ÿ✎ê✤ï➳ú✠å➄ä✥ø➀å✑✡✙û➑ï ✵ ➌✐ì❛ä➵ê✧ÿ❼✡☎ê✤è✧ø➑è✧ÿ✙è✧ï➤ø➩è➵ëíì➈ä❨å❙ù✂ø➀å✑✂❄✝ ì➈é✎å➄û✝î➓å✠è✥ä✧ø✙↔❢➌✙ì➈ä➔ê✧ÿ❼✡☎ê✤è✧ø➑è✧ÿ✙è✧ï➞ø➑è➔ëíì➈ä♣å➄é✺ï✖ê✤è✧ø➀î➓å✠è✧ï✌ì➈ë❇è✥ç✙ï➞ø➀é➇ú➈ï➊ä❿ê✧ï õ➔ï❞ê✧ê✧ø✓å➄é❤î➓å➄è✧ä✥ø➟↔ å➈ê❭ø➀é❑ñ➉ï➊ó✛è✧ì❛é❤ì❛ä✚✹♣ÿ✎å➈ê✧ø➟✝➠ñ➔ï✖ó✛è✧ì➈é❤î❭ï➊è✧ç✙ì✂ù✙ê✖ð ã✛ç✙ï➹☞✇ì➈é✬✓④ÿ❼✂❛å➄è✧ï â➳ä✥å❛ù✂ø➀ï➊é✐è✫î❭ï➊è✧ç✙ì✂ù ✞✺✠➉￾✖å➄é å➄û✓ê✤ì⑨✡◆ï ÿ☎ê✤ï❞ù☛ð õ➔ì✠ó➵ï➊ú❛ï➊ä★➌➑✕➉æ✙æ✎ï✖é☎ù✂ø✙↔❫✚❅ê✧ç✙ì✠ó➔ê❖è✧ç☎å➄è➷ù✂ï❞ê✤æ☎ø➩è✥ï î➓å➈é❩✘②￾➊û➀å➈ø➑î➓ê è✧ì➷è✥ç✙ï➔￾➊ì➈é✐è✧ä❿å➄ä❘✘ ø➑é➘è✥ç✙ï✫û➀ø➩è✥ï➊ä❿å✠è✧ÿ☎ä✧ï➎➌✇è✥ç✙ï➲ÿ☎ê✧ï❶ëíÿ☎û➑é✙ï❞ê✧ê➽ì➄ë➳è✧ç✙ï❞ê✤ï ê✧ï✒￾❶ì❛é☎ù☛✝⑥ì➈ä❿ù✂ï➊ä➻î➻ï❶è✥ç✙ì✂ù✙ê➻è✥ì û➀å➈ä❺✂❛ï✿û➀ï✖å➈ä✧é☎ø➑é❼✂➷î➓å✑￾❿ç☎ø➑é✙ï❞ê➻ø➀ê➽ú➈ï➊ä❘✘ û➀ø➑î➻ø➑è✧ï✖ù✝ð ✕✾æ◆ì➈æ☎ÿ✙û➀å➈ä✇î➻ø➀é✙ø➀î❭ø✠➽✖å➄è✧ø➀ì➈é✿æ✙ä✥ì☛￾➊ï✖ù✂ÿ☎ä✧ï♣ø➀ê➵è✥ç✙ï➳ê✤è✧ì✦￾❿ç☎å❛ê④è✥ø✠￾④✂➈ä❿å♦✝ ù✂ø➀ï➊é✐è➉å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➐➌✎å➄û✓ê✤ì➵￾➊å➈û➑û➀ï✖ù✶è✧ç☎ï➞ì➈é✦✝⑥û➀ø➑é✙ï➤ÿ✙æ✟ù✙å➄è✧ï➈ð➃➏⑥è④￾❶ì❛é☎ê✧ø➀ê✤è✥ê ø➀é➽ÿ✙æ✟ù✙å✠è✥ø➑é❼✂❙è✧ç☎ï➉æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä❫ú➈ï★￾↔è✥ì➈ä➏ÿ☎ê✧ø➀é❼✂❙å✒é✙ì➈ø✓ê❇✘➎➌❛ì❛ä✇å➈æ✙æ✙ä✥ì✪↔❩✝ ø➀î➻å➄è✧ï❞ù❢➌☎ú❛ï➊ä❿ê✤ø➀ì➈é✿ì➄ë❦è✧ç☎ï✒årú➈ï✖ä✥å✑✂➈ï✞✂➈ä❿å➈ù✂ø➀ï➊é✐è✖ð➃➏➠é✺è✧ç✙ï✒î❭ì✐ê④è✞￾❶ì➈î➚✝ î➻ì➈é❺ø➑é☎ê✤è✥å➈é✗￾❶ï➻ì➈ë❫ø➩è★➌ ❂ ø✓ê➤ÿ✙æ✟ù✙å✠è✥ï✖ù❺ì➈é❖è✧ç✙ï➝✡☎å❛ê✤ø✓ê➳ì➄ë✇å✺ê✧ø➑é❼✂❛û➑ï ê✥å➄î➻æ✙û➀ï✻✰ ❂✲ ✺ ❂✲✒✴ ✜ ✆✳✵ ✷❉❊✸✽✼☛➪✮❂➶ ✷❂ ➪✮❜➎➶ ✎ø➑è✧ç è✧ç☎ø➀ê æ☎ä✧ì✦￾❶ï❞ù✂ÿ✙ä✥ï✴è✥ç✙ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä ú➈ï✒￾❶è✧ì❛ä✿✾✎ÿ✗￾↔è✥ÿ☎å✠è✥ï✖ê å➄ä✥ì➈ÿ☎é☎ù➞å➄é❙årú➈ï➊ä❿å❄✂❛ï❯è✧ä❿å✓④ï✒￾❶è✧ì❛ä❺✘➎➌★✡☎ÿ✂è❯ÿ✎ê✤ÿ☎å➈û➑û✠✘✌￾❶ì❛é✐ú❛ï➊ä❘✂➈ï❞ê✏￾❶ì❛é☎ê✤ø✓ù☛✝ ï➊ä❿å❄✡☎û✙✘➓ë⑨å❛ê④è✥ï➊ä❨è✥ç☎å➄é✫ä✥ï✄✂❛ÿ✙û✓å➄ä✩✂➈ä❿å➈ù✙ø➑ï✖é❛è➔ù✂ï❞ê❺￾➊ï➊é✐è➉å➄é✎ù✿ê✧ï✒￾➊ì➈é☎ù✿ì❛ä❇✝ ù✂ï✖ä➏î➻ï❶è✧ç☎ì➇ù☎ê➏ì➈é➓û✓å➄ä❘✂➈ï➵è✥ä✥å➈ø➑é✙ø➀é❼✂➞ê✧ï❶è❿ê➏ó❨ø➑è✧ç➓ä✥ï✖ù✂ÿ✙é✎ù✙å➄é✐è✇ê✥å➄î➻æ✙û➀ï✖ê ➪⑨ê✧ÿ✗￾❿ç✫å➈ê✇è✥ç✙ì❛ê✧ï✌ï➊é✗￾➊ì➈ÿ✙é✐è✥ï➊ä✥ï✖ù➽ø➀é✫ê✤æ◆ï➊ï★￾❿ç✺ì➈ä✩￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä➵ä✥ï✒￾➊ì✑✂❛é✙ø➟✝ è✧ø➀ì➈é✈➶↔ð✶ã✛ç✙ï➓ä✥ï✖å❛ê✤ì❛é☎ê♣ëíì➈ä➤è✥ç✙ø➀ê✒å➄ä✥ï❭ï✄↔✂æ✙û➀å➈ø➑é☎ï✖ù❺ø➑é①✕➔æ☎æ✎ï✖é☎ù✂ø✙↔ ✚➳ð ã✛ç✙ï➳æ✙ä✥ì➈æ◆ï➊ä✧è✧ø➀ï✖ê➏ì➈ë❇ê✧ÿ✗￾❿ç✶å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê➵å➄æ☎æ✙û➑ø➀ï✖ù➓è✧ì❭û➑ï❞å➄ä✥é✙ø➑é✗✂➞ç☎årú➈ï ✡◆ï➊ï➊é❍ê✤è✧ÿ☎ù✙ø➑ï❞ù è✥ç✙ï➊ì❛ä✧ï➊è✧ø✁￾➊å➄û➀û✠✘❤ê✤ø➀é✗￾➊ï✢è✥ç✙ï ➾✽❀✗✓✻✘❂❁ ê✳✞❀✬✠✶➌✒✞➟➾✒✘✠❖➌✟✞✙➾✑➾❪✠✶➌ ✡✙ÿ✂è✌æ✙ä❿å✑￾❶è✧ø✁￾➊å➈û❦ê✧ÿ✗￾✒￾❶ï✖ê✥ê✧ï✖ê♣ëíì➈ä➤é✙ì❛é✦✝üè✥ä✧ø➀ú➇ø➀å➈û❯è✥å➈ê✧ô✂ê➤ù✂ø➀ù❺é✙ì➈è➤ì✦￾✒￾❶ÿ✙ä ÿ✙é✐è✧ø➀û☛è✥ç✙ï✌î➻ø➀ù✿ï➊ø✠✂➈ç✐è✥ø➑ï❞ê➊ð ❃✛ ➠✔➡❺➢ ✪✣ ➳✄➨✗➺ ✬ ➢✻❄✴➼✭✡❅✎➡❘➯❘➤✗➢❪✥❩➢♦➺❩✣ ➯♦➨ â➳ä✥å❛ù✂ø➑ï✖é✐è❇✝r✚➵å❛ê✤ï❞ù ✗✝ï❞å➄ä✥é✙ø➀é❼✂➶æ☎ä✧ì✦￾❶ï❞ù✂ÿ✙ä✥ï✖ê ç☎årú❛ï ✡◆ï➊ï✖éPÿ☎ê✧ï✖ù ê✧ø➑é✗￾➊ï➞è✥ç✙ï❙û✓å✠è✥ï ➾✒❀ ✙❆✘❇❁ ê✒➌✗✡☎ÿ✂è♣è✥ç✙ï✄✘✫ó➵ï➊ä✥ï✒î➻ì✐ê④è✥û✙✘✺û➀ø➑î➻ø➑è✧ï✖ù➲è✧ì✢û➑ø➀é✦✝ ï✖å➈ä✫ê❇✘✂ê✤è✧ï➊î➓ê▲✞✙➾✡✠⑥ð▼ã✛ç✙ï ê✧ÿ✙ä✥æ✙ä✧ø✓ê✧ø➑é❼✂❑ÿ☎ê✧ï❶ëíÿ✙û➀é✙ï✖ê✥ê✺ì➄ë➻ê✤ÿ✥￾❿ç✾ê✧ø➑î➚✝ æ✙û➀ï✩✂➈ä❿å➈ù✂ø➀ï➊é✐è➏ù✂ï❞ê❺￾➊ï➊é✐è❣è✧ï★￾❿ç✙é✙ø✁➍❛ÿ☎ï✖ê❦ëíì❛ä➜￾❶ì❛î❭æ☎û➑ï✄↔✒î➓å✑￾❿ç✙ø➀é✙ï❨û➀ï✖å➄ä✥é✦✝ ø➀é❼✂➓è✥å❛ê✤ô✂ê➉ó➵å❛ê❨é✙ì➄è➳ó❨ø➀ù✙ï➊û✠✘✢ä✥ï✖å➈û➑ø✠➽➊ï❞ù✺ÿ✙é✐è✧ø➀û❀è✥ç✙ï➞ëíì❛û➑û➀ì✠ó❨ø➑é✗✂➻è✧ç✙ä✥ï➊ï ï➊ú❛ï➊é✐è✥ê➳ì✦￾✄￾➊ÿ✙ä✥ä✧ï❞ù☛ð❙ã✛ç✙ï➚➞☎ä❿ê✤è➤ï✖ú➈ï✖é❛è✌ó✛å➈ê♣è✧ç☎ï❭ä✥ï✖å➈û➑ø✠➽✖å➄è✧ø➀ì➈é❖è✧ç☎å➄è✒➌ ù✂ï❞ê✤æ✙ø➑è✧ï✺ï❞å➄ä✥û✙✘➷ó✛å➄ä✥é✙ø➑é✗✂❛ê✌è✥ì è✧ç✙ï✫￾❶ì❛é❛è✥ä✥å➈ä❺✘✩✞✙➾ ✑✆✠❖➌❣è✥ç✙ï✺æ✙ä✧ï❞ê✤ï✖é✗￾❶ï ì➄ë✢û➑ì✦￾➊å➈û➻î❭ø➀é✙ø➀î➓å✾ø➀éòè✥ç✙ï➘û➑ì✐ê✧ê❺ëíÿ✙é✥￾↔è✧ø➀ì➈é▲ù✂ì➇ï✖ê é✙ì➈è➷ê✧ï➊ï✖î è✧ì ✡◆ï❖å î➓å✓④ì❛ä➽æ✙ä✥ì✑✡✙û➀ï➊î ø➑é✻æ☎ä✥å➎￾↔è✧ø✁￾❶ï❛ð❲ã✛ç☎ø➀ê ✡✎ï★￾➊å➄î➻ï❖å➈æ✙æ☎å➈ä✧ï✖é❛è ó❨ç✙ï✖é✻ø➩è✿ó➵å❛ê➓é✙ì➈è✧ø✁￾❶ï✖ù❍è✧ç☎å➄è✢û➀ì☛￾✖å➄û♣î❭ø➀é✙ø➀î➓å ù✂ø✓ù❍é✙ì➄è✺ê✤ï✖ï➊î è✧ì ✡◆ï➻å✿î➓å✓④ì❛ä➤ø➑î➻æ◆ï✖ù✂ø➀î➻ï➊é✐è✌è✧ì✿è✧ç☎ï➻ê✧ÿ✗￾✒￾❶ï✖ê✥ê➳ì➄ë➵ï✖å➄ä✥û✠✘✫é✙ì➈é✦✝⑥û➀ø➑é✙ï❞å➄ä ✂➈ä❿å➈ù✙ø➑ï✖é❛è❺✝✶✡✎å➈ê✧ï✖ù❊✗✝ï✖å➈ä✧é☎ø➑é❼✂➔è✧ï✒￾❿ç☎é✙ø✠➍✐ÿ✙ï❞ê❀ê✧ÿ✗￾❿ç➞å➈ê❳✚➵ì➈û➑è❺➽➊î➓å➈é✙é➞î➓å♦✝ ￾❿ç✙ø➀é✙ï✖ê ✞➟➾ ❜✠❖➌ ✞✙➾❪❝✬✠⑥ð✛ã✛ç✙ï✒ê✧ï✒￾❶ì❛é☎ù✿ï➊ú❛ï➊é✐è➔ó➵å❛ê✛è✧ç☎ï➞æ✎ì❛æ✙ÿ✙û✓å➄ä✥ø✙➽❞å✠è✥ø➑ì❛é ✡☛✘①✍❨ÿ✙î➻ï➊û➀ç☎å➄ä✧è✒➌❣õ➉ø➑é✐è✧ì❛é❤å➄é✎ù ✎ø➑û➀û➀ø➀å➈î➻ê✳✞✙➾ ✙✆✠✛å➄é☎ù ì➄è✧ç☎ï➊ä❿ê➞ì➄ë➔å ê✧ø➑î➻æ✙û➀ï å➄é☎ù✲ï❈✯➣￾❶ø➀ï➊é✐è✫æ✙ä✥ì☛￾➊ï✖ù✂ÿ☎ä✧ï➎➌➔è✧ç☎ï➒✡✎å✑￾❿ô❩✝♠æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ø➀ì➈é❍å➈û➟✝ ✂➈ì❛ä✧ø➑è✧ç☎î➐➌❦è✥ì➛￾➊ì➈î➻æ✙ÿ✂è✥ï➽è✧ç☎ï➙✂❛ä✥å❛ù✂ø➑ï✖é✐è✒ø➑é❑å➲é✙ì❛é✦✝♠û➀ø➀é✙ï✖å➈ä❙ê❺✘➇ê✤è✧ï✖î ￾❶ì❛î➻æ✎ì✐ê✤ï❞ù✫ì➄ë✇ê✧ï➊ú❛ï➊ä❿å➄û❯û➀å✪✘❛ï➊ä❿ê➔ì➈ë❫æ✙ä✥ì☛￾➊ï✖ê✥ê✤ø➀é❼✂✎ð➳ã✛ç✙ï❙è✧ç☎ø➑ä❿ù❖ï✖ú➈ï✖é❛è ó✛å➈ê❙è✧ç✙ï➲ù✙ï➊î➻ì➈é☎ê✤è✧ä❿å✠è✥ø➑ì❛é❤è✥ç☎å✠è➻è✥ç✙ï✆✡☎å✑￾❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é æ✙ä✥ì☛￾➊ï❯✝ ù✂ÿ✙ä✥ï➉å➈æ✙æ✙û➀ø➑ï❞ù➻è✧ì➞î❙ÿ✙û➑è✧ø✙✝♠û✓å✪✘➈ï✖ä➏é✙ï➊ÿ☎ä✥å➈û✙é✙ï❶è④ó➵ì➈ä✥ô✂ê❣ó❨ø➑è✧ç✶ê✧ø✙✂❛î➻ì➈ø✓ù✙å➄û ÿ✙é✙ø➑è✥ê✬￾➊å➄é✺ê✧ì➈û➀ú➈ï✌￾❶ì❛î❭æ☎û➑ø✁￾➊å➄è✧ï✖ù✢û➀ï✖å➄ä✥é✙ø➀é❼✂❙è✥å➈ê✧ô✂ê➊ð➏ã✛ç☎ï✞✡✎å➈ê✧ø✠￾➳ø✓ù✂ï✖å ì➄ë✇✡☎å➎￾❿ô➎✝⑥æ✙ä✥ì➈æ☎å✑✂❛å➄è✧ø➀ì➈é✌ø✓ê❯è✧ç☎å➄è❹✂➈ä❿å➈ù✂ø➀ï➊é✐è✥ê❨￾✖å➄é➚✡✎ï✛￾➊ì➈î➻æ✙ÿ✂è✥ï✖ù❭ï❈✯➝✝ ￾❶ø➀ï➊é✐è✥û✙✘➣✡☛✘➽æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ø➀ì➈é➽ëíä✧ì❛î è✧ç✙ï✌ì❛ÿ✂è✧æ☎ÿ✂è❨è✧ì❭è✧ç☎ï✌ø➑é✙æ☎ÿ✂è✖ð➏ã✛ç✙ø✓ê ø✓ù✂ï✖å✫ó✛å➈ê✌ù✙ï✖ê❘￾❶ä✥ø✙✡◆ï✖ù➷ø➀é➷è✧ç✙ï➙￾❶ì➈é✐è✥ä✧ì❛û❣è✧ç✙ï✖ì➈ä❘✘❖û➀ø➩è✥ï➊ä❿å✠è✥ÿ✙ä✧ï➓ì➈ë✛è✧ç✙ï ï✖å➈ä✧û✠✘➷ê✤ø✙↔➇è✧ø➀ï✖ê✳✞➟➾✒✓✠❖➌❜✡☎ÿ✂è❙ø➑è✥ê❭å➄æ✙æ☎û➑ø✁￾➊å➄è✧ø➀ì➈é è✥ì❖î➓å✑￾❿ç✙ø➀é✙ï➽û➀ï✖å➈ä✧é☎ø➑é❼✂ ó✛å➈ê➓é✙ì➈è ✂➈ï✖é✙ï➊ä❿å➄û➀û✠✘❤ä✥ï✖å➄û➀ø✠➽➊ï✖ù❑è✥ç✙ï➊é✝ð ➏➠é✐è✥ï➊ä✥ï✖ê✤è✧ø➀é❼✂➈û✠✘✑➌✛è✥ç✙ï➲ï✖å➈ä✧û✠✘ ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ì➈é✎ê➽ì➄ë✌✡☎å➎￾❿ô❩✝♠æ✙ä✥ì➈æ✎å❄✂❛å➄è✧ø➀ì➈é ø➑é✻è✧ç✙ï➒￾❶ì➈é✐è✥ï❯↔➇è✶ì➄ë➞é☎ï➊ÿ✙ä❿å➄û é✙ï➊è④ó✇ì❛ä✧ô û➑ï❞å➄ä✥é✙ø➀é❼✂❖ù✂ø✓ù é✙ì➈è❭ÿ✎ê✤ï➙✂➈ä❿å➈ù✂ø➀ï➊é✐è✥ê✒➌❨✡✙ÿ✂è✏☛✤ú➇ø➀ä✤è✥ÿ☎å➄û❫è✥å➈ä❇✝ ✂➈ï➊è✥ê❃✌➞ëíì❛ä❨ÿ✙é✙ø➑è✥ê✛ø➀é✺ø➑é✐è✧ï✖ä✧î➻ï❞ù✂ø➀å➄è✧ï➳û➀å✪✘❛ï➊ä❿ê✒✞➟➾✍✔✠❖➌ ✞✙➾✽✺✬✠✶➌✂ì❛ä✛î❭ø➀é✙ø➀î➓å➄û ù✂ø✓ê④è✥ÿ✙ä❘✡☎å➄é✗￾➊ï➳å➈ä❺✂❛ÿ✙î➻ï➊é✐è✥ê✟✞➟➾✒❀✠⑥ð❣ã✛ç✙ï❊✗❇å❄✂❛ä✥å➈é❼✂➈ï❨ëíì❛ä✧î➓å➈û➑ø✓ê✤î▲ÿ☎ê✧ï✖ù ø➀é❖è✥ç✙ï➣￾❶ì❛é❛è✥ä✧ì❛û❀è✥ç✙ï➊ì❛ä❺✘✫û➀ø➑è✧ï➊ä❿å✠è✥ÿ✙ä✥ï❙æ✙ä✥ì✠ú➇ø➀ù✙ï✖ê➳æ✎ï✖ä✧ç✎å➄æ☎ê♣è✧ç☎ï➵✡✎ï❞ê④è ä✥ø✙✂❛ì➈ä✥ì➈ÿ☎ê☛î➻ï➊è✧ç✙ì✂ù➤ëíì❛ä❇ù✂ï✖ä✧ø➀ú➇ø➑é❼✂④✡☎å✑￾❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é ✞✑❆✘✬✠✶➌✠å➄é✎ù➤ëíì➈ä ù✂ï✖ä✧ø➀ú➇ø➑é❼✂➒✂➈ï✖é✙ï➊ä❿å➄û➀ø✠➽✖å✠è✥ø➑ì❛é☎ê❙ì➄ë④✡✎å✑￾❿ô❩✝♠æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ø➀ì➈é➷è✥ì➷ä✧ï★￾❶ÿ✙ä✥ä✧ï✖é✐è

PROC.OF THE IEEE,NOVEMBER 1998 networks 21,and networks of heterogeneous modules 22 ferentiable,and therefore lends itself to the use of Gradient- A simple derivation for generic multi-layer systems is given Based Learning methods.Section V introduces the use of in Section I-E. directed acyclic graphs whose arcs carry numerical infor- The fact that local minima do not seem to be a problem mation as a way to represent the alternative hypotheses, for multi-layer neural networks is somewhat of a theoretical and introduces the idea of GTN. mystery.It is conPectured that if the network is oversized The second solution described in Section VII is to elim- for the task (as is usually the case in practice),the presence inate segmentation altogether.The idea is to sweep the of "extra dimensions"in parameter space reduces the risk recognizer over every possible location on the input image, of unattainable regions.Back-propagation is by far the and to rely on the "character spotting"property of the rec- most widely used neural-network learning algorithm,and ognizer,i.e.its ability to correctly recognize a well-centered probably the most widely used learning algorithm of any character in its input field,even in the presence of other form. characters besides it,while relecting images containing no centered characters [26],[27].The sequence of recognizer D.Learn Vig Vi F eal s andurig F ecogn on Systems outputs obtained by sweeping the recognizer over the in- Isolated handwritten character recognition has been ex- put is then fed to a Graph Transformer Network that takes tensively studied in the literature(see [23],[24]for reviews). linguistic constraints into account and finally extracts the and was one of the early successful applications of neural most likely interpretation.This GTN is somewhat similar networks [25].Comparative experiments on recognition of to Hidden Markov Models (HMM),which makes the ap- individual handwritten digits are reported in Section III. proach reminiscent of the classical speech recognition [28], They show that neural networks trained with Gradient- [29].While this technique would be quite expensive in Based Learning perform better than all other methods the general case,the use of Convolutional Neural Networks tested here on the same data.The best neural networks, makes it particularly attractive because it allows significant called Convolutional Networks,are designed to learn to savings in computational cost extract relevant features directly from pixel images (see Section II). 7.Glonally Travanle Systems One of the most difficult problems in handwriting recog- As stated earlier,most practical pattern recognition sys- nition,however,is not only to recognize individual charac- tems are composed of multiple modules.For example,a ters,but also to separate out characters from their neigh- document recognition system is composed of a field locator, bors within the word or sentence,a process known as seg- which extracts regions of interest,a field segmenter,which mentation.The technique for doing this that has become cuts the input image into images of candidate characters,a the "standard"is called eurVt Over-Segmentatin.It recognizer,which classifies and scores each candidate char- consists in generating a large number of potential cuts acter,and a contextual post-processor,generally based on between characters using heuristic image processing tech- a stochastic grammar,which selects the best grammatically niques,and subsequently selecting the best combination of correct answer from the hypotheses generated by the recog- cuts based on scores given for each candidate character by nizer.In most cases,the information carried from module the recognizer.In such a model,the accuracy of the sys- to module is best represented as graphs with numerical in- tem depends upon the quality of the cuts generated by the formation attached to the arcs.For example,the output heuristics,and on the ability of the recognizer to distin-of the recognizer module can be represented as an acyclic guish correctly segmented characters from pieces of char-graph where each arc contains the label and the score of acters,multiple characters,or otherwise incorrectly seg- a candidate character,and where each path represent a mented characters.Training a recognizer to perform this alternative interpretation of the input string.Typically, task poses a mapor challenge because of the difficulty in cre- each module is manually optimized,or sometimes trained, ating a labeled database of incorrectly segmented charac- outside of its context.For example,the character recog- ters.The simplest solution consists in running the images nizer would be trained on labeled images of pre-segmented of character strings through the segmenter,and then man- characters.Then the complete system is assembled,and ually labeling all the character hypotheses.Unfortunately,a subset of the parameters of the modules is manually ad- not only is this an extremely tedious and costly task,it is Pusted to maximize the overall performance.This last step also difficult to do the labeling consistently.For example, is extremely tedious,time-consuming,and almost certainly should the right half of a cut up 4 be labeled as a 1 or as suboptimal. a non-character2 should the right half of a cut up 8 be A better alternative would be to somehow train the en- labeled as a 32 tire system so as to minimize a global error measure such as The first solution,described in Section V consists in the probability of character misclassifications at the docu- training the system at the level of whole strings of char-ment level.Ideally,we would want to find a good minimum acters,rather than at the character level.The notion of of this global loss function with respect to all the param- Gradient-Based Learning can be used for this purpose.The eters in the system.If the loss function d measuring the system is trained to minimize an overall loss function which performance can be made differentiable with respect to the measures the probability of an erroneous answer.Section V system's tunable parameters W,we can find a local min- explores various ways to ensure that the loss function is dif- imum of d using Gradient-Based Learning.However,at

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ￾ é✙ï➊è④ó✇ì❛ä✧ô✂ê❅✞✑❼➾✡✠❖➌➈å➄é✎ù✒é✙ï➊è④ó✇ì❛ä✧ô✂ê❯ì➄ë✟ç✙ï➊è✧ï➊ä✥ì✑✂❛ï➊é✙ï✖ì➈ÿ☎ê❇î➻ì✂ù✂ÿ✙û➀ï✖ê ✞✑✫✑ ✠♠ð ✕✻ê✧ø➑î➻æ✙û➀ï➔ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ì➈é❭ëíì❛ä❷✂➈ï➊é☎ï➊ä✥ø✠￾✛î❙ÿ✙û➩è✥ø➟✝⑥û✓å✪✘➈ï➊ä❣ê❺✘✂ê④è✥ï➊î➓ê❣ø✓ê❷✂➈ø➀ú➈ï✖é ø➀é✫þ✂ï✒￾↔è✥ø➑ì❛é✫➏❖✝✭➉ð ã✛ç✙ï➉ë⑨å➎￾↔è✇è✧ç☎å➄è✇û➀ì☛￾✖å➄û◆î➻ø➑é✙ø➀î➓å❙ù✙ì✒é✙ì➈è➵ê✧ï➊ï➊î✭è✥ì➉✡◆ï➳å✒æ✙ä✧ì➎✡✙û➀ï➊î ëíì➈ä❯î✒ÿ☎û➩è✥ø➟✝⑥û➀å✪✘❛ï➊ä❇é✙ï➊ÿ✙ä❿å➄û✐é✙ï➊è④ó✇ì❛ä✧ô✂ê✝ø✓ê❦ê✤ì❛î➻ï➊ó❨ç☎å➄è❇ì➄ë◆å❨è✧ç☎ï➊ì➈ä✥ï❶è✥ø✠￾✖å➄û î➉✘✂ê✤è✧ï✖ä❺✘❛ð✬➏⑥è♣ø✓ê✞￾❶ì➈é✬✓④ï✒￾↔è✥ÿ✙ä✥ï✖ù✺è✥ç☎å✠è➳ø➩ë➏è✥ç✙ï❭é✙ï❶è④ó➵ì➈ä✥ô✶ø✓ê➉ì✠ú❛ï➊ä❿ê✤ø✠➽➊ï❞ù ëíì➈ä❯è✥ç✙ï❫è❿å➈ê✧ô➣➪⑨å❛ê❇ø✓ê❯ÿ☎ê✧ÿ☎å➄û➀û✙✘➳è✥ç✙ï✩￾➊å➈ê✧ï❫ø➀é✒æ☎ä✥å➎￾↔è✧ø✁￾❶ï✪➶✹➌✠è✥ç✙ï➵æ✙ä✧ï❞ê✤ï✖é✗￾❶ï ì➄ë✹☛✧ï❯↔➇è✧ä❿å➻ù✂ø➑î➻ï✖é☎ê✤ø➀ì➈é✎ê❴✌➻ø➀é✫æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä➔ê✧æ☎å➎￾❶ï✌ä✥ï✖ù✂ÿ✗￾➊ï✖ê✛è✥ç✙ï➞ä✥ø➀ê✧ô ì➄ë➞ÿ☎é☎å✠è✧è✥å➄ø➀é☎å✑✡✙û➑ï❺ä✧ï✒✂➈ø➀ì➈é☎ê✖ð②✚✛å✑￾❿ô❩✝⑥æ✙ä✧ì❛æ☎å❄✂✐å✠è✥ø➑ì❛é❤ø✓ê ✡☛✘❑ë⑨å➄ä✢è✧ç✙ï î➻ì❛ê✤è➞ó❨ø➀ù✙ï➊û✠✘❺ÿ☎ê✧ï✖ù é✙ï✖ÿ✙ä❿å➄û✙✝♠é✙ï➊è④ó✇ì❛ä✧ô✫û➀ï✖å➈ä✧é☎ø➑é❼✂✫å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î✫➌❀å➄é☎ù æ✙ä✥ì✑✡☎å✑✡✙û✠✘➲è✧ç☎ï➓î❭ì✐ê④è❙ó❨ø➀ù✂ï✖û✙✘❺ÿ☎ê✧ï✖ù➷û➀ï✖å➄ä✥é✙ø➀é❼✂✫å➄û✠✂➈ì❛ä✧ø➑è✧ç✙î❂ì➄ë❨å➄é☛✘ ëíì➈ä✥î✺ð ✧✛❊✢➳✴➢♦➡✹➨✤✣●➨✦✥❵✣●➨✂✁✬➳✴➢✲☎✄➑➢♦➨ ✪♦➻❷➡▼✣●➺❩✣●➨✵✥✆✁✬➳❄✴➯❴✥✑➨✤✣●➺❩✣ ➯♦➨✞✝☎✟✪➩✹➺❖➳✄➲➑➩ ➏④ê✤ì❛û➀å➄è✧ï✖ù✿ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï✖é✫￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä✛ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✿ç☎å❛ê✔✡✎ï✖ï➊é✫ï❯↔☛✝ è✧ï✖é☎ê✧ø➑ú❛ï➊û✠✘♣ê✤è✧ÿ☎ù✂ø➀ï✖ù➤ø➀é➤è✧ç☎ï❫û➀ø➩è✥ï➊ä❿å✠è✧ÿ☎ä✧ï✩➪⑨ê✧ï➊ï✟✞✑✫❜✠❖➌✵✞✑❝✫✠➄ëíì➈ä✝ä✥ï➊ú➇ø➀ï➊ó➔ê✴➶✹➌ å➄é✎ù❖ó✛å➈ê♣ì➈é✙ï➻ì➈ë➏è✧ç✙ï➻ï❞å➄ä✥û✙✘➲ê✧ÿ✗￾✄￾➊ï✖ê✥ê④ëíÿ☎û❦å➈æ✙æ✙û➀ø✠￾✖å✠è✥ø➑ì❛é☎ê➳ì➄ë✇é☎ï➊ÿ✙ä❿å➄û é✙ï➊è④ó✇ì❛ä✧ô✂ê✹✞✑✫✙✆✠⑥ð✎☞✇ì❛î❭æ✎å➄ä❿å✠è✧ø➀ú➈ï➳ï❯↔✂æ◆ï➊ä✥ø➑î➻ï✖é❛è❿ê❨ì➈é✺ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✢ì➈ë ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û✇ç✎å➄é☎ù✂ó❨ä✥ø➑è✤è✧ï✖é ù✂ø✠✂➈ø➑è✥ê✒å➈ä✧ï✶ä✧ï✖æ✎ì❛ä✤è✥ï✖ù➷ø➀é❤þ➇ï✒￾❶è✧ø➀ì➈é✢➏❺➏❇➏↔ð ã✛ç✙ï✒✘➘ê✧ç✙ì✠ó▼è✧ç✎å✠è✢é☎ï➊ÿ✙ä❿å➄û♣é✙ï❶è④ó➵ì➈ä✥ô✂ê➻è✧ä❿å➄ø➀é✙ï✖ù✻ó❨ø➩è✥ç✾â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝ ✚✛å➈ê✧ï✖ù✷✗❀ï✖å➈ä✧é✙ø➀é❼✂✾æ✎ï✖ä✤ëíì❛ä✧î ✡◆ï❶è✤è✥ï➊ä è✧ç✎å➄é➶å➈û➑û➻ì➈è✧ç✙ï✖ä❺î➻ï❶è✥ç✙ì✂ù✙ê è✧ï❞ê④è✥ï✖ù ç✙ï✖ä✧ï➻ì➈é❺è✧ç☎ï➽ê✧å➈î❭ï➓ù✙å➄è✥å☎ð➻ã✛ç☎ï➚✡◆ï✖ê✤è➞é✙ï✖ÿ✙ä✥å➈û❣é✙ï❶è④ó➵ì➈ä✥ô✂ê✄➌ ￾➊å➈û➑û➀ï✖ù✟☞✇ì➈é➇ú➈ì❛û➑ÿ✙è✧ø➀ì➈é☎å➈û♣ñ➔ï➊è④ó✇ì❛ä✧ô✂ê✒➌➉å➄ä✥ï❺ù✙ï✖ê✧ø✙✂❛é✙ï✖ù✻è✧ì û➑ï❞å➄ä✥é❍è✧ì ï❯↔➇è✥ä✥å➎￾↔è✢ä✥ï➊û➀ï➊ú✠å➈é❛è✢ëíï✖å➄è✧ÿ✙ä✥ï✖ê✺ù✂ø➀ä✧ï★￾↔è✥û✙✘ ëíä✥ì➈î æ✙ø✙↔✂ï➊û➤ø➑î➓å✑✂➈ï✖ê➛➪⑨ê✧ï➊ï þ➇ï★￾↔è✧ø➀ì➈é✫➏❺➏❇➶↔ð ý♣é✙ï➔ì➄ë✟è✥ç✙ï➉î➻ì❛ê✤è✇ù✂ø✯➣￾❶ÿ✙û➑è✇æ☎ä✧ì➎✡✙û➑ï✖î➓ê➏ø➀é➓ç☎å➄é☎ù✙ó❨ä✧ø➑è✧ø➀é❼✂➞ä✥ï✒￾➊ì✑✂❄✝ é✙ø➑è✧ø➀ì➈é✏➌➇ç✙ì✠ó➵ï➊ú❛ï➊ä★➌➈ø✓ê❫é☎ì➄è✛ì➈é✙û✠✘❭è✧ì✒ä✥ï✒￾➊ì✑✂❛é✙ø✙➽✖ï➔ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û✈￾❿ç✎å➄ä❿å✑￾✹✝ è✧ï✖ä✥ê✒➌✈✡✙ÿ✂è✌å➄û✓ê✤ì➻è✥ì✢ê✧ï➊æ☎å➈ä✥å➄è✧ï✌ì➈ÿ✙è✞￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê❨ëíä✧ì❛î✴è✥ç✙ï➊ø➀ä♣é☎ï➊ø✠✂➈ç✦✝ ✡◆ì➈ä❿ê❨ó❨ø➩è✥ç✙ø➀é✺è✧ç✙ï✌ó➵ì➈ä❿ù✶ì❛ä➔ê✤ï✖é✐è✧ï➊é✥￾❶ï✑➌◆å❭æ✙ä✧ì✦￾➊ï✖ê✥ê✛ô✐é☎ì✠ó❨é✺å❛ê❨ê✤ï✒✂❄✝ î➻ï➊é✐è✥å➄è✧ø➀ì➈é✝ð✌ã✛ç✙ï✒è✧ï★￾❿ç✙é✙ø✁➍❛ÿ☎ï✒ëíì❛ä➳ù✂ì❛ø➑é✗✂➽è✧ç☎ø➀ê♣è✧ç✎å✠è♣ç✎å➈ê④✡✎ï★￾❶ì➈î➻ï è✧ç☎ï ☛✧ê✤è✥å➈é☎ù✙å➈ä✥ù✌✿ø✓ê➉￾✖å➄û➀û➑ï❞ù✠✄➑➳☛✡✦➡▼✣✁➩✹➺❩✣✂❄✌☞✎✍♦➳✄➡▼✭✏✝✇➳✿✥❄➲➵➳✄➨✗➺r➢♦➺❩✣ ➯♦➨☎ð✫➏⑥è ￾❶ì❛é☎ê✧ø➀ê✤è✥ê❖ø➑é②✂❛ï➊é✙ï✖ä✥å➄è✧ø➀é❼✂✻å❍û✓å➄ä❘✂➈ï é➇ÿ✙î➑✡✎ï✖ä❖ì➈ë➽æ✎ì➈è✧ï➊é✐è✥ø➀å➈û➑￾❶ÿ✙è✥ê ✡◆ï❶è④ó➵ï➊ï➊é ￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê➉ÿ☎ê✧ø➑é❼✂✺ç✙ï✖ÿ✙ä✥ø➀ê✤è✧ø✁￾✒ø➀î➓å❄✂❛ï❙æ✙ä✥ì✦￾❶ï❞ê✧ê✧ø➑é✗✂✶è✥ï✒￾❿ç✦✝ é✙ø✁➍✐ÿ✙ï✖ê✒➌✂å➄é☎ù➓ê✧ÿ❼✡☎ê✧ï✒➍✐ÿ✙ï✖é❛è✥û✙✘➓ê✧ï➊û➀ï✒￾❶è✧ø➀é❼✂✌è✧ç☎ï④✡◆ï✖ê✤è✎￾➊ì➈î➉✡☎ø➑é☎å➄è✧ø➀ì➈é➓ì➈ë ￾❶ÿ✙è✥ê✔✡☎å❛ê✤ï❞ù➽ì➈é✿ê❺￾➊ì➈ä✥ï✖ê➃✂➈ø➀ú➈ï✖é➽ëíì➈ä✛ï❞å✑￾❿ç➙￾✖å➄é☎ù✙ø➀ù✙å➄è✧ït￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä➃✡☛✘ è✧ç☎ï➓ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ï➊ä❞ð➓➏➠é➷ê✤ÿ✥￾❿ç❺å✿î➻ì➇ù✙ï➊û✶➌☛è✧ç✙ï➽å➎￾✄￾➊ÿ✙ä✥å➎￾❯✘✺ì➄ë✇è✧ç✙ï➓ê❺✘✂ê❅✝ è✧ï✖î ù✂ï➊æ◆ï➊é☎ù☎ê✇ÿ☎æ✎ì❛é➓è✧ç✙ï✞➍✐ÿ☎å➄û➀ø➩è❅✘❭ì➄ë☛è✥ç✙ï④￾➊ÿ✂è✥ê➃✂➈ï✖é✙ï➊ä❿å✠è✥ï✖ù➚✡☛✘❭è✧ç✙ï ç✙ï✖ÿ✙ä✧ø✓ê✤è✧ø✁￾➊ê✒➌❫å➄é☎ù❤ì➈é❤è✧ç✙ï✺å❄✡✙ø➀û➀ø➩è❅✘ ì➈ë➔è✧ç✙ï✺ä✥ï✒￾➊ì✑✂➈é☎ø✙➽✖ï➊ä➞è✥ì ù✙ø➀ê✤è✧ø➀é✦✝ ✂➈ÿ☎ø➀ê✧ç①￾❶ì➈ä✥ä✥ï✒￾↔è✥û✙✘❺ê✤ï✒✂➈î➻ï➊é✐è✧ï❞ù➛￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê♣ëíä✧ì❛î❋æ✙ø➀ï✒￾➊ï✖ê✌ì➈ë✔￾❿ç☎å➈ä❇✝ å✑￾❶è✧ï✖ä✥ê✒➌➔î✒ÿ☎û➩è✥ø➑æ✙û➀ï➛￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê✄➌❨ì❛ä✶ì➈è✧ç✙ï✖ä✧ó❨ø✓ê✤ï❖ø➑é✗￾➊ì➈ä✥ä✧ï★￾↔è✥û✙✘➘ê✤ï✒✂❄✝ î➻ï➊é✐è✧ï❞ù ￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä❿ê✖ð❭ã❇ä✥å➈ø➑é☎ø➑é❼✂✫å✿ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï✖ä➳è✧ì✫æ◆ï➊ä✧ëíì➈ä✥î è✧ç✙ø✓ê è✥å❛ê✤ô➤æ◆ì❛ê✧ï✖ê❇å➔î➓å✓④ì❛ä❳￾❿ç☎å➈û➑û➀ï➊é❼✂❛ï❷✡✎ï★￾➊å➈ÿ☎ê✤ï✇ì➄ë✂è✥ç✙ï✇ù✂ø✯➣￾❶ÿ✙û➑è❅✘✌ø➀é➉￾❶ä✥ï❯✝ å✠è✥ø➑é✗✂✿å✿û✓å❄✡◆ï➊û➀ï✖ù ù✙å✠è❿å❄✡☎å❛ê✤ï❙ì➄ë✇ø➀é✗￾➊ì➈ä✥ä✧ï★￾↔è✧û✠✘✫ê✤ï✒✂➈î➻ï➊é✐è✥ï✖ù➔￾❿ç✎å➄ä❿å✑￾✹✝ è✧ï✖ä✥ê✖ð✛ã✛ç✙ï➞ê✤ø➀î➻æ✙û➀ï✖ê✤è♣ê✤ì❛û➑ÿ✙è✧ø➀ì➈é✆￾❶ì❛é☎ê✤ø✓ê✤è✥ê❨ø➀é✫ä✧ÿ☎é✙é✙ø➀é❼✂➻è✧ç✙ï✒ø➑î➓å❄✂❛ï✖ê ì➄ë❀￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä➵ê④è✥ä✧ø➀é❼✂❛ê➏è✧ç✙ä✥ì➈ÿ✗✂➈ç➻è✧ç☎ï➳ê✧ï✄✂❛î❭ï✖é✐è✧ï➊ä★➌➇å➄é☎ù➻è✥ç✙ï➊é✢î➓å➄é✦✝ ÿ☎å➈û➑û✠✘➓û➀å✑✡✎ï✖û➑ø➀é❼✂➓å➄û➀û◆è✧ç✙ï✌￾❿ç☎å➈ä✥å➎￾↔è✥ï➊ä❫ç☛✘➇æ◆ì➄è✧ç☎ï✖ê✧ï✖ê✖ð❨→➉é✂ëíì➈ä✧è✧ÿ☎é☎å✠è✥ï➊û✠✘✑➌ é✙ì➈è➉ì➈é✙û✠✘✶ø✓ê❨è✥ç✙ø✓ê➉å➄é✫ï✄↔➇è✧ä✥ï➊î➻ï➊û✠✘➽è✥ï✖ù✂ø➀ì➈ÿ☎ê♣å➄é☎ù✫￾➊ì❛ê✤è✧û✠✘➽è✥å❛ê✤ô✇➌✎ø➩è➉ø✓ê å➄û✓ê✧ì✶ù✂ø✯➵￾➊ÿ✙û➑è♣è✥ì✶ù✂ì➽è✥ç✙ï✒û✓å❄✡◆ï➊û➀ø➀é❼✂➙￾➊ì➈é☎ê✧ø➀ê✤è✧ï✖é✐è✧û✠✘➈ð✬✜☎ì➈ä➳ï❯↔✙å➄î➻æ✙û➀ï✑➌ ê✧ç✙ì➈ÿ✙û✓ù✺è✥ç✙ï✒ä✥ø✙✂❛ç✐è➉ç☎å➄û➑ë➏ì➄ë❫å ￾❶ÿ✂è➳ÿ✙æ ❝➣✡◆ï✒û✓å❄✡◆ï➊û➀ï✖ù❖å➈ê➉å➔➾➞ì➈ä➳å➈ê å é✙ì➈é❼✝❖￾❿ç☎å➈ä✥å➎￾↔è✥ï➊ä✒✑ ê✤ç✙ì❛ÿ✙û✓ù è✥ç✙ï✺ä✧ø✠✂➈ç✐è➻ç☎å➄û➑ë♣ì➈ë➳å➒￾❶ÿ✂è➓ÿ✙æ ✺➒✡✎ï û✓å❄✡◆ï➊û➀ï✖ù✺å➈ê❨å❑❜✓✑ ã✛ç✙ï①➞☎ä❿ê④è❖ê✧ì➈û➀ÿ✂è✥ø➑ì❛é✏➌➞ù✂ï✖ê❘￾❶ä✥ø✠✡✎ï❞ù✾ø➀é✹þ➇ï★￾↔è✧ø➀ì➈é✟✣ ￾➊ì➈é☎ê✧ø✓ê④è❿ê✫ø➑é è✧ä❿å➄ø➀é✙ø➀é❼✂✫è✥ç✙ï✢ê❺✘✂ê④è✥ï➊î å✠è❙è✧ç✙ï✢û➑ï✖ú➈ï➊û✇ì➄ë➔ó❨ç✙ì❛û➑ï✢ê④è✥ä✧ø➀é❼✂❛ê✒ì➄ë✬￾❿ç☎å➈ä❇✝ å✑￾❶è✧ï✖ä✥ê✒➌❦ä✥å➄è✧ç✙ï✖ä➞è✧ç✎å➄é❤å➄è➞è✧ç✙ï✫￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä➞û➀ï➊ú➈ï✖ûüð❺ã✛ç✙ï✶é✙ì➈è✧ø➀ì➈é ì➈ë â➳ä❿å➈ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù✒✗✝ï❞å➄ä✥é✙ø➀é❼✂✛￾➊å➈é➓✡✎ï✇ÿ☎ê✧ï✖ù➳ëíì➈ä❇è✧ç☎ø➀ê❀æ✙ÿ✙ä✥æ✎ì✐ê✤ï❛ð❇ã✛ç✙ï ê❺✘➇ê✤è✧ï✖î ø➀ê❀è✧ä❿å➄ø➀é✙ï❞ù➤è✧ì➳î➻ø➑é✙ø➀î➻ø✙➽✖ï➵å➈é✒ì✠ú❛ï➊ä❿å➄û➀û➄û➑ì✐ê✧ê❀ëíÿ✙é✗￾❶è✧ø➀ì➈é❙ó❨ç✙ø✠￾❿ç î➻ï✖å❛ê✤ÿ✙ä✥ï✖ê✝è✧ç✙ï✇æ✙ä✥ì✑✡☎å✑✡✙ø➑û➀ø➑è❅✘♣ì➈ë☎å➄é➞ï✖ä✧ä✥ì➈é✙ï✖ì➈ÿ☎ê❀å➄é☎ê✧ó✇ï✖ä✖ð❯þ➇ï✒￾❶è✧ø➀ì➈é➓✣ ï❯↔✂æ✙û➀ì➈ä✥ï✖ê❇ú✠å➄ä✥ø➑ì❛ÿ☎ê❇ó➵å✪✘✂ê☛è✥ì♣ï✖é☎ê✧ÿ✙ä✧ï❫è✧ç☎å➄è❦è✧ç✙ï➵û➑ì✐ê✧ê❀ëíÿ✙é✗￾❶è✧ø➀ì➈é❭ø➀ê❦ù✂ø➑ë●✝ ëíï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï➎➌✠å➄é☎ù➳è✥ç✙ï➊ä✥ï❶ëíì❛ä✧ï➏û➀ï➊é✎ù✙ê✝ø➑è✥ê✧ï➊û➑ë✂è✧ì❨è✥ç✙ï❫ÿ✎ê✤ï✇ì➄ë☎â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝ ✚✛å➈ê✧ï✖ù❖✗❀ï✖å➄ä✥é✙ø➀é❼✂➓î➻ï❶è✥ç✙ì✂ù✙ê➊ð➳þ➇ï✒￾❶è✧ø➀ì➈é✆✣✹ø➀é✐è✧ä✥ì✂ù✂ÿ✗￾❶ï❞ê❨è✥ç✙ï✒ÿ☎ê✧ï➞ì➈ë ù✂ø➀ä✧ï★￾↔è✥ï✖ù å✑￾✄✘✦￾❶û➀ø✠￾➵✂❛ä✥å➈æ✙ç☎ê✌ó❨ç☎ì❛ê✧ï✶å➄ä✴￾➊ê➓￾✖å➄ä✥ä❺✘❺é➇ÿ✙î➻ï➊ä✥ø✠￾✖å➄û❫ø➀é✂ëíì❛ä❇✝ î➓å✠è✥ø➑ì❛é å➈ê✒å➲ó➵å✪✘❺è✧ì➲ä✥ï➊æ☎ä✧ï❞ê✤ï✖é❛è✌è✥ç✙ï✢å➈û➩è✥ï➊ä✥é☎å✠è✥ø➑ú❛ï➽ç☛✘➇æ✎ì➈è✧ç✙ï❞ê✤ï❞ê✄➌ å➄é✎ù✶ø➀é✐è✧ä✥ì➇ù✙ÿ✗￾❶ï❞ê➵è✧ç✙ï✌ø✓ù✂ï✖å➻ì➈ë❦â➤ã❨ñ✒ð ã✛ç✙ï➞ê✤ï★￾❶ì❛é☎ù✺ê✤ì❛û➑ÿ✂è✥ø➑ì❛é✫ù✂ï✖ê❘￾❶ä✥ø✙✡◆ï✖ù✿ø➀é➲þ➇ï✒￾❶è✧ø➀ì➈é✫✣✬➏❺➏➵ø✓ê✛è✥ì➓ï➊û➀ø➑î➚✝ ø➀é☎å✠è✥ï✺ê✧ï✄✂❛î➻ï➊é✐è✥å➄è✧ø➀ì➈é❤å➈û➩è✥ì✑✂➈ï➊è✧ç✙ï✖ä✖ð❑ã✛ç✙ï✺ø➀ù✂ï❞å❖ø✓ê❭è✧ì ê✧ó✇ï✖ï➊æ❤è✧ç✙ï ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï✖ä➏ì✠ú➈ï✖ä➏ï➊ú❛ï➊ä❘✘❙æ◆ì❛ê✥ê✧ø✙✡✙û➀ï➉û➀ì✦￾➊å➄è✧ø➀ì➈é➓ì➈é➽è✧ç✙ï➳ø➑é☎æ✙ÿ✂è✛ø➑î➓å❄✂❛ï✑➌ å➄é✎ù➞è✥ì➳ä✥ï➊û✠✘➞ì❛é➞è✧ç☎ï ☛❺￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä❯ê✧æ◆ì➄è✤è✥ø➑é✗✂✌♣æ✙ä✧ì❛æ✎ï✖ä✤è❅✘✌ì➈ë☎è✥ç✙ï❨ä✧ï★￾✹✝ ì✑✂❛é✙ø✠➽➊ï➊ä★➌❞øüð ï➈ð❇ø➩è❿ê❀å❄✡☎ø➑û➀ø➩è❅✘➳è✥ì✬￾❶ì❛ä✧ä✥ï✒￾❶è✧û✠✘➉ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï❣å❨ó➵ï➊û➀û✙✝❖￾➊ï➊é✐è✧ï✖ä✧ï❞ù ￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✌ø➑é❤ø➩è❿ê➞ø➀é✙æ✙ÿ✂è➑➞☎ï➊û✓ù❢➌❦ï➊ú❛ï➊é ø➀é➷è✧ç✙ï✢æ✙ä✧ï❞ê✤ï✖é✗￾❶ï➓ì➈ë➔ì➄è✥ç✙ï➊ä ￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê✎✡◆ï✖ê✧ø➀ù✙ï✖ê➔ø➑è✒➌✎ó❨ç✙ø➀û➑ï➞ä✥ï✓④ï★￾↔è✥ø➑é❼✂➓ø➀î➓å❄✂❛ï✖ê✛￾➊ì➈é✐è✥å➈ø➑é✙ø➀é❼✂➻é✙ì ￾❶ï✖é✐è✧ï➊ä✥ï✖ù➒￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê❄✞✑❆✓✠❖➌ ✞✑✗✔✆✠♠ð➻ã✛ç✙ï➽ê✧ï✒➍✐ÿ✙ï➊é✥￾❶ï➻ì➄ë➵ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ï➊ä ì➈ÿ✙è✧æ✙ÿ✂è❿ê➞ì➎✡✂è✥å➈ø➑é✙ï❞ù ✡☛✘❺ê✧ó✇ï✖ï➊æ✙ø➀é❼✂✺è✧ç☎ï➽ä✧ï★￾❶ì➎✂➈é✙ø✠➽➊ï✖ä✌ì✠ú➈ï➊ä➤è✥ç✙ï➽ø➀é✦✝ æ✙ÿ✂è✇ø➀ê❯è✥ç✙ï➊é➻ëíï❞ù✒è✥ì➞å➞â➳ä✥å➈æ✙ç➻ã❀ä❿å➄é☎ê✤ëíì➈ä✥î➻ï➊ä❣ñ➔ï➊è④ó✇ì❛ä✧ô➞è✧ç☎å➄è❣è✥å➄ô❛ï✖ê û➀ø➑é❼✂❛ÿ✙ø✓ê④è✥ø✠￾➑￾❶ì➈é✎ê④è✥ä✥å➈ø➑é✐è✥ê➔ø➑é✐è✧ì✢å✑￾✄￾➊ì➈ÿ✙é✐è♣å➈é☎ù✫➞☎é✎å➄û➀û✙✘✿ï❯↔➇è✧ä❿å✑￾❶è✥ê➔è✧ç✙ï î➻ì❛ê✤è➔û➑ø➀ô➈ï✖û✙✘✢ø➑é✐è✥ï➊ä✥æ✙ä✧ï➊è✥å➄è✧ø➀ì➈é✝ð➵ã✛ç✙ø➀ê➳â➤ã❨ñ ø✓ê➔ê✤ì❛î➻ï➊ó❨ç☎å➄è➉ê✤ø➀î➻ø➑û✓å➄ä è✧ì õ➔ø✓ù✙ù✂ï✖é❑ö➲å➈ä✧ô❛ì✠ú❖ö✫ì✂ù✂ï➊û✓ê➐➪⑨õ➉ö❖ö➔➶✹➌❦ó❨ç✙ø✁￾❿ç î➓å➄ô❛ï✖ê✌è✥ç✙ï✺å➄æ✦✝ æ✙ä✥ì❛å➎￾❿ç✶ä✥ï➊î➻ø➑é☎ø➀ê❘￾❶ï✖é❛è❨ì➈ë❇è✥ç✙ï➓￾❶û✓å➈ê✥ê✤ø✁￾➊å➈û☛ê✤æ◆ï➊ï★￾❿ç✿ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é ✞✑❆✺✬✠✶➌ ✞✑❆❀✠⑥ð ✎ç✙ø➀û➀ï❖è✥ç✙ø➀ê✿è✥ï✒￾❿ç✙é☎ø✠➍✐ÿ✙ï➷ó➵ì➈ÿ✙û✓ù❭✡◆ï ➍✐ÿ✙ø➑è✧ï➷ï✄↔➇æ◆ï➊é✎ê✤ø➀ú➈ï ø➑é è✧ç☎ï✩✂➈ï➊é☎ï➊ä❿å➄û❼￾➊å❛ê✤ï➎➌✠è✧ç☎ï❨ÿ☎ê✤ï❨ì➈ë✏☞✇ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✂ñ➔ï✖ÿ✙ä❿å➄û☎ñ➉ï❶è④ó➵ì➈ä✥ô➇ê î➓å➄ô❛ï✖ê✝ø➑è❇æ✎å➄ä✧è✧ø✁￾❶ÿ✙û✓å➄ä✥û✙✘➤å✠è✧è✧ä❿å✑￾❶è✧ø➀ú➈ï❹✡◆ï✒￾✖å➄ÿ☎ê✧ï❫ø➑è❯å➈û➑û➀ì✠ó➔ê❀ê✧ø✙✂❛é✙ø✙➞✥￾➊å➈é❛è ê✥årú✐ø➀é❼✂✐ê✇ø➀é✆￾❶ì❛î❭æ☎ÿ✂è✥å➄è✧ø➀ì➈é☎å➈û✏￾❶ì❛ê✤è✖ð ✔✛ ➠❅✲✙➯✫✯✴➢✲ ✲✕✟ ➦❼➡❺➢✣●➨✇➢✫✯❪✲✙➳✖✝☎✟✪➩✹➺❖➳✄➲➑➩ ✕➉ê❫ê✤è✥å➄è✧ï❞ù➻ï✖å➄ä✥û➀ø➑ï✖ä✒➌➄î➻ì✐ê④è➏æ✙ä❿å✑￾❶è✧ø✁￾➊å➈û✙æ☎å✠è✧è✧ï✖ä✧é➓ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é❭ê❺✘✂ê❅✝ è✧ï✖î➓ê❙å➈ä✧ï➐￾❶ì➈î➻æ◆ì❛ê✧ï✖ù➷ì➈ë➔î✒ÿ✙û➑è✧ø➀æ✙û➀ï✢î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð➒✜☎ì➈ä❭ï❯↔✙å➄î➻æ✙û➀ï✑➌❣å ù✂ì✦￾❶ÿ☎î❭ï✖é✐è❯ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é➞ê❇✘✂ê✤è✧ï➊î ø✓ê❳￾➊ì➈î➻æ◆ì❛ê✧ï✖ù✌ì➈ë✎å✩➞☎ï✖û➀ù➞û➑ì✦￾➊å➄è✧ì❛ä✒➌ ó❨ç✙ø✁￾❿ç✶ï✄↔✐è✥ä✥å➎￾↔è❿ê❫ä✥ï✄✂❛ø➑ì❛é☎ê❫ì➈ë❀ø➀é❛è✥ï➊ä✥ï✖ê✤è✒➌✂å✌➞✎ï➊û✓ù✢ê✧ï✄✂❛î❭ï✖é✐è✧ï➊ä★➌✐ó❨ç✙ø✠￾❿ç ￾❶ÿ✙è✥ê❦è✥ç✙ï➔ø➀é✙æ✙ÿ✂è➏ø➀î➓å❄✂➈ï❨ø➀é✐è✧ì➤ø➀î➻å✑✂➈ï❞ê❦ì➄ë❢￾➊å➄é✎ù✂ø➀ù☎å✠è✧ï✛￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê✄➌✠å ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï✖ä✒➌➈ó❨ç✙ø✠￾❿ç➣￾➊û➀å❛ê✧ê✧ø✙➞☎ï✖ê➏å➈é☎ù➓ê❘￾❶ì➈ä✥ï✖ê❦ï❞å✑￾❿ç➣￾✖å➄é☎ù✂ø✓ù✙å➄è✧ï✬￾❿ç☎å➈ä❇✝ å✑￾❶è✧ï✖ä✒➌☎å➈é☎ù✺å➝￾❶ì➈é✐è✥ï❯↔➇è✧ÿ☎å➈û✝æ◆ì❛ê✤è❇✝⑥æ✙ä✧ì✦￾➊ï✖ê✥ê✤ì❛ä✒➌☛✂➈ï✖é✙ï➊ä❿å➄û➀û✠✘➣✡☎å➈ê✧ï✖ù✢ì➈é å➔ê✤è✧ì✦￾❿ç☎å❛ê④è✥ø✠￾❷✂❛ä✥å➈î➻î➻å➈ä✒➌✖ó❨ç✙ø✁￾❿ç➞ê✤ï✖û➑ï★￾↔è✥ê✝è✧ç✙ï➃✡✎ï❞ê④è❜✂➈ä❿å➄î➻î➓å✠è✥ø✠￾✖å➄û➀û✙✘ ￾❶ì❛ä✧ä✥ï✒￾❶è❇å➄é✎ê✤ó➵ï➊ä❇ëíä✧ì❛î❲è✧ç✙ï➵ç☛✘✐æ◆ì➄è✥ç✙ï✖ê✧ï✖ê❳✂➈ï✖é✙ï➊ä❿å✠è✥ï✖ù✌✡☛✘➳è✧ç☎ï✇ä✥ï✒￾➊ì✑✂❄✝ é✙ø✠➽➊ï✖ä✖ð➃➏➠é✿î➻ì✐ê④è✬￾✖å➈ê✧ï✖ê✒➌✂è✧ç✙ï✌ø➀é✂ëíì❛ä✧î➓å✠è✥ø➑ì❛é✆￾➊å➄ä✥ä✥ø➑ï❞ù➽ëíä✥ì➈îPî❭ì✂ù✂ÿ☎û➑ï è✧ì❙î➻ì➇ù✙ÿ✙û➑ï➳ø✓ê➜✡◆ï✖ê✤è✇ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✥ï✖ù➽å❛ê❹✂➈ä❿å➄æ☎ç☎ê➏ó❨ø➑è✧ç✢é✐ÿ☎î❭ï✖ä✧ø✁￾➊å➈û☎ø➀é✦✝ ëíì➈ä✥î➓å✠è✥ø➑ì❛é➷å➄è✤è❿å✑￾❿ç✙ï❞ù❖è✥ì✺è✧ç✙ï✢å➄ä✴￾➊ê✖ð ✜✙ì❛ä➞ï❯↔✙å➄î➻æ✙û➀ï✑➌❇è✧ç☎ï➽ì➈ÿ✂è✥æ✙ÿ✂è ì➄ë❣è✥ç✙ï❭ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ï➊ä➉î❭ì✂ù✂ÿ☎û➑ï➝￾➊å➈é↕✡✎ï❙ä✧ï✖æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù✫å➈ê➳å➄é❖å➎￾❯✘✦￾❶û➀ø✠￾ ✂➈ä❿å➄æ☎ç ó❨ç☎ï➊ä✥ï➓ï✖å✑￾❿ç å➈ä❘￾➵￾➊ì➈é✐è✥å➈ø➑é☎ê✌è✧ç✙ï✢û➀å✑✡✎ï✖û❫å➄é☎ù è✥ç✙ï✢ê❘￾❶ì❛ä✧ï➻ì➈ë å✢￾✖å➄é☎ù✂ø✓ù✙å➄è✧ï➔￾❿ç☎å➈ä✥å➎￾↔è✥ï➊ä★➌✛å➈é☎ù➘ó❨ç✙ï✖ä✧ï➲ï❞å✑￾❿ç❍æ☎å➄è✧ç❍ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✶å å➄û➑è✧ï✖ä✧é✎å✠è✧ø➀ú➈ï✺ø➀é✐è✧ï➊ä✥æ✙ä✥ï❶è❿å✠è✧ø➀ì➈é➘ì➈ë♣è✥ç✙ï➲ø➀é✙æ✙ÿ✂è✢ê✤è✧ä✥ø➀é❼✂☎ð❲ã➜✘➇æ✙ø✁￾➊å➈û➑û✠✘✑➌ ï✖å➎￾❿ç✢î➻ì✂ù✂ÿ✙û➀ï➤ø➀ê➔î➻å➈é➇ÿ☎å➄û➀û✙✘➓ì❛æ✂è✧ø➀î➻ø✙➽✖ï✖ù❢➌☎ì➈ä❨ê✤ì❛î➻ï❶è✧ø➀î➻ï✖ê➵è✥ä✥å➈ø➑é✙ï❞ù❢➌ ì➈ÿ✙è✥ê✧ø➀ù✂ï✶ì➄ë❨ø➩è❿ê➑￾➊ì➈é✐è✧ï✄↔➇è✖ð↕✜✙ì❛ä➞ï❯↔✙å➈î❭æ☎û➑ï➎➌❇è✥ç✙ï➙￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä✌ä✥ï✒￾➊ì✑✂❄✝ é✙ø✠➽➊ï✖ä✇ó➵ì➈ÿ✙û✓ù➵✡◆ï➉è✥ä✥å➈ø➑é☎ï✖ù➓ì❛é➽û➀å✑✡✎ï✖û➑ï❞ù➽ø➀î➓å❄✂➈ï❞ê➏ì➈ë✝æ✙ä✧ï✄✝⑥ê✧ï✄✂❛î➻ï➊é✐è✧ï❞ù ￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê✖ð➲ã✛ç✙ï✖é è✧ç✙ï✫￾❶ì❛î➻æ✙û➑ï➊è✧ï✶ê❺✘✂ê④è✥ï➊î÷ø✓ê❙å❛ê✧ê✧ï➊î➑✡✙û➑ï❞ù❢➌❣å➄é☎ù å➻ê✤ÿ✗✡☎ê✤ï➊è➔ì➄ë❇è✧ç✙ï✌æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê➵ì➈ë❇è✥ç✙ï✌î➻ì➇ù✙ÿ✙û➑ï❞ê✛ø➀ê➔î➻å➈é➇ÿ☎å➄û➀û✙✘✢å➈ù☛✝ ✓④ÿ☎ê✤è✧ï❞ù➓è✧ì❙î➓å♦↔✂ø➑î➻ø✠➽➊ï➉è✧ç✙ï♣ì✠ú➈ï➊ä❿å➄û➀û✙æ◆ï➊ä✧ëíì➈ä✥î➓å➄é✗￾➊ï➈ð❦ã✛ç✙ø➀ê✇û✓å➈ê✤è➵ê✤è✧ï➊æ ø✓ê❦ï❯↔➇è✧ä✥ï➊î➻ï➊û✠✘➤è✧ï❞ù✂ø➀ì➈ÿ☎ê✒➌✠è✧ø➀î➻ï❯✝r￾❶ì❛é☎ê✤ÿ☎î❭ø➀é❼✂✥➌➈å➄é☎ù❭å➄û➀î➻ì❛ê✤è❨￾➊ï➊ä✧è✥å➈ø➑é✙û✠✘ ê✧ÿ❼✡✎ì❛æ✂è✧ø➀î➓å➄û♠ð ✕ ✡✎ï➊è✤è✥ï➊ä♣å➈û➩è✥ï➊ä✥é☎å✠è✥ø➑ú❛ï✌ó✇ì❛ÿ✙û✓ù➙✡◆ï➞è✥ì➽ê✤ì❛î➻ï➊ç✙ì✠ó✾è✧ä❿å➄ø➀é✿è✧ç✙ï➞ï✖é✦✝ è✧ø➀ä✥ï✇ê❺✘➇ê✤è✧ï✖îòê✤ì➳å➈ê✝è✧ì➳î❭ø➀é✙ø➀î➻ø✙➽✖ï➵å✬✂➈û➀ì✑✡☎å➈û➈ï➊ä✥ä✥ì➈ä❀î➻ï❞å➈ê✧ÿ✙ä✧ï➵ê✤ÿ✥￾❿ç✒å➈ê è✧ç☎ï✌æ✙ä✧ì➎✡☎å❄✡☎ø➑û➀ø➩è❅✘✶ì➄ë❜￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä✛î➻ø➀ê❘￾❶û✓å➈ê✥ê✧ø➟➞✥￾✖å✠è✥ø➑ì❛é☎ê✛å✠è✛è✧ç✙ï➞ù✙ì☛￾➊ÿ✦✝ î➻ï➊é✐è❣û➑ï✖ú➈ï✖ûüð❳➏④ù✂ï✖å➈û➑û✠✘✑➌✠ó➵ï✇ó➵ì➈ÿ☎û➀ù✒ó➵å➈é❛è❯è✧ì④➞☎é☎ù❭å④✂➈ì➇ì✂ù✌î➻ø➑é✙ø➀î✒ÿ☎î ì➄ë❫è✧ç✙ø✓êt✂➈û➀ì✑✡☎å➈û❇û➀ì❛ê✥ê♣ëíÿ✙é✗￾❶è✧ø➀ì➈é ó❨ø➩è✥ç❖ä✥ï✖ê✧æ✎ï★￾↔è➳è✧ì✫å➄û➀û❯è✧ç✙ï❭æ☎å➄ä❿å➄î➚✝ ï❶è✥ï➊ä❿ê➳ø➑é❺è✧ç✙ï➽ê❺✘✂ê④è✥ï➊î✺ð➉➏⑥ë✇è✧ç✙ï➓û➀ì❛ê✥ê♣ëíÿ✙é✗￾❶è✧ø➀ì➈é▲❉✴î➻ï✖å❛ê✤ÿ☎ä✧ø➀é❼✂✢è✧ç✙ï æ◆ï➊ä✧ëíì➈ä✥î➻å➈é✗￾❶ï✩￾✖å➄é➚✡✎ï➔î➓å❛ù✂ï❨ù✂ø➟➘✟ï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï❨ó❨ø➑è✧ç➻ä✥ï✖ê✧æ✎ï★￾↔è❣è✥ì➤è✧ç✙ï ê❺✘➇ê✤è✧ï✖î❁ ê♣è✧ÿ☎é☎å❄✡✙û➀ï➓æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê✒❂➌✝ó➵ï➚￾✖å➄é➛➞☎é☎ù å✢û➀ì✦￾➊å➈û❯î➻ø➀é✦✝ ø➀î✒ÿ✙î ì➈ë❇❉ ÿ☎ê✧ø➑é❼✂➷â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù▲✗✝ï❞å➄ä✥é✙ø➑é✗✂☎ð❺õ➉ì✠ó✇ï✖ú➈ï✖ä✒➌❦å✠è

2Fs-s 4 OvE2EEE7Xs LEO AEF 1ii8 first glance,it appears that the sheer size and complexity tion system is best represented by graphs with numerical of the system would make this intractable. information attached to the arcs.In this case,each module, To ensure that the global loss function EP(ZP'W)is dif- called a Graph Transformer,takes one or more graphs as ferentiable,the overall system is built as a feed-forward net- input,and produces a graph as output.Networks of such work of differentiable modules.The function implemented modules are called Graph Transformer Networks (GTN). by each module must be continuous and differentiable alE Sections IV,VI and VIII develop the concept of GTNs, most ererywhere with respect to the internal parameters of and show that Gradient-Based Learning can be used to the module (e.g.the weights of a Neural Net character rec- train all the parameters in all the modules so as to mini- ognizer in the case of a character recognition module),and mize a global loss function.It may seem paradoxical that with respect to the module's inputs.If this is the case,a gradients can be computed when the state information is simple generalization of the well-known back-propagation represented by essentially discrete obfects such as graphs, procedure can be used to efficiently compute the gradients but that difficulty can be circumvented,as shown later. of the loss function with respect to all the parameters in the system [22].For example,let us consider a system II.Y ON VOLUTIONAL NEURAL NETWORKVFOR built as a cascade of modules,each of which implements a IVOLATED Y HARACTER RECOGNITION function Xn 8 Fn(Wn'Xn-1),where Xn is a vector rep- The ability of multi-layer networks trained with gradi- resenting the output of the module,Wn is the vector of ent descent to learn complex,high-dimensional,non-linear tunable parameters in the module (a subset of W),and mappings from large collections of examples makes them Xn-1 is the module's input vector (as well as the previous obvious candidates for image recognition tasks.In the tra- module's output vector).The input X-to the first module ditional model of pattern recognition,a hand-designed fea- is the input pattern Zp.If the partial derivative of Ep with ture extractor gathers relevant information from the input respect to Xn is known,then the partial derivatives of Ep and eliminates irrelevant variabilities.A trainable classifier with respect to Wn and Xn-1 can be computed using the then categorizes the resulting feature vectors into classes. backward recurrence In this scheme,standard,fully-connected multi-layer net- 8EP 8 8 W w(W'Xn-1) EP works can be used as classifiers.A potentially more inter- esting scheme is to rely on as much as possible on learning 8EP 8F EP in the feature extractor itself.In the case of character -8 8Xn-1 (W'X-1) Xn (4) recognition,a network could be fed with almost raw in- puts (e.g.size-normalized images).While this can be done where(W)is the Jacobian of F with respect to with an ordinary fully connected feed-for ward network with W evaluated at the point (Wn'Xn-1),andWXn-) some success for tasks such as character recognition,there is the Jacobian of F with respect to X.The Jacobian of are problems. a vector function is a matrix containing the partial deriva- Firstly,typical images are large,often with several hun- tives of all the outputs with respect to all the inputs. dred variables (pixels).A fully-connected first layer with, The first equation computes some terms of the gradient say one hundred hidden units in the first layer,would al- of EP(W),while the second equation generates a back-ready contain several tens of thousands of weights.Such ward recurrence,as in the well-known back-propagation a large number of parameters increases the capacity of the procedure for neural networks.We can average the gradi- system and therefore requires a larger training set.In ad- ents over the training patterns to obtain the full gradient.dition,the memory requirement to store so many weights It is interesting to note that in many instances there is may rule out certain hardware implementations.But,the no need to explicitly compute the Jacobian matrix.The main deficiency of unstructured nets for image or speech above formula uses the product of the Jacobian with a vec- applications is that they have no built-in invariance with tor of partial derivatives,and it is often easier to compute respect to translations,or local distortions of the inputs. this product directly without computing the Jacobian be- Before being sent to the fixed-size input layer of a neural forehand.In By analogy with ordinary multi-layer neural net,character images,or other 2D or 1D signals,must be networks,all but the last module are called hidden layers approximately size-normalized and centered in the input because their outputs are not observable from the outside. field.Unfortunately,no such preprocessing can be perfect: more complex situations than the simple cascade of mod- handwriting is often normalized at the word level,which ules described above,the partial derivative notation be- can cause size,slant,and position variations for individual comes somewhat ambiguous and awkward.A completely characters.This,combined with variability in writing style, rigorous derivation in more general cases can be done using will cause variations in the position of distinctive features Lagrange functions [20],[21],[22]. in input obbects.In principle,a fully-connected network of Traditional multi-layer neural networks are a special case sufficient size could learn to produce outputs that are in- of the above where the state information xn is represented variant with respect to such variations.However,learning with fixed-sized vectors,and where the modules are al- such a task would probably result in multiple units with ternated lavers of matrix multiplications (the weights)and similar weight patterns positioned at various locations in component-wise sigmoid functions (the neurons).However, the input so as to detect distinctive features wherever they as stated ear lier,the state information in complex recogni- appear on the input.Learning these weight configurations

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ￾ ➞☎ä❿ê④è✞✂➈û✓å➄é✗￾➊ï✑➌◆ø➩è✌å➄æ☎æ✎ï❞å➄ä❿ê❨è✧ç✎å✠è➳è✧ç✙ï➻ê✤ç☎ï➊ï➊ä➤ê✧ø✙➽✖ï❙å➄é✎ù↕￾❶ì❛î❭æ☎û➑ï✄↔➇ø➑è❅✘ ì➄ë❯è✧ç☎ï➞ê❇✘✂ê✤è✧ï➊î✴ó✇ì❛ÿ✙û➀ù✢î➓å➄ô❛ï♣è✥ç✙ø✓ê❨ø➑é✐è✥ä✥å➎￾↔è✥å✑✡✙û➀ï➈ð ã❀ì➞ï✖é☎ê✧ÿ✙ä✧ï✛è✥ç☎å✠è✇è✧ç✙ï✬✂➈û➀ì✑✡☎å➈û✙û➑ì✐ê✧ê❣ëíÿ✙é✗￾↔è✥ø➑ì❛é✘❉❊✸❼➪❩✾❅✸ ❁❃❂➶❦ø✓ê❫ù✂ø➑ë●✝ ëíï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï➎➌❞è✧ç✙ï➏ì✠ú❛ï➊ä❿å➄û➀û➄ê❇✘✂ê✤è✧ï✖î✾ø✓ê✏✡✙ÿ☎ø➑û➑è❯å❛ê❀å✛ëíï✖ï✖ù☛✝♠ëíì➈ä✥ó➵å➈ä✥ù♣é✙ï➊è❇✝ ó➵ì➈ä✥ô➓ì➄ë❣ù✂ø➟➘✟ï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï➤î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð➏ã✛ç✙ï➤ëíÿ✙é✗￾❶è✧ø➀ì➈é✺ø➀î❭æ☎û➑ï✖î❭ï✖é✐è✧ï✖ù ✡☛✘✫ï✖å➎￾❿ç❖î➻ì✂ù✂ÿ✙û➀ï❭î❙ÿ☎ê✤è✞✡◆ï➵￾❶ì➈é✐è✥ø➑é➇ÿ✙ì❛ÿ☎ê♣å➈é☎ù ù✂ø➟➘✟ï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï✆➢✲✭ ➲➵➯✪➩✹➺❹➳☛✍♦➳❯➡✒✟♦➻❳➥✗➳❯➡❘➳➏ó❨ø➑è✧ç➓ä✥ï✖ê✧æ✎ï★￾↔è❣è✥ì✌è✧ç✙ï➉ø➑é✐è✧ï✖ä✧é✎å➄û✙æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê❯ì➈ë è✧ç☎ï❨î❭ì✂ù✂ÿ☎û➑ï➓➪⑨ï➈ð ✂☎ð❯è✧ç✙ï❨ó➵ï➊ø✠✂➈ç✐è❿ê❦ì➄ë◆å✌ñ➉ï➊ÿ✙ä❿å➄û✙ñ➉ï❶è➜￾❿ç☎å➈ä✥å➎￾↔è✥ï➊ä❦ä✧ï★￾✹✝ ì✑✂❛é✙ø✠➽➊ï➊ä❫ø➑é➓è✥ç✙ï④￾✖å➈ê✧ï➔ì➄ë❀å➉￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä❣ä✥ï✒￾➊ì✑✂❛é✙ø➩è✥ø➑ì❛é➓î➻ì➇ù✙ÿ✙û➑ï✪➶✹➌➇å➄é☎ù ó❨ø➑è✧ç ä✧ï❞ê✤æ◆ï✒￾❶è♣è✥ì✶è✥ç✙ï➻î❭ì✂ù✂ÿ☎û➑ï✗❁ ê➳ø➀é✙æ✙ÿ✂è❿ê➊ð➓➏⑥ë✇è✧ç✙ø✓ê➤ø✓ê➉è✥ç✙ï➝￾➊å➈ê✧ï✑➌✝å ê✧ø➑î➻æ✙û➀ï➵✂➈ï✖é✙ï➊ä❿å➄û➀ø✙➽❞å✠è✥ø➑ì❛é❖ì➄ë➵è✧ç☎ï➽ó✇ï✖û➑û✙✝⑥ô✐é☎ì✠ó❨é➛✡☎å➎￾❿ô❩✝♠æ✙ä✥ì➈æ✎å❄✂❛å➄è✧ø➀ì➈é æ✙ä✥ì✦￾❶ï✖ù✙ÿ✙ä✧ï✞￾➊å➈é ✡◆ï♣ÿ☎ê✧ï✖ù➽è✧ì❭ï❈✯➣￾❶ø➀ï➊é✐è✥û✙✘➣￾➊ì➈î➻æ✙ÿ✂è✥ï➉è✥ç✙ït✂➈ä❿å➈ù✂ø➀ï➊é✐è✥ê ì➄ë✛è✥ç✙ï➽û➀ì❛ê✥ê➤ëíÿ✙é✗￾❶è✧ø➀ì➈é ó❨ø➑è✧ç ä✥ï✖ê✧æ✎ï★￾↔è✌è✧ì❺å➄û➀û❣è✧ç✙ï✶æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê➤ø➑é è✧ç☎ï ê❺✘✂ê④è✥ï➊î ✞✑ ✑✆✠⑥ð➧✜✙ì❛ä✢ï✄↔✙å➄î➻æ✙û➀ï✑➌➔û➀ï❶è✿ÿ☎ê✫￾❶ì➈é✎ê✤ø✓ù✂ï➊ä✿å ê❺✘➇ê✤è✧ï✖î ✡✙ÿ✙ø➀û➑è➉å➈ê❨å➝￾➊å❛ê❺￾✖å➈ù✂ï➳ì➄ë❦î➻ì➇ù✙ÿ✙û➑ï❞ê✄➌✂ï❞å✑￾❿ç✿ì➄ë❯ó❨ç✙ø✠￾❿ç✺ø➀î➻æ✙û➑ï✖î➻ï➊é✐è✥ê➔å ëíÿ✙é✗￾❶è✧ø➀ì➈é✂✁❳✏✺ ✼❳❢➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶❯➌❯ó❨ç☎ï➊ä✥ï☎✁❑❳❺ø✓ê✒å✺ú➈ï★￾↔è✥ì➈ä✌ä✥ï➊æ✦✝ ä✥ï✖ê✧ï➊é✐è✧ø➀é❼✂❺è✧ç✙ï✫ì❛ÿ✂è✧æ✙ÿ✙è➻ì➈ë➉è✥ç✙ï✫î➻ì➇ù✙ÿ✙û➑ï➎➌❀❂❳❤ø✓ê❭è✧ç☎ï✫ú➈ï✒￾❶è✧ì❛ä❙ì➈ë è✧ÿ☎é☎å❄✡✙û➀ï✫æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê❭ø➀é❑è✥ç✙ï✫î➻ì✂ù✂ÿ✙û➀ï✢➪⑨å ê✧ÿ❼✡☎ê✧ï❶è➓ì➄ë✜❂➶✹➌✛å➄é☎ù ✁❳✸✴ ✜➉ø✓ê✇è✧ç✙ï➤î➻ì✂ù✂ÿ✙û➀ï✻❁ ê✛ø➑é✙æ☎ÿ✂è❨ú➈ï✒￾❶è✧ì❛ät➪üå➈ê➵ó➵ï➊û➀û✟å❛ê✇è✧ç✙ï➤æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê î➻ì✂ù✂ÿ✙û➀ï✻❁ ê➏ì➈ÿ✙è✧æ✙ÿ✂è➵ú➈ï★￾↔è✧ì❛ä✴➶❶ð❯ã✛ç☎ï➔ø➑é☎æ✙ÿ✂è✆✁✞✝❨è✥ì✌è✧ç☎ï✛➞☎ä❿ê④è✇î❭ì✂ù✂ÿ☎û➑ï ø✓ê❯è✧ç✙ï❨ø➀é✙æ✙ÿ✂è➏æ✎å✠è✤è✥ï➊ä✥é✘✾❀✸✂ð❳➏⑥ë◆è✧ç☎ï➵æ✎å➄ä✧è✧ø✓å➄û✙ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ú➈ï➵ì➄ë ❉❊✸♣ó❨ø➑è✧ç ä✥ï✖ê✧æ✎ï★￾↔è❨è✧ì☎✁❳✢ø➀ê➔ô✐é☎ì✠ó❨é✏➌➇è✧ç✙ï✖é✿è✧ç✙ï➞æ✎å➄ä✧è✧ø✓å➄û✝ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ú➈ï❞ê✛ì➄ë ❉❊✸ ó❨ø➑è✧ç➲ä✥ï✖ê✧æ✎ï★￾↔è➔è✥ì✳❂❳ å➄é☎ù✟✁❳✸✴ ✜t￾➊å➈é✆✡✎ï➚￾❶ì❛î❭æ☎ÿ✂è✧ï❞ù✫ÿ☎ê✧ø➑é❼✂➽è✧ç✙ï ✡☎å➎￾❿ô✐ó✛å➄ä❿ù➽ä✥ï✒￾➊ÿ✙ä✥ä✧ï✖é✗￾❶ï ✷❉❊✸ ✷❂❳ ✺ ✷✼ ✷❂ ➪❩❂❳ ❁✄✁❳✸✴ ✜ ➶ ✷❉❊✸ ✷✁❳ ✷❉❊✸ ✷✁❳✸✴ ✜ ✺ ✷✼ ✷✁ ➪❩❂❳ ❁✄✁❳✸✴ ✜✄➶ ✷❉❊✸ ✷✁❳ ➪❏❝❩➶ ó❨ç✙ï✖ä✧ï✡✠☞☛ ✠✍✌ ➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶❫ø➀ê✇è✧ç☎ï✏✎✐å✑￾➊ì✑✡✙ø✓å➄é➓ì➈ë ✼ ó❨ø➑è✧ç✶ä✥ï✖ê✧æ◆ï✒￾↔è✇è✧ì ❂ ï✖úrå➈û➑ÿ✎å✠è✧ï❞ù➽å✠è✇è✧ç✙ï➳æ◆ì➈ø➀é❛èt➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶❯➌➇å➈é☎ù✑✠✒☛ ✠✒✓ ➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶ ø✓ê➳è✧ç✙ï☎✎✐å✑￾❶ì➎✡✙ø✓å➄é❖ì➈ë❀✼✴ó❨ø➩è✥ç❺ä✥ï✖ê✧æ✎ï★￾↔è➤è✧ì✟✁ ð➻ã✛ç✙ï☎✎✐å✑￾❶ì➎✡✙ø✓å➄é➲ì➈ë å❙ú➈ï✒￾❶è✧ì❛ä➏ëíÿ✙é✗￾❶è✧ø➀ì➈é✿ø➀ê✛å✒î➻å➄è✧ä✥ø➟↔ ￾➊ì➈é✐è✥å➈ø➑é☎ø➑é❼✂✒è✧ç✙ï➳æ☎å➈ä✤è✥ø➀å➈û◆ù✂ï✖ä✧ø➀ú✠å♦✝ è✧ø➀ú➈ï❞ê❖ì➈ë✢å➄û➀û❙è✧ç✙ï ì➈ÿ✂è✥æ✙ÿ✂è❿ê❖ó❨ø➑è✧ç➶ä✧ï❞ê✤æ◆ï✒￾❶è❖è✧ì å➄û➀û❙è✧ç✙ï ø➑é☎æ✙ÿ✂è✥ê✖ð ã✛ç✙ï✆➞✎ä✥ê✤è➽ï✒➍✐ÿ☎å➄è✧ø➀ì➈é❭￾❶ì❛î❭æ☎ÿ✂è✧ï❞ê✶ê✧ì➈î➻ï✺è✧ï✖ä✧î➓ê➓ì➈ë♣è✥ç✙ï↕✂➈ä❿å➈ù✙ø➑ï✖é❛è ì➄ë❑❉❊✸❼➪✮❂➶✹➌➤ó❨ç✙ø➀û➀ï❖è✥ç✙ï ê✤ï★￾❶ì➈é✎ù✻ï✒➍✐ÿ☎å➄è✧ø➀ì➈é ✂➈ï➊é☎ï➊ä❿å✠è✧ï❞ê✶å⑧✡☎å✑￾❿ô❩✝ ó✛å➄ä❿ù❍ä✧ï★￾❶ÿ✙ä✥ä✥ï➊é✗￾➊ï✑➌➔å➈ê✢ø➀é✻è✧ç☎ï❺ó➵ï➊û➀û➟✝⑥ô➇é✙ì✠ó❨é ✡☎å➎￾❿ô❩✝♠æ✙ä✥ì➈æ✎å❄✂❛å➄è✧ø➀ì➈é æ✙ä✥ì✦￾❶ï✖ù✙ÿ✙ä✧ï➤ëíì❛ä➉é✙ï✖ÿ✙ä❿å➄û❀é☎ï❶è④ó➵ì➈ä✥ô➇ê✖ð ✎ï➚￾➊å➄é➲årú❛ï➊ä❿å❄✂❛ï➉è✥ç✙ï➉✂❛ä✥å❛ù✂ø➟✝ ï➊é✐è❿ê❨ì✠ú➈ï✖ä✛è✧ç✙ï✌è✥ä✥å➈ø➑é☎ø➑é❼✂➓æ✎å✠è✤è✥ï➊ä✥é☎ê✛è✥ì➓ì✑✡✂è❿å➄ø➀é✢è✥ç✙ï✌ëíÿ✙û➀û❀✂❛ä✥å❛ù✂ø➀ï➊é✐è✖ð ➏⑥è✢ø✓ê➽ø➀é✐è✧ï✖ä✧ï❞ê④è✥ø➑é❼✂ è✧ì é✙ì➈è✧ï✫è✥ç☎å✠è✿ø➑é✲î➻å➈é☛✘❤ø➀é☎ê④è❿å➄é✗￾➊ï✖ê➽è✧ç✙ï✖ä✧ï➲ø✓ê é✙ì❺é✙ï➊ï❞ù è✧ì❖ï✄↔✂æ✙û➑ø✁￾❶ø➑è✧û✠✘①￾❶ì❛î❭æ☎ÿ✂è✧ï➓è✥ç✙ï✔✎✐å✑￾➊ì✑✡✙ø✓å➄é î➓å➄è✧ä✥ø➟↔☛ð➲ã✛ç✙ï å❄✡◆ì✠ú➈ï➵ëíì❛ä✧î❙ÿ✙û➀å➳ÿ☎ê✧ï✖ê❣è✧ç✙ï➔æ☎ä✧ì✂ù✂ÿ✗￾❶è➏ì➄ë◆è✧ç✙ï✕✎❛å➎￾❶ì✑✡☎ø➀å➈é❙ó❨ø➑è✧ç➓å➤ú➈ï★￾✹✝ è✧ì❛ä❨ì➄ë❦æ☎å➄ä✧è✧ø✓å➄û✝ù✙ï➊ä✥ø➑ú✠å✠è✥ø➑ú❛ï✖ê✒➌✂å➄é☎ù✿ø➑è➔ø➀ê✛ì➈ë➺è✧ï➊é✺ï❞å➈ê✧ø➑ï✖ä✛è✧ì➵￾❶ì❛î❭æ☎ÿ✂è✧ï è✧ç☎ø➀ê♣æ✙ä✥ì➇ù✙ÿ✗￾↔è➤ù✂ø➑ä✥ï✒￾❶è✧û✠✘✢ó❨ø➑è✧ç☎ì➈ÿ✂èt￾❶ì➈î➻æ✙ÿ✙è✧ø➀é❼✂➽è✥ç✙ï✖✎✐å✑￾➊ì✑✡✙ø✓å➄é✫✡◆ï❯✝ ëíì➈ä✥ï➊ç✎å➄é☎ù☛ð✞➏➠é➛✚✎✘✫å➄é✎å➄û➀ì✑✂✑✘✢ó❨ø➩è✥ç❖ì❛ä✥ù✂ø➀é☎å➈ä❺✘✿î✒ÿ✙û➑è✧ø✙✝⑥û➀å✪✘❛ï➊ä➉é☎ï➊ÿ✙ä❿å➄û é✙ï➊è④ó✇ì❛ä✧ô✂ê✒➌◆å➈û➑û❜✡✙ÿ✂è➤è✧ç☎ï❙û✓å➈ê✤è♣î➻ì✂ù✂ÿ✙û➀ï❭å➈ä✧ï➚￾➊å➈û➑û➀ï✖ù✫ç☎ø➀ù✙ù✙ï➊é❖û✓å✪✘➈ï✖ä✥ê ✡◆ï✒￾➊å➈ÿ☎ê✧ï➉è✥ç✙ï➊ø➀ä❨ì➈ÿ✂è✥æ✙ÿ✂è❿ê➵å➈ä✧ï➳é✙ì➈è➵ì➎✡☎ê✤ï✖ä✧ú✠å✑✡✙û➑ï➉ëíä✧ì❛î▲è✧ç✙ï➤ì❛ÿ✂è✥ê✧ø➀ù✙ï➈ð î➻ì➈ä✥ï➑￾➊ì➈î➻æ✙û➀ï❯↔➲ê✤ø➑è✧ÿ✎å✠è✧ø➀ì➈é✎ê➔è✥ç☎å➄é❖è✧ç✙ï➻ê✤ø➀î➻æ✙û➀ï➚￾✖å➈ê❘￾➊å➈ù✙ï➞ì➄ë➏î➻ì✂ù☛✝ ÿ✙û➀ï✖ê✢ù✂ï✖ê❘￾❶ä✥ø✙✡◆ï✖ù❍å✑✡✎ì✠ú❛ï✑➌➵è✧ç☎ï❖æ☎å➈ä✤è✥ø➀å➈û➉ù✂ï➊ä✥ø➀úrå➄è✧ø➀ú➈ï✫é✙ì➈è✥å➄è✧ø➀ì➈é❭✡◆ï❯✝ ￾❶ì❛î➻ï✖ê➞ê✧ì➈î➻ï➊ó❨ç✎å✠è➞å➄î➑✡✙ø✙✂❛ÿ✙ì➈ÿ✎ê✌å➄é☎ù➷åró❨ô➇ó➵å➈ä✥ù✝ð➣✕ ￾❶ì➈î➻æ✙û➀ï❶è✥ï➊û✠✘ ä✥ø✙✂❛ì➈ä✥ì➈ÿ☎ê❯ù✂ï➊ä✥ø➀úrå➄è✧ø➀ì➈é❭ø➑é❭î➻ì➈ä✥ï➃✂❛ï➊é✙ï✖ä✥å➈û✦￾➊å➈ê✧ï✖ê❨￾✖å➄é➚✡✎ï➔ù✂ì❛é✙ï✛ÿ☎ê✧ø➑é❼✂ ✗❀å✑✂➈ä❿å➄é❼✂❛ï➔ëíÿ✙é✥￾↔è✧ø➀ì➈é✎ê✜✞✑✻✘✠❖➌ ✞✑❼➾✡✠❖➌ ✞✑ ✑✆✠⑥ð ã❀ä❿å➈ù✂ø➑è✧ø➀ì➈é✎å➄û➈î❙ÿ✙û➑è✧ø✙✝♠û✓å✪✘➈ï✖ä❀é✙ï✖ÿ✙ä✥å➈û❛é☎ï❶è④ó➵ì➈ä✥ô➇ê❇å➄ä✥ï❫å➉ê✧æ◆ï✒￾❶ø✓å➄û☛￾➊å❛ê✤ï ì➄ë✟è✥ç✙ï➉å❄✡◆ì✠ú➈ï✛ó❨ç✙ï✖ä✧ï✛è✥ç✙ï♣ê✤è✥å➄è✧ï❨ø➀é✂ëíì➈ä✥î➓å✠è✥ø➑ì❛é✗✁❳ ø✓ê❣ä✧ï✖æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù ó❨ø➑è✧ç ➞❼↔✂ï✖ù☛✝➠ê✤ø✠➽➊ï❞ù✻ú❛ï✒￾↔è✥ì➈ä❿ê✄➌♣å➄é☎ù✲ó❨ç✙ï➊ä✥ï❖è✥ç✙ï➷î➻ì➇ù✙ÿ✙û➑ï❞ê✫å➄ä✥ï❺å➈û➟✝ è✧ï✖ä✧é✎å✠è✧ï❞ù➓û✓å✪✘➈ï➊ä❿ê❦ì➄ë✝î➓å➄è✧ä✥ø➟↔➻î✒ÿ✙û➑è✧ø➀æ✙û➀ø✠￾✖å✠è✥ø➑ì❛é☎ê④➪íè✧ç✙ï♣ó✇ï✖ø✙✂❛ç✐è✥ê✴➶❣å➄é☎ù ￾❶ì❛î➻æ✎ì❛é✙ï➊é✐è❇✝⑥ó❨ø✓ê✤ï❫ê✧ø✠✂➈î➻ì➈ø✓ù➳ëíÿ✙é✗￾❶è✧ø➀ì➈é☎ê✎➪➺è✥ç✙ï❫é✙ï✖ÿ✙ä✥ì➈é☎ê✴➶↔ð❯õ➔ì✠ó➵ï➊ú➈ï✖ä✒➌ å➈ê✇ê④è❿å✠è✥ï✖ù➓ï❞å➄ä✥û➑ø➀ï➊ä★➌➈è✥ç✙ï♣ê✤è✥å➄è✧ï♣ø➑é✂ëíì❛ä✧î➓å➄è✧ø➀ì➈é➓ø➀é ￾❶ì❛î❭æ☎û➑ï✄↔➻ä✥ï✒￾➊ì✑✂❛é✙ø➟✝ è✧ø➀ì➈é➷ê❺✘➇ê✤è✧ï✖î❋ø✓êt✡✎ï❞ê④è➞ä✥ï➊æ☎ä✧ï❞ê✤ï✖é❛è✥ï✖ù➔✡❩✘➔✂➈ä❿å➄æ✙ç☎ê➳ó❨ø➑è✧ç➷é➇ÿ✙î➻ï➊ä✥ø✠￾✖å➄û ø➀é✂ëíì➈ä✥î➓å✠è✧ø➀ì➈é➞å➄è✤è❿å✑￾❿ç✙ï❞ù➳è✧ì➔è✥ç✙ï✇å➈ä❘￾✖ê➊ð❀➏➠é✌è✧ç✙ø✓ê❜￾➊å➈ê✧ï✑➌rï❞å✑￾❿ç✌î➻ì✂ù✂ÿ✙û➀ï✑➌ ￾➊å➈û➑û➀ï✖ù❺å✫â➳ä✥å➈æ✙ç❺ã❇ä✥å➈é☎ê✤ëíì➈ä✥î❭ï✖ä✒➌◆è❿å➄ô➈ï❞ê➉ì❛é✙ï➻ì➈ä➤î➻ì➈ä✥ï➉✂❛ä✥å➈æ✙ç☎ê➤å➈ê ø➀é✙æ✙ÿ✂è★➌✝å➄é☎ù✺æ✙ä✥ì✂ù✂ÿ✗￾➊ï✖ê➳å➣✂➈ä❿å➄æ☎ç➲å➈ê➔ì❛ÿ✂è✧æ☎ÿ✂è✖ð➤ñ➔ï➊è④ó✇ì❛ä✧ô✂ê❨ì➈ë❫ê✤ÿ✗￾❿ç î➻ì✂ù✂ÿ✙û➀ï✖ê❭å➄ä✥ï➣￾➊å➄û➀û➀ï✖ù❤â➳ä❿å➄æ✙ç ã❇ä❿å➄é☎ê✤ëíì➈ä✥î➻ï➊ä✒ñ➉ï❶è④ó➵ì➈ä✥ô➇ê ➪üâ➤ã❨ñ④➶❶ð þ➇ï★￾↔è✧ø➀ì➈é✎ê➵➏r✣➉➌✔✣✬➏➻å➄é✎ù⑧✣④➏❇➏❺➏➻ù✙ï➊ú➈ï✖û➑ì❛æ❤è✥ç✙ï↕￾❶ì❛é✗￾❶ï✖æ✂è➽ì➄ë➞â➤ã❨ñ➉ê✒➌ å➄é✎ù✻ê✤ç☎ì✠ó✴è✥ç☎å✠è➲â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù ✗✝ï✖å➈ä✧é☎ø➑é❼✂⑧￾➊å➄é❭✡◆ï❖ÿ✎ê✤ï❞ù➘è✧ì è✧ä❿å➄ø➀é❺å➈û➑û❦è✥ç✙ï➻æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê♣ø➑é➷å➄û➀û❦è✧ç✙ï➻î➻ì✂ù✂ÿ✙û➀ï✖ê✌ê✧ì✿å❛ê♣è✧ì✺î➻ø➑é✙ø✙✝ î➻ø✙➽✖ï❙å➵✂❛û➑ì➎✡☎å➄û❀û➑ì✐ê✧ê➔ëíÿ✙é✗￾↔è✥ø➑ì❛é✝ð✬➏⑥è♣î➻å✪✘✺ê✧ï➊ï➊î æ☎å➈ä✥å❛ù✂ì✪↔✂ø✠￾✖å➄û✟è✥ç☎å✠è ✂➈ä❿å➈ù✙ø➑ï✖é❛è❿êt￾➊å➄é➒✡◆ï➣￾❶ì➈î➻æ✙ÿ✙è✧ï✖ù❺ó❨ç✙ï✖é❺è✥ç✙ï➓ê④è❿å✠è✥ï➓ø➑é✂ëíì❛ä✧î➓å➄è✧ø➀ì➈é❖ø✓ê ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù✫✡☛✘✢ï❞ê✧ê✧ï➊é✐è✧ø✓å➄û➀û✠✘✿ù✂ø✓ê❘￾❶ä✥ï❶è✧ï❙ì✑✡✔✓④ï★￾↔è❿ê➉ê✧ÿ✗￾❿ç❖å➈ê✬✂➈ä❿å➄æ✙ç✎ê✄➌ ✡✙ÿ✂è❨è✥ç☎å✠è➉ù✙ø✯➣￾❶ÿ☎û➩è❅✘➐￾➊å➄é➐✡◆ï➓￾❶ø➀ä❘￾➊ÿ✙î✒ú❛ï➊é✐è✧ï❞ù❢➌✙å➈ê❨ê✧ç✙ì✠ó❨é✿û➀å➄è✧ï➊ä❞ð ➇✄➇★➈☎✘➳×❀Ú✚✙✝×❇Ü✐ß☛Þ❢➋í×❀Ú☛Ý❇Ü✢Ö❭Ù✙ß✝à✟Ý❀Ü✺Ö❭Ù✂Þ✜✛➻×❇à✚✢✤✣✗✥✐×❀à ➇✦✣✖×❇Ü❛Ý✂Þ✟Ù✗➊✧✘✕★☛Ý❇à✟Ý☛Û✎Þ☛Ù✙à✪✩➞Ù☎Û✟×✬✫❇Ú✏➋✓Þ❢➋í×❇Ú ã✛ç✙ï✢å✑✡✙ø➑û➀ø➑è❅✘❺ì➈ë➔î✒ÿ✙û➑è✧ø✙✝♠û✓å✪✘➈ï✖ä➞é✙ï➊è④ó✇ì❛ä✧ô✂ê➤è✥ä✥å➈ø➑é✙ï❞ù➷ó❨ø➩è✥ç➹✂❛ä✥å❛ù✂ø➟✝ ï➊é✐è❨ù✂ï✖ê❘￾❶ï✖é✐è✇è✥ì❙û➀ï✖å➈ä✧é➐￾❶ì➈î➻æ✙û➀ï❯↔❢➌➇ç✙ø✠✂➈ç✦✝➠ù✂ø➀î➻ï➊é☎ê✧ø➑ì❛é☎å➄û✶➌✐é✙ì➈é✦✝⑥û➀ø➑é✙ï❞å➄ä î➓å➄æ✙æ☎ø➑é❼✂✐ê✌ëíä✥ì➈î÷û✓å➄ä❘✂➈ï➣￾❶ì❛û➑û➀ï✒￾❶è✧ø➀ì➈é☎ê❙ì➄ë❨ï✄↔✂å➈î➻æ✙û➑ï❞ê✒î➓å➈ô➈ï✖ê✌è✧ç✙ï✖î ì✑✡➇ú➇ø➀ì➈ÿ☎ê➃￾➊å➄é✎ù✂ø➀ù☎å✠è✧ï❞ê❣ëíì❛ä✇ø➀î➓å❄✂➈ï➉ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é➻è✥å❛ê✤ô✂ê✖ð❳➏➠é➽è✧ç✙ï♣è✧ä❿å♦✝ ù✂ø➑è✧ø➀ì➈é☎å➈û☎î➻ì✂ù✂ï➊û✎ì➄ë✟æ☎å✠è✧è✧ï✖ä✧é➓ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✏➌➈å✌ç☎å➈é☎ù☛✝➠ù✂ï✖ê✧ø✠✂➈é✙ï❞ù✒ëíï❞å♦✝ è✧ÿ☎ä✧ï➳ï❯↔➇è✧ä❿å✑￾❶è✧ì❛ä➃✂✐å✠è✧ç☎ï➊ä❿ê❫ä✥ï➊û➀ï➊ú✠å➄é✐è✛ø➑é✙ëíì➈ä✥î➻å➄è✧ø➀ì➈é➓ëíä✥ì➈î▲è✥ç✙ï➤ø➑é☎æ✙ÿ✂è å➄é✎ù➞ï➊û➀ø➀î❭ø➀é☎å➄è✧ï✖ê❯ø➀ä✧ä✥ï➊û➀ï➊ú✠å➈é❛è❯ú✠å➄ä✥ø➀å✑✡✙ø➑û➀ø➑è✧ø➀ï✖ê✖ð❳✕❤è✧ä❿å➄ø➀é☎å❄✡✙û➀ï➃￾➊û➀å❛ê✧ê✧ø✙➞☎ï➊ä è✧ç☎ï➊é ￾➊å✠è✥ï✄✂❛ì➈ä✥ø✙➽✖ï✖ê➉è✧ç✙ï➻ä✥ï✖ê✧ÿ✙û➩è✥ø➑é✗✂✶ëíï❞å✠è✧ÿ☎ä✧ï➻ú➈ï★￾↔è✥ì➈ä❿ê➉ø➀é✐è✧ì✫￾❶û✓å➈ê✥ê✤ï❞ê➊ð ➏➠é è✧ç✙ø✓ê✌ê❘￾❿ç✙ï➊î➻ï✑➌❇ê④è❿å➄é☎ù✙å➈ä✥ù✏➌☛ëíÿ✙û➀û✙✘❩✝r￾❶ì➈é☎é✙ï✒￾❶è✧ï✖ù❺î✒ÿ☎û➩è✥ø➟✝⑥û➀å✪✘❛ï➊ä➤é✙ï➊è❇✝ ó➵ì➈ä✥ô➇ê✩￾✖å➄é✫✡◆ï➞ÿ☎ê✧ï✖ù✫å➈ê✩￾➊û➀å❛ê✧ê✧ø✙➞☎ï➊ä❿ê➊ð✎✕ æ◆ì➄è✧ï✖é✐è✧ø✓å➄û➀û✙✘✢î❭ì❛ä✧ï✌ø➀é✐è✧ï✖ä❇✝ ï✖ê✤è✧ø➀é❼✂➻ê❺￾❿ç☎ï➊î➻ï➉ø✓ê✇è✧ì✒ä✥ï➊û✠✘➓ì➈é✢å➈ê✇î✒ÿ✗￾❿ç✿å➈ê✇æ✎ì✐ê✧ê✧ø✙✡☎û➑ï➳ì➈é➽û➀ï✖å➈ä✧é☎ø➑é❼✂ ø➀é✲è✥ç✙ï❖ëíï❞å✠è✥ÿ✙ä✧ï ï❯↔➇è✥ä✥å➎￾↔è✧ì❛ä✢ø➑è✥ê✧ï➊û➑ë④ð ➏➠é✾è✧ç✙ï ￾➊å❛ê✤ï ì➄ë➚￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❀➌✇å é✙ï❶è④ó➵ì➈ä✥ô✢￾❶ì❛ÿ✙û✓ù➹✡◆ï✺ëíï✖ù ó❨ø➑è✧ç❍å➄û➀î➻ì❛ê✤è➻ä❿åró ø➀é✦✝ æ✙ÿ✂è❿ê✬➪⑨ï➈ð ✂☎ð❣ê✤ø✠➽➊ï✄✝♠é☎ì➈ä✥î➻å➈û➑ø✠➽➊ï❞ù✒ø➀î➓å❄✂❛ï✖ê✴➶↔ð ✎ç✙ø➑û➀ï❨è✥ç✙ø➀ê✎￾➊å➈é➝✡✎ï➉ù✙ì➈é✙ï ó❨ø➑è✧ç✒å➈é✌ì➈ä❿ù✂ø➑é✎å➄ä❘✘❨ëíÿ✙û➀û✙✘✌￾❶ì❛é✙é✙ï★￾↔è✧ï❞ù➳ëíï➊ï❞ù☛✝üëíì❛ä✧ó✛å➄ä❿ù➳é✙ï❶è④ó➵ì➈ä✥ô➉ó❨ø➑è✧ç ê✧ì➈î➻ï➳ê✧ÿ✗￾✒￾❶ï✖ê✥ê➵ëíì➈ä➵è❿å➈ê✧ô➇ê✛ê✧ÿ✗￾❿ç✺å➈ê✔￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä✇ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✏➌✐è✧ç✙ï✖ä✧ï å➄ä✥ï➤æ✙ä✥ì✑✡✙û➀ï➊î➓ê✖ð ✜❇ø➀ä❿ê④è✥û✙✘➎➌➇è❅✘✐æ☎ø✠￾✖å➄û✝ø➀î➓å❄✂➈ï❞ê✛å➄ä✥ï➳û✓å➄ä❘✂➈ï➎➌➇ì➄ë➺è✧ï✖é✢ó❨ø➑è✧ç➲ê✧ï➊ú❛ï➊ä❿å➄û◆ç➇ÿ✙é✦✝ ù✂ä✥ï✖ù✫ú✠å➈ä✧ø✓å❄✡✙û➀ï✖ê➑➪íæ✙ø✙↔✂ï➊û✓ê❘➶❶ð✌✕òëíÿ✙û➀û✠✘➎✝r￾❶ì❛é✙é✙ï★￾↔è✧ï❞ù✆➞☎ä❿ê✤è♣û✓å✪✘➈ï➊ä➉ó❨ø➩è✥ç✏➌ ê✥å✪✘✫ì➈é✙ï❭ç➇ÿ✙é☎ù✂ä✥ï✖ù❖ç✙ø➀ù☎ù✂ï➊é➷ÿ✙é✙ø➑è✥ê➤ø➀é❺è✧ç✙ï➚➞☎ä❿ê④è➤û✓å✪✘➈ï➊ä★➌✟ó✇ì❛ÿ✙û✓ù❺å➈û➟✝ ä✥ï✖å➈ù❼✘➛￾❶ì❛é✐è✥å➄ø➀é➷ê✤ï✖ú➈ï➊ä❿å➄û❦è✥ï➊é☎ê✒ì➄ë✛è✧ç☎ì➈ÿ☎ê✥å➄é☎ù☎ê✌ì➄ë✛ó✇ï✖ø✙✂❛ç✐è✥ê✖ð✿þ➇ÿ✗￾❿ç å❙û➀å➈ä❺✂❛ï➔é➇ÿ✙î➉✡◆ï➊ä✛ì➄ë❇æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê❫ø➀é✗￾➊ä✧ï❞å➈ê✧ï✖ê❣è✥ç✙ï✌￾➊å➈æ☎å✑￾➊ø➩è❅✘➻ì➄ë❀è✧ç✙ï ê❺✘➇ê✤è✧ï✖î▼å➈é☎ù✿è✧ç☎ï➊ä✥ï❶ëíì➈ä✥ï✌ä✥ï✒➍✐ÿ✙ø➀ä✧ï❞ê❨å➓û➀å➈ä❺✂❛ï➊ä✛è✥ä✥å➈ø➑é✙ø➀é❼✂➽ê✧ï❶è❞ð✎➏➠é❺å➈ù☛✝ ù✂ø➑è✧ø➀ì➈é✏➌◆è✥ç✙ï❙î➻ï✖î❭ì❛ä❺✘✺ä✥ï✒➍✐ÿ✙ø➀ä✧ï✖î➻ï➊é✐è➉è✥ì✢ê✤è✧ì➈ä✥ï✒ê✧ì✶î➓å➄é☛✘✺ó➵ï➊ø✠✂➈ç✐è✥ê î➓å✪✘✢ä✥ÿ✙û➀ï➞ì➈ÿ✙è✞￾❶ï✖ä✤è❿å➄ø➀é✫ç☎å➄ä❿ù✂ó✛å➄ä✥ï➤ø➑î➻æ✙û➀ï➊î➻ï➊é✐è❿å✠è✧ø➀ì➈é✎ê➊ð✬✚➵ÿ✂è★➌◆è✧ç✙ï î➓å➄ø➀é❤ù✂ï✄➞✥￾❶ø➀ï➊é✗￾✄✘❺ì➈ë➔ÿ✙é☎ê✤è✧ä✥ÿ✗￾↔è✥ÿ✙ä✥ï✖ù➷é✙ï➊è✥ê✒ëíì➈ä✒ø➀î➓å❄✂❛ï➽ì➈ä✒ê✧æ◆ï➊ï✒￾❿ç å➄æ☎æ✙û➑ø✁￾➊å➄è✧ø➀ì➈é☎ê➤ø✓ê✌è✥ç☎å✠è✌è✥ç✙ï✄✘ ç☎årú❛ï❭é☎ì✆✡✙ÿ✙ø➀û➩è❺✝♠ø➀é ø➑é➇ú✠å➄ä✥ø✓å➄é✗￾➊ï❭ó❨ø➑è✧ç ä✥ï✖ê✧æ✎ï★￾↔è➞è✥ì✫è✧ä❿å➄é☎ê✧û✓å✠è✧ø➀ì➈é✎ê✄➌❯ì➈ä➞û➀ì✦￾➊å➈û❫ù✂ø✓ê④è✥ì➈ä✧è✧ø➀ì➈é☎ê✒ì➄ë✛è✧ç✙ï✶ø➑é✙æ☎ÿ✂è✥ê✖ð ✚➵ï❶ëíì❛ä✧ï➚✡✎ï✖ø➑é✗✂✺ê✧ï➊é✐è➳è✧ì✿è✧ç☎ï➑➞✗↔➇ï❞ù☛✝➠ê✤ø✠➽➊ï❭ø➑é☎æ✙ÿ✂è➞û✓å✪✘➈ï➊ä➳ì➄ë➵å✢é☎ï➊ÿ✙ä❿å➄û é✙ï➊è✒➌✇￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä➉ø➑î➓å❄✂❛ï✖ê✒➌☎ì❛ä➉ì➄è✥ç✙ï➊ä❊✑❄✧➶ì➈ä➉➾★✧✹ê✧ø✙✂❛é☎å➄û✓ê✒➌☎î✒ÿ✎ê④è✞✡✎ï å➄æ☎æ✙ä✧ì✪↔✂ø➀î➓å✠è✧ï✖û✙✘ ê✧ø✠➽➊ï❯✝⑥é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✖ù❤å➄é☎ù ￾❶ï✖é✐è✧ï➊ä✥ï✖ù❑ø➀é❤è✥ç✙ï✫ø➑é☎æ✙ÿ✂è ➞☎ï✖û➀ù☛ð❹→➉é✂ëíì➈ä✧è✧ÿ☎é☎å✠è✥ï➊û✠✘✑➌➇é✙ì➻ê✤ÿ✗￾❿ç✢æ✙ä✥ï➊æ✙ä✥ì✦￾❶ï✖ê✥ê✧ø➑é❼✂➑￾➊å➄é➙✡✎ï➤æ◆ï➊ä✧ëíï✒￾❶è✽✰ ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é❼✂❖ø➀ê✒ì➄ë➺è✧ï✖é é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✖ù å➄è➞è✧ç☎ï✶ó➵ì➈ä❿ù❺û➀ï➊ú❛ï➊û✶➌❦ó❨ç✙ø✠￾❿ç ￾➊å➈é ￾➊å➈ÿ☎ê✧ï♣ê✧ø✙➽✖ï✑➌✂ê✤û✓å➄é✐è★➌➇å➈é☎ù➽æ◆ì❛ê✧ø➑è✧ø➀ì➈é➽ú✠å➄ä✥ø✓å✠è✧ø➀ì➈é✎ê➏ëíì❛ä✇ø➀é☎ù✂ø➀ú➇ø➀ù✂ÿ✎å➄û ￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê✖ð✝ã✛ç✙ø✓ê✄➌✪￾❶ì❛î➉✡✙ø➀é✙ï❞ù♣ó❨ø➑è✧ç✒úrå➈ä✧ø✓å❄✡☎ø➑û➀ø➩è❅✘➉ø➑é✌ó❨ä✥ø➩è✥ø➑é✗✂➔ê④è❅✘➇û➀ï✑➌ ó❨ø➀û➑û❨￾➊å➈ÿ☎ê✤ï✒úrå➈ä✧ø✓å✠è✥ø➑ì❛é☎ê❨ø➀é✫è✧ç☎ï✒æ◆ì❛ê✧ø➩è✥ø➑ì❛é✫ì➄ë❫ù✂ø✓ê④è✥ø➑é✥￾↔è✧ø➀ú➈ï✒ëíï✖å➄è✧ÿ✙ä✥ï✖ê ø➀é➽ø➑é☎æ✙ÿ✂è✛ì✑✡✔✓④ï★￾↔è✥ê✖ð❜➏➠é✶æ✙ä✥ø➑é✥￾❶ø➀æ✙û➑ï➎➌✐å➞ëíÿ✙û➀û✙✘❩✝r￾❶ì➈é☎é✙ï✒￾❶è✧ï✖ù➽é✙ï❶è④ó➵ì➈ä✥ô✒ì➈ë ê✧ÿ✯➣￾❶ø➀ï➊é✐è➞ê✤ø✠➽➊ï➵￾❶ì❛ÿ✙û✓ù❖û➀ï✖å➈ä✧é❖è✧ì✺æ✙ä✥ì➇ù✙ÿ✗￾❶ï❭ì➈ÿ✂è✥æ✙ÿ✂è❿ê➤è✧ç☎å➄è➞å➄ä✥ï❙ø➀é✦✝ ú✠å➄ä✥ø➀å➈é✐è✛ó❨ø➑è✧ç✫ä✥ï✖ê✧æ◆ï✒￾↔è❨è✥ì✶ê✧ÿ✗￾❿ç✺úrå➈ä✧ø✓å✠è✥ø➑ì❛é☎ê✖ð❫õ➔ì✠ó➵ï➊ú❛ï➊ä★➌✐û➀ï✖å➈ä✧é☎ø➑é❼✂ ê✧ÿ✗￾❿ç å✫è❿å➈ê✧ô❺ó➵ì➈ÿ✙û✓ù æ✙ä✥ì✑✡✎å❄✡✙û✠✘❖ä✥ï✖ê✧ÿ✙û➑è❙ø➀é î✒ÿ✙û➑è✧ø➀æ✙û➀ï✶ÿ✙é☎ø➩è❿ê❙ó❨ø➑è✧ç ê✧ø➑î➻ø➀û➀å➈ä➞ó✇ï✖ø✙✂❛ç✐è✒æ☎å✠è✧è✧ï✖ä✧é☎ê✒æ✎ì✐ê✤ø➑è✧ø➀ì➈é☎ï✖ù å✠è❙úrå➈ä✧ø➀ì➈ÿ✎ê✌û➑ì✦￾✖å✠è✧ø➀ì➈é✎ê✌ø➑é è✧ç☎ï➉ø➀é✙æ✙ÿ✂è✛ê✤ì➞å❛ê❣è✧ì❙ù✂ï❶è✥ï✒￾↔è✛ù✂ø✓ê④è✥ø➑é✗￾❶è✧ø➀ú➈ï❨ëíï❞å✠è✥ÿ✙ä✧ï❞ê➏ó❨ç✙ï✖ä✧ï✖ú➈ï✖ä❣è✧ç✙ï✒✘ å➄æ☎æ✎ï❞å➄ä✛ì➈é✢è✧ç✙ï➤ø➀é✙æ✙ÿ✂è❞ð ✗✝ï❞å➄ä✥é✙ø➑é✗✂✒è✥ç✙ï✖ê✧ï➤ó✇ï✖ø✙✂❛ç❛è✩￾❶ì❛é✦➞✗✂❛ÿ✙ä✥å➄è✧ø➀ì➈é☎ê

CXC.Ob CRE IEEE,AOVEy BEXFV requires a very large number of training instances to cover planes,each of which is a feature map.A unit in a feature the space of possible variations.In convolutional networks. map has I 5 inputs connected to a 5 by 5 area in the input, described below,shift invariance is automatically obtained called the receptave rel of the unit.Each unit has 15 in- by forcing the replication of weight configurations across puts,and therefore I5 tiainable coefficients plus a trainable space. bias.The receptive fields of contiguous units in a feature Secondly,a deficiency of fully-connected architectures is map are centered on correspondingly contiguous units in that the topology of the input is entirely ignored.The in-the previous layer.Therefore receptive fields of neighbor- put variables can be presented in any(fixed)order without ing units overlap.For example,in the first hidden layer affecting the outcome of the training.On the contrary, of veNet-5,the receptive fields of horizontally contiguous images (or time-frequency representations of speech)have units overlap by t columns and 5 rows.As stated earlier, a strong ID local structure:variables (or pixels)that are all the units in a feature map share the same set of 15 spatially or temporally nearby are highly correlated.vocal weights and the same bias so they detect the same feature correlations are the reasons for the well-known advantages at all possible locations on the input.The other feature of extracting and combining local features before recogniz- maps in the layer use different sets of weights and biases, ing spatial or temporal objects,because configurations of thereby extracting different types of local features.In the neighboring variables can be classified into a small number case of veNet-5,at each input location six different types of categories (e.g.edges,corners...).Convolutsonal Net- of features are extracted by six units in identical locations works force the extraction of local features by restricting in the six feature maps.A sequential implementation of the receptive fields of hidden units to be local. a feature map would scan the input image with a single unit that has a local receptive field,and store the states B.Convolutgonal Networks of this unit at corresponding locations in the feature map. Convolutional Networks combine three architectural This operation is equivalent to a convolution,followed by ideas to ensure some degree of shift,scale,and distor- an additive bias and squashing function,hence the name tion invariance:local receptge rel s,share wegghts (or convolutgonal network.The kernel of the convolution is the weight replication),and spatial or temporal ab-s amplang. set of connection weights used by the units in the feature A typical convolutional network for recognizing characters, map.An interesting property of convolutional layers is that dubbed veNet-5,is shown in figure I.The input plane if the input image is shifted,the feature map output will receives images of characters that are approximately size- be shifted by the same amount,but will be left unchanged normalized and centered.Each unit in a layer receives in- otherwise.This property is at the basis of the robustness puts from a set of units located in a small neighborhood of convolutional networks to shifts and distortions of the in the previous layer.The idea of connecting units to local input. receptive fields on the input goes back to the Perceptron in Once a feature has been detected.its exact location the early 60s,and was almost simultaneous with Hubel and becomes less important.Only its approximate position Wiesel's discovery of locally-sensitive,orientation-selective relative to other features is relevant.For example,once neurons in the cat's visual system.].vocal connections we know that the input image contains the endpoint of a have been used many times in neural models of visual learn- roughly horizontal segment in the upper left area,a corner ingBl,1,F],.I].With local receptive in the upper right area,and the endpoint of a roughly ver- fields,neurons can extract elementary visual features such tical segment in the lower portion of the image,we can tell as oriented edges,end-points,corners (or similar features in the input image is a 7.Not only is the precise position of other signals such as speech spectrograms).These features each of those features irrelevant for identifying the pattern, are then combined by the subsequent layers in order to de- it is potentially harmful because the positions are likely to tect higher-order features.As stated earlier,distortions or vary for different instances of the character.A simple way shifts of the input can cause the position of salient features to reduce the precision with which the position of distinc- to vary.In addition,elementary feature detectors that are tive features are encoded in a feature map is to reduce the useful on one part of the image are likely to be useful across spatial resolution of the feature map.This can be achieved the entire image.This knowledge can be applied by forcing with a so-called sub-samplang layers which performs a local a set of units,whose receptive fields are located at different averaging and a sub-sampling,reducing the resolution of places on the image,to have identical weight vectors.Bl, the feature map,and reducing the sensitivity of the output 15],t Units in a layer are organized in planes within to shifts and distortions.The second hidden layer of veNet- which all the units share the same set of weights.The set 5 is a sub-sampling layer.This layer comprises six feature of outputs of the units in such a plane is called a feature maps,one for each feature map in the previous layer.The map.Units in a feature map are all constrained to per- receptive field of each unit is a i by I area in the previous form the same operation on different parts of the image. layer's corresponding feature map.Each unit computes the A complete convolutional layer is composed of several fea-average of its four inputs,multiplies it by a trainable coef- ture maps (with different weight vectors),so that multiple ficient,adds a trainable bias,and passes the result through features can be extracted at each location.A concrete ex- a sigmoid function.Contiguous units have non-overlapping ample of this is the first layer of yeNet-5 shown in Figure I. contiguous receptive fields.Consequently,a sub-sampling Units in the first hidden layer of veNet-5 are organized in 6 layer feature map has half the number of rows and columns

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ￾ ä✥ï✒➍✐ÿ✙ø➀ä✧ï❞ê✇å❭ú➈ï✖ä❺✘➻û✓å➄ä❘✂➈ï♣é✐ÿ☎î➉✡◆ï➊ä✛ì➄ë❇è✧ä❿å➄ø➀é✙ø➑é✗✂❙ø➀é☎ê✤è✥å➄é✥￾❶ï✖ê✇è✥ì➚￾➊ì✠ú➈ï➊ä è✧ç☎ï➔ê✤æ☎å➎￾❶ï❨ì➈ë✎æ◆ì❛ê✥ê✤ø✠✡✙û➀ï❨úrå➈ä✧ø✓å✠è✥ø➑ì❛é☎ê✖ð❀➏➠é➵￾❶ì❛é✐ú❛ì➈û➀ÿ✂è✧ø➀ì➈é✎å➄û➇é✙ï❶è④ó➵ì➈ä✥ô✂ê✄➌ ù✂ï❞ê❺￾➊ä✧ø✠✡✎ï❞ù➵✡◆ï➊û➀ì✠ó✌➌➇ê✧ç✙ø➑ë➺è➵ø➀é➇ú✠å➄ä✥ø➀å➈é✗￾❶ï➔ø✓ê➵å➄ÿ✂è✥ì➈î➓å✠è✥ø✠￾✖å➄û➀û✙✘➻ì✑✡✙è✥å➄ø➀é✙ï❞ù ✡☛✘➷ëíì➈ä✴￾❶ø➀é❼✂➲è✥ç✙ï✿ä✧ï✖æ✙û➑ø✁￾➊å➄è✧ø➀ì➈é ì➈ë➉ó➵ï➊ø✠✂➈ç✐è➚￾➊ì➈é✦➞✗✂❛ÿ✙ä❿å✠è✧ø➀ì➈é✎ê✒å✑￾➊ä✧ì✐ê✧ê ê✧æ☎å✑￾➊ï➈ð þ➇ï✒￾➊ì➈é☎ù✂û✠✘✑➌✙å➻ù✂ï✄➞✥￾❶ø➀ï➊é✥￾❯✘➽ì➈ë❀ëíÿ✙û➀û✙✘❩✝r￾❶ì❛é✙é✙ï✒￾❶è✧ï❞ù✢å➈ä❘￾❿ç✙ø➑è✧ï★￾↔è✥ÿ✙ä✧ï❞ê❫ø✓ê è✧ç✎å✠è➔è✥ç✙ï✌è✧ì❛æ✎ì❛û➑ì➎✂✑✘➽ì➈ë❦è✧ç✙ï➞ø➀é✙æ✙ÿ✙è➉ø➀ê➔ï➊é✐è✧ø➀ä✧ï✖û✙✘✢ø✙✂❛é✙ì➈ä✥ï✖ù✝ð✇ã✛ç✙ï✌ø➀é✦✝ æ✙ÿ✂è✇úrå➈ä✧ø✓å❄✡☎û➑ï❞ê❷￾➊å➄é➝✡◆ï➔æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù❭ø➑é➓å➄é☛✘➐➪●➞❼↔✂ï✖ù✥➶❦ì➈ä❿ù✂ï➊ä❣ó❨ø➑è✧ç☎ì➈ÿ✂è å♦➘✟ï✒￾❶è✧ø➀é❼✂ è✥ç✙ï❖ì❛ÿ✂è❘￾➊ì➈î➻ï➲ì➄ë➤è✥ç✙ï➲è✥ä✥å➈ø➑é✙ø➀é❼✂✎ðòý♣é➘è✥ç✙ï➛￾➊ì➈é✐è✧ä❿å➄ä❘✘✑➌ ø➀î➻å✑✂➈ï❞ê➉➪íì❛ä➔è✥ø➑î➻ï❯✝♠ëíä✥ï✒➍✐ÿ✙ï➊é✥￾❯✘✿ä✧ï✖æ✙ä✧ï❞ê✤ï✖é✐è✥å✠è✥ø➑ì❛é☎ê✛ì➄ë❫ê✧æ◆ï➊ï✒￾❿ç✈➶✛ç☎årú➈ï å✿ê④è✥ä✧ì❛é❼✂✳✑✑✧▲û➑ì✦￾➊å➈û❯ê✤è✧ä✥ÿ✗￾❶è✧ÿ✙ä✥ï✻✰♣úrå➈ä✧ø✓å❄✡☎û➑ï❞ê➑➪⑨ì➈ä➤æ✙ø✙↔➇ï✖û➀ê✴➶➉è✧ç☎å➄è➤å➈ä✧ï ê✧æ☎å✠è✥ø➀å➈û➑û✠✘✒ì❛ä❣è✧ï✖î❭æ◆ì➈ä❿å➄û➀û✠✘❙é✙ï❞å➄ä❘✡☛✘✒å➄ä✥ï❨ç✙ø✠✂➈ç✙û✠✘➚￾➊ì➈ä✥ä✧ï✖û➀å➄è✧ï❞ù☛ð ✗❀ì☛￾✖å➄û ￾❶ì❛ä✧ä✥ï➊û✓å✠è✥ø➑ì❛é☎ê❫å➈ä✧ï♣è✧ç☎ï➳ä✥ï✖å❛ê✤ì❛é☎ê❫ëíì➈ä➵è✧ç✙ï➤ó➵ï➊û➀û➟✝⑥ô➇é✙ì✠ó❨é✶å❛ù✂ú✠å➄é✐è✥å✑✂➈ï✖ê ì➄ë❯ï❯↔➇è✧ä❿å✑￾❶è✧ø➀é❼✂➻å➄é☎ù➙￾❶ì➈î➑✡✙ø➀é✙ø➑é✗✂ ✲✙➯✒❄✴➢✲✐ëíï✖å➄è✧ÿ✙ä✥ï✖ê✔✡◆ï❶ëíì❛ä✧ï➳ä✥ï✒￾❶ì➎✂➈é✙ø✠➽❯✝ ø➀é❼✂✫ê✤æ☎å➄è✧ø✓å➄û❣ì❛ä➤è✧ï➊î➻æ◆ì➈ä❿å➄û❣ì➎✡✔✓④ï✒￾❶è✥ê✒➌✏✡✎ï★￾➊å➈ÿ☎ê✤ï➵￾➊ì➈é✦➞✗✂❛ÿ✙ä❿å✠è✧ø➀ì➈é✎ê➉ì➈ë é✙ï✖ø✙✂❛ç❩✡◆ì➈ä✥ø➀é❼✂✌ú✠å➄ä✥ø➀å✑✡✙û➑ï❞ê❹￾➊å➈é➣✡◆ï✬￾❶û✓å➈ê✥ê✤ø✙➞☎ï❞ù➻ø➑é✐è✧ì❙å➞ê✤î➓å➈û➑û✎é✐ÿ☎î➉✡◆ï➊ä ì➄ë✞￾➊å✠è✥ï✄✂❛ì➈ä✥ø➑ï❞ê➙➪⑨ï➈ð ✂☎ð ï✖ù❼✂➈ï✖ê✒➌➜￾➊ì➈ä✥é✙ï➊ä❿ê➊ð➀ð➀ð ➶❶ð ❃ ➯❄➨✍♦➯ ✲✡✦➺❩✣ ➯♦➨✇➢✲➜➸➑➳❯➺❩✭ ➻➜➯♦➡❺➼♦➩➤ëíì➈ä✴￾❶ï➓è✥ç✙ï✶ï✄↔➇è✧ä❿å✑￾↔è✥ø➑ì❛é➷ì➄ë➔û➀ì☛￾✖å➄û❣ëíï❞å✠è✥ÿ✙ä✧ï❞ê➓✡☛✘ ä✧ï❞ê④è✥ä✧ø✁￾↔è✥ø➑é❼✂ è✧ç☎ï✌ä✧ï★￾❶ï➊æ✙è✧ø➀ú➈ï✞➞☎ï➊û✓ù✙ê❨ì➈ë❇ç✙ø✓ù✙ù✂ï✖é✺ÿ✙é✙ø➑è✥ê✛è✥ì➵✡✎ï✌û➀ì✦￾➊å➄û♠ð ✚✜✛ ❃ ➯♦➨☎✍✪➯✬✲✡✦➺✮✣ ➯❄➨✈➢✬✲✈➸➉➳✄➺✶➻➜➯♦➡❺➼♦➩ ☞✇ì➈é➇ú➈ì❛û➑ÿ✂è✥ø➑ì❛é☎å➄û➘ñ➔ï➊è④ó✇ì❛ä✧ô✂ê ￾❶ì❛î➉✡✙ø➀é✙ïPè✧ç✙ä✥ï➊ï å➄ä✴￾❿ç✙ø➑è✧ï✒￾❶è✧ÿ✙ä❿å➄û ø✓ù✂ï✖å❛ê✺è✧ì✻ï✖é☎ê✧ÿ✙ä✧ï ê✧ì➈î➻ï ù✂ï✒✂➈ä✥ï➊ï ì➈ë➓ê✧ç✙ø➑ë➺è✒➌✒ê❘￾➊å➈û➑ï➎➌✒å➄é✎ù❲ù✙ø➀ê✤è✧ì❛ä❇✝ è✧ø➀ì➈é✾ø➑é➇ú✠å➄ä✥ø✓å➄é✗￾➊ï✻✰ ✲✙➯✒❄✴➢✲t➡❺➳❄✴➳❖➤✇➺✮✣✍♦➳✂✁✎➳✡✲❨✪♦➩✹➌➉➩❺➥❼➢❄➡❺➳✱✪ ➻➜➳❪✣✥★➥✦➺➩➛➪⑨ì➈ä ó➵ï➊ø✠✂➈ç✐è➉ä✥ï➊æ✙û➀ø✠￾✖å✠è✥ø➑ì❛é✥➶✹➌✟å➄é☎ù✫ê✧æ☎å➄è✧ø✓å➄û❀ì❛ä❨è✧ï✖î➻æ✎ì❛ä✥å➈û❹➩✒✡✤✯✡✭❖➩✄➢❄➲✛➤ ✲✣●➨✵✥➈ð ✕➘è❅✘➇æ✙ø✁￾➊å➈û❼￾❶ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✂é✙ï❶è④ó➵ì➈ä✥ô➤ëíì➈ä❫ä✧ï★￾❶ì✑✂❛é✙ø✠➽➊ø➀é❼✂✞￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê✄➌ ù✂ÿ❼✡✗✡✎ï❞ù ✗✝ï❞ñ➔ï❶è❺✝✝✙❼➌✛ø➀ê➽ê✧ç✙ì✠ó❨é➘ø➀é ➞✗✂➈ÿ✙ä✥ï ✑✂ð ã✛ç✙ï➲ø➀é✙æ✙ÿ✙è✶æ✙û✓å➄é✙ï ä✥ï✒￾❶ï✖ø➑ú❛ï✖ê♣ø➑î➓å✑✂➈ï✖ê➉ì➄ë➃￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê❨è✥ç☎å✠è✌å➄ä✥ï❙å➈æ✙æ✙ä✥ì✪↔➇ø➀î➓å✠è✥ï➊û✠✘✿ê✧ø✙➽✖ï❯✝ é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✖ù✺å➈é☎ù✆￾❶ï✖é❛è✥ï➊ä✥ï✖ù☛ð ✭❫å➎￾❿ç✺ÿ✙é✙ø➑è➉ø➑é➲å➽û➀å✪✘❛ï➊ä❨ä✥ï✒￾❶ï✖ø➑ú❛ï✖ê✛ø➀é✦✝ æ✙ÿ✂è❿ê➞ëíä✥ì➈î å➲ê✧ï❶è✒ì➈ë✛ÿ☎é✙ø➩è❿ê✒û➀ì✦￾➊å➄è✧ï✖ù➷ø➀é å➲ê✧î➓å➄û➀û➏é✙ï➊ø✠✂➈ç☛✡◆ì➈ä✥ç✙ì➇ì✂ù ø➀é➓è✧ç✙ï➳æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê❫û➀å✪✘❛ï➊ä❞ð❇ã✛ç✙ï♣ø➀ù✂ï❞å➞ì➈ë❀￾❶ì➈é☎é✙ï✒￾❶è✧ø➀é❼✂➞ÿ✙é✙ø➑è✥ê✇è✥ì✒û➀ì☛￾✖å➄û ä✥ï✒￾❶ï✖æ✂è✧ø➀ú➈ï✔➞✎ï➊û✓ù✙ê❣ì➈é❭è✥ç✙ï❨ø➑é☎æ✙ÿ✂è➜✂❛ì✐ï❞ê❷✡☎å➎￾❿ô✌è✧ì➤è✥ç✙ï✬✓❦ï➊ä✴￾❶ï✖æ✂è✧ä✥ì➈é❙ø➑é è✧ç☎ï➵ï❞å➄ä✥û✙✘ ✓✗✘➈ê✒➌➄å➄é☎ù✒ó➵å❛ê❇å➈û➑î➻ì❛ê✤è❣ê✤ø➀î✒ÿ✙û➑è✥å➈é✙ï➊ì❛ÿ☎ê❇ó❨ø➩è✥ç❭õ➉ÿ❼✡✎ï✖û✙å➄é☎ù ✎ø➀ï✖ê✧ï➊û ❁ ê✛ù✂ø✓ê❺￾➊ì✠ú➈ï✖ä❺✘❭ì➄ë❯û➑ì✦￾➊å➈û➑û✠✘❩✝⑥ê✧ï➊é✎ê✤ø➑è✧ø➀ú➈ï➎➌❛ì❛ä✧ø➀ï➊é✐è✥å➄è✧ø➀ì➈é✦✝➠ê✧ï➊û➀ï✒￾↔è✥ø➑ú❛ï é✙ï✖ÿ✙ä✧ì❛é☎ê➔ø➀é✫è✥ç✙ï➑￾✖å✠è✽❁ ê♣ú✐ø✓ê✧ÿ☎å➄û❯ê❇✘✂ê✤è✧ï➊î ✞❜✻✘✠⑥ð❇✗❀ì☛￾✖å➄û❜￾❶ì➈é☎é✙ï✒￾❶è✧ø➀ì➈é☎ê ç☎årú❛ï❷✡✎ï✖ï➊é➞ÿ☎ê✧ï✖ù➤î➓å➈é❩✘♣è✧ø➀î➻ï✖ê❇ø➑é✌é☎ï➊ÿ✙ä❿å➄û❛î❭ì✂ù✂ï✖û➀ê❀ì➄ë✙ú➇ø✓ê✤ÿ☎å➈û➄û➀ï✖å➄ä✥é✦✝ ø➀é❼✂✷✞❜✗➾✡✠❖➌❵✞❜✵✑✆✠❖➌❵✞➟➾✒✺✠❖➌ ✞❜ ❜✠❖➌❵✞❜✫❝✬✠❖➌❵✞✑ ✠♠ð ✎ø➩è✥ç û➑ì✦￾➊å➈û➞ä✥ï✒￾❶ï✖æ✂è✧ø➀ú➈ï ➞☎ï✖û➀ù✙ê✒➌✙é✙ï✖ÿ✙ä✥ì➈é☎ê✔￾➊å➈é✢ï✄↔➇è✧ä❿å✑￾↔è✛ï➊û➀ï➊î➻ï✖é❛è❿å➄ä❘✘➓ú➇ø➀ê✧ÿ☎å➄û✟ëíï❞å✠è✥ÿ✙ä✧ï❞ê❨ê✤ÿ✥￾❿ç å➈ê❀ì➈ä✥ø➑ï✖é✐è✧ï✖ù➤ï❞ù✦✂➈ï❞ê✄➌rï✖é☎ù☛✝⑥æ✎ì❛ø➑é✐è✥ê✒➌♦￾❶ì❛ä✧é✙ï✖ä✥ê➜➪íì➈ä❦ê✧ø➑î➻ø➀û➀å➈ä✝ëíï✖å➄è✧ÿ✙ä✥ï✖ê❀ø➑é ì➄è✥ç✙ï➊ä➵ê✤ø✠✂➈é✎å➄û✓ê➏ê✤ÿ✥￾❿ç➽å➈ê❫ê✧æ◆ï➊ï✒￾❿ç➽ê✧æ◆ï✒￾↔è✥ä✧ì➎✂➈ä❿å➄î➓ê✴➶↔ð❇ã✛ç☎ï✖ê✧ï❨ëíï✖å➄è✧ÿ✙ä✥ï✖ê å➄ä✥ï❨è✥ç✙ï➊é ￾➊ì➈î➑✡✙ø➑é☎ï✖ù➝✡☛✘❭è✧ç✙ï➳ê✤ÿ❼✡✎ê✤ï★➍❛ÿ☎ï➊é✐è❫û✓å✪✘➈ï➊ä❿ê❣ø➀é➽ì➈ä❿ù✂ï✖ä❣è✧ì✒ù✙ï❯✝ è✧ï★￾↔è➔ç✙ø✠✂➈ç☎ï➊ä❺✝♠ì❛ä✥ù✂ï✖ä➏ëíï❞å✠è✧ÿ☎ä✧ï❞ê➊ð❷✕➉ê❨ê✤è✥å➄è✧ï❞ù✶ï❞å➄ä✥û➑ø➀ï➊ä★➌✂ù✂ø➀ê✤è✧ì❛ä✤è✥ø➑ì❛é☎ê➵ì➈ä ê✧ç✙ø➩ë➺è❿ê➏ì➈ë◆è✥ç✙ï➔ø➀é✙æ✙ÿ✂è✎￾➊å➄é➣￾✖å➄ÿ☎ê✧ï✛è✧ç☎ï➔æ✎ì✐ê✤ø➑è✧ø➀ì➈é➻ì➈ë☛ê✥å➄û➀ø➀ï➊é✐è❣ëíï✖å➄è✧ÿ✙ä✥ï✖ê è✧ì➻ú✠å➄ä❘✘➈ð❨➏➠é✺å➈ù✙ù✙ø➩è✥ø➑ì❛é✏➌✙ï➊û➀ï➊î➻ï✖é❛è❿å➄ä❘✘➻ëíï✖å➄è✧ÿ✙ä✥ï➳ù✙ï❶è✧ï★￾↔è✥ì➈ä❿ê➵è✧ç☎å➄è❨å➈ä✧ï ÿ☎ê✧ï❶ëíÿ✙û✂ì❛é➞ì➈é☎ï✇æ☎å➈ä✤è❦ì➄ë☎è✧ç✙ï➵ø➑î➓å✑✂➈ï✛å➄ä✥ï❫û➀ø➑ô❛ï➊û✠✘♣è✥ì④✡◆ï✛ÿ☎ê✧ï❶ëíÿ✙û✂å✑￾➊ä✧ì✐ê✧ê è✧ç☎ï❨ï➊é✐è✧ø➀ä✧ï✛ø➀î➓å❄✂➈ï❛ð❇ã✛ç✙ø✓ê❦ô➇é✙ì✠ó❨û➀ï✖ù✦✂❛ï✎￾✖å➄é➚✡✎ï➔å➈æ✙æ✙û➀ø➑ï❞ù➉✡☛✘✌ëíì❛ä❘￾➊ø➑é❼✂ å➤ê✤ï➊è➏ì➄ë✟ÿ✙é✙ø➑è✥ê✒➌➈ó❨ç✙ì✐ê✤ï✛ä✥ï✒￾➊ï➊æ✂è✥ø➑ú❛ï➃➞✎ï➊û✓ù✙ê➏å➄ä✥ï➵û➑ì✦￾➊å➄è✧ï❞ù❭å➄è➏ù✂ø➟➘✟ï➊ä✥ï➊é✐è æ✙û✓å✑￾➊ï✖ê➔ì❛é✫è✧ç☎ï❙ø➀î➓å❄✂➈ï➎➌☎è✥ì➽ç☎årú❛ï➞ø➀ù✙ï➊é✐è✧ø✁￾➊å➈û❇ó➵ï➊ø✠✂➈ç✐è➉ú❛ï✒￾↔è✥ì➈ä❿ê ✞❜ ✑ ✠✶➌ ✞✙➾ ✙✆✠❖➌ ✞❜✬❝✫✠♠ð➵→➔é☎ø➩è❿ê✌ø➑é➷å✿û➀å✪✘❛ï➊ä✌å➄ä✥ï❭ì➈ä❘✂❛å➄é☎ø✙➽✖ï✖ù➲ø➀é➷æ✙û✓å➄é✙ï❞ê➤ó❨ø➩è✥ç✙ø➑é ó❨ç✙ø✁￾❿ç➲å➄û➀û☛è✧ç☎ï✒ÿ✙é✙ø➑è✥ê♣ê✤ç✎å➄ä✥ï➤è✧ç✙ï✒ê✥å➄î➻ï➞ê✧ï❶è➉ì➈ë❣ó✇ï✖ø✙✂❛ç❛è❿ê➊ð➵ã✛ç✙ï✒ê✧ï❶è ì➄ë✛ì➈ÿ✙è✧æ✙ÿ✂è❿ê✌ì➄ë➵è✧ç☎ï➓ÿ✙é✙ø➑è✥ê✌ø➀é ê✤ÿ✗￾❿ç å✿æ✙û✓å➄é✙ï➓ø✓ê✌￾➊å➈û➑û➀ï✖ù➷å➝➫❯➳✴➢♦➺✡✦➡❺➳ ➲➵➢❘➤◆ð⑧→➔é✙ø➑è✥ê➻ø➀é❑å❖ëíï✖å➄è✧ÿ✙ä✥ï✢î➓å➈æ❑å➄ä✥ï✢å➈û➑û✩￾➊ì➈é☎ê✤è✧ä❿å➄ø➀é✙ï✖ù è✧ì æ✎ï✖ä❇✝ ëíì➈ä✥î è✧ç✙ï✿ê✧å➈î➻ï➽ì➈æ◆ï➊ä❿å✠è✥ø➑ì❛é ì➈é❤ù✂ø✙➘✟ï➊ä✥ï➊é✐è❙æ☎å➈ä✤è❿ê➞ì➈ë❨è✧ç✙ï✢ø➑î➓å❄✂❛ï➈ð ✕ ￾❶ì❛î➻æ✙û➑ï➊è✧ï✌￾❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✟û✓å✪✘➈ï➊ä✛ø✓ê✩￾❶ì❛î❭æ◆ì❛ê✧ï✖ù✢ì➄ë❦ê✤ï✖ú➈ï✖ä✥å➈û✎ëíï❞å♦✝ è✧ÿ☎ä✧ï➤î➓å➄æ☎êt➪⑨ó❨ø➩è✥ç✿ù✂ø✙➘✟ï➊ä✥ï➊é✐è❨ó✇ï✖ø✙✂❛ç❛è➵ú➈ï★￾↔è✧ì❛ä✥ê✴➶✹➌✂ê✧ì✒è✥ç☎å✠è❨î❙ÿ✙û➩è✥ø➑æ☎û➑ï ëíï✖å➄è✧ÿ✙ä✥ï✖ê✬￾➊å➄é✫✡◆ï✌ï❯↔➇è✧ä❿å✑￾❶è✧ï❞ù✿å➄è➔ï✖å➎￾❿ç✢û➀ì✦￾➊å✠è✥ø➑ì❛é✝ð➜✕ ￾➊ì➈é✗￾➊ä✧ï➊è✧ï✌ï❯↔☛✝ å➄î➻æ✙û➀ï➵ì➄ë☎è✥ç✙ø✓ê❦ø➀ê❇è✧ç☎ï➃➞✎ä✥ê✤è❯û✓å✪✘➈ï✖ä❇ì➈ë✴✗✝ï❞ñ➔ï➊è❇✝ ✙➉ê✧ç✙ì✠ó❨é✒ø➀é➝✜❇ø✠✂➈ÿ☎ä✧ï❙✑✂ð →➔é☎ø➩è❿ê❣ø➑é❭è✥ç✙ï✎➞☎ä✥ê✤è❣ç✙ø✓ù✙ù✂ï➊é➻û✓å✪✘➈ï✖ä❯ì➈ë ✗❀ï✖ñ➔ï➊è❇✝ ✙➳å➄ä✥ï✇ì❛ä❺✂✐å➄é✙ø✠➽➊ï❞ù➞ø➀é ✓ æ✙û✓å➄é✙ï❞ê✄➌➇ï❞å✑￾❿ç✶ì➈ë❀ó❨ç✙ø✁￾❿ç✶ø✓ê✛å➞ëíï❞å✠è✧ÿ☎ä✧ï➳î➓å➄æ✝ð❷✕❲ÿ✙é✙ø➑è❨ø➑é✺å✌ëíï❞å✠è✧ÿ☎ä✧ï î➓å➄æ➽ç✎å➈ê✄✑ ✙✌ø➀é✙æ✙ÿ✂è❿ê➃￾➊ì➈é✙é☎ï✒￾↔è✥ï✖ù➻è✥ì❙å❄✙t✡☛✘ ✙✒å➄ä✥ï✖å➤ø➀é➽è✥ç✙ï➉ø➀é✙æ✙ÿ✂è★➌ ￾➊å➈û➑û➀ï✖ù✫è✧ç✙ï➐➡❺➳❄✴➳❖➤✇➺✮✣✍♦➳✄✁➃➳❪✲❨✪✒ì➄ë➏è✥ç✙ï❭ÿ✙é✙ø➑è✖ð ✭❫å➎￾❿ç✫ÿ✙é✙ø➑è➤ç✎å➈ê✒✑✫✙➻ø➀é✦✝ æ✙ÿ✂è❿ê✄➌➄å➈é☎ù✌è✥ç✙ï➊ä✥ï❶ëíì❛ä✧ï❀✑✫✙✛è✥ä✥å➈ø➑é✎å❄✡✙û➀ï➃￾➊ì✐ï✱✯➣￾❶ø➀ï➊é✐è✥ê❇æ✙û➀ÿ☎ê❦å❨è✧ä❿å➄ø➀é☎å✑✡✙û➑ï ✡✙ø✓å➈ê✖ð✒ã✛ç✙ï❭ä✧ï★￾❶ï✖æ✂è✧ø➀ú➈ï➑➞☎ï➊û✓ù✙ê➳ì➄ë✔￾❶ì❛é❛è✥ø✙✂❛ÿ✙ì➈ÿ✎ê➉ÿ✙é☎ø➩è❿ê➤ø➑é å➽ëíï❞å✠è✧ÿ☎ä✧ï î➓å➄æ❤å➈ä✧ï➙￾❶ï➊é✐è✥ï➊ä✥ï✖ù ì❛é➹￾❶ì❛ä✧ä✥ï✖ê✧æ✎ì❛é☎ù✂ø➀é❼✂➈û✠✘➛￾➊ì➈é✐è✧ø✠✂➈ÿ☎ì➈ÿ☎ê✒ÿ✙é✙ø➑è✥ê❭ø➑é è✧ç☎ï❙æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê♣û✓å✪✘➈ï➊ä❞ð✌ã✛ç✙ï➊ä✥ï❶ëíì❛ä✧ï✒ä✧ï★￾❶ï➊æ✙è✧ø➀ú➈ï➓➞✎ï➊û✓ù✙ê➤ì➄ë➏é✙ï✖ø✙✂❛ç☛✡✎ì❛ä❇✝ ø➀é❼✂❺ÿ☎é✙ø➩è❿ê❭ì✠ú➈ï➊ä✥û✓å➄æ✝ð➒✜✙ì➈ä❭ï❯↔✙å➈î❭æ☎û➑ï➎➌➏ø➑é❤è✧ç✙ï➙➞☎ä❿ê④è❭ç✙ø✓ù✙ù✂ï➊é û✓å✪✘➈ï➊ä ì➄ë❅✗✝ï❞ñ➔ï❶è❺✝✝✙❼➌✝è✥ç✙ï➻ä✧ï★￾❶ï➊æ✙è✧ø➀ú➈ï➚➞☎ï➊û✓ù✙ê➤ì➈ë➵ç✙ì❛ä✧ø✠➽➊ì❛é✐è✥å➄û➀û✠✘✫￾➊ì➈é✐è✧ø✠✂➈ÿ☎ì➈ÿ☎ê ÿ✙é✙ø➑è✥ê➳ì✠ú❛ï➊ä✥û➀å➈æ➐✡☛✘◆❝➙￾➊ì➈û➀ÿ✙î➻é☎ê➳å➄é☎ù ✙➽ä✧ì✠ó➔ê✖ð✬✕♣ê♣ê✤è✥å➄è✧ï✖ù➲ï✖å➄ä✥û➀ø➑ï✖ä✒➌ å➄û➀û➔è✥ç✙ï❖ÿ☎é✙ø➩è❿ê✶ø➀é✻å➷ëíï❞å✠è✧ÿ☎ä✧ï➲î➓å➈æ✻ê✤ç☎å➈ä✧ï➲è✧ç✙ï❖ê✥å➄î➻ï❖ê✧ï❶è✿ì➄ë ✑✫✙ ó➵ï➊ø✠✂➈ç✐è✥ê✛å➄é✎ù➽è✥ç✙ï✌ê✧å➈î❭ï✞✡✙ø✓å➈ê❨ê✧ì✒è✥ç✙ï✄✘✢ù✂ï❶è✥ï✒￾↔è❨è✥ç✙ï✌ê✧å➈î➻ï➉ëíï❞å✠è✧ÿ☎ä✧ï å✠è➻å➄û➀û✇æ◆ì❛ê✥ê✧ø✙✡✙û➀ï✶û➀ì✦￾➊å➄è✧ø➀ì➈é☎ê✒ì➈é è✥ç✙ï✢ø➀é✙æ✙ÿ✙è✖ð➷ã✛ç✙ï✢ì➈è✧ç✙ï✖ä➞ëíï❞å✠è✧ÿ☎ä✧ï î➓å➄æ☎ê➳ø➀é➲è✥ç✙ï❙û✓å✪✘➈ï✖ä➉ÿ☎ê✧ï➻ù✂ø➟➘✟ï➊ä✥ï➊é✐è✌ê✤ï➊è✥ê➳ì➄ë✇ó✇ï✖ø✙✂❛ç❛è❿ê♣å➈é☎ù↕✡☎ø➀å❛ê✤ï❞ê✄➌ è✧ç☎ï➊ä✥ï✄✡☛✘✢ï✄↔➇è✧ä❿å✑￾↔è✥ø➑é✗✂➽ù✂ø➟➘✟ï➊ä✥ï➊é✐è➔è❅✘➇æ◆ï✖ê♣ì➄ë❣û➀ì☛￾✖å➄û❀ëíï✖å✠è✥ÿ✙ä✥ï✖ê✖ð✎➏➠é➲è✧ç✙ï ￾➊å❛ê✤ï❙ì➄ë✄✗❀ï✖ñ➉ï❶è❇✝ ✙✦➌☛å➄è➳ï❞å✑￾❿ç➲ø➀é✙æ✙ÿ✂è➤û➀ì☛￾✖å✠è✥ø➑ì❛é❺ê✧ø➟↔➲ù✂ø✙➘◆ï✖ä✧ï✖é✐è♣è❅✘➇æ◆ï✖ê ì➄ë❇ëíï✖å➄è✧ÿ✙ä✥ï✖ê❨å➈ä✧ï➤ï❯↔➇è✥ä✥å➎￾↔è✧ï❞ù➣✡☛✘✶ê✧ø➟↔✢ÿ✙é✙ø➑è✥ê❨ø➀é✺ø➀ù✂ï✖é✐è✧ø✁￾➊å➄û☛û➀ì✦￾➊å➄è✧ø➀ì➈é☎ê ø➀é è✧ç☎ï✿ê✧ø➟↔➷ëíï❞å✠è✧ÿ☎ä✧ï✢î➻å➈æ☎ê✖ð ✕▼ê✧ï✒➍✐ÿ✙ï➊é✐è✥ø➀å➈û❫ø➀î❭æ☎û➑ï✖î❭ï✖é✐è✥å✠è✥ø➑ì❛é❤ì➈ë å❖ëíï✖å✠è✥ÿ✙ä✥ï✢î➓å➄æ❑ó✇ì❛ÿ✙û✓ù❤ê❘￾➊å➄é❤è✧ç✙ï✿ø➀é✙æ✙ÿ✂è➓ø➀î➻å✑✂➈ï✿ó❨ø➩è✥ç å ê✧ø➑é❼✂❛û➑ï ÿ✙é✙ø➑è❭è✧ç☎å➄è❙ç☎å❛ê✒å✫û➀ì✦￾➊å➄û➵ä✥ï✒￾❶ï✖æ✂è✧ø➀ú➈ï➣➞☎ï➊û✓ù❢➌➏å➄é✎ù ê④è✥ì➈ä✥ï➓è✧ç✙ï✺ê✤è✥å✠è✥ï✖ê ì➄ë❇è✧ç✙ø✓ê❨ÿ✙é☎ø➩è➉å➄è✬￾❶ì➈ä✥ä✥ï✖ê✧æ✎ì❛é☎ù✂ø➀é❼✂✒û➀ì✦￾➊å✠è✥ø➑ì❛é☎ê✛ø➀é✶è✥ç✙ï➤ëíï✖å➄è✧ÿ✙ä✥ï➤î➻å➈æ✝ð ã✛ç✙ø✓ê➳ì➈æ◆ï➊ä❿å✠è✧ø➀ì➈é❖ø➀ê♣ï✒➍✐ÿ✙ø➀ú✠å➄û➀ï➊é✐è♣è✥ì✿å➙￾❶ì❛é✐ú❛ì➈û➀ÿ✂è✧ø➀ì➈é❀➌✎ëíì❛û➑û➀ì✠ó➵ï✖ù✫✡☛✘ å➄é å❛ù✙ù✂ø➑è✧ø➀ú➈ï➣✡✙ø✓å➈ê✒å➄é☎ù➷ê❘➍❛ÿ✎å➈ê✧ç✙ø➑é✗✂✺ëíÿ✙é✗￾↔è✥ø➑ì❛é✏➌❦ç✙ï➊é✥￾❶ï➻è✧ç☎ï➽é☎å➄î➻ï ❄✴➯♦➨☎✍✪➯✬✲✡✦➺✮✣ ➯❄➨✈➢✬✲✥➨✇➳❯➺✶➻➜➯♦➡❺➼➄ð❣ã✛ç✙ï❨ô➈ï✖ä✧é✙ï✖û✂ì➄ë◆è✧ç✙ï④￾❶ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é❭ø➀ê❣è✧ç✙ï ê✧ï❶è➤ì➄ë➜￾❶ì❛é✙é✙ï★￾↔è✧ø➀ì➈é❖ó✇ï✖ø✙✂❛ç❛è❿ê➉ÿ✎ê✤ï❞ù↕✡☛✘✿è✧ç✙ï❭ÿ✙é✙ø➑è✥ê➳ø➀é➲è✧ç☎ï✒ëíï❞å✠è✧ÿ☎ä✧ï î➓å➄æ✝ð❜✕➔é➞ø➀é✐è✧ï✖ä✧ï❞ê④è✥ø➑é❼✂❨æ☎ä✧ì❛æ✎ï✖ä✤è❅✘♣ì➄ë❼￾❶ì❛é✐ú❛ì➈û➀ÿ✂è✧ø➀ì➈é✎å➄û➄û✓å✪✘➈ï✖ä✥ê✟ø✓ê☛è✥ç☎å✠è ø➑ë✇è✥ç✙ï➻ø➑é✙æ☎ÿ✂è➞ø➑î➓å✑✂➈ï➻ø➀ê✌ê✧ç✙ø➑ë➺è✧ï❞ù❢➌✝è✥ç✙ï➻ëíï✖å➄è✧ÿ✙ä✥ï➻î➻å➈æ❺ì❛ÿ✂è✧æ☎ÿ✂è➞ó❨ø➑û➀û ✡◆ï➤ê✧ç✙ø➑ë➺è✧ï❞ù➣✡☛✘➻è✧ç✙ï✌ê✥å➄î➻ï♣å➈î➻ì➈ÿ✙é✐è✒➌☛✡✙ÿ✙è✛ó❨ø➀û➀û✈✡◆ï➤û➑ï➊ë➺è❨ÿ✙é✗￾❿ç☎å➈é❼✂➈ï❞ù ì➄è✥ç✙ï➊ä✥ó❨ø✓ê✤ï❛ð➳ã✛ç✙ø✓ê♣æ✙ä✧ì❛æ✎ï✖ä✤è❅✘✺ø✓ê♣å➄è♣è✥ç✙ï➚✡☎å➈ê✧ø➀ê♣ì➄ë❣è✥ç✙ï❭ä✧ì➎✡✙ÿ☎ê✤è✧é✙ï❞ê✧ê ì➄ë✛￾➊ì➈é➇ú➈ì❛û➑ÿ✂è✥ø➑ì❛é☎å➄û❫é✙ï❶è④ó➵ì➈ä✥ô✂ê➤è✧ì❖ê✧ç✙ø➑ë➺è✥ê❭å➄é☎ù❤ù✂ø➀ê✤è✧ì❛ä✤è✥ø➑ì❛é☎ê✌ì➈ë❨è✧ç✙ï ø➀é✙æ✙ÿ✂è❞ð ý♣é✗￾❶ï å❍ëíï✖å➄è✧ÿ✙ä✥ï❤ç✎å➈ê➛✡✎ï✖ï➊é➶ù✙ï❶è✧ï★￾↔è✥ï✖ù❢➌❭ø➩è❿ê ï❯↔✙å✑￾❶è❺û➀ì☛￾✖å✠è✥ø➑ì❛é ✡◆ï✒￾❶ì❛î➻ï✖ê✺û➀ï✖ê✥ê✢ø➀î➻æ✎ì❛ä✤è❿å➄é✐è✖ð ý♣é✙û✠✘✻ø➑è✥ê✫å➄æ☎æ✙ä✧ì✪↔✂ø➀î➓å✠è✧ï❺æ✎ì✐ê✤ø➑è✧ø➀ì➈é ä✥ï➊û✓å✠è✧ø➀ú➈ï✢è✧ì➷ì➈è✧ç✙ï✖ä➻ëíï✖å✠è✥ÿ✙ä✥ï✖ê❭ø➀ê➽ä✧ï✖û➑ï✖úrå➈é✐è✖ð ✜✙ì➈ä➻ï✄↔✂å➈î➻æ✙û➑ï➎➌✇ì❛é✗￾❶ï ó➵ï❙ô➇é✙ì✠ó è✥ç☎å✠è➤è✥ç✙ï➻ø➑é✙æ☎ÿ✂è✌ø➑î➓å❄✂❛ï➚￾➊ì➈é✐è✥å➈ø➑é✎ê➔è✥ç✙ï➻ï➊é☎ù✂æ◆ì➈ø➀é✐è✌ì➄ë✇å ä✥ì➈ÿ❼✂❛ç✙û✙✘❭ç✙ì❛ä✧ø✠➽➊ì❛é❛è❿å➄û☎ê✧ï✄✂❛î➻ï➊é✐è✇ø➀é➓è✧ç☎ï♣ÿ✙æ☎æ✎ï✖ä✇û➀ï❶ë➺è❨å➄ä✥ï✖å✗➌❛å➉￾➊ì➈ä✥é✙ï➊ä ø➀é➽è✧ç☎ï➉ÿ✙æ✙æ◆ï➊ä✛ä✥ø✙✂❛ç✐è➵å➄ä✥ï✖å✗➌❛å➈é☎ù➻è✧ç☎ï♣ï✖é☎ù✂æ◆ì➈ø➀é❛è✛ì➄ë❀å✒ä✧ì❛ÿ❼✂➈ç☎û✙✘❭ú➈ï✖ä❇✝ è✧ø✁￾➊å➈û✎ê✧ï✄✂❛î❭ï✖é✐è❫ø➑é➽è✧ç✙ï♣û➑ì✠ó➵ï➊ä➏æ◆ì➈ä✧è✧ø➀ì➈é➓ì➈ë☛è✧ç✙ï♣ø➑î➓å✑✂➈ï✑➌❛ó✇ï✞￾➊å➈é➻è✧ï➊û➀û è✧ç☎ï✒ø➀é✙æ✙ÿ✂è➳ø➑î➓å❄✂❛ï➞ø✓ê➉å✜✔✂ð➔ñ➔ì➄è➳ì➈é✙û✠✘✿ø➀ê➔è✧ç✙ï❙æ✙ä✧ï★￾❶ø✓ê✤ï✒æ✎ì✐ê✤ø➑è✧ø➀ì➈é✫ì➈ë ï✖å➎￾❿ç❙ì➈ë☎è✥ç✙ì❛ê✧ï➵ëíï✖å✠è✥ÿ✙ä✥ï✖ê❦ø➀ä✧ä✥ï➊û➀ï➊ú✠å➄é✐è❦ëíì➈ä❣ø✓ù✂ï✖é❛è✥ø➩ë➭✘➇ø➀é❼✂➳è✧ç✙ï❨æ✎å✠è✤è✥ï➊ä✥é✏➌ ø➑è✛ø✓ê➵æ✎ì➈è✧ï✖é❛è✥ø➀å➈û➑û✠✘➻ç☎å➄ä✥î❭ëíÿ✙û❢✡◆ï✒￾➊å➈ÿ☎ê✧ï➔è✧ç☎ï➳æ◆ì❛ê✧ø➑è✧ø➀ì➈é☎ê➵å➄ä✥ï♣û➀ø➀ô➈ï➊û✠✘❭è✧ì ú✠å➄ä❘✘➻ëíì➈ä➔ù✂ø✙➘✟ï➊ä✥ï➊é✐è❨ø➑é☎ê✤è✥å➈é✗￾❶ï❞ê✛ì➄ë❇è✧ç✙ï✌￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✖ð❨✕ ê✤ø➀î➻æ✙û➀ï➳ó✛å✪✘ è✧ì✶ä✧ï❞ù✂ÿ✗￾❶ï➤è✥ç✙ï➞æ✙ä✥ï✒￾➊ø➀ê✧ø➀ì➈é✺ó❨ø➩è✥ç✫ó❨ç✙ø✁￾❿ç✺è✧ç✙ï➞æ◆ì❛ê✧ø➑è✧ø➀ì➈é✺ì➄ë➏ù✂ø✓ê④è✥ø➑é✥￾✹✝ è✧ø➀ú➈ï♣ëíï✖å➄è✧ÿ✙ä✥ï✖ê✛å➈ä✧ï➳ï➊é✗￾➊ì✂ù✂ï✖ù✶ø➑é✺å➞ëíï❞å✠è✥ÿ✙ä✧ï➳î➓å➄æ✢ø➀ê✇è✧ì➻ä✥ï✖ù✂ÿ✥￾❶ï♣è✧ç✙ï ê✧æ☎å✠è✥ø➀å➈û✙ä✧ï❞ê✤ì❛û➑ÿ✙è✧ø➀ì➈é❭ì➄ë✟è✧ç✙ï➔ëíï✖å✠è✥ÿ✙ä✥ï❨î➻å➈æ✝ð❣ã✛ç✙ø✓ê❹￾➊å➈é➚✡◆ï➉å➎￾❿ç✙ø➑ï✖ú➈ï❞ù ó❨ø➑è✧ç➓å✌ê✤ì✑✝❖￾✖å➄û➀û➑ï❞ù➵➩✒✡✤✯❪✭❖➩❯➢❄➲✛➤ ✲✣●➨✵✥ ✲✙➢✟✑➳❯➡✴➩❦ó❨ç✙ø✁￾❿ç➻æ✎ï✖ä✤ëíì❛ä✧î➓ê❣å➤û➀ì☛￾✖å➄û årú➈ï✖ä✥å✑✂➈ø➀é❼✂✫å➄é☎ù❤å➲ê✤ÿ✗✡✦✝⑥ê✥å➄î➻æ✙û➀ø➀é❼✂✗➌❦ä✥ï✖ù✂ÿ✗￾➊ø➑é✗✂➲è✧ç☎ï✶ä✥ï✖ê✧ì➈û➀ÿ✂è✧ø➀ì➈é ì➈ë è✧ç☎ï❨ëíï✖å➄è✧ÿ✙ä✥ï❨î➓å➄æ✏➌➇å➄é☎ù❭ä✥ï✖ù✂ÿ✥￾❶ø➀é❼✂➤è✧ç✙ï♣ê✤ï✖é☎ê✤ø➑è✧ø➀ú➇ø➩è❅✘❙ì➄ë✟è✧ç☎ï➔ì➈ÿ✂è✥æ✙ÿ✂è è✧ì♣ê✤ç✙ø➑ë➺è✥ê❯å➄é✎ù✌ù✂ø➀ê✤è✧ì❛ä✤è✥ø➑ì❛é☎ê✖ð❇ã✛ç✙ï❫ê✧ï✒￾➊ì➈é☎ù➤ç☎ø➀ù✙ù✙ï➊é➞û✓å✪✘➈ï✖ä✝ì➄ë ✗✝ï❞ñ➔ï➊è❇✝ ✙❭ø✓ê➔å➻ê✤ÿ❼✡❼✝⑥ê✥å➄î➻æ✙û➀ø➑é✗✂➓û➀å✪✘❛ï➊ä❞ð❦ã✛ç☎ø➀ê❨û✓å✪✘➈ï✖ä✛￾❶ì❛î❭æ☎ä✧ø✓ê✤ï❞ê❨ê✤ø✙↔➽ëíï❞å✠è✧ÿ☎ä✧ï î➓å➄æ☎ê✒➌✂ì➈é☎ï♣ëíì❛ä✛ï✖å✑￾❿ç✢ëíï✖å➄è✧ÿ✙ä✥ï➤î➓å➄æ✿ø➑é✢è✧ç✙ï✌æ☎ä✧ï✖ú✐ø➀ì➈ÿ✎ê✇û✓å✪✘➈ï✖ä✖ð❦ã✛ç✙ï ä✥ï✒￾❶ï✖æ✂è✧ø➀ú➈ï✞➞☎ï✖û➀ù✢ì➄ë❯ï✖å➎￾❿ç✢ÿ✙é☎ø➩è➔ø✓ê❨å ✑➉✡☛✘ ✑❭å➈ä✧ï❞å✒ø➀é✶è✥ç✙ï✌æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê û✓å✪✘➈ï➊ä✒❁ ê❀￾❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✂ø➀é❼✂➉ëíï✖å✠è✥ÿ✙ä✥ï✇î➓å➄æ❀ð ✭❫å➎￾❿ç✒ÿ✙é☎ø➩è❹￾❶ì❛î➻æ✙ÿ✂è✧ï❞ê❇è✧ç✙ï ➢✍♦➳✄➡❺➢❪✥❩➳✛ì➄ë❇ø➩è❿ê❫ëíì❛ÿ✙ä✛ø➑é☎æ✙ÿ✂è✥ê✒➌✂î✒ÿ☎û➩è✥ø➑æ✙û➀ø➀ï✖ê✛ø➑è✎✡☛✘➽å✒è✧ä❿å➄ø➀é☎å❄✡✙û➀ï④￾➊ì➇ï❶ë●✝ ➞✥￾➊ø➑ï✖é❛è★➌❛å❛ù✙ù✙ê➏å➳è✥ä✥å➈ø➑é☎å✑✡✙û➀ï✎✡☎ø➀å❛ê✄➌✐å➄é☎ù❭æ☎å❛ê✧ê✧ï✖ê❯è✥ç✙ï➔ä✥ï✖ê✧ÿ✙û➩è❣è✥ç✙ä✥ì➈ÿ❼✂❛ç å➳ê✤ø✠✂➈î➻ì➈ø✓ù✌ëíÿ☎é✗￾↔è✥ø➑ì❛é✝ð❷☞✇ì❛é✐è✧ø✠✂➈ÿ✙ì❛ÿ☎ê❀ÿ☎é✙ø➩è❿ê❦ç☎årú➈ï✇é✙ì➈é❼✝♠ì✠ú❛ï➊ä✥û➀å➈æ✙æ✙ø➀é❼✂ ￾❶ì❛é✐è✧ø✠✂➈ÿ✙ì❛ÿ☎ê➉ä✥ï✒￾➊ï➊æ✂è✥ø➑ú❛ï➉➞✎ï➊û✓ù✙ê➊ð➝☞✇ì➈é☎ê✧ï✒➍✐ÿ✙ï✖é❛è✥û✙✘➎➌✝å✢ê✧ÿ❼✡✦✝➠ê✥å➄î➻æ✙û➀ø➑é❼✂ û✓å✪✘➈ï➊ä❯ëíï❞å✠è✧ÿ☎ä✧ï✛î➓å➄æ❭ç☎å➈ê❦ç✎å➄û➑ë☎è✧ç☎ï❨é✐ÿ☎î➉✡◆ï➊ä❣ì➄ë◆ä✧ì✠ó➔ê❦å➈é☎ù➑￾➊ì➈û➀ÿ✙î➻é☎ê

2Fs-s u OvE 2EE7X LEO AEF 1ii8 C3:f.maps 16@10x10 INPUT C1:feature maps S4:f.maps 16@5x5 32x32 6@28x28 S2:f.maps 6@14x14 layer F:layer OUTPUT 10 Full connection Gaussian connections Convolutions Subsampling Convolutions Subsampling Full connection ofLConyoltional Nual Ntwok foitcognition.Eacffi ana fatu maat of unit 62街eg。coa护ain:Fto b7Enka. as the feature maps in the previous la0er.The trainable LeNet-c coeb cient and bias control the effect of the sigmoid non- linearitO.If the coeb cient is small,then the unit operates This section describes in more detail the architecture of in a quasi-linear mode,and the sub-sampling la0er merelO LeNet-5,the Convolutional Neural Network used in the blurs the input.If the coeb cient is large,sub-sampling experiments.LeNet-5 comprises,la0ers,not counting the units can be seen as performing a“nois0OR”ora“nois0 input,all of which contain trainable parameters (weights). AND"function depending on the value of the bias.Succes- The input is a 32x32 pixel image.This is significantl0 larger sive la0ers of convolutions and sub-sampling are topicallo than the largest character in the database (at most 2MK2M alternated,resulting in a"bi-poramid"at each la0er,the pixels centered in a 28x28 field).The reason is that it is number of feature maps is increased as the spatial resolu- desirable that potential distinctive features such as stroke tion is decreased.-ach unit in the third hidden la0er in fig- end-points or corner can appear v the enter of the recep- ure 2 ma0 have input connections from several feature maps tive field of the highest-level feature detedtors.In LeNet-5 in the previous la0er.The convolutionosub-sampling com- the set of centers of the receptive fields of the last convolu- bination,inspired bo Hubel and Wiesel s notions of "sim- tional la0er(C3,see below)form a 2NK2Marea in the center ple”and“complex”cels,was implemente in Fukushima s of the 32x32 input.The values of the input pixels are nor- Neocognitron [32],though no globallo spervised learnin malized so that the background level (white)corresponds procedure such as back-propagation was available then.to a value ofMl and the foreground (black)corresponds large degree of invariance to geometric transformations of to 1.1,5.This makes the mean input roughlo M and the the input can be achieved with this progressive reduction variance roughlo 1 which accelerates learning [4j]. of spatial resolution compensated b0 a progressive increase In the following,convolutionallaOers are labeled Cx,sub- of the richness of the representation (the number of feature sampling la0ers are labeled Sx,and full0-connected la0ers maps). are labeled Fx,where x is the la0er index. Since all the weights are learned with back-propagation, LaOer Cl is a convolutional laOer with j feature maps. convolutional networks can be seen as sOnthesizing their ach unit in each feature map is connected to a 5x5 neigh- own feature extractor.The weight sharing technique has borhood in the input.The size of the feature maps is 28x28 the interesting side effect of reducing the number of free which prevents connection from the input from falling off parameters,therebo reducing the "capacito"of the ma- the boundar0.Cl contains 15j trainable parameters,and chine and reducing the gap between test error and training 122,3M connections. error [34].The network in figure 2 contains 34M9N8 con- LaOer S2 is a sub-sampling la0er with i feature maps of nections,but onl0 jMNMMtrainable free parameters because size 14x14.-ach unit in each feature map is connected to a of the weight sharing. 2x2 neighborhood in the corresponding feature map in C1. Fixed-size Convolutional Networks have been applied The four inputs to a unit in S2 are added,then multiplied to mano applications,among other handwriting recogni-bo a trainable coeb cient,and added to a trainable bias. tion [35],[3j],machine-printed character recognition [3,] The result is passed through a sigmoidal function.The on-line handwriting recognition [38],and face recogni-2x2 receptive fields are non-overlapping,therefore feature tion [39].Fixed-size convolutional networks that share maps in S2 have half the number of rows and column as weights along a single temporal dimension are known as feature maps in C1.LaOer S2 has 12 trainable parameters Time-Dela0 Neural Networks(TDNNs).TDNNs have been and 5,88M connections. used in phoneme recognition(without sub-sampling)[4M, LaOer C3 is a convolutional laOer with 1j feature maps. [41],spoken word recognition (with sub-sampling)[42],-ach unit in each feature map is connected to several 5x5 [43],on-line recognition of isolated handwritten charac- neighborhoods at identical locations in a subset of S2s ters [44],and signature verification [45]. feature maps.Table I shows the set of $2 featurea

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ￾ INPUT 32x32 Convolutions Convolutions Subsampling C1: feature maps 6@28x28 Subsampling S2: f. maps 6@14x14 S4: f. maps 16@5x5 C5: layer 120 C3: f. maps 16@10x10 F6: layer 84 Full connection Full connection Gaussian connections OUTPUT 10 ✁❼✿▲❍✪❦✪❾★❦➝❱❜✸✶❆✶✯✪✿ ✵✶✰❅❆r✵✻✳✪✸✶✰❷✷✴⑥✦P❩✰❺❤❀✰r✵✻❑✂✁★❚♦✱✎❻✇✷✹❈❯▼❯✷✹❴▲✳★✵✶✿▲✷✹❈✪✱✴❴✦❤❳✰❅✳✪✸✶✱✴❴☛❤❳✰r✵●✽❢✷✹✸✻❣✑❚★✯✪✰❅✸✶✰❜⑥✙✷✹✸❢❉✪✿▲❍✹✿ ✵✻✺✏✸✶✰❅❆❅✷✹❍✹❈✪✿ ✵✶✿▲✷✹❈➎❦✥❧✈✱✴❆❖✯✞❃★❴➂✱✴❈✪✰❨✿▲✺✏✱➜⑥✙✰❇✱❘✵✶✳★✸✶✰❷❋✛✱✴❃❩❚✄✿✁❦ ✰✹❦✪✱➜✺✶✰r✵✏✷✴⑥❼✳✪❈✪✿ ✵✻✺ ✽❳✯★✷✹✺✶✰❨✽❢✰❅✿▲❍✹✯❯✵✶✺❜✱✴✸✶✰❨❆❅✷✹❈✪✺✵✶✸✶✱✴✿▲❈✪✰❅❉✞✵✶✷✎◗✑✰❷✿▲❉★✰❇❈❯✵✻✿▲❆❺✱✴❴✁❦ å➈ê➞è✧ç✙ï✶ëíï✖å➄è✧ÿ✙ä✥ï➽î➓å➄æ☎ê✒ø➑é➷è✥ç✙ï✶æ☎ä✧ï✖ú✐ø➀ì➈ÿ✎ê✌û➀å✪✘❛ï➊ä❞ð✫ã✛ç✙ï➓è✧ä❿å➄ø➀é☎å✑✡✙û➑ï ￾❶ì➇ï✱✯➵￾➊ø➑ï✖é✐è✒å➄é☎ù➒✡✙ø✓å➈ê✌￾➊ì➈é✐è✧ä✥ì➈û❦è✥ç✙ï➓ï❯➘✟ï✒￾❶è➞ì➄ë➵è✧ç✙ï✶ê✧ø✠✂➈î➻ì➈ø✓ù❖é☎ì➈é✦✝ û➀ø➑é✙ï❞å➄ä✥ø➩è❅✘❛ð➃➏⑥ë❦è✥ç✙ï➉￾❶ì➇ï✱✯➵￾➊ø➑ï✖é✐è➔ø➀ê♣ê✤î➓å➄û➀û✶➌✙è✧ç☎ï➊é✺è✧ç☎ï➞ÿ✙é✙ø➑è➉ì➈æ◆ï➊ä❿å✠è✥ï✖ê ø➀é✺å➵➍✐ÿ☎å❛ê✤ø✙✝⑥û➑ø➀é✙ï✖å➈ä✛î❭ì✂ù✂ï➎➌☎å➄é✎ù✶è✥ç✙ï➞ê✧ÿ❼✡✦✝➠ê✧å➈î❭æ☎û➑ø➀é❼✂➓û✓å✪✘➈ï➊ä✛î➻ï➊ä✥ï➊û✠✘ ✡✙û➀ÿ✙ä❿ê➻è✧ç✙ï❖ø➑é✙æ☎ÿ✂è✖ð ➏⑥ë➳è✥ç✙ï↕￾❶ì➇ï✱✯➵￾➊ø➑ï✖é✐è✶ø➀ê➽û➀å➈ä❺✂❛ï✑➌✇ê✧ÿ❼✡✦✝➠ê✥å➄î➻æ✙û➀ø➑é❼✂ ÿ✙é✙ø➑è✥ê④￾➊å➈é✆✡✎ï✒ê✧ï➊ï✖é✫å➈ê❨æ◆ï➊ä✧ëíì➈ä✥î➻ø➑é❼✂➽å ☛✤é✙ì❛ø➀ê❺✘✿ý✞✍✌➻ì❛ä➉å▲☛✤é✙ì❛ø➀ê❺✘ ✕➉ñ④✧✟✌➳ëíÿ☎é✗￾↔è✥ø➑ì❛é➻ù✙ï➊æ◆ï➊é☎ù✂ø➀é❼✂➤ì➈é❙è✧ç☎ï❨úrå➈û➑ÿ☎ï➵ì➈ë☎è✥ç✙ï✔✡✙ø➀å❛ê➊ð❣þ➇ÿ✗￾✒￾❶ï❞ê❅✝ ê✧ø➑ú❛ï❙û✓å✪✘➈ï✖ä✥ê♣ì➄ë✔￾❶ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎ê➳å➄é✎ù❺ê✧ÿ❼✡✦✝➠ê✧å➈î➻æ✙û➑ø➀é❼✂✺å➄ä✥ï✒è❅✘➇æ✙ø✁￾➊å➈û➑û✠✘ å➄û➑è✧ï✖ä✧é✎å✠è✧ï❞ù❢➌☎ä✥ï✖ê✧ÿ✙û➑è✧ø➀é❼✂➽ø➀é❖å✏☛❺✡✙ø➟✝⑥æ☛✘➇ä✥å➈î❭ø✓ù✌❇✰✇å✠è♣ï✖å✑￾❿ç✫û✓å✪✘➈ï✖ä✒➌✙è✧ç✙ï é➇ÿ✙î➉✡◆ï➊ä➳ì➄ë❣ëíï❞å✠è✥ÿ✙ä✧ï❙î➓å➄æ☎ê➉ø➀ê♣ø➑é✥￾❶ä✥ï✖å➈ê✧ï✖ù✫å❛ê➔è✥ç✙ï❙ê✧æ☎å➄è✧ø✓å➄û❯ä✧ï❞ê✤ì❛û➑ÿ✦✝ è✧ø➀ì➈é❭ø✓ê❯ù✙ï✒￾❶ä✥ï✖å❛ê✤ï❞ù☛ð ✭✇å✑￾❿ç➞ÿ☎é✙ø➩è❣ø➀é✒è✥ç✙ï➵è✧ç✙ø➀ä❿ù✒ç✙ø✓ù✙ù✂ï✖é❙û✓å✪✘➈ï✖ä❇ø➀é➑➞✗✂❄✝ ÿ✙ä✥ï✄✑➵î➓å✪✘➉ç☎årú❛ï❣ø➑é☎æ✙ÿ✂è❜￾➊ì➈é✙é☎ï✒￾↔è✥ø➑ì❛é☎ê✟ëíä✥ì➈î ê✤ï✖ú➈ï✖ä✥å➈ûrëíï✖å✠è✥ÿ✙ä✥ï➏î➓å➄æ☎ê ø➀é✢è✥ç✙ï✌æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê✛û✓å✪✘➈ï✖ä✖ð❦ã✛ç✙ï➓￾❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎✄✠ê✧ÿ❼✡✦✝➠ê✧å➈î➻æ✙û➑ø➀é❼✂➚￾❶ì➈î➚✝ ✡✙ø➀é☎å✠è✥ø➑ì❛é✏➌☛ø➀é☎ê✤æ☎ø➑ä✥ï✖ù✆✡☛✘✫õ➔ÿ✗✡✎ï✖û➏å➄é☎ù ✎ø➑ï❞ê✤ï✖û✡❁ ê➉é☎ì➄è✧ø➀ì➈é✎ê➉ì➈ë ☛✧ê✧ø➑î➚✝ æ✙û➀ï✖✌➤å➄é✎ù ☛❘￾❶ì❛î❭æ☎û➑ï✄↔✔✌✞￾❶ï✖û➑û✓ê✒➌➄ó➵å❛ê❦ø➑î➻æ✙û➀ï➊î➻ï✖é❛è✥ï✖ù❭ø➀é➚✜☎ÿ✙ô➇ÿ☎ê✤ç☎ø➑î➓å❂❁ ê ñ➔ï✖ì✦￾❶ì✑✂❛é✙ø➑è✧ä✥ì➈é ✞❜ ✑✆✠❖➌✎è✧ç✙ì❛ÿ❼✂➈ç✫é✙ì➙✂➈û➀ì✑✡☎å➈û➑û✠✘✺ê✤ÿ✙æ◆ï➊ä✥ú➇ø➀ê✧ï✖ù✺û➀ï✖å➈ä✧é☎ø➑é❼✂ æ✙ä✥ì✦￾❶ï✖ù✙ÿ✙ä✧ï✌ê✧ÿ✗￾❿ç✺å➈ê✎✡☎å✑￾❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é➓ó➵å❛ê❨årú✠å➄ø➀û➀å✑✡✙û➑ï♣è✧ç☎ï➊é✝ð❹✕ û✓å➄ä❘✂➈ï✒ù✂ï✒✂➈ä✥ï➊ï➞ì➈ë❫ø➀é✐ú✠å➈ä✧ø✓å➄é✗￾➊ï✌è✧ì➙✂➈ï✖ì➈î➻ï❶è✥ä✧ø✁￾➤è✧ä❿å➄é✎ê④ëíì❛ä✧î➓å✠è✥ø➑ì❛é☎ê❨ì➈ë è✧ç☎ï❭ø➀é✙æ✙ÿ✙è✌￾✖å➄é➛✡✎ï➓å✑￾❿ç☎ø➑ï✖ú➈ï✖ù➲ó❨ø➩è✥ç❖è✥ç✙ø➀ê➤æ☎ä✧ì➎✂➈ä✥ï✖ê✥ê✤ø➀ú➈ï➞ä✥ï✖ù✙ÿ✗￾↔è✥ø➑ì❛é ì➄ë✝ê✧æ☎å✠è✥ø➀å➈û✙ä✥ï✖ê✧ì➈û➀ÿ✂è✧ø➀ì➈é➵￾❶ì❛î➻æ✎ï✖é☎ê✧å➄è✧ï❞ù➝✡❩✘❭å✌æ✙ä✥ì✑✂❛ä✧ï❞ê✧ê✧ø➑ú❛ï❫ø➀é✗￾❶ä✥ï✖å❛ê✤ï ì➄ë◆è✧ç☎ï➔ä✧ø✁￾❿ç✙é✙ï❞ê✧ê❯ì➈ë✎è✥ç✙ï➔ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✥å➄è✧ø➀ì➈é➐➪➺è✧ç☎ï➔é➇ÿ✙î➉✡◆ï➊ä➏ì➈ë✎ëíï❞å✠è✥ÿ✙ä✧ï î➓å➄æ☎ê✴➶↔ð þ➇ø➑é✥￾❶ï✌å➄û➀û☎è✥ç✙ï➤ó✇ï✖ø✙✂❛ç✐è✥ê➵å➄ä✥ï➉û➀ï✖å➄ä✥é✙ï❞ù➽ó❨ø➑è✧ç➐✡☎å✑￾❿ô❩✝⑥æ✙ä✧ì❛æ☎å❄✂✐å✠è✥ø➑ì❛é✏➌ ￾❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✛é✙ï➊è④ó✇ì❛ä✧ô✂ê➝￾➊å➈é⑧✡◆ï➲ê✤ï✖ï➊é➘å➈ê➓ê❺✘➇é❛è✥ç✙ï✖ê✧ø✠➽➊ø➀é❼✂❖è✥ç✙ï➊ø➀ä ì✠ó❨é ëíï✖å➄è✧ÿ✙ä✥ï➓ï❯↔➇è✧ä❿å✑￾❶è✧ì❛ä✖ð➽ã✛ç✙ï✶ó✇ï✖ø✙✂❛ç❛è➞ê✧ç☎å➄ä✥ø➀é❼✂✿è✧ï✒￾❿ç☎é✙ø✠➍✐ÿ✙ï✶ç☎å➈ê è✧ç☎ï✢ø➀é✐è✧ï➊ä✥ï✖ê✤è✧ø➀é❼✂ ê✤ø✓ù✂ï✿ï❯➘✟ï✒￾↔è➻ì➄ë➔ä✥ï✖ù✙ÿ✗￾❶ø➀é❼✂➲è✥ç✙ï✿é✐ÿ☎î➉✡◆ï➊ä➻ì➄ë❨ëíä✥ï➊ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê✒➌✛è✧ç✙ï✖ä✧ï✒✡☛✘ ä✥ï✖ù✙ÿ✗￾❶ø➀é❼✂ è✥ç✙ï ☛❺￾➊å➈æ☎å✑￾➊ø➩è❅✘✔✌➷ì➈ë➞è✧ç✙ï î➓å♦✝ ￾❿ç✙ø➀é✙ï➉å➈é☎ù➻ä✧ï❞ù✂ÿ✗￾➊ø➑é❼✂➤è✥ç✙ï✛✂✐å➄æ➝✡✎ï➊è④ó✇ï✖ï➊é➻è✥ï✖ê✤è❫ï➊ä✥ä✧ì❛ä➏å➄é☎ù❭è✥ä✥å➈ø➑é☎ø➑é❼✂ ï➊ä✥ä✥ì➈ä ✞❜✫❝✬✠⑥ð➽ã✛ç✙ï➓é✙ï➊è④ó✇ì❛ä✧ô➲ø➀é➒➞✗✂➈ÿ☎ä✧ï❵✑✫￾❶ì❛é✐è✥å➄ø➀é☎ê✹❜✫❝✗✘❼➌ ❀✻✘✗✺ ￾➊ì➈é✦✝ é✙ï★￾↔è✧ø➀ì➈é✎ê✄➌♦✡✙ÿ✙è❦ì❛é✙û✠✘ ✓✗✘❼➌ ✘✻✘✗✘✛è✧ä❿å➄ø➀é☎å❄✡☎û➑ï❫ëíä✧ï✖ï❫æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê❳✡✎ï★￾➊å➈ÿ☎ê✤ï ì➄ë❯è✧ç☎ï✌ó✇ï✖ø✙✂❛ç❛è➔ê✧ç☎å➄ä✥ø➀é❼✂☎ð ✜❇ø✙↔✂ï✖ù☛✝➠ê✧ø✙➽✖ï❭☞✇ì❛é✐ú❛ì➈û➀ÿ✂è✧ø➀ì➈é✎å➄û❙ñ➉ï❶è④ó➵ì➈ä✥ô✂ê❖ç☎årú❛ï➹✡◆ï➊ï✖é➶å➄æ☎æ✙û➑ø➀ï✖ù è✧ì î➓å➈é❩✘❑å➄æ✙æ✙û➀ø✁￾➊å✠è✥ø➑ì❛é☎ê✒➌➵å➈î❭ì❛é❼✂ ì➄è✧ç☎ï➊ä➽ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é✗✂ ä✥ï✒￾➊ì✑✂❛é✙ø➟✝ è✧ø➀ì➈é✩✞❜ ✙✆✠❖➌ ✞❜✗✓✠❖➌☛î➓å✑￾❿ç✙ø➀é✙ï❯✝⑥æ✙ä✥ø➑é✐è✥ï✖ù➔￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä♣ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é ✞❜ ✔✆✠✶➌ ì➈é❼✝♠û➀ø➑é☎ï✾ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é✗✂➶ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é ✞❜✗✺✠❖➌➲å➄é☎ù✴ë⑨å✑￾➊ï✾ä✥ï✒￾➊ì✑✂❛é✙ø➟✝ è✧ø➀ì➈é ✞❜✗❀✠⑥ð ✜❇ø✙↔✂ï✖ù☛✝➠ê✧ø✙➽✖ï➹￾❶ì❛é✐ú❛ì➈û➀ÿ✂è✧ø➀ì➈é✎å➄û➞é✙ï➊è④ó✇ì❛ä✧ô✂ê✿è✥ç☎å✠è➷ê✤ç☎å➈ä✧ï ó➵ï➊ø✠✂➈ç✐è✥ê➻å➄û➀ì➈é✗✂ å ê✤ø➀é❼✂❛û➑ï✿è✧ï✖î➻æ✎ì❛ä✥å➈û✛ù✙ø➑î➻ï➊é✎ê✤ø➀ì➈é➘å➄ä✥ï✢ô➇é✙ì✠ó❨é å➈ê ã✛ø➀î❭ï✄✝❖✧♣ï➊û✓å✪✘♣ñ➉ï➊ÿ✙ä❿å➄û✐ñ➔ï➊è④ó✇ì❛ä✧ô✂ê❷➪⑨ã✩✧➳ñ➉ñ♣ê❘➶❶ð❯ã✩✧➳ñ➉ñ♣ê❀ç✎årú➈ï❷✡◆ï➊ï➊é ÿ☎ê✧ï✖ù❖ø➑é➲æ☎ç✙ì➈é✙ï✖î➻ï❙ä✥ï✒￾➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✢➪íó❨ø➑è✧ç✙ì❛ÿ✂è➤ê✧ÿ❼✡✦✝➠ê✥å➄î➻æ✙û➀ø➑é❼✂☛➶✜✞❝✗✘✬✠✶➌ ✞❝✗➾✡✠❖➌➻ê✤æ◆ì➈ô❛ï➊éòó➵ì➈ä❿ù ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é ➪⑨ó❨ø➩è✥ç✭ê✧ÿ❼✡✦✝➠ê✥å➄î➻æ✙û➀ø➑é❼✂☛➶ ✞❝✵✑ ✠✶➌ ✞❝❜✠❖➌✌ì➈é❼✝♠û➀ø➑é☎ï ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é ì➄ë➻ø➀ê✧ì➈û✓å✠è✥ï✖ù❲ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï➊é②￾❿ç✎å➄ä❿å✑￾✹✝ è✧ï✖ä✥ê✹✞❝✫❝✫✠✶➌✙å➈é☎ù✺ê✤ø✠✂➈é✎å✠è✧ÿ☎ä✧ï➳ú➈ï✖ä✧ø✙➞✥￾➊å➄è✧ø➀ì➈é ✞❝✦✙✆✠⑥ð ✬✛❊✢➳❺➸➑➳❯➺❩✭✝✆ ã✛ç✙ø➀ê➉ê✤ï★￾↔è✧ø➀ì➈é✫ù✂ï❞ê❺￾➊ä✧ø✠✡◆ï✖ê❨ø➀é✺î❭ì❛ä✧ï✌ù✂ï➊è✥å➈ø➑û❀è✧ç✙ï➞å➈ä❘￾❿ç✙ø➑è✧ï★￾↔è✥ÿ✙ä✧ï➤ì➈ë ✗✝ï❞ñ➔ï➊è❇✝ ✙✦➌❨è✥ç✙ï①☞✇ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û➳ñ➔ï➊ÿ☎ä✥å➈û♣ñ➔ï➊è④ó✇ì❛ä✧ô➘ÿ☎ê✧ï✖ù✻ø➑é✲è✧ç✙ï ï❯↔✂æ◆ï➊ä✥ø➑î➻ï✖é❛è❿ê➊ð ✗❀ï✖ñ➔ï➊è❇✝ ✙➉￾➊ì➈î➻æ✙ä✥ø➀ê✧ï✖ê ✔➞û➀å✪✘❛ï➊ä❿ê✄➌✐é✙ì➄è✩￾❶ì❛ÿ✙é✐è✧ø➀é❼✂❙è✧ç✙ï ø➀é✙æ✙ÿ✂è★➌☎å➄û➀û✟ì➄ë❀ó❨ç✙ø✁￾❿ç➙￾➊ì➈é✐è✥å➈ø➑é✶è✧ä❿å➄ø➀é☎å❄✡☎û➑ï➳æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê✬➪⑨ó✇ï✖ø✙✂❛ç✐è✥ê✴➶↔ð ã✛ç✙ï➏ø➀é✙æ✙ÿ✙è❀ø➀ê❀å❙❜ ✑✪↔❜✵✑❫æ☎ø➟↔✂ï➊û➈ø➑î➓å❄✂❛ï➈ð❀ã✛ç☎ø➀ê❀ø➀ê✝ê✧ø✠✂➈é✙ø✙➞✥￾➊å➈é✐è✧û✠✘➉û✓å➄ä❘✂➈ï➊ä è✧ç✎å➄é✺è✧ç☎ï➞û➀å➈ä❺✂❛ï✖ê✤è✬￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä❨ø➑é✺è✥ç✙ï✒ù✙å➄è✥å❄✡✎å➈ê✧ï➣➪⑨å✠è♣î❭ì✐ê④è✒✑❆✘♦↔✤✑✻✘ æ✙ø✙↔✂ï➊û✓ê➓￾❶ï✖é❛è✥ï➊ä✥ï✖ù❺ø➑é å❈✑✻✺♦↔✤✑❆✺ ➞☎ï➊û✓ù✗➶❶ð✶ã✛ç✙ï➓ä✥ï✖å➈ê✧ì➈é ø✓ê➤è✧ç☎å➄è➞ø➩è➞ø✓ê ù✂ï❞ê✤ø➀ä✥å✑✡✙û➀ï✌è✧ç☎å➄è♣æ◆ì➄è✥ï➊é✐è✧ø✓å➄û❦ù✂ø✓ê④è✥ø➑é✗￾❶è✧ø➀ú➈ï✒ëíï✖å✠è✥ÿ✙ä✥ï✖ê➉ê✧ÿ✗￾❿ç➲å❛ê♣ê✤è✧ä✥ì➈ô❛ï ï➊é✎ù☛✝♠æ◆ì➈ø➀é✐è✥ê➵ì➈ä✎￾❶ì❛ä✧é✙ï✖ä✎￾✖å➄é✶å➈æ✙æ◆ï✖å➄ä❊✣●➨✆➺➭➥❼➳ ❄✴➳✄➨✗➺r➳❯➡❣ì➈ë✝è✧ç☎ï♣ä✥ï✒￾➊ï➊æ✦✝ è✧ø➀ú➈ï✞➞☎ï✖û➀ù✿ì➈ë❀è✧ç☎ï✌ç✙ø✙✂❛ç✙ï✖ê✤è❇✝⑥û➀ï➊ú➈ï✖û◆ëíï✖å✠è✥ÿ✙ä✥ï➤ù✙ï❶è✧ï★￾↔è✥ì➈ä❿ê➊ð❨➏➠é❖✗✝ï❞ñ➔ï❶è❺✝✝✙ è✧ç☎ï♣ê✧ï❶è✇ì➈ë✏￾❶ï➊é✐è✥ï➊ä❿ê➏ì➄ë☛è✥ç✙ï➉ä✥ï✒￾➊ï➊æ✂è✥ø➑ú❛ï✩➞☎ï✖û➀ù✙ê✇ì➄ë✟è✥ç✙ï➉û✓å➈ê✤è✎￾➊ì➈é➇ú➈ì❛û➑ÿ✦✝ è✧ø➀ì➈é✎å➄û✂û✓å✪✘➈ï➊ä✛➪❖☞❀❜❼➌➈ê✧ï➊ï✩✡◆ï➊û➀ì✠ó✬➶❀ëíì➈ä✥î➶å✹✑❆✘❄↔✤✑❆✘➳å➄ä✥ï✖å♣ø➑é❭è✧ç☎ï✬￾❶ï✖é❛è✥ï➊ä ì➄ë❀è✧ç✙ï✹❜ ✑♦↔❜ ✑➤ø➀é✙æ✙ÿ✂è❞ð❫ã✛ç✙ï➳ú✠å➄û➀ÿ✙ï✖ê➵ì➄ë❇è✧ç✙ï➳ø➀é✙æ✙ÿ✂è➔æ✙ø✙↔✂ï➊û✓ê✛å➄ä✥ï➉é✙ì❛ä❇✝ î➓å➄û➀ø✙➽✖ï✖ù❺ê✤ì✢è✧ç☎å➄è➤è✧ç✙ï➝✡☎å➎￾❿ô❩✂❛ä✧ì❛ÿ✙é☎ù✫û➀ï➊ú❛ï➊û✩➪íó❨ç☎ø➩è✥ï★➶t￾❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✙ê è✧ì✫å✿ú✠å➄û➀ÿ✙ï➻ì➄ë➃✝ ✘☎ð✙➾❭å➄é☎ù❖è✧ç✙ï➻ëíì❛ä✧ï✒✂➈ä✥ì➈ÿ✙é✎ù✢➪➭✡✙û✓å✑￾❿ô❼➶✞￾❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✙ê è✧ì①➾❛ð✙➾✍✔✬✙✙ð➻ã✛ç☎ø➀ê✌î➓å➈ô➈ï✖ê➳è✥ç✙ï➓î➻ï✖å➈é ø➀é✙æ✙ÿ✙è✒ä✧ì❛ÿ❼✂➈ç☎û✙✘✦✘✗➌❇å➈é☎ù❺è✧ç✙ï ú✠å➄ä✥ø➀å➈é✗￾❶ï➳ä✥ì➈ÿ❼✂❛ç✙û✙✘➔➾➳ó❨ç☎ø✠￾❿ç✺å✑￾✒￾❶ï➊û➀ï➊ä❿å✠è✥ï✖ê➵û➀ï✖å➄ä✥é✙ø➀é❼✂❖✞❝✓✠⑥ð ➏➠é➤è✧ç✙ï❫ëíì➈û➀û➑ì✠ó❨ø➀é❼✂✗➌✪￾❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✠û✓å✪✘➈ï✖ä✥ê✝å➈ä✧ï❣û✓å❄✡◆ï➊û➀ï✖ù➉☞➜↔❢➌✠ê✧ÿ❼✡✦✝ ê✥å➄î➻æ✙û➀ø➑é❼✂✢û➀å✪✘❛ï➊ä❿ê➉å➈ä✧ï➞û✓å❄✡◆ï➊û➀ï✖ù þ❩↔❢➌☛å➄é☎ù➲ëíÿ✙û➑û✠✘❩✝❖￾➊ì➈é✙é☎ï✒￾↔è✥ï✖ù✫û✓å✪✘➈ï✖ä✥ê å➄ä✥ï➤û➀å✑✡✎ï✖û➑ï❞ù➙✜❳↔❢➌✙ó❨ç✙ï✖ä✧ï✞↔✿ø➀ê➵è✥ç✙ï✌û➀å✪✘❛ï➊ä✛ø➀é☎ù✂ï❯↔☛ð ✗❀å✪✘➈ï✖ä➑☞④➾➽ø✓ê✒å✆￾❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û❣û✓å✪✘➈ï✖ä➞ó❨ø➩è✥ç✮✓✺ëíï✖å➄è✧ÿ✙ä✥ï➓î➓å➄æ☎ê✖ð ✭❫å➎￾❿ç➻ÿ✙é✙ø➑è➵ø➀é➓ï✖å➎￾❿ç➻ëíï✖å✠è✥ÿ✙ä✥ï➔î➓å➄æ➽ø✓ê➜￾➊ì➈é✙é✙ï★￾↔è✥ï✖ù➻è✥ì✒å ✙✪↔✤✙➤é☎ï➊ø✠✂➈ç✦✝ ✡◆ì➈ä✥ç✙ì➇ì➇ù✒ø➑é✒è✧ç✙ï✛ø➀é✙æ✙ÿ✂è❞ð❯ã✛ç☎ï✛ê✧ø✠➽➊ï❫ì➈ë☎è✥ç✙ï❫ëíï❞å✠è✥ÿ✙ä✧ï➵î➓å➄æ☎ê❯ø✓ê ✑❆✺♦↔✤✑✻✺ ó❨ç✙ø✁￾❿ç❖æ✙ä✥ï➊ú❛ï➊é✐è✥êt￾➊ì➈é✙é✙ï★￾↔è✥ø➑ì❛é➲ëíä✥ì➈î è✧ç☎ï❭ø➀é✙æ✙ÿ✙è✌ëíä✧ì❛î ë⑨å➄û➀û➑ø➀é❼✂✿ì❄➘ è✧ç☎ï➑✡◆ì➈ÿ✙é✎ù✙å➄ä❘✘➈ð➓☞④➾➑￾➊ì➈é✐è✥å➈ø➑é☎ê➉➾ ✙✻✓➻è✧ä❿å➄ø➀é☎å❄✡☎û➑ï❙æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê✄➌◆å➄é☎ù ➾ ✑ ✑✦➌ ❜✻✘✫❝➑￾❶ì❛é✙é✙ï★￾↔è✧ø➀ì➈é✎ê➊ð ✗❀å✪✘➈ï✖ä➔þ ✑❭ø➀ê➉å➓ê✤ÿ❼✡❼✝⑥ê✥å➄î➻æ✙û➀ø➑é✗✂❭û✓å✪✘➈ï✖ä❨ó❨ø➩è✥ç ✓❙ëíï✖å✠è✥ÿ✙ä✥ï✌î➻å➈æ☎ê❨ì➈ë ê✧ø✙➽✖ï✞➾✖❝❄↔❢➾❪❝✎ð★✭❫å➎￾❿ç✒ÿ✙é☎ø➩è❣ø➀é❭ï❞å✑￾❿ç✒ëíï✖å✠è✥ÿ✙ä✥ï➵î➓å➈æ❙ø✓ê❷￾❶ì❛é✙é✙ï✒￾❶è✧ï❞ù➞è✥ì➳å ✑✪↔✤✑➤é☎ï➊ø✠✂➈ç☛✡✎ì❛ä✧ç☎ì✐ì✂ù➻ø➀é➓è✧ç☎ï④￾❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✂ø➀é❼✂➤ëíï❞å✠è✧ÿ☎ä✧ï♣î➻å➈æ➓ø➑é✫☞④➾➈ð ã✛ç✙ï➳ëíì➈ÿ☎ä❨ø➑é✙æ☎ÿ✂è✥ê✛è✥ì➓å✒ÿ✙é☎ø➩è➔ø➀é➲þ ✑✒å➈ä✧ï✌å❛ù✙ù✂ï✖ù✏➌➇è✧ç✙ï✖é✢î❙ÿ✙û➩è✥ø➑æ☎û➑ø➀ï✖ù ✡☛✘ å✫è✧ä❿å➄ø➀é☎å✑✡✙û➑ï➐￾❶ì➇ï❈✯➣￾➊ø➑ï✖é❛è★➌➏å➄é☎ù❑å➈ù✙ù✙ï✖ù➷è✥ì å➲è✧ä❿å➄ø➀é☎å❄✡✙û➀ï➣✡✙ø✓å➈ê✖ð ã✛ç✙ï➲ä✥ï✖ê✧ÿ✙û➑è➽ø➀ê✶æ☎å➈ê✥ê✧ï✖ù❑è✧ç✙ä✥ì➈ÿ✗✂➈ç❍å ê✤ø✠✂➈î➻ì❛ø➀ù✙å➈û✛ëíÿ✙é✗￾❶è✧ø➀ì➈é✝ð ã✛ç✙ï ✑✪↔✤✑✶ä✧ï★￾❶ï✖æ✂è✧ø➀ú➈ï➑➞☎ï➊û✓ù✙ê✌å➈ä✧ï❙é✙ì➈é✦✝⑥ì✠ú➈ï✖ä✧û✓å➄æ☎æ✙ø➑é✗✂✗➌✎è✧ç✙ï✖ä✧ï➊ëíì➈ä✥ï✒ëíï❞å✠è✧ÿ☎ä✧ï î➓å➄æ☎ê✒ø➑é❤þ✤✑✺ç☎årú➈ï➻ç☎å➈û➩ë✛è✥ç✙ï✶é➇ÿ✙î➑✡✎ï✖ä✒ì➄ë✛ä✥ì✠ó➔ê➞å➄é☎ù①￾➊ì➈û➀ÿ✙î➻é å➈ê ëíï✖å➄è✧ÿ✙ä✥ï➤î➓å➄æ☎ê❨ø➀é➔☞④➾➈ð ✗❇å✪✘➈ï➊ä❨þ ✑❭ç☎å❛êt➾ ✑✒è✧ä❿å➄ø➀é☎å✑✡✙û➑ï➤æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê å➄é✎ù✳✙❼➌ ✺✗✺✻✘➚￾❶ì➈é☎é✙ï✒￾❶è✧ø➀ì➈é☎ê✖ð ✗❀å✪✘➈ï✖ä✞☞❀❜➻ø✓ê♣å➣￾➊ì➈é➇ú➈ì❛û➑ÿ✙è✧ø➀ì➈é☎å➈û☛û➀å✪✘❛ï➊ä➔ó❨ø➑è✧ç✢➾✒✓❭ëíï✖å➄è✧ÿ✙ä✥ï➞î➓å➄æ☎ê✖ð ✭❫å➎￾❿ç✺ÿ✙é✙ø➑è➉ø➑é✫ï❞å✑￾❿ç✿ëíï✖å➄è✧ÿ✙ä✥ï➞î➓å➄æ✺ø✓ê✬￾➊ì➈é✙é✙ï★￾↔è✥ï✖ù✢è✧ì✶ê✧ï➊ú❛ï➊ä❿å➄û ✙♦↔✙ é✙ï✖ø✙✂❛ç❩✡◆ì➈ä✥ç✙ì➇ì✂ù✙ê✺å✠è✫ø✓ù✂ï➊é✐è✧ø✁￾➊å➈û➤û➑ì✦￾➊å➄è✧ø➀ì➈é☎ê✺ø➀é å❑ê✧ÿ❼✡☎ê✧ï❶è➲ì➈ë➻þ ✑❂❁ ê ëíï✖å➄è✧ÿ✙ä✥ï✿î➻å➈æ☎ê✖ð❑ã❯å❄✡✙û➀ï➙➏❭ê✤ç☎ì✠ó➔ê✒è✥ç✙ï✫ê✤ï➊è❭ì➈ë➤þ ✑➲ëíï✖å➄è✧ÿ✙ä✥ï✿î➻å➈æ☎ê

CXC.Ob CRE IEEE,AOVEy BE XFV 8 y c 2 3 4 5 P 7 s 9 cccdd where A is the amplitude of the funption and S determines A AAA AAAA AA its slope at the origin.The funption f is odd with horizon- C AA AAA AAAA A tal asvmptotes at VA and-A.The ponstant A is phosen 2 AAA to re c759.The rationale for this phoipe of a skuashing 3 A AA AAAA A funption is given in Nppendix N. 4 AAAA AA A 1 inallvothe output laver is pomposed of-uplidean H adial 5 AA A A A A Basis i unption units IB1)oone for eaph plasso with s4 LBLi inputs eaph.The outputs of eaph HB unit yi is romputed EACH CULUMN INDICATEO WHIEH FEAT GRE MAP IN S2 ARE CUMNED as followse THE UNITOIN A PARTICUUAR FEATURE MAP VF -3 班8∑Ei-u)C b) In other wordsoeaph output HBI unit pomputes the-u- plidean distanpe Fetween its input veptor and its parameter pomrined FM eaph T3 feature map.WhM not ponnept ev- veptor.The further awam is the input from the parameter erM S2 feature map to everMT3 feature map2 The rea- son is twofold.I irsto a non-pomplete ponneption spheme veptoro the larger is the HBI output.The output of a keeps the numrer of ponneptions within reasonarle rounds. partipular HB pan Fe interpreted as a penaltm term mea- More importantlyoit forpes a Freak of symmetrmin the net- suring the fit retween the input pattern and a model of the plass assopiated with the HB1.In prorarilistip termsothe work.Different feature maps are forped to extrapt different opefullM pomplementar)features Fepause them get dif- HBI output pan Fe interpreted as the unnormalized nega- ferent sets of inputs.The rationale rehind the ponneption tive log-likelihood of a Gaussian distripution in the spape of ponfigurations of laver I P.Given an input patternothe spheme in tarle I is the following.The first six r 3 feature loss funption should Fe designed so as to get the ponfigu- maps take inputs from everM pontiguous supsets of three ration of 1 P as plose as possiple to the parameter veptor feature maps in S2.The next six take input from everM of the that porresponds to the patterns desired plass. pontiguous supset of four.The next three take input from The parameter veptors of these units were phosen Fy hand some dispontinuous supsets of four.I inallM the last one takes input from all S2 feature maps.Laver r3 has cod and kept fixed rat least initiall).The pomponents of those parameters veptors were set to -cor VG While thempould trainarle parameters and ooPyy ponneptions. have Feen phosen at random with ekual prorarilities for-c Laver S4 is a sur-sampling laver with dfeature maps of and Vooor even phosen to form an error porrepting pode size 5x5.-aph unit in eaph feature map is ponnepted to a as suggested FM [47]otheM were instead designed to repre- 2x2 neighrorhood in the porresponding feature map in T3o sent a stMized image of the porresponding pharapter plass in a similar wam as T cand S2.Laver S4 has 32 trainarle drawn on a 7x Fitmap ihenpe the numper s4).Suph a parameters and 2oyy ponneptions. representation is not partipularlM useful for repognizing iso- Laver T5 is a ponvolutional laver with dy feature maps. lated digitsorut it is kuite useful for repognizing strings of aph unit is ponnepted to a 5x5 neighrorhood on all pharapters taken from the full printarle NSTII set.The of S4s feature maps.Hereorepause the size of S4 is also rationale is that pharapters that are similaroand therefore 5x5o the size of r5s feature maps is cce this amounts ponfusapleosuph as upper pase oolowerpase Ooand zerooor to a full ponneption Fetween S4 and r5.T5 is lapeled lowerpase lodigit Coskuare Frarketsoand upperpase Io will as a ronvolutional laerinstead of a fullroneted ar have similar output rodes.This is partipularlMuseful if the repause if LeNet-5 input were made Figger with ever thing system is pompined with a linguistip post-propessor that else kept ponstant.the feature map dimension would re larger than &G This propess of dmnamipallM inpreasing the pan porrept suph ponfusions.Bepause the podes for ponfus- aple plasses are similarothe output of the porresponding size of a ponvolutional network is despriged in the seption Seption 9 II.Laver T5 has 4s cy trainarle ponneptions. HBIs for an ampiguous pharapter will Fe similaroand the Laver 1 Popontains s4 units ithe reason for this numper post-propessor will re arle to pipk the appropriate interpre- tation.1 igure 3 gives the output podes for the full NSrII pomes from the design of the output laveroexplained Fe- set. low)and is fullM ponnepted to T5.It has cyo4 trainarle N nother reason for using suph distriputed podesorather parameters. than the more pommon "cof N"pode ralso palled plape Ns in plassipal neural networksounits in lavers up to 1 P pompute a dot produpt Fetween their input veptor and their podeo or grand-mother pell pode)for the outputs is that weight veptoroto whiph a rias is added.This weighted sumo non distriputed podes tend to rehave FadlM when the num- Fer of plasses is larger than a few dozens.The reason is denoted a;for unit io is then passed through a sigmoid skuashing funption to produpe the state of unit iodenoted that output units in a non-distriputed pode must Fe off most of the time.This is kuite din pult to aphieve with FMTie sigmoid units.Yet another reason is that the plassifiers are Ti8 fi) ) often used to not onlmrepognize pharaptersoFut also to re- The skuashing funption is a spaled hmperrolip tangente 6ept non-pharapters.HBIs with distriruted podes are more appropriate for that purpose Fepause unlike sigmoidsothem f)8 Atanha) D are aptivated within a well pirpumsprig ed region of their in-

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✦ ✘ ➾ ✑ ❜ ❝ ✙ ✓ ✔ ✺ ❀ ➾✽✘❭➾➎➾✢➾ ✑❭➾✖❜ ➾✖❝ ➾ ✙ ✘ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ➾ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✑ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ❜ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ❝ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✙ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✮✦❱❨❞✇P☛❧ ❊ ￾✂✁☎✄✝✆✞✄✠✟☎✡☞☛✍✌✏✎✞✑✒✎✍✓☎✑✔✄✝✁✖✕✠✗✙✘✛✚✜✆✍✑✔✄✙✆✣✢✤✗✖✁✙✕✝☛✍✥✦✗✧✌✏✁☎★✧✑✒✎✪✩☎✫✬✁☎✥✦✗✭✄✝✟✮✌✏✯✠✑✒✎☎✗✖✓ ✯✝✰✞✕✠✆☎✗✭☛☎✎☎✑✒✕✦✘✛✑✒✎✞✁✞★✱✁☎✥✤✕✠✑✔✄✝☛☎✡☞✁☎✥✲✢✤✗✙✁✖✕✠☛☎✥✦✗✧✌✏✁☎★✳✟☎✢✭✴✶✵✦✷ ￾❶ì❛î➉✡✙ø➀é✙ï❞ù➔✡☛✘➲ï❞å✑￾❿ç✢☞❀❜✶ëíï❞å✠è✥ÿ✙ä✧ï➻î➓å➄æ❀ð ✎ç❩✘❖é✙ì➄è➓￾❶ì❛é✙é✙ï★￾↔è➞ï➊ú❩✝ ï➊ä❘✘❑þ ✑❺ëíï❞å✠è✥ÿ✙ä✧ï✺î➓å➄æ è✥ì ï✖ú➈ï✖ä❺✘⑨☞❀❜❺ëíï❞å✠è✧ÿ☎ä✧ï✺î➓å➄æ☎✑✯ã✛ç✙ï✫ä✥ï✖å♦✝ ê✧ì➈é❤ø✓ê❙è④ó✇ì➈ëíì➈û✓ù☛ð①✜❯ø➑ä❿ê④è★➌❫å❺é✙ì➈é✦✝r￾❶ì❛î➻æ✙û➑ï➊è✧ï➐￾❶ì➈é☎é✙ï✒￾❶è✧ø➀ì➈é❑ê❘￾❿ç✙ï✖î❭ï ô➈ï✖ï➊æ☎ê❀è✧ç☎ï➵é➇ÿ✙î➑✡✎ï✖ä❦ì➄ë✥￾➊ì➈é✙é✙ï★￾↔è✥ø➑ì❛é☎ê❀ó❨ø➑è✧ç☎ø➑é❭ä✧ï❞å➈ê✧ì➈é☎å✑✡✙û➀ï❹✡✎ì❛ÿ✙é☎ù✙ê✖ð ö✫ì❛ä✧ï➵ø➑î➻æ◆ì➈ä✧è✥å➈é❛è✥û✙✘➎➌➄ø➑è❯ëíì❛ä❘￾➊ï✖ê❦å④✡✙ä✧ï❞å➄ô✌ì➈ë◆ê❺✘✐î➻î➻ï❶è✥ä❺✘✌ø➀é❙è✧ç✙ï✛é✙ï➊è❇✝ ó➵ì➈ä✥ô◆ð❜✧➉ø✙➘✟ï➊ä✥ï➊é✐è❣ëíï✖å➄è✧ÿ✙ä✥ï❨î➓å➄æ☎ê❣å➈ä✧ï➵ëíì➈ä✴￾❶ï❞ù➞è✥ì✌ï❯↔➇è✧ä❿å✑￾❶è➏ù✂ø➟➘✟ï➊ä✥ï➊é✐è ➪íç☎ì➈æ◆ï❶ëíÿ✙û➀û✙✘①￾➊ì➈î➻æ✙û➀ï➊î➻ï➊é✐è✥å➈ä❺✘❼➶♣ëíï❞å✠è✧ÿ☎ä✧ï❞ê✌✡✎ï★￾➊å➈ÿ☎ê✤ï➽è✧ç✙ï✒✘➛✂❛ï❶è❙ù✂ø➑ë●✝ ëíï➊ä✥ï➊é✐è♣ê✧ï❶è❿ê❨ì➄ë❣ø➀é✙æ✙ÿ✂è❿ê➊ð❨ã✛ç✙ï✒ä✥å➄è✧ø➀ì➈é☎å➈û➑ï✌✡◆ï➊ç✙ø➀é☎ù✿è✥ç✙ï➑￾➊ì➈é✙é✙ï★￾↔è✥ø➑ì❛é ê❘￾❿ç✙ï➊î➻ï✌ø➀é✢è❿å❄✡✙û➀ï✌➏✛ø➀ê❨è✧ç✙ï➤ëíì❛û➑û➀ì✠ó❨ø➑é✗✂☎ð❫ã✛ç✙ït➞✎ä✥ê✤è➉ê✤ø✙↔✆☞❀❜❭ëíï❞å✠è✥ÿ✙ä✧ï î➓å➄æ☎ê❙è❿å➄ô➈ï✢ø➑é☎æ✙ÿ✂è✥ê❙ëíä✥ì➈î ï✖ú➈ï➊ä❘✘ ￾❶ì➈é✐è✥ø✙✂❛ÿ✙ì➈ÿ☎ê❭ê✧ÿ❼✡☎ê✧ï❶è✥ê❭ì➈ë❨è✧ç✙ä✥ï➊ï ëíï✖å➄è✧ÿ✙ä✥ï✿î➻å➈æ☎ê❙ø➑é➘þ ✑✂ð ã✛ç✙ï✿é✙ï❯↔➇è➓ê✤ø✙↔➷è✥å➈ô➈ï✿ø➑é✙æ☎ÿ✂è❙ëíä✥ì➈î ï✖ú➈ï✖ä❺✘ ￾❶ì❛é✐è✧ø✠✂➈ÿ✙ì❛ÿ☎ê❨ê✤ÿ✗✡☎ê✤ï➊è➔ì➄ë❯ëíì➈ÿ✙ä❞ð❫ã✛ç✙ï✌é☎ï❯↔➇è❨è✧ç✙ä✥ï➊ï➤è❿å➄ô❛ï✌ø➑é✙æ☎ÿ✂è➔ëíä✥ì➈î ê✧ì➈î➻ï✿ù✙ø➀ê❘￾❶ì❛é❛è✥ø➑é➇ÿ✙ì❛ÿ☎ê➻ê✤ÿ✗✡☎ê✤ï➊è✥ê➻ì➈ë➉ëíì➈ÿ☎ä✖ð⑨✜❇ø➀é☎å➈û➑û✠✘ è✧ç☎ï✿û✓å➈ê✤è➓ì➈é✙ï è✥å➈ô➈ï❞ê➔ø➀é✙æ✙ÿ✂è➳ëíä✧ì❛î▼å➈û➑û❣þ ✑❭ëíï❞å✠è✥ÿ✙ä✧ï❙î➓å➄æ☎ê✖ð❇✗❀å✪✘❛ï➊ä✞☞❀❜✶ç☎å➈ê➉➾✑➌ ✙✦➾✒✓ è✧ä❿å➄ø➀é☎å✑✡✙û➑ï➤æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê✛å➈é☎ù➔➾✆✙✦➾➎➌ ✓✗✘✻✘➉￾➊ì➈é✙é☎ï✒￾↔è✥ø➑ì❛é☎ê✖ð ✗❀å✪✘➈ï✖ä❫þ✦❝➞ø✓ê❫å➞ê✧ÿ❼✡✦✝➠ê✥å➄î➻æ✙û➀ø➑é❼✂✌û✓å✪✘➈ï✖ä❣ó❨ø➩è✥ç↕➾✽✓➳ëíï❞å✠è✥ÿ✙ä✧ï➉î➻å➈æ☎ê❣ì➈ë ê✧ø✙➽✖ï ✙✪↔✤✙✂ð ✭✇å✑￾❿ç✫ÿ✙é☎ø➩è➳ø➑é➲ï❞å✑￾❿ç✺ëíï❞å✠è✧ÿ☎ä✧ï➞î➓å➈æ✫ø➀ê✞￾❶ì❛é✙é✙ï✒￾❶è✧ï❞ù✿è✧ì✢å ✑✪↔✤✑➤é☎ï➊ø✠✂➈ç☛✡✎ì❛ä✧ç☎ì✐ì✂ù➻ø➀é➓è✧ç☎ï④￾❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✂ø➀é❼✂➤ëíï❞å✠è✧ÿ☎ä✧ï♣î➻å➈æ➓ø➑é✆☞❀❜❼➌ ø➀é❺å✶ê✧ø➀î❭ø➀û✓å➄ä➉ó✛å✪✘✺å➈ê✌☞④➾❙å➄é✎ù❖þ ✑✂ð✒✗❀å✪✘❛ï➊ä➤þ✦❝✶ç☎å❛ê✟❜✵✑❭è✧ä❿å➄ø➀é☎å✑✡✙û➑ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê✛å➄é✎ù✳✑❼➌ ✘✗✘✻✘➚￾❶ì➈é☎é✙ï✒￾❶è✧ø➀ì➈é☎ê✖ð ✗❀å✪✘➈ï✖ä✎☞❅✙➤ø✓ê✇å➉￾❶ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✙û✓å✪✘➈ï➊ä❫ó❨ø➩è✥ç➔➾ ✑✻✘➳ëíï✖å➄è✧ÿ✙ä✥ï➉î➓å➄æ✎ê➊ð ✭❫å➎￾❿ç❑ÿ☎é✙ø➩è➽ø✓ê➵￾❶ì❛é✙é✙ï★￾↔è✧ï❞ù❑è✧ì å ✙✪↔✤✙❖é✙ï➊ø✠✂➈ç☛✡◆ì➈ä✥ç✙ì➇ì➇ù❤ì➈é❍å➄û➀û✌➾✽✓ ì➄ë❨þ✦❝❇❁ ê➳ëíï✖å➄è✧ÿ✙ä✥ï➻î➓å➄æ☎ê✖ð❭õ➉ï➊ä✥ï✑➌✏✡◆ï✒￾✖å➄ÿ☎ê✧ï❙è✧ç✙ï➽ê✧ø✙➽✖ï❙ì➈ë❨þ✔❝✢ø✓ê✌å➄û✓ê✤ì ✙✪↔✤✙❼➌❫è✥ç✙ï❖ê✧ø✙➽✖ï✫ì➄ë➉☞❅✙ ❁ ê❭ëíï✖å✠è✥ÿ✙ä✥ï✫î➻å➈æ☎ê➓ø✓ê✆➾❯↔❢➾✗✰✺è✧ç✙ø✓ê✶å➈î❭ì❛ÿ✙é✐è✥ê è✧ì➘å ëíÿ☎û➑û✌￾❶ì❛é✙é✙ï★￾↔è✧ø➀ì➈é❝✡✎ï➊è④ó✇ï✖ï➊é✾þ✦❝ å➄é☎ù✟☞❅✙✂ð ☞❅✙ ø✓ê✶û✓å❄✡◆ï➊û➀ï✖ù å➈ê➵å➚￾❶ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✙û✓å✪✘➈ï➊ä★➌✐ø➑é☎ê✤è✧ï❞å➈ù➓ì➈ë❇å✌ëíÿ☎û➑û✠✘❩✝❖￾➊ì➈é✙é✙ï★￾↔è✥ï✖ù➽û✓å✪✘➈ï✖ä✒➌ ✡◆ï✒￾➊å➈ÿ☎ê✧ï➵ø➑ë ✗✝ï✖ñ➉ï❶è❺✝✝✙♣ø➀é✙æ✙ÿ✂è➏ó➵ï➊ä✥ï✛î➻å❛ù✂ï✔✡✙ø✠✂✑✂➈ï✖ä❦ó❨ø➩è✥ç❭ï✖ú➈ï✖ä❺✘✐è✧ç☎ø➑é❼✂ ï➊û✓ê✧ï✫ô➈ï➊æ✙è ￾❶ì➈é✎ê④è❿å➄é✐è✒➌✇è✧ç✙ï✺ëíï❞å✠è✥ÿ✙ä✧ï✫î➓å➈æ➘ù✙ø➑î➻ï➊é✎ê✤ø➀ì➈é➘ó➵ì➈ÿ✙û✓ù⑨✡✎ï û✓å➄ä❘✂➈ï➊ä❇è✧ç✎å➄é ➾✄↔❢➾➈ð❇ã✛ç☎ø➀ê❯æ✙ä✥ì✦￾❶ï✖ê✥ê❇ì➄ë✎ù✦✘➇é☎å➈î➻ø✠￾✖å➄û➀û✙✘✌ø➀é✗￾❶ä✥ï✖å❛ê✤ø➀é❼✂➉è✧ç✙ï ê✧ø✙➽✖ï✒ì➄ë➵å ￾❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✝é☎ï❶è④ó➵ì➈ä✥ô✢ø✓ê➤ù✂ï✖ê❘￾❶ä✥ø✙✡◆ï✖ù➲ø➑é❖è✧ç✙ï➻ê✤ï★￾↔è✥ø➑ì❛é þ➇ï★￾↔è✧ø➀ì➈é✫✣④➏❇➏↔ð ✗❀å✪✘❛ï➊ä✬☞❅✙❙ç☎å➈ê❅❝✺❼➌✠➾ ✑✻✘✌è✧ä❿å➄ø➀é☎å✑✡✙û➑ï✌￾➊ì➈é✙é✙ï★￾↔è✥ø➑ì❛é☎ê➊ð ✗❀å✪✘➈ï✖ä✛✜✓❼➌✥￾➊ì➈é✐è✥å➈ø➑é✎ê ✺✬❝➓ÿ✙é☎ø➩è❿ê➉➪➺è✥ç✙ï✒ä✥ï✖å❛ê✤ì❛é✶ëíì❛ä❨è✧ç☎ø➀ê➉é✐ÿ☎î➉✡◆ï➊ä ￾❶ì❛î➻ï✖ê➤ëíä✥ì➈î❂è✧ç✙ï✢ù✂ï✖ê✧ø✙✂❛é➷ì➄ë✛è✧ç✙ï✶ì➈ÿ✂è✥æ✙ÿ✂è✒û✓å✪✘➈ï✖ä✒➌❇ï❯↔✂æ✙û✓å➄ø➀é✙ï✖ù➒✡◆ï❯✝ û➀ì✠ó✬➶➔å➄é✎ù✺ø➀ê➉ëíÿ✙û➀û✙✘✆￾❶ì❛é✙é✙ï★￾↔è✧ï❞ù✺è✧ì✆☞❅✙✂ð✛➏⑥è➳ç☎å➈ê➑➾✽✘✗➌✙➾✒✓✬❝❭è✧ä❿å➄ø➀é☎å✑✡✙û➑ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê✖ð ✕➉ê➔ø➀é↕￾❶û✓å➈ê✥ê✧ø✠￾✖å➄û☛é✙ï✖ÿ✙ä✥å➈û✝é✙ï➊è④ó✇ì❛ä✧ô✂ê✒➌✂ÿ✙é✙ø➑è✥ê➔ø➀é✫û✓å✪✘➈ï➊ä❿ê➵ÿ✙æ✺è✧ì➣✜✓ ￾❶ì❛î➻æ✙ÿ✂è✧ï➵å➔ù✂ì➈è❀æ✙ä✥ì➇ù✙ÿ✗￾↔è❀✡◆ï❶è④ó➵ï➊ï✖é✌è✧ç☎ï➊ø➀ä❀ø➀é✙æ✙ÿ✂è❯ú➈ï✒￾❶è✧ì❛ä❀å➄é☎ù➳è✥ç✙ï➊ø➀ä ó➵ï➊ø✠✂➈ç✐è❇ú❛ï✒￾↔è✥ì➈ä★➌➊è✥ì➉ó❨ç✙ø✁￾❿ç✒å✬✡✙ø➀å❛ê✝ø✓ê❇å❛ù✙ù✂ï❞ù☛ð❯ã✛ç☎ø➀ê❇ó✇ï✖ø✙✂❛ç✐è✧ï✖ù✌ê✧ÿ✙î✫➌ ù✂ï✖é✙ì➄è✥ï✖ù✹✸❲ ëíì❛ä✶ÿ✙é☎ø➩è✻✺❘➌❨ø✓ê➓è✥ç✙ï➊é✲æ☎å➈ê✥ê✤ï❞ù è✧ç✙ä✥ì➈ÿ✗✂➈ç✻å ê✧ø✙✂❛î➻ì➈ø✓ù ê❘➍❛ÿ✎å➈ê✧ç✙ø➑é✗✂❙ëíÿ☎é✗￾↔è✥ø➑ì❛é✺è✧ì➽æ✙ä✥ì✂ù✂ÿ✗￾➊ï➤è✧ç✙ï➞ê✤è✥å➄è✧ï✌ì➄ë❣ÿ✙é☎ø➩è✧✺✴➌☎ù✙ï➊é✙ì➈è✧ï✖ù ✡☛✘✽✼✴❲ ✰ ✼❲ ✺✿✾❨➪❀✸✦❲✻➶ ➪❩✙✑➶ ã✛ç✙ï➞ê❘➍✐ÿ☎å➈ê✧ç✙ø➀é❼✂✒ëíÿ☎é✗￾↔è✥ø➑ì❛é✿ø✓ê➔å➻ê❘￾➊å➄û➀ï✖ù✢ç☛✘➇æ✎ï✖ä❺✡◆ì➈û➀ø✠￾♣è❿å➄é❼✂❛ï➊é✐è✽✰ ✾❨➪❁✸✦➶ ✺✿❂✺è✥å➈é✙ç❀➪❀❃✜✸☛➶ ➪✡✓➎➶ ó❨ç✙ï✖ä✧ï❄❂✲ø➀ê➏è✧ç✙ï➉å➈î➻æ✙û➑ø➑è✧ÿ✎ù✂ï➉ì➈ë◆è✥ç✙ï❨ëíÿ✙é✥￾↔è✧ø➀ì➈é✢å➄é☎ù❅❃➷ù✂ï➊è✧ï➊ä✥î➻ø➑é☎ï✖ê ø➑è✥ê➏ê✧û➑ì❛æ✎ï❨å➄è❯è✥ç✙ï❨ì➈ä✥ø✠✂➈ø➀é✝ð❇ã✛ç✙ï➵ëíÿ✙é✥￾↔è✧ø➀ì➈é✽✾➽ø➀ê❦ì✂ù✙ù❢➌❛ó❨ø➩è✥ç➻ç✙ì➈ä✥ø✙➽✖ì➈é✦✝ è✥å➈û❯å❛ê❇✘➇î➻æ✂è✥ì➄è✧ï❞ê➉å➄è ✣✭❂✹å➈é☎ù ✆ ❂✒ð➳ã✛ç✙ï➚￾❶ì❛é☎ê④è❿å➄é✐è✳❂➶ø➀ê✞￾❿ç✙ì✐ê✤ï✖é è✧ì➐✡◆ï➙➾ ❭✕✔✦➾ ✙❆❀☎ð♣ã✛ç✙ï❭ä❿å✠è✧ø➀ì➈é✎å➄û➀ï✌ëíì➈ä♣è✧ç✙ø✓êt￾❿ç✙ì➈ø✁￾❶ï❙ì➄ë✇å✢ê❺➍✐ÿ☎å❛ê✤ç☎ø➑é❼✂ ëíÿ✙é✗￾❶è✧ø➀ì➈é✺ø✓ê✩✂➈ø➀ú➈ï✖é✶ø➀é✫✕➉æ✙æ◆ï➊é☎ù✂ø✙↔✆✕➞ð ✜❇ø➀é☎å➈û➑û✠✘✑➌❞è✧ç☎ï❫ì➈ÿ✙è✧æ✙ÿ✂è❯û➀å✪✘❛ï➊ä☛ø✓ê❳￾➊ì➈î➻æ◆ì❛ê✧ï✖ù➳ì➄ë❇✭❫ÿ✗￾❶û➀ø➀ù✙ï✖å➄é➓✍➉å➈ù✂ø✓å➄û ✚✛å➈ê✧ø➀ê➵✜☎ÿ✙é✗￾❶è✧ø➀ì➈é➘ÿ✙é✙ø➑è✥ê➛➪✍✛✚✔✜❷➶✹➌✇ì➈é✙ï✿ëíì➈ä✶ï✖å✑￾❿ç ￾❶û✓å➈ê✥ê✒➌❫ó❨ø➑è✧ç ✺✬❝ ø➀é✙æ✙ÿ✂è❿ê❫ï❞å✑￾❿ç✝ð❦ã✛ç✙ï➉ì❛ÿ✂è✧æ☎ÿ✂è✥ê❫ì➄ë✝ï✖å➎￾❿ç➣✍✛✚✔✜❖ÿ☎é✙ø➩è❇❆✫❲☛ø✓ê✎￾❶ì➈î➻æ✙ÿ✙è✧ï✖ù å➈ê➵ëíì❛û➑û➀ì✠ó➔ê✽✰ ❆✫❲ ✺❉❈✠❊ ➪❋✼❊ ✆❍●❲❊ ➶ ￾ ❭ ➪ ✔❄➶ ➏➠é➷ì➄è✥ç✙ï➊ä➞ó➵ì➈ä❿ù✙ê✒➌✝ï✖å➎￾❿ç➷ì➈ÿ✂è✥æ✙ÿ✂è➑✍✛✚✔✜✲ÿ✙é✙ø➑è➑￾❶ì❛î❭æ☎ÿ✂è✧ï❞ê➤è✧ç✙ï✜✭➏ÿ✦✝ ￾❶û➀ø✓ù✂ï✖å➈é❭ù✂ø✓ê✤è✥å➄é✥￾❶ï✎✡✎ï➊è④ó✇ï✖ï➊é❭ø➩è❿ê❦ø➑é✙æ☎ÿ✂è➏ú➈ï★￾↔è✧ì❛ä❣å➄é☎ù✒ø➩è❿ê❣æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä ú➈ï★￾↔è✥ì➈ä❞ð➵ã✛ç✙ï➞ëíÿ✙ä✤è✥ç✙ï➊ä➤åró✛å✪✘➽ø➀ê➔è✧ç✙ï✒ø➑é✙æ☎ÿ✂è➉ëíä✥ì➈îPè✧ç✙ï❙æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä ú➈ï★￾↔è✥ì➈ä★➌✛è✧ç✙ï❺û➀å➈ä❺✂❛ï➊ä➽ø✓ê➽è✥ç✙ï➛✍✬✚✎✜✭ì❛ÿ✂è✧æ✙ÿ✙è✖ð➶ã✛ç☎ï❖ì➈ÿ✙è✧æ✙ÿ✂è✺ì➄ë➞å æ☎å➈ä✤è✥ø✠￾➊ÿ✙û➀å➈ä✩✍✛✚✔✜✢￾➊å➈é ✡◆ï➳ø➀é✐è✧ï✖ä✧æ✙ä✥ï❶è✥ï✖ù✿å➈ê✛å❙æ✎ï✖é☎å➄û➑è❅✘➓è✧ï✖ä✧î✴î❭ï❞å♦✝ ê✧ÿ✙ä✧ø➀é❼✂➳è✧ç☎ï✔➞✙è❹✡✎ï➊è④ó✇ï✖ï➊é❭è✧ç☎ï❨ø➑é✙æ☎ÿ✂è❫æ☎å➄è✤è✧ï✖ä✧é➻å➄é✎ù❭å➳î➻ì✂ù✂ï➊û☎ì➄ë◆è✧ç✙ï ￾❶û✓å➈ê✥ê➉å❛ê✧ê✧ì✦￾❶ø✓å✠è✥ï✖ù✿ó❨ø➩è✥ç✫è✧ç☎ï➑✍✛✚✔✜➵ð✥➏➠é❖æ✙ä✧ì➎✡☎å❄✡☎ø➑û➀ø➀ê✤è✧ø✁￾➤è✥ï➊ä✥î➻ê✒➌✎è✧ç✙ï ✍✛✚✔✜ ì❛ÿ✂è✧æ✙ÿ✙è✩￾✖å➄é➙✡✎ï➤ø➀é❛è✥ï➊ä✥æ✙ä✥ï❶è✧ï❞ù✶å❛ê❫è✥ç✙ï➤ÿ✙é✙é✙ì❛ä✧î➓å➈û➑ø✠➽➊ï❞ù➓é✙ï✄✂✐å♦✝ è✧ø➀ú➈ï❭û➀ì✑✂❄✝⑥û➀ø➑ô❛ï➊û➀ø➑ç✙ì➇ì✂ù➲ì➈ë✇å✺â➤å➄ÿ☎ê✥ê✤ø✓å➄é ù✂ø✓ê④è✥ä✧ø✠✡✙ÿ✂è✥ø➑ì❛é❺ø➀é❖è✥ç✙ï➓ê✤æ✎å✑￾❶ï ì➄ë❹￾➊ì➈é✦➞✗✂❛ÿ✙ä❿å✠è✧ø➀ì➈é✎ê❨ì➄ë❣û✓å✪✘➈ï➊ä④✜✝✓☎ð➉â➳ø➑ú❛ï➊é❖å➈é✫ø➑é☎æ✙ÿ✂è♣æ✎å✠è✤è✥ï➊ä✥é✏➌✎è✧ç✙ï û➀ì❛ê✥ê➉ëíÿ☎é✗￾↔è✥ø➑ì❛é➷ê✧ç✙ì❛ÿ✙û➀ù➔✡✎ï➽ù✙ï✖ê✧ø✙✂❛é✙ï✖ù❺ê✤ì✫å➈ê♣è✥ì➐✂❛ï❶è➤è✧ç☎ï➝￾➊ì➈é✦➞✥✂➈ÿ✦✝ ä❿å✠è✧ø➀ì➈é ì➈ë✬✜✓➲å➈ê➑￾❶û➀ì❛ê✧ï✶å➈ê❙æ✎ì✐ê✧ê✧ø✠✡✙û➑ï➽è✧ì➲è✥ç✙ï✶æ✎å➄ä❿å➄î➻ï❶è✥ï➊ä➞ú❛ï✒￾↔è✥ì➈ä ì➄ë✟è✥ç✙ï④✍✛✚✔✜❖è✧ç☎å➄è✎￾➊ì➈ä✥ä✧ï❞ê✤æ◆ì➈é☎ù☎ê❯è✧ì➞è✥ç✙ï➔æ☎å➄è✤è✥ï➊ä✥é ❁ ê✇ù✂ï✖ê✧ø➀ä✧ï❞ù➵￾❶û✓å➈ê✥ê➊ð ã✛ç✙ï➤æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä➵ú❛ï✒￾❶è✧ì➈ä❿ê✇ì➄ë❀è✥ç✙ï✖ê✧ï➤ÿ✙é✙ø➑è✥ê❨ó➵ï➊ä✥ït￾❿ç✙ì❛ê✧ï➊é➙✡❩✘✶ç☎å➄é☎ù å➄é✎ù✒ô❛ï➊æ✂è❨➞❼↔✂ï✖ù➐➪⑨å➄è➏û➑ï❞å➈ê✤è❣ø➑é✙ø➑è✧ø✓å➄û➀û✠✘✦➶❶ð❯ã✛ç☎ï✛￾❶ì➈î➻æ◆ì➈é✙ï✖é✐è✥ê❯ì➄ë✎è✧ç✙ì✐ê✤ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê✇ú➈ï★￾↔è✧ì❛ä✥ê✇ó➵ï➊ä✥ï➳ê✧ï❶è➵è✧ì➚✝✴➾♣ì❛ä ✣➑➾❛ð ✎ç☎ø➑û➀ï➳è✧ç✙ï✒✘ ￾❶ì❛ÿ✙û➀ù ç☎årú❛ï✩✡✎ï✖ï➊é➙￾❿ç☎ì❛ê✧ï➊é➓å✠è➵ä✥å➈é☎ù✂ì➈î✭ó❨ø➑è✧ç➽ï★➍❛ÿ✎å➄û☎æ☎ä✧ì➎✡☎å❄✡✙ø➀û➀ø➩è✥ø➑ï❞ê❣ëíì➈ä➜✝❘➾ å➄é✎ù ✣➑➾✑➌❦ì➈ä➞ï✖ú➈ï✖é✢￾❿ç✙ì❛ê✧ï➊é➷è✥ì✫ëíì➈ä✥î å➈é ï➊ä✥ä✧ì❛ä➉￾❶ì❛ä✧ä✥ï✒￾❶è✧ø➀é❼✂✆￾❶ì✂ù✂ï å➈ê➳ê✧ÿ❼✂✑✂❛ï✖ê✤è✧ï✖ù✫✡☛✘✏✞❝✸✔✠❖➌✎è✧ç✙ï✒✘✫ó✇ï✖ä✧ï➞ø➀é☎ê✤è✧ï❞å➈ù➲ù✂ï❞ê✤ø✠✂➈é☎ï✖ù✺è✥ì✶ä✥ï➊æ✙ä✥ï❯✝ ê✧ï➊é✐è➤å✿ê✤è❅✘✐û➀ø✠➽➊ï✖ù❖ø➑î➓å✑✂➈ï❭ì➄ë➏è✥ç✙ï➚￾➊ì➈ä✥ä✧ï❞ê✤æ◆ì➈é✎ù✂ø➑é✗✂➙￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä④￾➊û➀å❛ê✧ê ù✂ä❿åró❨é❺ì❛é å✦✔★↔❢➾ ✑➙✡✙ø➑è✧î➓å➄æ ➪⑨ç✙ï➊é✥￾❶ï➓è✥ç✙ï➓é➇ÿ✙î➉✡◆ï➊ä ✺✫❝❩➶❶ð✿þ➇ÿ✗￾❿ç å ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✥å➄è✧ø➀ì➈é❙ø➀ê❦é✙ì➈è➏æ☎å➄ä✧è✧ø✁￾❶ÿ☎û➀å➈ä✧û✠✘✌ÿ☎ê✧ï❶ëíÿ✙û✂ëíì❛ä❣ä✧ï★￾❶ì➎✂➈é✙ø✠➽➊ø➀é❼✂➳ø➀ê✧ì❄✝ û✓å✠è✧ï❞ù✢ù✙ø✙✂❛ø➩è❿ê✄➌✦✡✙ÿ✙è✛ø➑è➔ø✓ê✎➍✐ÿ✙ø➑è✧ï➤ÿ☎ê✧ï❶ëíÿ☎û☛ëíì➈ä✛ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ø➀é❼✂❭ê④è✥ä✧ø➀é❼✂❛ê➵ì➈ë ￾❿ç☎å➈ä✥å➎￾↔è✧ï✖ä✥ê➞è✥å➈ô➈ï➊é❤ëíä✧ì❛î è✧ç☎ï✢ëíÿ☎û➑û❨æ✙ä✥ø➀é❛è❿å❄✡✙û➀ï✫✕♣þ✗☞➃➏❺➏❭ê✧ï❶è❞ð❑ã✛ç✙ï ä❿å✠è✧ø➀ì➈é✎å➄û➀ï✌ø➀ê✛è✥ç☎å✠èt￾❿ç☎å➄ä❿å✑￾❶è✧ï✖ä✥ê✇è✧ç✎å✠è➳å➈ä✧ï➞ê✧ø➀î❭ø➀û✓å➄ä★➌☎å➄é☎ù✿è✥ç✙ï➊ä✥ï❶ëíì❛ä✧ï ￾❶ì❛é✂ëíÿ☎ê✥å❄✡✙û➀ï✑➌➇ê✧ÿ✗￾❿ç✶å➈ê✇ÿ✙æ✙æ◆ï➊ä✴￾➊å❛ê✤ï✌ý➑➌❛û➀ì✠ó➵ï➊ä✴￾➊å➈ê✧ï♣ý➑➌➇å➄é☎ù➵➽✖ï➊ä✥ì✗➌✐ì➈ä û➀ì✠ó✇ï✖ä❘￾✖å➈ê✧ï➳û✶➌☛ù✂ø✠✂➈ø➑è➑➾✑➌◆ê❺➍✐ÿ☎å➈ä✧ï➉✡✙ä✥å➎￾❿ô➈ï➊è✥ê✒➌☎å➄é☎ù➲ÿ✙æ✙æ◆ï➊ä✴￾➊å➈ê✧ï➓➏✹➌◆ó❨ø➑û➀û ç☎årú❛ï➔ê✤ø➀î➻ø➑û✓å➄ä❣ì❛ÿ✂è✧æ✙ÿ✙è✎￾❶ì✂ù✂ï❞ê➊ð❦ã✛ç✙ø✓ê➏ø✓ê❣æ☎å➈ä✤è✥ø✠￾➊ÿ✙û➀å➈ä✧û✠✘➞ÿ☎ê✧ï❶ëíÿ☎û☎ø➑ë☛è✧ç✙ï ê❺✘➇ê✤è✧ï✖î❅ø✓ê➣￾❶ì❛î➉✡✙ø➀é✙ï✖ù❑ó❨ø➑è✧ç✻å û➑ø➀é❼✂❛ÿ✙ø➀ê✤è✧ø✁￾✺æ◆ì❛ê✤è❇✝⑥æ✙ä✧ì✦￾➊ï✖ê✥ê✤ì❛ä✒è✥ç☎å✠è ￾➊å➈é➐￾➊ì➈ä✥ä✧ï★￾↔è✛ê✤ÿ✗￾❿ç✫￾❶ì❛é✂ëíÿ☎ê✧ø➑ì❛é☎ê✖ð❨✚➵ï★￾➊å➄ÿ✎ê✤ï♣è✧ç✙ï✌￾➊ì➇ù✙ï✖ê➵ëíì➈ä✛￾➊ì➈é✂ëíÿ✎ê❅✝ å❄✡☎û➑ï✆￾❶û✓å➈ê✥ê✧ï✖ê➻å➄ä✥ï✿ê✧ø➀î❭ø➀û✓å➄ä★➌➏è✧ç☎ï✫ì➈ÿ✂è✥æ✙ÿ✂è➽ì➄ë♣è✧ç✙ï↕￾➊ì➈ä✥ä✧ï❞ê✤æ◆ì➈é✎ù✂ø➑é✗✂ ✍✛✚✔✜❦ê❨ëíì❛ä♣å➄é❺å➄î➉✡☎ø✙✂❛ÿ✙ì➈ÿ☎ê✬￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä❨ó❨ø➀û➀û❳✡◆ï❙ê✧ø➑î➻ø➀û➀å➈ä✒➌◆å➄é☎ù✫è✧ç✙ï æ◆ì❛ê✤è❇✝⑥æ✙ä✧ì✦￾➊ï✖ê✥ê✤ì❛ä✝ó❨ø➑û➀û☛✡✎ï➵å❄✡✙û➀ï➏è✥ì♣æ☎ø✠￾❿ô➳è✧ç☎ï➵å➈æ✙æ✙ä✥ì➈æ✙ä✥ø➀å➄è✧ï❣ø➀é✐è✧ï➊ä✥æ✙ä✥ï❯✝ è✥å➄è✧ø➀ì➈é✝ð✩✜❯ø✠✂➈ÿ✙ä✥ï✜❜➵✂➈ø➀ú➈ï❞ê✛è✧ç☎ï➞ì➈ÿ✂è✥æ✙ÿ✂è④￾➊ì✂ù✂ï✖ê➔ëíì➈ä❨è✥ç✙ï➞ëíÿ✙û➀û❳✕➳þ✗☞➃➏❇➏ ê✧ï❶è✖ð ✕➔é✙ì➈è✧ç✙ï✖ä♣ä✥ï✖å❛ê✤ì❛é✺ëíì➈ä➳ÿ☎ê✤ø➀é❼✂✿ê✤ÿ✗￾❿ç❺ù✂ø➀ê✤è✧ä✥ø✠✡✙ÿ✂è✧ï❞ù↕￾❶ì✂ù✂ï❞ê✄➌✟ä❿å✠è✥ç✙ï➊ä è✧ç✎å➄é❑è✧ç✙ï✺î➻ì➈ä✥ï➙￾➊ì➈î➻î➻ì➈é✻☛✒➾✢ì➈ë➳ñ✟✌➛￾➊ì➇ù✙ï ➪üå➄û✓ê✤ì➒￾➊å➈û➑û➀ï✖ù❤æ✙û➀å➎￾❶ï ￾❶ì✂ù✂ï➎➌✇ì❛ä➵✂➈ä❿å➄é☎ù✦✝♠î➻ì➄è✥ç✙ï➊ä➵￾➊ï➊û➀û✛￾❶ì✂ù✂ï★➶❙ëíì❛ä➻è✧ç✙ï✫ì❛ÿ✂è✧æ☎ÿ✂è✥ê➓ø✓ê➻è✥ç☎å✠è é✙ì❛é➻ù✙ø➀ê✤è✧ä✥ø✙✡☎ÿ✂è✧ï❞ù➵￾❶ì✂ù✂ï✖ê❯è✥ï➊é☎ù❭è✥ì✌✡✎ï✖ç☎årú➈ï✔✡☎å❛ù✂û✠✘✒ó❨ç✙ï✖é➻è✧ç✙ï❨é➇ÿ✙î➚✝ ✡◆ï➊ä❭ì➄ë✛￾❶û✓å➈ê✥ê✤ï❞ê✌ø✓ê✒û✓å➄ä❘✂➈ï✖ä✌è✧ç☎å➈é å✺ëíï➊ó ù✙ì✑➽➊ï✖é☎ê✖ð❖ã✛ç✙ï➽ä✥ï✖å❛ê✤ì❛é ø✓ê è✧ç✎å✠è✶ì❛ÿ✂è✧æ✙ÿ✙è✶ÿ✙é✙ø➑è✥ê✶ø➑é✲å➷é☎ì➈é✦✝➠ù✂ø✓ê④è✥ä✧ø✠✡✙ÿ✂è✥ï✖ù ￾❶ì✂ù✂ï➲î❙ÿ☎ê✤è ✡✎ï❖ì❄➘ î➻ì❛ê✤è➓ì➄ë➉è✥ç✙ï✿è✧ø➀î➻ï➈ð➘ã✛ç✙ø➀ê➻ø✓ê➵➍✐ÿ✙ø➑è✧ï✫ù✂ø✯➵￾➊ÿ✙û➑è➓è✧ì å✑￾❿ç✙ø➀ï➊ú❛ï✢ó❨ø➑è✧ç ê✧ø✙✂❛î❭ì❛ø➀ù❭ÿ✙é☎ø➩è❿ê➊ð❏■➏ï❶è✛å➄é✙ì➈è✧ç✙ï✖ä➏ä✥ï✖å➈ê✧ì➈é❭ø✓ê❦è✧ç☎å➄è❫è✥ç✙ï✬￾❶û✓å➈ê✥ê✧ø➟➞☎ï✖ä✥ê❣å➈ä✧ï ì➄ë➺è✥ï➊é✿ÿ☎ê✧ï✖ù➓è✥ì❙é☎ì➄è✛ì➈é✙û✠✘➓ä✧ï★￾❶ì➎✂➈é✙ø✠➽➊ï✞￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä❿ê✒➌✑✡✙ÿ✂è➔å➈û➀ê✧ì➞è✥ì❙ä✥ï❯✝ ✓④ï✒￾❶è❫é✙ì❛é✦✝❖￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê➊ð❳✍✛✚✔✜❯ê➏ó❨ø➩è✥ç➽ù✂ø➀ê✤è✧ä✥ø✠✡✙ÿ✂è✧ï❞ù➵￾❶ì✂ù✂ï❞ê❣å➄ä✥ï❨î❭ì❛ä✧ï å➄æ☎æ✙ä✧ì❛æ✙ä✥ø➀å➄è✧ï➵ëíì➈ä❣è✥ç☎å✠è➵æ✙ÿ✙ä✥æ✎ì✐ê✤ï✩✡◆ï✒￾✖å➄ÿ☎ê✧ï❨ÿ✙é✙û➀ø➀ô➈ï➉ê✧ø✙✂❛î➻ì➈ø✓ù✙ê✄➌➈è✧ç✙ï✒✘ å➄ä✥ï✛å✑￾↔è✥ø➑ú✠å➄è✧ï✖ù❭ó❨ø➑è✧ç✙ø➀é➓å➳ó✇ï✖û➑û❼￾➊ø➑ä✴￾❶ÿ✙î➓ê❘￾❶ä✥ø✙✡◆ï✖ù✒ä✧ï✒✂➈ø➀ì➈é❭ì➄ë✎è✧ç✙ï✖ø➑ä❣ø➀é✦✝

CXOC.Ob CRE IEEE,AOVEy BEXFV 口四日H露图8口I】器田B日日 penalties,it means that in addition to pushins down the p1234§日☑日9日且☒目33 penalto of the 2orre2t 2lass like the MS-2riterion,this 2riterion also pulls up the penalties of the ingorreat 2lassese gH日日日EF回日四T口可万可 p日Bs0g▣893ⅢIJ▣ C(UDP(Z W)(e-iv>e-vi(z,W))) 9目回gEE日5旦H巴网五可 (9) The nesative of the se2ond term pla0s a"zompetitive"role. p9FsE口9▣893图Π图日0 It is nezessarilo smaller than (or equal to)the first term, a密gee9女nh therefore this loss funation is positive.The 2onstant j is positive,and prevents the penalties of 2lasses that are als reado vero larse from ueins pushed further up.The posS terior prouauilito of this ruuuish 2lass lauel would ue the put spaze that non gOpizal patterns are more likel0 to fall ratio of e and e-Vie-wi(z,w).This diszriminas tive zriterion prevents the previous10 mentioned "zollapsS outside of. The parameter veators of the v Ai s plao the role of tarnet in effe2t"when the v A parameters are learned uezause ve2tors for laOer 1 P.It is worth pointins out that the 2omS it keeps the vA1 zenters apart from eazh other.In Se2S ponents of those veators are V1 or S1,whizh is well within tion iI,we present a seneralization of this zriterion for the ranse of the sismoid of P,and therefore prevents those sOstems that learn to 2lassifo multiple ouje2ts in the input sismoids from setting saturated.In fa2t,V1 and Sl are the (e.s.,2harazters in words or in dozuments). points of maximum 2ur vature of the sigmoids.This forzes Oomputins the fradient of the loss funztion with respe2t the 1 Punits to operate in their maximall0 non ginear ranse. to all the weishts in all the la0ers of the 2onvolutional Saturation of the sismoids must ue avoided uezause it is network is done with uazkpropanation.The standard als known to lead to slow 2onversenze and illSonditionins of forithm must ue slishtlo modified to take azzount of the the loss funation. weisht sharins.Xn eas0 wao to implement it is to first 2omS pute the partial derivatives of the loss funztion with respe2t P.Loss Fbnctgon to eah connectgon,as if the network were a 2onventional multigaOer network without weisht sharins.Then the parS The simplest output loss funztion that 2an ue used with tial derivatives of all the 2onneztions that share a same the auove network is the Maximum vikelihood-stimation parameter are added to form the derivative with respe2t to 2riterion(Mv-),whigh in our 2ase is equivalent to the Mins that parameter. imum Mean Squared -rror (MS-).The zriterion for a set Suzh a larse arzhitezture 2an ue trained vero en 2ient10, of trainins samples is simplO uut doins so requires the use of a few te2hniques that are P deszrived in the appendix.Se2tion X of the appendix W)8 Z W (s) deszriues details suzh as the partizular sismoid used,and the weisht initialization.Seation A and o deszriue the minimization prozedure used,whizh is a stozhasti2 version where yDp is the output of the Sth v unit,i.e.the of a diasonal approximation to the vevenuers Marquardt one that zorresponds to the zorrlass of input pattern prozedure. Z. While this 2ost funztion is appropriate for most zases, itaks three important properties.irst,if we allow the III.RESULTS AND COMPARISON WITH OTHER parameters of theA to adapt,W)has a trivial,uut METHODS totallo unaz2eptaule,solution.In this solution,all theAl While re2osnizing individual disits is onlo one of mano parameter veators are equal,and the state of Pis 2onstant proulems involved in desisning a praztizal rezosnition sOsS and equal to that parameter ve2tor.In this 2ase the nets tem,it is an exzellent uenzhmark for 2omparins shape work happilo isnores the input,and all the outputs rezornition methods.Thoush man0 existins method 2omS are equal to zero.This 2ollapsing phenomenon does not uine a handrafted feature extraztor and a trainaule 2lass o22ur if the weishts are not allowed to adapt.The sifier,this studo 2onzentrates on adaptive methods that sezond proulem is that there is no 2ompetition uetween operate direztlo on size ormalized imases. the 2lasses.Suzh a 2ompetition 2an ue outained u0 usS inn a more diszriminative training zriterion,duuued the B.Database:the Mo gfie NI-T set MXj (maximum a posteriori)2riterion,similar to Maxis The datavase used ko ain and test the sOstems des mum Mutual Information zriterion sometimes used to train szriued in this paper was 2onstruzted from the NISTss SpeS HMMs.ts],.t9],5y].It 2orresponds to maximizin the 2ial Datauase Band Spe2ial Datauase 1 2ontaining uinaro posterior prouauilito of the orre2t 2lass (or minimizS imases of handwritten disits.NIST orisinallo desisnated ins the losarithm of the prouauilito of thepprre2t 2lass),SDSas their trainins set and SDsl as their test set.Hows iven that the input imane 2an 2ome from one of the 2lasses ever,SDSis muzh2leaner and easier to rezonize than SDS or from a uazkaround "ruuuish"2lass lauel.In terms of 1.The reason for this 2an ue found on the faat that SDS

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✣ ! " # $ % & ’ ( ) * + , − . / 0 1 2 3 4 5 6 7 8 9 : ; ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ‘ a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ ✁❼✿▲❍✪❦❼➆★❦ ❊●❈✪✿ ✵✻✿➂✱✴❴✇❃♦✱✴✸✶✱✴❋✩✰r✵✶✰❅✸✶✺➜✷✴⑥❀✵✶✯✪✰✛✷✹✳★✵✻❃✪✳★✵✩◆❳❞✁❼✺❹⑥✠✷✹✸➜✸✶✰❅❆❅✷✹❍✹❈✪✿▲❵❅✿▲❈✪❍t✵✻✯✪✰ ⑥✠✳✪❴▲❴➎❱❷❁✪❻✇❊ ❊✇✺✶✰r✵❇❦ æ✙ÿ✂è➤ê✤æ✎å✑￾❶ï➞è✧ç☎å➄è♣é✙ì❛é✦✝üè❅✘➇æ✙ø✁￾➊å➈û✝æ☎å➄è✤è✧ï✖ä✧é✎ê➉å➄ä✥ï➞î➻ì➈ä✥ï➞û➀ø➑ô❛ï➊û✠✘✶è✥ì➓ë⑨å➄û➀û ì➈ÿ✙è✥ê✧ø➀ù✂ï✌ì➈ë④ð ã✛ç✙ï➏æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä☛ú➈ï✒￾❶è✧ì❛ä✥ê☛ì➈ë❛è✥ç✙ï➃✍✬✚✎✜❦ê☛æ☎û➀å✪✘➉è✧ç✙ï✇ä✧ì❛û➑ï❦ì➄ë➇è❿å➄ä❘✂➈ï❶è ú➈ï★￾↔è✥ì➈ä❿ê❦ëíì➈ä➏û✓å✪✘➈ï✖ä❹✜✓✙ð❨➏⑥è❫ø✓ê➏ó➵ì➈ä✧è✧ç➻æ✎ì❛ø➑é✐è✥ø➑é❼✂✒ì➈ÿ✂è➏è✥ç☎å✠è❫è✧ç✙ï✞￾❶ì➈î➚✝ æ◆ì➈é✙ï✖é❛è❿ê✛ì➄ë❇è✧ç✙ì✐ê✤ï➤ú➈ï★￾↔è✥ì➈ä❿ê➵å➈ä✧ï✛✣➑➾➳ì➈ä✔✝✴➾✑➌➇ó❨ç✙ø✁￾❿ç✢ø✓ê❨ó➵ï➊û➀û☛ó❨ø➩è✥ç✙ø➑é è✧ç☎ï✇ä❿å➄é❼✂❛ï➏ì➄ë☎è✧ç✙ï➵ê✤ø✠✂➈î➻ì➈ø✓ù✌ì➈ë✥✜✓❼➌✠å➄é☎ù➞è✧ç✙ï✖ä✧ï➊ëíì➈ä✥ï➏æ✙ä✥ï➊ú❛ï➊é✐è✥ê❀è✧ç✙ì✐ê✤ï ê✧ø✙✂❛î❭ì❛ø➀ù☎ê❯ëíä✧ì❛î➧✂➈ï➊è✤è✥ø➑é❼✂➞ê✥å✠è✥ÿ✙ä❿å✠è✧ï❞ù☛ð❀➏➠é➻ë⑨å➎￾↔è★➌✸✣➑➾❨å➄é☎ù➑✝✴➾➔å➄ä✥ï➵è✧ç✙ï æ◆ì➈ø➀é❛è❿ê❨ì➄ë❣î➓å♦↔✂ø➀î✒ÿ✙î ￾❶ÿ✙ä✥ú✠å✠è✥ÿ✙ä✧ï➤ì➈ë❇è✥ç✙ï✒ê✧ø✙✂❛î❭ì❛ø➀ù☎ê➊ð❫ã✛ç☎ø➀ê✛ëíì❛ä❘￾➊ï✖ê è✧ç☎ï➃✜✓❨ÿ✙é✙ø➑è✥ê✝è✧ì➔ì❛æ✎ï✖ä✥å➄è✧ï➏ø➀é➤è✧ç☎ï➊ø➀ä❀î➓å♦↔✂ø➀î➻å➈û➑û✠✘➳é✙ì➈é✦✝⑥û➀ø➑é✙ï❞å➄ä✝ä❿å➄é✗✂➈ï➈ð þ✂å➄è✧ÿ✙ä❿å✠è✥ø➑ì❛é ì➄ë✛è✧ç✙ï✿ê✤ø✠✂➈î➻ì❛ø➀ù✙ê➞î✒ÿ☎ê✤è➉✡◆ï✢årú➈ì❛ø➀ù✙ï✖ù➒✡✎ï★￾➊å➈ÿ☎ê✤ï✶ø➩è✒ø✓ê ô➇é✙ì✠ó❨é➲è✥ì✢û➀ï✖å❛ù✫è✥ì✿ê✧û➑ì✠ó❫￾❶ì➈é➇ú❛ï➊ä❘✂➈ï➊é✥￾❶ï➞å➄é✎ù❖ø➀û➑û✙✝r￾❶ì➈é✎ù✂ø➩è✥ø➑ì❛é✙ø➀é❼✂✶ì➈ë è✧ç☎ï✌û➑ì✐ê✧ê➵ëíÿ✙é✥￾↔è✧ø➀ì➈é❀ð ❃✛❊✢➯✪➩✴➩✁￾✡✦➨❄✄➺✮✣ ➯❄➨ ã✛ç✙ï➳ê✧ø➀î❭æ☎û➑ï❞ê④è✛ì➈ÿ✂è✥æ✙ÿ✂è❨û➀ì❛ê✥ê➏ëíÿ✙é✥￾↔è✧ø➀ì➈é✶è✧ç☎å➄è✔￾✖å➄é➙✡✎ï➳ÿ☎ê✧ï✖ù➽ó❨ø➑è✧ç è✧ç☎ï➤å✑✡✎ì✠ú❛ï➉é✙ï➊è④ó✇ì❛ä✧ô➻ø✓ê➵è✧ç✙ï✌ö➲å❄↔✂ø➑î❙ÿ✙î ✗✝ø➀ô➈ï✖û➑ø➀ç✙ì➇ì✂ù✙✭❫ê✤è✧ø➀î➓å✠è✥ø➑ì❛é ￾❶ä✥ø➑è✧ï➊ä✥ø➀ì➈é➣➪⑨ö✗★✭➃➶❯➌ró❨ç✙ø✠￾❿ç✒ø➑é❙ì➈ÿ✙ä❜￾➊å➈ê✧ï❫ø✓ê❇ï✒➍✐ÿ✙ø➀úrå➈û➑ï✖é✐è❀è✧ì➉è✧ç✙ï✛ö✫ø➀é✦✝ ø➀î✒ÿ✙î ö✫ï✖å➈é❖þ✦➍✐ÿ☎å➄ä✥ï✖ù ✭❫ä✧ä✥ì➈ä➓➪üö❖þ✭➃➶❶ð✎ã✛ç✙ï➑￾❶ä✥ø➩è✥ï➊ä✥ø➑ì❛é✢ëíì❛ä♣å➓ê✧ï❶è ì➄ë❯è✧ä❿å➄ø➀é✙ø➀é❼✂➓ê✧å➈î➻æ✙û➑ï❞ê✛ø➀ê❨ê✧ø➀î❭æ☎û✙✘☞✰ ❉➣➪❩❂➶ ✺ ➾ ✌ ❫ ❈ ✸✄✂ ✜ ❆✆☎✞✝❼➪❩✾✸ ❁❃❂➶ ➪✡✺➎➶ ó❨ç✙ï✖ä✧ï❅❆☎✝ ø✓ê➤è✧ç✙ï✶ì➈ÿ✂è✥æ✙ÿ✂è➞ì➄ë➵è✧ç☎ï✘■ ✸ ✝♠è✧ç①✍✛✚✔✜✾ÿ✙é✙ø➑è✒➌❯øüð ï➈ð✶è✧ç✙ï ì➈é☎ï➞è✧ç✎å✠è✌￾❶ì➈ä✥ä✥ï✖ê✧æ✎ì❛é☎ù✙ê✛è✥ì✶è✥ç✙ï➑￾➊ì➈ä✥ä✧ï★￾↔è④￾➊û➀å❛ê✧ê♣ì➄ë➏ø➀é✙æ✙ÿ✂è➤æ☎å➄è✤è✧ï✖ä✧é ✾❅✸➇ð ✎ç✙ø➀û➀ï➉è✥ç✙ø➀ê✔￾➊ì❛ê✤è✇ëíÿ✙é✥￾↔è✧ø➀ì➈é✢ø➀ê✛å➈æ✙æ✙ä✥ì➈æ✙ä✥ø➀å➄è✧ï❨ëíì❛ä✛î❭ì✐ê④è✔￾➊å❛ê✤ï❞ê✄➌ ø➑è➤û✓å✑￾❿ô✂ê♣è✧ç☎ä✧ï✖ï❙ø➀î➻æ✎ì❛ä✤è❿å➄é✐è➳æ☎ä✧ì❛æ✎ï✖ä✤è✥ø➑ï❞ê➊ð➉✜❯ø➀ä✥ê✤è✒➌❀ø➩ë➵ó✇ï➻å➄û➀û➑ì✠óòè✧ç✙ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê❨ì➈ë❣è✧ç✙ï➚✍✛✚✔✜❑è✧ì✢å❛ù✙å➄æ✙è✒➌ ❉➣➪✮❂➶➉ç☎å➈ê♣å➓è✥ä✧ø➀ú➇ø➀å➈û✻➌✈✡✙ÿ✂è è✧ì➈è✥å➈û➑û✠✘♣ÿ☎é☎å✑￾✒￾❶ï➊æ✙è✥å❄✡☎û➑ï➎➌rê✧ì➈û➀ÿ✂è✥ø➑ì❛é✝ð❀➏➠é✌è✥ç✙ø✓ê❇ê✧ì➈û➀ÿ✂è✧ø➀ì➈é❀➌rå➈û➑û➈è✧ç✙ï✎✍✛✚✔✜ æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä❇ú❛ï✒￾↔è✥ì➈ä❿ê❀å➈ä✧ï➵ï✒➍✐ÿ☎å➈û✻➌➄å➈é☎ù➤è✧ç☎ï✛ê✤è✥å➄è✧ï➵ì➄ë✥✜✓♣ø➀ê❨￾➊ì➈é☎ê✤è✥å➈é❛è å➄é✎ù➲ï✒➍✐ÿ☎å➈û❯è✧ì✢è✧ç☎å➄è➤æ✎å➄ä❿å➄î➻ï❶è✥ï➊ä➳ú➈ï✒￾❶è✧ì❛ä✖ð✌➏➠é❖è✧ç✙ø✓ê✌￾➊å❛ê✤ï✒è✧ç✙ï➻é✙ï➊è❇✝ ó➵ì➈ä✥ô✫ç☎å➄æ✙æ☎ø➑û✠✘➲ø✠✂➈é✙ì❛ä✧ï❞ê➉è✥ç✙ï➓ø➀é✙æ✙ÿ✂è★➌❇å➄é✎ù å➈û➑û❦è✥ç✙ï➵✍✛✚✔✜✻ì❛ÿ✂è✧æ✙ÿ✙è✥ê å➄ä✥ï✶ï★➍✐ÿ☎å➄û✇è✧ì➒➽➊ï✖ä✧ì✎ð ã✛ç✙ø✓ê➝￾❶ì❛û➑û✓å➄æ☎ê✧ø➀é❼✂➲æ✙ç✙ï✖é✙ì➈î➻ï✖é✙ì➈é ù✂ì➇ï✖ê❙é✙ì➄è ì✦￾✄￾➊ÿ✙ä✒ø➑ë✛è✧ç✙ï➐✍✛✚✔✜✲ó➵ï➊ø✠✂➈ç✐è✥ê❙å➄ä✥ï➓é✙ì➄è❭å➄û➀û➑ì✠ó➵ï✖ù❺è✧ì❺å➈ù✙å➈æ✂è✖ð✫ã✛ç✙ï ê✧ï✒￾❶ì❛é☎ù➘æ✙ä✥ì✑✡✙û➀ï➊î ø✓ê➽è✥ç☎å✠è➽è✥ç✙ï➊ä✥ï❖ø✓ê➽é☎ì➹￾❶ì❛î❭æ◆ï❶è✥ø➩è✥ø➑ì❛é ✡◆ï❶è④ó➵ï➊ï➊é è✧ç☎ï➔￾❶û✓å➈ê✥ê✤ï❞ê➊ð✲þ✂ÿ✗￾❿ç❍å①￾❶ì➈î➻æ◆ï❶è✥ø➩è✥ø➑ì❛é ￾✖å➄é ✡◆ï➲ì✑✡✂è❿å➄ø➀é✙ï✖ù⑨✡☛✘❤ÿ✎ê❅✝ ø➀é❼✂❤å➷î➻ì➈ä✥ï✫ù✂ø➀ê❘￾❶ä✥ø➀î❭ø➀é☎å➄è✧ø➀ú➈ï✺è✧ä❿å➄ø➀é✙ø➀é❼✂✢￾❶ä✥ø➑è✧ï➊ä✥ø➀ì➈é✏➌✛ù✂ÿ❼✡❼✡◆ï✖ù❍è✧ç✙ï ö↕✕④✓❡➪íî➓å❄↔➇ø➀î✒ÿ☎î❅å✫æ✎ì✐ê④è✥ï➊ä✥ø➑ì❛ä✧ø●➶✌￾➊ä✧ø➑è✧ï✖ä✧ø➀ì➈é✏➌➏ê✧ø➀î❭ø➀û✓å➄ä➞è✥ì❺ö➲å❄↔✂ø➟✝ î✒ÿ☎îòö✫ÿ✂è✧ÿ✎å➄û☛➏➠é✂ëíì➈ä✥î➓å✠è✥ø➑ì❛é➉￾❶ä✥ø➩è✥ï➊ä✥ø➑ì❛é✒ê✧ì➈î➻ï❶è✥ø➑î➻ï✖ê❇ÿ☎ê✧ï✖ù➞è✧ì♣è✧ä❿å➄ø➀é õ➉ö❖ö➲ê◆✞❝✺✠❖➌❅✞❝✗❀✬✠✶➌ ✞✙❆✘✬✠♠ð➒➏⑥è➚￾➊ì➈ä✥ä✧ï❞ê✤æ◆ì➈é✎ù✙ê✌è✥ì❖î➓å♦↔✂ø➀î❭ø✠➽➊ø➀é❼✂❖è✧ç✙ï æ◆ì❛ê✤è✧ï➊ä✥ø➀ì➈ä✌æ✙ä✥ì✑✡☎å✑✡✙ø➀û➑ø➑è❅✘➲ì➄ë➵è✧ç☎ï➣￾❶ì➈ä✥ä✥ï✒￾↔è✌￾❶û✓å➈ê✥ê✹■ ✸ ➪⑨ì➈ä✌î➻ø➑é☎ø➑î➻ø✠➽❯✝ ø➀é❼✂✫è✧ç☎ï➽û➑ì➎✂❛å➈ä✧ø➑è✧ç✙î❋ì➄ë✛è✧ç✙ï✶æ✙ä✧ì➎✡☎å❄✡☎ø➑û➀ø➩è❅✘❖ì➄ë✛è✧ç✙ï➙￾❶ì➈ä✥ä✥ï✒￾↔è➓￾➊û➀å❛ê✧ê✴➶✹➌ ✂➈ø➀ú➈ï✖é✌è✧ç✎å✠è❯è✧ç✙ï➵ø➑é☎æ✙ÿ✂è❣ø➑î➓å✑✂➈ï➃￾✖å➄é➉￾➊ì➈î➻ï✇ëíä✧ì❛î❲ì❛é✙ï➵ì➄ë✂è✧ç☎ï✔￾➊û➀å❛ê✧ê✧ï✖ê ì➈ä❙ëíä✥ì➈î å↕✡☎å➎￾❿ô☛✂➈ä✥ì➈ÿ✙é☎ù ☛✧ä✧ÿ❼✡✗✡✙ø➀ê✧ç✤✌↕￾➊û➀å❛ê✧ê❙û✓å❄✡◆ï➊û♠ð➛➏➠é è✥ï➊ä✥î➓ê✒ì➈ë æ◆ï➊é☎å➈û➩è✥ø➑ï❞ê✄➌❯ø➩è❙î❭ï❞å➄é☎ê✌è✧ç☎å➄è✒ø➑é å❛ù✙ù✂ø➑è✧ø➀ì➈é è✧ì✫æ☎ÿ☎ê✤ç☎ø➑é❼✂❺ù✂ì✠ó❨é è✧ç✙ï æ◆ï➊é☎å➈û➩è❅✘➘ì➄ë✌è✥ç✙ï➛￾➊ì➈ä✥ä✧ï★￾↔è➙￾➊û➀å❛ê✧ê✶û➑ø➀ô➈ï➲è✧ç✙ï➷ö❖þ✭ ￾❶ä✥ø➩è✥ï➊ä✥ø➑ì❛é✏➌✛è✧ç✙ø✓ê ￾❶ä✥ø➑è✧ï➊ä✥ø➀ì➈é➻å➄û✓ê✤ì➤æ✙ÿ✙û➀û✓ê❣ÿ✙æ➻è✧ç☎ï➔æ✎ï✖é☎å➄û➑è✧ø➀ï✖ê➏ì➄ë◆è✧ç✙ï❨ø➀é✗￾❶ì❛ä✧ä✥ï✒￾❶è❹￾❶û✓å➈ê✥ê✤ï❞ê✱✰ ❉ ➪✮❂➶ ✺ ➾ ✌ ❫ ❈ ✸✟✂ ✜ ➪❁❆✆☎✞✝❼➪❩✾✸ ❁❃❂➶★✣ û➑ì➎✂✥➪✡✠ ✴ ❊ ✣ ❈ ❲ ✠ ✴☞☛✍✌✏✎✒✑✝✔✓✌✖✕ ➶❺➶ ➪✡❀➎➶ ã✛ç✙ï➵é✙ï✄✂✐å✠è✥ø➑ú❛ï❫ì➄ë✂è✥ç✙ï❨ê✤ï★￾❶ì❛é☎ù➤è✧ï✖ä✧î æ☎û➀å✪✘✂ê❯å❵☛❺￾➊ì➈î➻æ◆ï❶è✧ø➑è✧ø➀ú➈ï ✌➔ä✥ì➈û➀ï➈ð ➏⑥è➞ø✓ê➤é✙ï✒￾➊ï✖ê✥ê✧å➈ä✧ø➀û✙✘➲ê✧î➓å➄û➀û➑ï✖ä➤è✧ç☎å➈é❭➪íì➈ä✌ï★➍✐ÿ☎å➄û❦è✥ì❩➶♣è✧ç☎ï➝➞☎ä✥ê✤è➤è✧ï✖ä✧î✫➌ è✧ç☎ï➊ä✥ï❶ëíì➈ä✥ï➻è✧ç☎ø➀ê✒û➑ì✐ê✧ê➤ëíÿ☎é✗￾↔è✥ø➑ì❛é ø➀ê✒æ✎ì✐ê✤ø➑è✧ø➀ú➈ï❛ð✿ã✛ç✙ï➙￾❶ì➈é✎ê④è❿å➄é✐è✁✗➲ø✓ê æ◆ì❛ê✧ø➩è✥ø➑ú❛ï✑➌☛å➄é☎ù❖æ✙ä✥ï➊ú➈ï✖é✐è✥ê➉è✧ç✙ï❭æ✎ï✖é☎å➄û➑è✧ø➀ï✖ê➳ì➈ë➜￾❶û✓å➈ê✥ê✤ï❞ê➔è✥ç☎å✠è✌å➄ä✥ï❙å➈û➟✝ ä✥ï✖å➈ù❼✘✫ú➈ï➊ä❘✘✺û➀å➈ä❺✂❛ï➞ëíä✥ì➈î ✡◆ï➊ø➀é❼✂✿æ✙ÿ☎ê✧ç✙ï✖ù❖ëíÿ✙ä✧è✧ç✙ï✖ä✌ÿ✙æ✝ð❭ã✛ç✙ï➻æ✎ì✐ê❅✝ è✧ï✖ä✧ø➀ì➈ä➤æ✙ä✥ì✑✡✎å❄✡✙ø➀û➑ø➑è❅✘✫ì➄ë✇è✧ç✙ø✓ê➤ä✥ÿ❼✡❼✡✙ø✓ê✤ç ￾❶û✓å➈ê✥ê➳û➀å✑✡✎ï✖û❣ó✇ì❛ÿ✙û✓ù↕✡◆ï➻è✧ç✙ï ä❿å✠è✧ø➀ì ì➈ë✘✠ ✴ ❊ å➄é☎ù✙✠ ✴ ❊ ✣✛✚❲ ✠ ✴☞☛✌ ✎✒✑✝✔✓✌✖✕ ð✭ã✛ç✙ø✓ê✢ù✂ø✓ê❘￾❶ä✥ø➑î➻ø➀é☎å♦✝ è✧ø➀ú➈ï➣￾➊ä✧ø➑è✧ï✖ä✧ø➀ì➈é❺æ✙ä✥ï➊ú➈ï✖é✐è✥ê➳è✧ç☎ï➓æ✙ä✧ï✖ú➇ø➑ì❛ÿ☎ê✤û✠✘✫î➻ï➊é✐è✧ø➀ì➈é☎ï✖ù ☛❺￾➊ì➈û➀û➀å➈æ☎ê❅✝ ø➀é❼✂➽ï❯➘✟ï✒￾❶è❃✌➓ó❨ç☎ï➊é✫è✥ç✙ï➑✍✛✚✔✜❑æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê➔å➈ä✧ï✌û➀ï✖å➈ä✧é✙ï❞ù➐✡✎ï★￾➊å➄ÿ✎ê✤ï ø➑è❙ô➈ï✖ï➊æ☎ê✌è✧ç✙ï ✍✬✚✎✜✟￾➊ï➊é✐è✧ï✖ä✥ê➞å➈æ☎å➄ä✧è✌ëíä✥ì➈î ï✖å✑￾❿ç➷ì➈è✧ç✙ï✖ä✖ð✫➏➠é❤þ➇ï★￾✹✝ è✧ø➀ì➈é ✣④➏✹➌❨ó✇ï✫æ✙ä✥ï✖ê✧ï➊é✐è✶å ✂❛ï➊é✙ï✖ä✥å➈û➑ø✠➽✖å➄è✧ø➀ì➈é❤ì➈ë➤è✧ç✙ø✓ê➣￾❶ä✥ø➑è✧ï➊ä✥ø➀ì➈é ëíì➈ä ê❺✘➇ê✤è✧ï✖î➓ê❫è✥ç☎å✠è✛û➀ï✖å➄ä✥é➓è✥ì➚￾➊û➀å❛ê✧ê✧ø➩ë➭✘➻î❙ÿ✙û➩è✥ø➑æ☎û➑ï➤ì✑✡✓④ï✒￾↔è❿ê➵ø➑é✶è✧ç✙ï➳ø➑é☎æ✙ÿ✂è ➪íï❛ð ✂✎ð✙➌✥￾❿ç☎å➄ä❿å✑￾❶è✧ï➊ä❿ê✇ø➑é✺ó➵ì➈ä❿ù✙ê➵ì➈ä❨ø➀é✿ù✂ì✦￾➊ÿ✙î➻ï➊é✐è✥ê✴➶↔ð ☞✇ì➈î➻æ✙ÿ✂è✥ø➑é❼✂➤è✥ç✙ï✩✂➈ä❿å➈ù✙ø➑ï✖é❛è❣ì➈ë☎è✧ç☎ï➔û➑ì✐ê✧ê❯ëíÿ☎é✗￾↔è✥ø➑ì❛é❭ó❨ø➑è✧ç➻ä✥ï✖ê✧æ✎ï★￾↔è è✧ì❍å➈û➑û✌è✥ç✙ï ó➵ï➊ø✠✂➈ç✐è✥ê✿ø➀é å➈û➑û➤è✥ç✙ï û✓å✪✘➈ï➊ä❿ê✿ì➄ë❙è✧ç✙ï✢￾➊ì➈é➇ú➈ì❛û➑ÿ✂è✥ø➑ì❛é☎å➄û é✙ï➊è④ó✇ì❛ä✧ô✿ø➀ê➤ù✂ì❛é✙ï❭ó❨ø➩è✥ç➔✡☎å➎￾❿ô➎✝⑥æ✙ä✥ì➈æ☎å✑✂❛å➄è✧ø➀ì➈é✝ð➉ã✛ç☎ï❭ê✤è✥å➈é☎ù✙å➄ä❿ù➲å➈û➟✝ ✂➈ì❛ä✧ø➑è✧ç☎î❋î✒ÿ✎ê④è➓✡◆ï✶ê✧û➑ø✠✂➈ç✐è✧û✠✘❖î➻ì✂ù✂ø✙➞☎ï✖ù è✥ì✺è✥å➈ô➈ï➽å✑￾✒￾❶ì❛ÿ✙é✐è➤ì➈ë✛è✧ç✙ï ó➵ï➊ø✠✂➈ç✐è❇ê✧ç☎å➄ä✥ø➑é✗✂☎ð❀✕➉é✌ï✖å➈ê❺✘➳ó➵å✪✘➉è✧ì➔ø➀î➻æ✙û➑ï✖î➻ï➊é✐è❇ø➑è❇ø✓ê☛è✥ì✛➞☎ä❿ê④è❜￾❶ì➈î➚✝ æ✙ÿ✂è✥ï❫è✥ç✙ï❫æ✎å➄ä✧è✧ø✓å➄û➇ù✂ï➊ä✥ø➑ú✠å➄è✧ø➀ú➈ï✖ê❀ì➄ë✂è✥ç✙ï❫û➀ì❛ê✥ê✝ëíÿ☎é✗￾↔è✥ø➑ì❛é✒ó❨ø➑è✧ç➞ä✥ï✖ê✧æ✎ï★￾↔è è✧ì✺ï❞å✑￾❿ç ❄✴➯❄➨✗➨✈➳❄✄➺✮✣ ➯❄➨✗➌❀å❛ê➳ø➩ë➵è✧ç✙ï➻é✙ï➊è④ó✇ì❛ä✧ô✫ó➵ï➊ä✥ï❙å✫￾➊ì➈é➇ú➈ï✖é❛è✥ø➑ì❛é☎å➄û î✒ÿ☎û➩è✥ø➟✝⑥û➀å✪✘❛ï➊ä❣é✙ï➊è④ó✇ì❛ä✧ô✒ó❨ø➩è✥ç✙ì➈ÿ✂è✇ó✇ï✖ø✙✂❛ç❛è❫ê✧ç☎å➈ä✧ø➀é❼✂☎ð❯ã✛ç✙ï➊é❭è✥ç✙ï➔æ☎å➈ä❇✝ è✧ø✓å➄û➳ù✂ï➊ä✥ø➀úrå➄è✧ø➀ú➈ï❞ê➓ì➄ë➞å➄û➀û➔è✥ç✙ï➛￾➊ì➈é✙é☎ï✒￾↔è✥ø➑ì❛é☎ê➻è✥ç☎å✠è✺ê✤ç☎å➈ä✧ï➲å❤ê✧å➈î❭ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä➏å➄ä✥ï➔å➈ù✙ù✂ï❞ù❙è✧ì✌ëíì❛ä✧î✹è✧ç✙ï➉ù✙ï➊ä✥ø➑ú✠å✠è✥ø➑ú❛ï❨ó❨ø➩è✥ç➓ä✧ï❞ê✤æ◆ï✒￾❶è❣è✧ì è✧ç✎å✠è➔æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✖ð þ➇ÿ✗￾❿ç✫å❭û➀å➈ä❺✂❛ï➤å➈ä❘￾❿ç☎ø➩è✥ï✒￾↔è✥ÿ✙ä✥ï✞￾✖å➄é✫✡✎ï➤è✥ä✥å➈ø➑é☎ï✖ù✢ú➈ï➊ä❘✘➽ï✱✯➵￾➊ø➑ï✖é✐è✧û✠✘✑➌ ✡✙ÿ✂è➤ù✂ì➈ø➀é❼✂✶ê✧ì✶ä✥ï✒➍✐ÿ✙ø➀ä✥ï✖ê✛è✥ç✙ï✒ÿ☎ê✧ï✒ì➈ë➏å➓ëíï➊ó è✥ï✒￾❿ç✙é☎ø✠➍✐ÿ✙ï❞ê❨è✧ç✎å✠è➳å➈ä✧ï ù✂ï❞ê❺￾➊ä✧ø✠✡✎ï❞ù❍ø➀é❍è✧ç✙ï å➄æ✙æ◆ï➊é✎ù✂ø➟↔☛ð▲þ➇ï✒￾❶è✧ø➀ì➈é ✕❂ì➄ë✌è✥ç✙ï❺å➈æ✙æ✎ï✖é☎ù✂ø✙↔ ù✂ï❞ê❺￾➊ä✧ø✠✡✎ï❞ê➤ù✂ï❶è❿å➄ø➀û➀ê➳ê✤ÿ✥￾❿ç❺å➈ê♣è✥ç✙ï❙æ✎å➄ä✧è✧ø✁￾❶ÿ✙û✓å➄ä➤ê✤ø✠✂➈î➻ì❛ø➀ù✫ÿ✎ê✤ï❞ù❢➌✝å➄é☎ù è✧ç☎ï❺ó➵ï➊ø✠✂➈ç✐è✶ø➀é✙ø➑è✧ø✓å➄û➀ø✙➽❞å✠è✧ø➀ì➈é❀ð▲þ➇ï★￾↔è✥ø➑ì❛é ✚ å➄é☎ù✖☞ ù✂ï✖ê❘￾❶ä✥ø✠✡✎ï❖è✧ç✙ï î➻ø➑é☎ø➑î➻ø✠➽✖å✠è✥ø➑ì❛é➽æ✙ä✥ì✦￾❶ï✖ù✙ÿ✙ä✧ï♣ÿ☎ê✧ï✖ù❢➌✐ó❨ç✙ø✁￾❿ç➽ø➀ê➵å✒ê✤è✧ì✦￾❿ç☎å❛ê④è✥ø✠￾➉ú➈ï✖ä✥ê✧ø➑ì❛é ì➄ë✛å✺ù✂ø➀å✑✂➈ì❛é☎å➄û❣å➄æ☎æ✙ä✧ì✪↔✂ø➀î➓å✠è✧ø➀ì➈é❖è✧ì✿è✧ç☎ï❑✗❀ï➊ú❛ï➊é☛✡✎ï✖ä❺✂✑✝⑥ö❖å➄ä✴➍❛ÿ✎å➄ä❿ù➇è æ✙ä✥ì✦￾❶ï✖ù✙ÿ✙ä✧ï❛ð ➇❯➇❯➇✪➈✖✩➞Ù✣✖ß✝Ü❶Þ✣➻Ý❀Ú❀➊✧✘➳×❇Ø✢✜rÝ❀à❢➋ ✣❞×❇Ú ✛➣➋✓Þ✚★✤✣✒Þ✚★✝Ù✙à ✥Ù✙Þ★❇×❀➊✤✣ ✎ç✙ø➑û➀ï➻ä✧ï★￾❶ì➎✂➈é✙ø✠➽➊ø➀é❼✂➽ø➀é☎ù✂ø➀ú➇ø➀ù✂ÿ✎å➄û❣ù✂ø✠✂➈ø➑è✥ê➳ø➀ê➳ì❛é✙û✙✘✺ì❛é✙ï❙ì➈ë❫î➓å➄é☛✘ æ✙ä✥ì✑✡✙û➀ï➊î➓ê➵ø➀é✐ú❛ì➈û➀ú➈ï❞ù➓ø➑é✺ù✂ï❞ê✤ø✠✂➈é✙ø➀é❼✂➻å✒æ☎ä✥å➎￾↔è✧ø✁￾➊å➈û✟ä✧ï★￾❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✶ê❺✘✂ê❅✝ è✧ï✖î✫➌✒ø➑è❖ø➀ê❺å➄é ï❯↔❼￾➊ï➊û➀û➑ï✖é❛è➛✡✎ï✖é✗￾❿ç✙î➓å➄ä✥ô✻ëíì❛ä➛￾❶ì❛î➻æ☎å➄ä✥ø➑é✗✂➘ê✧ç☎å➈æ✎ï ä✥ï✒￾❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✢î➻ï❶è✧ç☎ì➇ù☎ê➊ð➏ã✛ç✙ì❛ÿ❼✂➈ç✢î➓å➄é☛✘➽ï❯↔✂ø✓ê④è✥ø➑é✗✂❙î➻ï❶è✥ç✙ì✂ù✫￾❶ì➈î➚✝ ✡✙ø➀é✙ï✌å❭ç☎å➄é☎ù✦✝❖￾➊ä✥å➄ë➺è✧ï✖ù✶ëíï✖å➄è✧ÿ✙ä✥ï➳ï❯↔➇è✧ä❿å✑￾❶è✧ì➈ä❨å➈é☎ù✢å❙è✥ä✥å➈ø➑é☎å✑✡✙û➀ï✞￾➊û➀å❛ê❅✝ ê✧ø➟➞☎ï✖ä✒➌➵è✥ç✙ø➀ê✿ê✤è✧ÿ☎ù✦✘ ￾❶ì❛é✗￾❶ï✖é✐è✧ä❿å✠è✧ï❞ê➓ì➈é❍å❛ù✙å➄æ✂è✥ø➑ú❛ï✫î➻ï❶è✧ç☎ì➇ù☎ê➽è✥ç☎å✠è ì➈æ◆ï➊ä❿å✠è✥ï✌ù✂ø➑ä✥ï✒￾❶è✧û✠✘➽ì➈é✺ê✧ø✙➽✖ï❯✝⑥é✙ì➈ä✥î➓å➄û➀ø✙➽✖ï✖ù➽ø➀î➓å❄✂❛ï✖ê✖ð ✚✜✛ ✧➚➢♦➺r➢✫✯✴➢✪➩✄➳✧✦✬➺➭➥❼➳✩★➔➯✖✪✬✣✁➃➳✱✪➚➸✢✪✝❹➦✢➩❯➳✄➺ ã✛ç✙ï ù✙å✠è❿å❄✡☎å❛ê✤ï ÿ☎ê✧ï✖ù✲è✧ì è✧ä❿å➄ø➀é å➄é☎ù✲è✧ï❞ê④è✫è✥ç✙ï ê❇✘✂ê✤è✧ï✖î➻ê✫ù✙ï❯✝ ê❘￾❶ä✥ø✙✡◆ï✖ù❙ø➑é❙è✧ç☎ø➀ê❦æ☎å➈æ✎ï✖ä❣ó➵å❛ê❳￾➊ì➈é☎ê✤è✧ä✥ÿ✗￾❶è✧ï✖ù✒ëíä✥ì➈î è✧ç☎ï➔ñ✛➏✤þ✂ã ❁ ê➏þ➇æ◆ï❯✝ ￾❶ø✓å➄û❜✧♣å➄è✥å✑✡☎å➈ê✧ï✜❜➽å➄é✎ù➲þ➇æ✎ï★￾❶ø✓å➄û❜✧♣å➄è✥å❄✡✎å➈ê✧ï➣➾➉￾❶ì❛é✐è✥å➄ø➀é✙ø➀é❼✂➵✡✙ø➀é☎å➄ä❘✘ ø➀î➻å✑✂➈ï❞ê➤ì➄ë✇ç✎å➄é☎ù✂ó❨ä✥ø➑è✤è✧ï✖é ù✂ø✙✂❛ø➩è❿ê➊ð➽ñ✬➏✤þ✂ã➶ì➈ä✥ø✙✂❛ø➑é☎å➈û➑û✠✘➲ù✂ï❞ê✤ø✠✂➈é☎å➄è✧ï❞ù þ✦✧✬✝✿❜❭å➈ê❫è✧ç☎ï➊ø➀ä❫è✥ä✥å➈ø➑é✙ø➀é❼✂➻ê✤ï➊è➵å➈é☎ù✢þ✦✧✬✝✴➾➳å❛ê➏è✥ç✙ï➊ø➀ä➵è✧ï✖ê✤è❨ê✤ï➊è✖ð❣õ➔ì✠ó✔✝ ï➊ú❛ï➊ä★➌➄þ✦✧✛✝✝❜♣ø➀ê❯î✒ÿ✥￾❿ç➑￾❶û➀ï✖å➈é✙ï➊ä❦å➄é✎ù➞ï✖å❛ê✤ø➀ï➊ä❇è✧ì➳ä✥ï✒￾❶ì➎✂➈é✙ø✠➽➊ï❫è✧ç☎å➈é❭þ✦✧✬✝ ➾➈ð✛ã✛ç✙ï✌ä✥ï✖å❛ê✤ì❛é✶ëíì❛ä❨è✧ç✙ø✓ê✬￾✖å➄é✫✡◆ï✌ëíì➈ÿ✙é✎ù✺ì➈é✺è✧ç☎ï✌ë⑨å✑￾❶è❨è✧ç☎å➄è➳þ✦✧✬✝✿❜

PROC.OF THE IEEE,NOVEMBER 1998 was collected among Census Bureau employees,while SD-1 was collected among high-school students.Drawing sensi- > 68/79b641 ble conclusions from learning experiments requires that the b 757863485 result be independent of the choice of training set and test among the complete set of samples.Therefore it was nec- 2【79 7 a 86 essary to build a new database by mixing NISTss datasets. SD-1 contains 5s,517 digit images written by 5yy dif- 4819018894 ferent writers.In contrast to SD-3,where blocks of data 子61&6 4/5b0 from each writer appeared in sequence,the data in SD-1 is scrambled.N riter identities for SD-1 are available and we 7592658197 used this information to unscramble the writers.N e then split sD-1 in twoe characters written by the first 5y writers 2久22234480 went into our new training set.The remaining 5y writers 038073857 were placed in our test set.Thus we had two sets with nearly 3y,yyy examples each.The new training set was 01446024a completed with enough examples from SD-3,starting at pattern c y,to make a full set of Py,yyy training patterns. 7/281o98b Similarly,the new test set was completed with SD-3 exam- ples starting at pattern c 35,yyy to make a full set with Aig..Size-normalized examples from the MNIST database. Py,yyy test patterns.In the experiments described here,we only used a subset of 1y,yyy test images(5,yyy from SD-1 and 5,yyy from SD-3),but we used the full Py,yyy training three,y.yyyl for the next three,y.yyyy5 for the next 4, samples.The resulting database was called the Modified and y.yyyyl thereafter.Before each iteration,the diagonal NIST,or MNIST,dataset. Hessian approximation was reevaluated on 5yy samples,as The original black and white (bilevel)images were size described in Appendix C and kept fixed during the entire normalized to fit in a I yxy pixel box while preserving iteration.The parameter l was set to y.y.The resulting their aspect ratio.The resulting images contain grey lev- effective learning rates during the first pass varied between els as result of the anti-aliasing (image interpolation)tech- approximately 7 x 1y-2 and y.ylP over the set of parame- nique used by the normalization algorithm.Three ver- ters.The test error rate stabilizes after around ly passes sions of the database were used.In the first version, through the training set at y.95%.The error rate on the the images were centered in a IsxIs image by comput- training set reaches y.35%after 19 passes.Many authors ing the center of mass of the pixels,and translating the have reported observing the common phenomenon of over- image so as to position this point at the center of the training when training neural networks or other adaptive IsxIs field.In some instances,this IsxIs field was ex- algorithms on various tasks.N hen over-training occurs, tended to 31 x3 with background pixels.This version of the training error keeps decreasing over time,but the test the database will be referred to as the regular database. error goes through a minimum and starts increasing after In the second version of the database,the character im- a certain number of iterations.N hile this phenomenon is ages were deslanted and cropped down to yxy pixels im- very common,it was not observed in our case as the learn- ages.The deslanting computes the second moments of in- ing curves in figure 5 show.A possible reason is that the ertia of the pixels(counting a foreground pixel as 1 and a learning rate was kept relatively large.The effect of this is background pixel as y),and shears the image by horizon- that the weights never settle down in the local minimum tally shifting the lines so that the principal axis is verti- but keep oscillating randomly.Because of those fluctua- cal.This version of the database will be referred to as the tions,the average cost will be lower in a broader minimum. deslanted database.In the third version of the database Therefore,stochastic gradient will have a similar effect as used in some early experiments,the images were reduced a regularization term that favors broader minima.Broader to 1Px1P pixels.The regular database (Py,yyy training minima correspond to solutions with large entropy of the examples,ly,yyy test examples size-normalized to I yx y, parameter distribution,which is beneficial to the general- and centered by center of mass in IsxIs fields)is avail- ization error. able at http]“F.2676L25h.Ltt.5 om"yInn“o52“mi7t. The influence of the training set size was measured by Figure 4 shows examples randomly picked from the test set.training the network with 15,yyy,3y,yyy,and Py,yyy exam- ples.The resulting training error and test error are shown B.Results in figure P.It is clear that,even with specialized architec- Several versions of EeNet-5 were trained on the regular tures such as EeNet-5,more training data would improve MNIST database.y iterations through the entire train-the accuracy. ing data were performed for each session.The values of To verify this hypothesis,we artificially generated more the global learning rate n(see-quation1 in Appendix C training examples by randomly distorting the original for a definition)was decreased using the following sched-training images.The increased training set was composed ulee y.yyy5 for the first two passes,y.yyyI for the next of the Py,yyy original patterns plus 54y,yyy instances of

￾✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✜ ✝ ó✛å➈ê❨￾❶ì❛û➑û➀ï✒￾❶è✧ï✖ù❭å➄î➻ì❛é❼✂➉☞✇ï➊é☎ê✧ÿ☎ê❷✚➵ÿ✙ä✥ï✖å➄ÿ❙ï➊î➻æ✙û➀ì♦✘➈ï✖ï✖ê✒➌✠ó❨ç✙ø➑û➀ï➔þ✦✧✬✝❘➾ ó✛å➈ê✬￾➊ì➈û➀û➑ï★￾↔è✥ï✖ù✫å➄î➻ì❛é❼✂➓ç✙ø✠✂➈ç✦✝➠ê❺￾❿ç☎ì✐ì❛û❀ê✤è✧ÿ☎ù✂ï✖é✐è✥ê✖ð✬✧➉ä❿åró❨ø➀é❼✂➽ê✤ï✖é☎ê✧ø➟✝ ✡✙û➀ï✩￾❶ì➈é✥￾❶û➀ÿ☎ê✤ø➀ì➈é✎ê✝ëíä✥ì➈îòû➀ï✖å➈ä✧é☎ø➑é❼✂♣ï❯↔✂æ◆ï➊ä✥ø➑î➻ï➊é✐è❿ê❦ä✧ï★➍❛ÿ☎ø➑ä✥ï✖ê❀è✧ç☎å➄è❦è✧ç✙ï ä✥ï✖ê✧ÿ✙û➩è✎✡✎ï➤ø➀é☎ù✂ï✖æ✎ï✖é☎ù✂ï➊é✐è✛ì➄ë✝è✥ç✙ït￾❿ç✙ì➈ø✁￾❶ï➔ì➈ë✝è✧ä❿å➄ø➀é✙ø➀é❼✂❙ê✧ï❶è✛å➄é☎ù➽è✧ï❞ê④è å➄î➻ì❛é❼✂➻è✧ç✙ï➚￾❶ì❛î❭æ☎û➑ï➊è✧ï✒ê✧ï❶è➳ì➄ë➏ê✧å➈î➻æ✙û➑ï❞ê➊ð❨ã✛ç✙ï✖ä✧ï➊ëíì➈ä✥ï✌ø➩è➳ó➵å❛ê➔é✙ï★￾✹✝ ï✖ê✥ê✥å➄ä❘✘➞è✧ì➑✡✙ÿ✙ø➀û➀ù✶å➞é✙ï✖ó➘ù☎å✠è✥å✑✡☎å➈ê✧ï✛✡☛✘❭î❭ø✙↔✂ø➑é✗✂❙ñ✛➏✤þ✙ã ❁ ê➵ù✙å✠è❿å➈ê✧ï❶è❿ê➊ð þ✦✧✛✝✴➾➔￾➊ì➈é✐è✥å➈ø➑é☎ê ✙❆✺❼➌ ✙✫✑ ✔❺ù✂ø✙✂❛ø➩è✿ø➑î➓å✑✂➈ï✖ê➽ó❨ä✧ø➑è✤è✥ï➊é❭✡☛✘✩✙✻✘✻✘ ù✂ø➑ë●✝ ëíï➊ä✥ï➊é✐è➞ó❨ä✥ø➩è✥ï➊ä❿ê➊ð ➏➠é✢￾❶ì➈é✐è✥ä✥å❛ê④è➤è✧ì❖þ❼✧✛✝✝❜❼➌❇ó❨ç✙ï➊ä✥ï➵✡✙û➀ì☛￾❿ô✂ê➤ì➈ë❨ù☎å✠è✥å ëíä✥ì➈î▲ï✖å➎￾❿ç➓ó❨ä✧ø➑è✧ï✖ä✇å➄æ☎æ✎ï❞å➄ä✥ï✖ù❭ø➑é✢ê✤ï★➍❛ÿ☎ï➊é✗￾➊ï✑➌❛è✧ç✙ï➳ù✙å✠è❿å➞ø➀é✢þ✦✧✬✝❘➾➔ø✓ê ê❘￾❶ä❿å➄î➉✡☎û➑ï❞ù☛ð ✎ä✥ø➩è✥ï➊ä❨ø✓ù✂ï➊é✐è✥ø➩è✥ø➑ï❞ê✛ëíì➈ä➉þ✦✧✬✝✴➾➤å➈ä✧ï✌årú✠å➄ø➀û✓å❄✡✙û➀ï➳å➈é☎ù✿ó✇ï ÿ☎ê✧ï✖ù✿è✥ç✙ø➀ê♣ø➀é✂ëíì➈ä✥î➓å✠è✧ø➀ì➈é✿è✥ì✶ÿ✙é☎ê❘￾❶ä❿å➄î➑✡✙û➀ï➤è✧ç✙ï➞ó❨ä✥ø➑è✧ï➊ä❿ê✖ð ✎ï➞è✥ç✙ï➊é ê✧æ✙û➑ø➑è❯þ✦✧✬✝✴➾➏ø➑é➤è④ó➵ì❇✰✏￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❿ê✟ó❨ä✧ø➑è✤è✥ï➊é✌✡☛✘♣è✧ç✙ï❷➞✎ä✥ê✤è ✑✫✙✻✘➵ó❨ä✧ø➑è✧ï✖ä✥ê ó➵ï➊é✐è➵ø➀é✐è✧ì➻ì➈ÿ☎ä➵é☎ï➊ó❍è✥ä✥å➈ø➑é✙ø➀é❼✂➻ê✤ï➊è✖ð➏ã✛ç✙ï➳ä✥ï➊î➓å➄ø➀é✙ø➀é❼✂❑✑ ✙❆✘✒ó❨ä✧ø➑è✧ï✖ä✥ê ó➵ï➊ä✥ï✢æ✙û✓å✑￾➊ï✖ù❑ø➑é ì➈ÿ☎ä❙è✥ï✖ê✤è➽ê✤ï➊è✖ð❑ã✛ç➇ÿ☎ê➻ó➵ï✿ç☎å➈ù❤è④ó✇ì➷ê✧ï❶è✥ê➻ó❨ø➑è✧ç é✙ï❞å➄ä✥û✙✘ ❜✻✘❼➌ ✘✻✘✗✘❖ï✄↔✙å➄î➻æ✙û➀ï✖ê➓ï❞å✑￾❿ç✝ð❲ã✛ç✙ï✫é✙ï➊ó▼è✥ä✥å➈ø➑é✙ø➀é❼✂ ê✧ï❶è➽ó✛å➈ê ￾❶ì❛î➻æ✙û➑ï➊è✧ï❞ù❑ó❨ø➑è✧ç➘ï✖é✙ì➈ÿ❼✂❛ç➘ï❯↔✙å➄î➻æ✙û➀ï✖ê❭ëíä✥ì➈î þ❼✧✛✝✝❜❼➌✛ê④è❿å➄ä✧è✧ø➀é❼✂ å✠è æ☎å➄è✤è✧ï✖ä✧é✁￾ ✘❼➌❛è✧ì❭î➻å➈ô➈ï➉å➞ëíÿ✙û➀û✟ê✧ï❶è✛ì➄ë★✓✗✘❼➌ ✘✻✘✗✘➳è✧ä❿å➄ø➀é✙ø➑é✗✂✒æ☎å➄è✤è✧ï✖ä✧é✎ê➊ð þ➇ø➀î➻ø➑û✓å➄ä✥û✙✘➎➌➄è✥ç✙ï➉é✙ï✖ó è✧ï✖ê✤è✇ê✧ï❶è❫ó✛å➈ê➜￾❶ì➈î➻æ✙û➀ï❶è✥ï✖ù➻ó❨ø➑è✧ç✶þ✦✧✬✝✝❜✌ï❯↔✙å➄î➚✝ æ✙û➀ï✖ê➞ê✤è✥å➈ä✤è✥ø➑é✗✂✺å➄è➞æ☎å✠è✧è✧ï✖ä✧é✂￾ ❜ ✙✦➌ ✘✻✘✗✘➽è✥ì✫î➓å➄ô➈ï➓å✿ëíÿ✙û➀û✇ê✧ï❶è➞ó❨ø➑è✧ç ✓✻✘✗➌ ✘✗✘✻✘✛è✥ï✖ê✤è❣æ☎å✠è✧è✧ï➊ä✥é☎ê✖ð❀➏➠é❙è✧ç✙ï✛ï✄↔➇æ◆ï➊ä✥ø➀î❭ï✖é✐è✥ê❦ù✂ï✖ê❘￾❶ä✥ø✠✡✎ï❞ù➞ç✙ï✖ä✧ï➎➌➄ó✇ï ì➈é☎û✙✘✢ÿ☎ê✤ï❞ù✺å➓ê✤ÿ❼✡✎ê✤ï➊è➔ì➄ë✔➾✽✘❼➌ ✘✻✘✗✘➞è✥ï✖ê✤è➉ø➀î➻å✑✂➈ï❞ê✌➪✮✙✦➌ ✘✻✘✗✘❙ëíä✧ì❛îPþ✦✧✬✝❘➾ å➄é✎ù ✙✦➌ ✘✻✘✗✘➳ëíä✧ì❛î✴þ✦✧✬✝✿❜❩➶✹➌☛✡✙ÿ✂è✛ó✇ï♣ÿ☎ê✧ï✖ù➓è✥ç✙ï➔ëíÿ☎û➑û ✓✗✘❼➌ ✘✻✘✻✘➳è✥ä✥å➈ø➑é☎ø➑é❼✂ ê✥å➄î➻æ✙û➀ï✖ê✖ð✿ã✛ç✙ï✶ä✧ï❞ê✤ÿ✙û➑è✧ø➀é❼✂➲ù✙å➄è✥å✑✡☎å➈ê✧ï➓ó➵å❛ê✌￾➊å➈û➑û➀ï✖ù è✥ç✙ï✶ö✫ì✂ù✂ø✙➞☎ï✖ù ñ✛➏✤þ✙ãt➌✙ì➈ä➔ö➲ñ✬➏✤þ✂ãt➌☎ù✙å➄è✥å❛ê✤ï➊è✖ð ã✛ç✙ï➓ì➈ä✥ø✙✂❛ø➑é✎å➄û❜✡✙û➀å➎￾❿ô❖å➄é✎ù❖ó❨ç✙ø➑è✧ï↕➪✡✙ø➀û➑ï✖ú➈ï➊û●➶➳ø➀î➻å✑✂➈ï❞ê♣ó➵ï➊ä✥ï➓ê✤ø✠➽➊ï é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✖ù✻è✧ì⑨➞✙è✫ø➀é❲å☞✑❆✘❄↔✤✑❆✘ æ☎ø➟↔✂ï➊ût✡◆ì✪↔✲ó❨ç✙ø➀û➑ï➷æ✙ä✥ï✖ê✧ï➊ä✥ú➇ø➑é✗✂ è✧ç☎ï➊ø➀ä➤å❛ê✤æ◆ï✒￾❶è➳ä❿å✠è✥ø➑ì✎ð➞ã✛ç✙ï❭ä✧ï❞ê✤ÿ✙û➑è✧ø➀é❼✂✢ø➑î➓å❄✂❛ï✖ê✞￾❶ì❛é❛è❿å➄ø➀é➔✂❛ä✧ï✒✘✿û➀ï➊ú❩✝ ï➊û✓ê➵å➈ê➏ä✥ï✖ê✧ÿ✙û➑è❫ì➄ë☛è✥ç✙ï♣å➈é✐è✧ø✙✝⑥å➈û➑ø✓å➈ê✧ø➑é✗✂➣➪íø➀î➻å✑✂➈ï➔ø➀é✐è✧ï✖ä✧æ◆ì➈û✓å✠è✥ø➑ì❛é✥➶❦è✥ï✒￾❿ç✦✝ é✙ø✁➍✐ÿ✙ï➷ÿ☎ê✤ï❞ù ✡☛✘❍è✥ç✙ï➷é✙ì➈ä✥î➓å➄û➀ø✙➽❞å✠è✧ø➀ì➈é✾å➄û✠✂➈ì➈ä✥ø➑è✧ç✙î✺ð ã✛ç✙ä✥ï➊ï➷ú➈ï✖ä❇✝ ê✧ø➑ì❛é☎ê➷ì➄ë✢è✧ç✙ï✻ù✙å➄è✥å✑✡☎å➈ê✧ï❑ó➵ï➊ä✥ï ÿ☎ê✤ï❞ù☛ð ➏➠é✹è✥ç✙ï ➞☎ä❿ê④è ú❛ï➊ä❿ê✤ø➀ì➈é✏➌ è✧ç☎ï❺ø➀î➓å❄✂➈ï❞ê✶ó➵ï➊ä✥ï➔￾➊ï➊é✐è✧ï✖ä✧ï❞ù❍ø➑é❲å ✑✻✺♦↔✤✑❆✺➷ø➀î➓å❄✂❛ï➔✡☛✘❭￾❶ì➈î➻æ✙ÿ✙è❇✝ ø➀é❼✂ è✧ç✙ï✆￾❶ï✖é✐è✧ï➊ä➻ì➈ë♣î➓å➈ê✥ê❭ì➄ë➉è✥ç✙ï✺æ✙ø✙↔➇ï✖û➀ê✒➌✇å➈é☎ù❤è✧ä❿å➄é☎ê✧û➀å➄è✧ø➀é❼✂❺è✧ç✙ï ø➀î➻å✑✂➈ï ê✧ì❍å➈ê✺è✥ì❍æ✎ì✐ê✤ø➑è✧ø➀ì➈é❲è✧ç☎ø➀ê✫æ◆ì➈ø➀é✐è❺å✠è✫è✥ç✙ï✢￾❶ï✖é✐è✧ï➊ä❖ì➄ë❭è✧ç✙ï ✑❆✺❄↔✤✑❆✺➛➞☎ï➊û✓ù☛ð✖➏➠é✻ê✤ì❛î➻ï✫ø➑é☎ê✤è✥å➈é✗￾❶ï❞ê✄➌➵è✥ç✙ø➀ê✳✑❆✺❄↔✑✻✺➛➞☎ï➊û✓ù➘ó➵å❛ê➓ï❯↔☛✝ è✧ï✖é☎ù✂ï❞ù è✧ì ❜✵✑✪↔❜ ✑✢ó❨ø➑è✧ç①✡☎å➎￾❿ô❩✂❛ä✧ì❛ÿ✙é☎ù❺æ✙ø✙↔➇ï✖û➀ê✖ð✢ã✛ç☎ø➀ê✒ú➈ï➊ä❿ê✧ø➑ì❛é❺ì➈ë è✧ç☎ï➲ù✙å✠è❿å❄✡☎å❛ê✤ï✢ó❨ø➑û➀û✩✡✎ï✫ä✥ï❶ëíï✖ä✧ä✥ï✖ù è✧ì å➈ê❙è✥ç✙ï ➡❺➳✿✥✡✔✲✙➢❄➡❙ù✙å➄è✥å✑✡☎å➈ê✧ï➈ð ➏➠é❑è✧ç✙ï✫ê✧ï✒￾❶ì❛é☎ù ú❛ï➊ä❿ê✤ø➀ì➈é❤ì➄ë➉è✥ç✙ï✫ù✙å✠è❿å❄✡☎å❛ê✤ï➎➌❣è✧ç☎ï✫￾❿ç✎å➄ä❿å✑￾↔è✥ï➊ä❙ø➑î➚✝ å❄✂❛ï✖ê✇ó✇ï✖ä✧ï➳ù✂ï✖ê✧û✓å➄é✐è✧ï❞ù➽å➄é☎ù➙￾❶ä✥ì➈æ✙æ◆ï✖ù✶ù✂ì✠ó❨é➽è✥ì❑✑✻✘♦↔✤✑❆✘✌æ☎ø➟↔✂ï➊û✓ê✇ø➑î➚✝ å❄✂❛ï✖ê✖ð✇ã✛ç✙ï➞ù✙ï✖ê✧û➀å➈é❛è✥ø➑é✗✂➣￾❶ì➈î➻æ✙ÿ✙è✧ï✖ê➔è✧ç✙ï❙ê✤ï★￾❶ì➈é✎ù✶î➻ì➈î➻ï✖é❛è❿ê➔ì➈ë❯ø➀é✦✝ ï➊ä✧è✧ø✓å➻ì➄ë❦è✥ç✙ï✒æ✙ø✙↔✂ï➊û✓ê➉➪✻￾❶ì➈ÿ☎é❛è✥ø➑é✗✂➽å➻ëíì➈ä✥ï✄✂❛ä✧ì❛ÿ✙é☎ù✢æ✙ø➟↔✂ï✖û❇å➈ê➓➾✒å➄é☎ù✫å ✡☎å➎￾❿ô❩✂❛ä✧ì❛ÿ✙é☎ù✺æ✙ø✙↔✂ï➊û➏å❛ê✤✘❩➶✹➌✝å➄é✎ù❖ê✧ç✙ï✖å➈ä✥ê➉è✧ç✙ï❭ø➀î➻å✑✂➈ï➚✡☛✘✫ç✙ì➈ä✥ø✙➽✖ì➈é✦✝ è✥å➈û➑û✠✘ ê✤ç☎ø➩ë➺è✥ø➑é❼✂❖è✧ç✙ï✿û➀ø➑é✙ï❞ê❙ê✧ì✫è✥ç☎å✠è❙è✧ç✙ï✿æ✙ä✥ø➑é✗￾➊ø➑æ✎å➄û✇å❄↔✂ø➀ê✒ø➀ê❙ú➈ï✖ä✤è✥ø➟✝ ￾➊å➈ûüð➏ã✛ç☎ø➀ê✛ú❛ï➊ä❿ê✤ø➀ì➈é✢ì➄ë❯è✧ç✙ï➞ù✙å➄è✥å✑✡☎å➈ê✧ï♣ó❨ø➀û➀û✏✡✎ï✌ä✥ï❶ëíï✖ä✧ä✥ï✖ù➽è✧ì➽å❛ê➵è✧ç✙ï ✪✑➳❯➩▼✲✙➢♦➨✥➺❖➳✱✪✶ù✙å✠è❿å❄✡☎å❛ê✤ï❛ð ➏➠é➷è✧ç☎ï➓è✧ç✙ø➀ä❿ù➷ú➈ï➊ä❿ê✧ø➑ì❛é❺ì➈ë➵è✥ç✙ï✶ù✙å➄è✥å✑✡☎å➈ê✧ï✑➌ ÿ☎ê✧ï✖ù❺ø➑é ê✧ì➈î➻ï❭ï✖å➄ä✥û✠✘✫ï❯↔✂æ✎ï✖ä✧ø➀î➻ï➊é✐è✥ê✒➌☛è✥ç✙ï➻ø➑î➓å❄✂❛ï✖ê➳ó➵ï➊ä✥ï❭ä✥ï✖ù✙ÿ✗￾❶ï❞ù è✧ì✟➾✒✓♦↔❢➾✽✓❑æ✙ø✙↔➇ï✖û➀ê✖ð ã✛ç✙ï➷ä✧ï✒✂➈ÿ✙û✓å➄ä✫ù✙å➄è✥å✑✡☎å➈ê✧ï⑨➪✡✓✻✘✗➌ ✘✗✘✻✘ è✥ä✥å➈ø➑é☎ø➑é❼✂ ï❯↔✙å➈î❭æ☎û➑ï❞ê✄➌✎➾✽✘✗➌ ✘✗✘✻✘✶è✧ï✖ê✤è✒ï✄↔✂å➈î➻æ✙û➑ï❞ê➞ê✧ø✙➽✖ï❯✝⑥é✙ì➈ä✥î➓å➄û➀ø✙➽✖ï✖ù❖è✧ì ✑❆✘❄↔✤✑❆✘❼➌ å➄é✎ù⑨￾➊ï➊é✐è✧ï✖ä✧ï❞ù➹✡☛✘⑨￾❶ï➊é✐è✥ï➊ä➽ì➈ë➳î➓å➈ê✥ê➻ø➀é☞✑✻✺♦↔✤✑❆✺➔➞☎ï✖û➀ù✙ê✴➶➻ø✓ê➓årú✠å➄ø➀û➟✝ å❄✡☎û➑ï❫å➄è☎✄✝✆✞✆✞✟✡✠☞☛✌☛✎✍✌✍✌✍✡✏✒✑✝✓✕✔✖✓✌✗✘✑✕✙✚✄✡✏✒✗✘✆✞✆✛✏✜✙✣✢✚✤✥☛✧✦★✝✗✖✩✞✩✕☛✞✢✝✙✎✑✕☛✪✤✫✩✭✬✞✔✎✆✎ð ✜❯ø✙✂❛ÿ✙ä✥ï ❝➉ê✧ç✙ì✠ó➔ê✝ï✄↔✙å➄î➻æ✙û➀ï✖ê✝ä❿å➄é✎ù✂ì➈î➻û✠✘♣æ✙ø✁￾❿ô➈ï❞ù➤ëíä✥ì➈î✾è✧ç✙ï✇è✧ï❞ê④è❦ê✤ï➊è✖ð ✬✛ ✁④➳✹➩✒✡✔✲➂➺✻➩ þ➇ï➊ú❛ï➊ä❿å➄û❯ú➈ï➊ä❿ê✧ø➑ì❛é☎ê➔ì➈ë✄✗✝ï❞ñ➔ï❶è❺✝✝✙➽ó✇ï✖ä✧ï✒è✧ä❿å➄ø➀é✙ï✖ù❖ì➈é❖è✧ç✙ï❭ä✧ï✒✂➈ÿ✙û✓å➄ä ö➲ñ✬➏✤þ✂ã ù✙å✠è❿å❄✡☎å❛ê✤ï❛ð◆✑✻✘✺ø➩è✥ï➊ä❿å✠è✥ø➑ì❛é☎ê➤è✧ç☎ä✧ì❛ÿ❼✂➈ç➷è✥ç✙ï➽ï➊é✐è✥ø➑ä✥ï➓è✧ä❿å➄ø➀é✦✝ ø➀é❼✂ ù☎å✠è✥å❖ó✇ï✖ä✧ï✢æ✎ï✖ä✤ëíì❛ä✧î➻ï❞ù ëíì➈ä❭ï✖å➎￾❿ç❑ê✧ï✖ê✥ê✤ø➀ì➈é❀ð➷ã✛ç☎ï✢ú✠å➄û➀ÿ✙ï❞ê✒ì➈ë è✧ç☎ï✞✂❛û➑ì➎✡☎å➄û✟û➀ï✖å➈ä✧é✙ø➀é❼✂❙ä✥å➄è✧ï✯✮➔➪⑨ê✧ï➊ï ✭➜➍✐ÿ☎å➄è✧ø➀ì➈é◆✑✦➾➳ø➑é✫✕➔æ✙æ◆ï➊é✎ù✂ø➟↔✆☞ ëíì➈ä❙å✫ù✂ï❯➞☎é☎ø➩è✥ø➑ì❛é✥➶➤ó✛å➈ê➞ù✂ï★￾❶ä✥ï✖å❛ê✤ï❞ù❖ÿ☎ê✧ø➀é❼✂✺è✧ç✙ï➽ëíì➈û➀û➑ì✠ó❨ø➀é❼✂✫ê❺￾❿ç☎ï✖ù☛✝ ÿ✙û➀ï✻✰✶✘✙ð ✘✻✘✗✘ ✙❖ëíì➈ä✶è✧ç✙ï✆➞✎ä✥ê✤è➽è④ó➵ì æ☎å➈ê✥ê✤ï❞ê✄➌ ✘☎ð ✘✗✘✻✘ ✑❖ëíì❛ä➽è✧ç☎ï➲é✙ï❯↔➇è ✁✗✿▲❍✪❦✱✰✪❦➙❁✒✿▲❵❅✰r❑●❈✪✷✹✸✶❋✛✱✴❴▲✿▲❵❅✰❅❉✬✰☎★✱✴❋✩❃✪❴▲✰❅✺❢⑥✠✸✻✷✹❋①✵✶✯✪✰➜❸✌❤❀❊ ❁✒✮✆❉♦✱❘✵✶✱✴◗♦✱✴✺✶✰✹❦ è✧ç☎ä✧ï✖ï✑➌ ✘✙ð ✘✻✘✻✘✗➾➽ëíì❛ä➓è✧ç✙ï✫é☎ï❯↔➇è➻è✧ç✙ä✥ï➊ï➎➌ ✘☎ð ✘✗✘✻✘✻✘✵✙✺ëíì❛ä➓è✧ç✙ï✫é☎ï❯↔➇è ❝✗➌ å➄é✎ù✚✘✙ð ✘✻✘✗✘✻✘❼➾➵è✥ç✙ï➊ä✥ï✖å➄ë➺è✧ï➊ä❞ð❷✚➵ï❶ëíì➈ä✥ï➔ï❞å✑￾❿ç➓ø➑è✧ï➊ä❿å✠è✥ø➑ì❛é✏➌❛è✧ç✙ï➤ù✂ø✓å❄✂➈ì❛é☎å➄û õ➔ï❞ê✧ê✧ø✓å➄é➽å➄æ☎æ✙ä✧ì✪↔✂ø➀î➓å✠è✧ø➀ì➈é➻ó✛å➈ê❫ä✧ï✖ï➊ú✠å➄û➀ÿ☎å✠è✥ï✖ù➻ì❛é ✙❆✘✗✘✌ê✧å➈î❭æ☎û➑ï❞ê✄➌➇å➈ê ù✂ï❞ê❺￾➊ä✧ø✠✡✎ï❞ù✫ø➀é➔✕➔æ✙æ◆ï➊é✎ù✂ø➟↔➒☞❲å➈é☎ù✫ô❛ï➊æ✂è④➞❼↔✂ï✖ù➲ù✂ÿ☎ä✧ø➀é❼✂➽è✥ç✙ï❭ï➊é✐è✧ø➀ä✧ï ø➑è✧ï➊ä❿å✠è✥ø➑ì❛é✝ð✛ã✛ç✙ï➞æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✳✲ ó✛å➈ê❨ê✧ï❶è➉è✧ì✙✘☎ð ✘✵✑✂ð✇ã✛ç☎ï➞ä✧ï❞ê✤ÿ☎û➩è✥ø➑é❼✂ ï❯➘✟ï✒￾❶è✧ø➀ú➈ï➔û➀ï✖å➈ä✧é☎ø➑é❼✂✌ä❿å✠è✥ï✖ê➏ù✙ÿ✙ä✧ø➀é❼✂✌è✥ç✙ï✩➞☎ä❿ê✤è❫æ☎å➈ê✥ê➏ú✠å➈ä✧ø➀ï✖ù➚✡◆ï❶è④ó➵ï➊ï➊é å➄æ☎æ✙ä✧ì✪↔✂ø➀î➓å✠è✧ï✖û✙✘✜✔✵✴➛➾✒✘ ✴ ￾ å➈é☎ù ✘ ❭ ✘❼➾✽✓❭ì✠ú❛ï➊ä➵è✧ç✙ï➞ê✧ï❶è♣ì➄ë❦æ☎å➄ä❿å➄î➻ï❯✝ è✧ï✖ä✥ê✖ð✒ã✛ç✙ï❭è✥ï✖ê✤è➤ï✖ä✧ä✥ì➈ä➳ä❿å✠è✧ï➻ê✤è✥å❄✡☎ø➑û➀ø✙➽✖ï✖ê➤å✠ë➺è✥ï➊ä✌å➄ä✥ì➈ÿ☎é☎ù✢➾✒✘➽æ☎å❛ê✧ê✧ï✖ê è✧ç☎ä✧ì❛ÿ❼✂➈ç❺è✧ç✙ï➻è✥ä✥å➈ø➑é☎ø➑é❼✂✺ê✧ï❶è✒å✠è✛✘☎ð ❀✵✙✘✶✶ð➓ã✛ç✙ï➻ï➊ä✥ä✧ì❛ä➳ä❿å✠è✥ï❙ì❛é è✧ç✙ï è✧ä❿å➄ø➀é✙ø➀é❼✂✺ê✤ï➊è➤ä✥ï✖å➎￾❿ç✙ï✖ê ✘✙ð ❜ ✙✞✶På✠ë➺è✧ï✖ä➝➾✽❀✢æ☎å➈ê✥ê✤ï❞ê➊ð❙ö➲å➄é☛✘➲å➄ÿ✙è✧ç✙ì❛ä✥ê ç☎årú❛ï➔ä✧ï✖æ✎ì❛ä✤è✥ï✖ù➻ì➎✡☎ê✤ï✖ä✧ú➇ø➀é❼✂✌è✥ç✙ï✞￾➊ì➈î➻î➻ì➈é➓æ✙ç☎ï➊é✙ì❛î❭ï✖é✙ì➈é✶ì➄ë✝ì✠ú➈ï✖ä❇✝ è✧ä❿å➄ø➀é✙ø➀é❼✂✺ó❨ç✙ï➊é è✥ä✥å➈ø➑é✙ø➀é❼✂✫é✙ï✖ÿ✙ä❿å➄û❣é✙ï➊è④ó✇ì❛ä✧ô✂ê➳ì➈ä➞ì➈è✧ç✙ï✖ä➞å➈ù✙å➈æ✂è✧ø➀ú➈ï å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê❙ì➈é ú✠å➄ä✥ø➑ì❛ÿ☎ê✒è✥å➈ê✧ô✂ê➊ð ✎ç✙ï➊é ì✠ú➈ï✖ä❇✝♠è✧ä❿å➄ø➀é✙ø➀é❼✂➲ì✦￾✄￾➊ÿ✙ä❿ê✄➌ è✧ç☎ï✌è✧ä❿å➄ø➀é✙ø➀é❼✂➓ï➊ä✥ä✧ì❛ä❨ô➈ï✖ï➊æ☎ê❨ù✂ï★￾❶ä✥ï✖å❛ê✤ø➀é❼✂➻ì✠ú➈ï✖ä✛è✧ø➀î❭ï➎➌✥✡☎ÿ✂è➔è✧ç☎ï✌è✧ï❞ê④è ï➊ä✥ä✥ì➈ä✛✂❛ì✐ï❞ê➔è✥ç✙ä✥ì➈ÿ❼✂❛ç❖å➽î➻ø➀é✙ø➑î❙ÿ✙î å➄é✎ù➲ê④è❿å➄ä✧è✥ê➉ø➑é✗￾➊ä✧ï❞å➈ê✧ø➑é✗✂✶å✠ë➺è✥ï➊ä å ￾➊ï➊ä✧è✥å➈ø➑é✫é➇ÿ✙î➑✡✎ï✖ä♣ì➈ë➏ø➩è✥ï➊ä❿å✠è✧ø➀ì➈é✎ê➊ð ✎ç☎ø➑û➀ï➞è✥ç✙ø➀ê➳æ✙ç☎ï➊é✙ì❛î❭ï✖é✙ì➈é✫ø✓ê ú➈ï✖ä❺✘➝￾❶ì❛î➻î❭ì❛é✏➌✐ø➩è➵ó➵å❛ê➏é✙ì➈è❫ì✑✡☎ê✧ï➊ä✥ú➈ï❞ù➻ø➑é✶ì➈ÿ✙ä✎￾➊å➈ê✧ï➉å❛ê❣è✧ç☎ï➉û➀ï✖å➄ä✥é✦✝ ø➀é❼✂➐￾➊ÿ✙ä✥ú➈ï✖ê♣ø➀é↕➞✗✂❛ÿ✙ä✧ï❑✙✶ê✧ç✙ì✠ó✌ð➓✕➶æ✎ì✐ê✧ê✧ø✠✡✙û➑ï❭ä✥ï✖å➈ê✧ì➈é✫ø✓ê♣è✧ç☎å➄è➤è✧ç✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂➞ä❿å✠è✥ï➉ó✛å➈ê➏ô❛ï➊æ✂è✛ä✥ï➊û✓å✠è✧ø➀ú➈ï✖û✙✘❭û✓å➄ä❘✂➈ï➈ð❦ã✛ç✙ï♣ï❯➘✟ï✒￾↔è✛ì➄ë☛è✥ç✙ø✓ê✇ø✓ê è✧ç✎å✠è➞è✥ç✙ï➽ó➵ï➊ø✠✂➈ç✐è✥ê➤é✙ï✖ú➈ï✖ä➞ê✤ï➊è✤è✧û➀ï✶ù✂ì✠ó❨é➷ø➀é➷è✧ç✙ï✶û➑ì✦￾➊å➈û❣î➻ø➑é✙ø➀î✒ÿ☎î ✡✙ÿ✂è➓ô❛ï➊ï➊æ❤ì❛ê❘￾❶ø➀û➑û✓å✠è✥ø➑é❼✂❺ä✥å➈é☎ù✂ì❛î❭û✠✘➈ð✢✚➵ï★￾➊å➄ÿ✎ê✤ï✢ì➄ë➔è✥ç✙ì❛ê✧ï✚✾☎ÿ✥￾↔è✧ÿ✎å♦✝ è✧ø➀ì➈é✎ê✄➌✠è✥ç✙ï➔årú➈ï✖ä✥å✑✂➈ï✎￾❶ì❛ê✤è❣ó❨ø➑û➀û❼✡◆ï❨û➑ì✠ó➵ï➊ä❦ø➀é❭åt✡☎ä✧ì✐å➈ù✂ï✖ä❇î➻ø➀é✙ø➑î❙ÿ✙î✺ð ã✛ç✙ï✖ä✧ï➊ëíì➈ä✥ï✑➌✟ê④è✥ì☛￾❿ç✎å➈ê✤è✧ø✁￾✌✂➈ä❿å➈ù✂ø➀ï➊é✐è➉ó❨ø➀û➀û❇ç☎årú❛ï✒å✶ê✧ø➀î❭ø➀û✓å➄ä➉ï✄➘◆ï★￾↔è✌å➈ê å➤ä✥ï✄✂➈ÿ☎û➀å➈ä✧ø✠➽✖å➄è✧ø➀ì➈é➞è✥ï➊ä✥îòè✥ç☎å✠è❣ë⑨årú❛ì➈ä❿ê❳✡☎ä✧ì✐å➈ù✂ï✖ä❣î❭ø➀é✙ø➀î➓å✙ð❨✚✇ä✥ì❛å❛ù✂ï➊ä î➻ø➑é☎ø➑î➓å✫￾❶ì❛ä✧ä✥ï✖ê✧æ✎ì❛é☎ù✫è✥ì✫ê✤ì❛û➑ÿ✂è✥ø➑ì❛é☎ê➤ó❨ø➑è✧ç➷û➀å➈ä❺✂❛ï✒ï✖é❛è✥ä✧ì❛æ☛✘✫ì➄ë➵è✧ç✙ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä♣ù✂ø✓ê④è✥ä✧ø✠✡✙ÿ✂è✥ø➑ì❛é✏➌☛ó❨ç✙ø✁￾❿ç➲ø✓ê④✡◆ï➊é☎ï❯➞✥￾➊ø➀å➈û❀è✥ì✶è✥ç✙ï➑✂❛ï➊é✙ï✖ä✥å➈û➟✝ ø✠➽✖å✠è✥ø➑ì❛é✢ï✖ä✧ä✥ì➈ä❞ð ã✛ç✙ï➽ø➀é✾☎ÿ✙ï✖é✗￾❶ï✢ì➄ë✛è✧ç✙ï➽è✧ä❿å➄ø➀é✙ø➑é✗✂➲ê✤ï➊è❙ê✤ø✠➽➊ï✶ó➵å❛ê✌î➻ï✖å➈ê✧ÿ✙ä✥ï✖ù➒✡☛✘ è✧ä❿å➄ø➀é✙ø➀é❼✂➔è✥ç✙ï✛é✙ï❶è④ó➵ì➈ä✥ô➳ó❨ø➩è✥ç ➾ ✙❼➌ ✘✗✘✻✘✗➌ ❜✗✘❼➌ ✘✻✘✻✘✗➌✠å➄é☎ù✛✓✗✘❼➌ ✘✻✘✗✘❨ï❯↔✙å➄î➚✝ æ✙û➀ï✖ê✖ð➏ã✛ç✙ï➤ä✧ï❞ê✤ÿ☎û➩è✥ø➑é❼✂❙è✧ä❿å➄ø➀é✙ø➀é❼✂❭ï➊ä✥ä✧ì❛ä➵å➈é☎ù➽è✥ï✖ê✤è❨ï➊ä✥ä✧ì❛ä➵å➈ä✧ï➤ê✤ç☎ì✠ó❨é ø➀é➙➞✥✂➈ÿ✙ä✥ï✛✓✙ð❹➏⑥è➔ø✓ê✛￾➊û➑ï❞å➄ä✛è✧ç✎å✠è✒➌✎ï➊ú❛ï➊é✿ó❨ø➩è✥ç➲ê✤æ◆ï✒￾➊ø➀å➈û➑ø✠➽➊ï❞ù✢å➈ä❘￾❿ç✙ø➑è✧ï★￾✹✝ è✧ÿ☎ä✧ï❞ê➳ê✧ÿ✗￾❿ç➷å➈ê✒✗✝ï❞ñ➔ï➊è❇✝ ✙✦➌☛î➻ì➈ä✥ï✒è✥ä✥å➈ø➑é☎ø➑é❼✂✺ù✙å➄è✥å✢ó✇ì❛ÿ✙û➀ù❖ø➑î➻æ✙ä✥ì✠ú➈ï è✧ç☎ï➞å✑￾✄￾➊ÿ✙ä❿å✑￾❯✘❛ð ã❀ì➓ú❛ï➊ä✥ø➩ë➭✘✶è✧ç✙ø✓ê➔ç☛✘➇æ✎ì➈è✧ç✙ï❞ê✤ø✓ê✒➌☎ó➵ï➞å➄ä✧è✧ø✙➞✥￾❶ø✓å➄û➀û✙✘ ✂➈ï➊é☎ï➊ä❿å✠è✧ï❞ù✿î❭ì❛ä✧ï è✧ä❿å➄ø➀é✙ø➀é❼✂ ï✄↔✂å➈î➻æ✙û➑ï❞ê❭✡☛✘Pä❿å➄é✎ù✂ì➈î➻û✠✘Pù✂ø✓ê④è✥ì➈ä✧è✧ø➀é❼✂ è✥ç✙ï✹ì❛ä✧ø✠✂➈ø➀é☎å➄û è✧ä❿å➄ø➀é✙ø➀é❼✂➓ø➀î➻å✑✂➈ï❞ê➊ð❫ã✛ç☎ï✌ø➑é✗￾➊ä✧ï❞å➈ê✧ï✖ù✶è✧ä❿å➄ø➀é✙ø➀é❼✂➽ê✤ï➊è➔ó➵å❛ê✛￾❶ì❛î➻æ✎ì✐ê✤ï❞ù ì➄ë➤è✥ç✙ï✦✓✻✘✗➌ ✘✗✘✻✘ ì➈ä✥ø✠✂➈ø➀é☎å➄û❨æ☎å➄è✤è✥ï➊ä✥é☎ê➽æ✙û➀ÿ☎ê✳✙❝✗✘✗➌ ✘✗✘✻✘❺ø➑é✎ê④è❿å➄é✗￾➊ï✖ê➓ì➈ë