ch. 8: Artificial neural networks Introduction to Back Propagation Neural Networks bpnn By KH Wong Neural Networks Ch9
Ch. 8: Artificial Neural networks Introduction to Back Propagation Neural Networks BPNN By KH Wong Neural Networks Ch9. , ver. 9b 1
Introduction Neural Network research is are very hot a high performance classifier(multi-class Successful in handwritten optical character OCR recognition speech recognition image Random Sampling of MNIST noise removal etc ° Easy to implement 图四DB Slow in learning 085 Fast in classification Example and dataset http://yann.lecun.com/exdb/mnist/ Neural Networks Ch9
Introduction • Neural Network research is are very hot • A high performance Classifier (multi-class) • Successful in handwritten optical character OCR recognition, speech recognition, image noise removal etc. • Easy to implement – Slow in learning – Fast in classification Neural Networks Ch9. , ver. 9b 2 Example and dataset: http://yann.lecun.com/exdb/mnist/
Motivation Biological findings inspire the development of Neural net -nput→ weights >Logic function→ output Neuron(Logic function) Biological relation Input Dendrites INputs Output W=weights Human computes using a net Output https://www.ninds.nihgov/disorders/patient-caregiver-education/life-and-death-neuron Neural Networks Ch9. ver. 9b
Motivation • Biological findings inspire the development of Neural Net – Input →weights →Logic function→ output • Biological relation – Input – Dendrites – Output – Human computes using a net Neural Networks Ch9. , ver. 9b 3 X=inputs W=weights Neuron(Logic function) Output https://www.ninds.nih.gov/Disorders/Patient-Caregiver-Education/Life-and-Death-Neuron
Applications 家例,个人都有 人工爱机题人, ,区一开来馆 Microsoft xiaolce. Al 我理划二代人,这次 化,出有见新冠 部的日,考一下受的,在 httpiimage SVRC 20 200 net. org/challenges/LSVRC/2015, Number of object cla )机号码、作周于们 作考一一证在 Num images 养一个,占出 传,人的容机 200 categories: accordion Training Num objects airplane, ant antelope Num images 20121 dishwasher dog domestic Validatio Num objects 55502 cat dragonfly, drum dumbbell Num images 40152 , etc. Testing Num objects · Tensor flov person motorcycle 侧画湿 Car 崮一 erson Neural IycLwuIna LI. vEl, Ju
Applications • Microsoft: XiaoIce. AI • http://imagenet.org/challenges/LSVRC/2015/ – 200 categories: accordion, airplane ,ant ,antelope ….dishwasher ,dog ,domestic cat ,dragonfly ,drum ,dumbbell , etc. • Tensor flow Neural Networks Ch9. , ver. 9b 4 ILSVRC 2015 Number of object classes 200 Training Num images 456567 Num objects 478807 Validation Num images 20121 Num objects 55502 Testing Num images 40152 Num objects ---
Different types of artificial neural networks Autoencoder DNN Deep neural network & Deep learning MLP Multilayer perceptron RNN(Recurrent Neural Networks), LSTM(Long Short-term memory) RBM Restricted boltzmann machine SOM Self-organizing map Convolutional neural network cnn Fromhttps://en.wikipedia.org/wiki/artificial_neuRal_network The method discussed in this power point can be applied to many of the above n Neural Networks Ch9
Different types of artificial neural networks • Autoencoder • DNN Deep neural network & Deep learning • MLP Multilayer perceptron • RNN (Recurrent Neural Networks), LSTM (Long Short-term memory) • RBM Restricted Boltzmann machine • SOM (Self-organizing map) • Convolutional neural network CNN • From https://en.wikipedia.org/wiki/Artificial_neural_network • The method discussed in this power point can be applied to many of the above nets. Neural Networks Ch9. , ver. 9b 5
Theory of Back Propagation Neural Net (BPnn Use many samples to train the weights w)& Biases(b), so it can be used to classify an unknown input into different classes Will explain How to use it after training forward pass (classify yor the recognition of the input How to train it how to train the weights and biases ( using forward and backward passes Neural Networks Ch9, ver. 9b
Theory of Back Propagation Neural Net (BPNN) • Use many samples to train the weights (W) & Biases (b), so it can be used to classify an unknown input into different classes • Will explain – How to use it after training: forward pass (classify /or the recognition of the input ) – How to train it: how to train the weights and biases (using forward and backward passes) Neural Networks Ch9. , ver. 9b 6
Back propagation is an essential step in many artificial network designs Used to train an artificial neural network For each training example xi, a supervised ( teacher) output t; is given For each ith training sample x: X )Feed forward propagation feed x, to the neural net, obtain output y Error e, ac t-y 2)Back propagation feed e i back to net from the output side and adjust weight w (by finding Aw to minimize e Repeat 1 and 2 ) for all samples until E is o or very small Neural Networks Ch9. ver. 9b
Back propagation is an essential step in many artificial network designs • Used to train an artificial neural network • For each training example xi , a supervised (teacher) output ti is given. • For each i th training sample x: xi 1) Feed forward propagation: feed xi to the neural net, obtain output yi . Error ei |ti -yi| 2 2) Back propagation: feed ei back to net from the output side and adjust weight w (by finding ∆w) to minimize e. • Repeat 1) and 2) for all samples until E is 0 or very small. Neural Networks Ch9. , ver. 9b 7
Example: Optical character recognition OCR Training: train the system first by presenting a lot of samples with known classes to the network Random Sampling of MNIST Training up the network: 3四DB weights(W)and bias ( b) Neural net 8[5 Recognition When an image is input to the system, it will tell what character it is Neural Net Output3 =1, other outputs= Neural Networks Ch9. ver. 9b 8
Example :Optical character recognition OCR • Training: Train the system first by presenting a lot of samples with known classes to the network • Recognition: When an image is input to the system, it will tell what character it is Neural Networks Ch9. , ver. 9b 8 Neural Net Output3=‘1’, other outputs=‘0’ Neural Net Training up the network: weights (W) and bias (b)
Overview of this document Back Propagation Neural Networks bpnn) Part 1: Feed forward processing(classification or Recognition) Part 2: Back propagation(Training the network), also include forward processing backward processing and update weights Appendix A MATLAB example is explained %source http://www.mathworks.com/matlabcentral/fileexchange/19997 neural- network-for-pattern- recognition-tutorial Neural Networks Ch9. ver. 9b
Overview of this document • Back Propagation Neural Networks (BPNN) – Part 1: Feed forward processing (classification or Recognition) – Part 2: Back propagation (Training the network), also include forward processing, backward processing and update weights • Appendix: • A MATLAB example is explained • %source : http://www.mathworks.com/matlabcentral/fileexchange/19997 -neural-network-for-pattern-recognition-tutorial Neural Networks Ch9. , ver. 9b 9
Part 1(classification in action / or the Recognition process Forward pass of Back Propagation Neural Net (BPNN Assume weights(W)and bias(b) are found by training already to be discussed in part2 Neural Networks Ch9. ver. 9b
Part 1 (classification in action /or the Recognition process) Forward pass of Back Propagation Neural Net (BPNN) Assume weights (W) and bias (b) are found by training already (to be discussed in part2) Neural Networks Ch9. , ver. 9b 10