Deep Learning and Backpropagation
文章目录
- 一、Deep learning attracts lots of attention.
- Ups and downs of Deep Learning
- Three Steps for Deep Learning
- Neural Network
- Output Layer as Multi-Class Classifier
- FAQ
- 二、Step 2: goodness of function
- Loss for an Example：
- Total Loss：
- 三、Step 3: pick the best function
- Gradient Descent
- Backpropagation
- 四、Backpropagation
- Backpropagation – Forward pass
- Backpropagation – Summary
- 总结
一、Deep learning attracts lots of attention.
Deep learning trends at Google. Source: SIGMOD 2016/Jeff Dean
Ups and downs of Deep Learning
1958: Perceptron (linear model)
1969: Perceptron has limitation
1980s: Multi-layer perceptron
Do not have significant difference from DNN today
1986: Backpropagation
Usually more than 3 hidden layers is not helpful
1989: 1 hidden layer is “good enough”, why deep?
2006: RBM initialization
2009: GPU
2011: Start to be popular in speech recognition
2012: win ILSVRC image competition
2015.2: Image recognition surpassing human-level performance
2016.3: Alpha GO beats Lee Sedol
2016.10: Speech recognition system as good as humans
Three Steps for Deep Learning
Neural Network
Network parameter θ: all the weights and biases in the “neurons” 。
Fully Connect Feedforward Network：
Deep = Many hidden layers：
Matrix Operation：
Neural Network ：
Output Layer as Multi-Class Classifier
Example Application–Handwriting Digit Recognition
ou need to decide the network structure to let a good function in your function set.
FAQ
Q: How many layers? How many neurons for each layer?
Q: Can the structure be automatically determined?
E.g. Evolutionary Artificial Neural Networks
Q: Can we design the network structure?
二、Step 2: goodness of function
Loss for an Example：
Total Loss：
Find a function in function set that minimizes total loss L.
Find the network parameters θ^∗ that minimize total loss L.
三、Step 3: pick the best function
Gradient Descent
Backpropagation
Backpropagation: an efficient way to compute ∂L∕∂w in neural network.
四、Backpropagation
Backpropagation – Forward pass
Compute ∂z∕∂w for all parameters
Backpropagation – Summary
