Lecture 3: CNN

By: Dr. Eman Ahmed

1 CNN Layer
2 Consecutive CNN Layers
Rules
Learning
• Backpropagation
• Gradient Descent
• For Convolution layers: dot product between the filter and the image
then passing through an activation (linear or Relu).
• Dot Product:
𝐴 𝐵

𝑎𝑟,𝑐 = ෍ ෍ 𝑥𝑟−𝑖,𝑐−𝑗 . 𝜔𝑖𝑗

Weights: Filter Values
𝑖=−𝐴 𝑗=−𝐵
Input Pixel Value
• Activation is similar to linear layer.
• Gradient w.r.to the parameters (weights):
𝑑𝑎𝑟𝑐
= σ𝐴𝑖=−𝐴 σ𝐵𝑗=−𝐵 𝑥𝑟−𝑖,𝑐−𝑗
𝑑𝜔𝑖𝑗
Pooling
• Average Pooling
• Max Pooling
• Doesn’t affect the number of channels (depth).
• Apply 2x2 Max Pooling
1 3 2 1

4 5 9 3

3 8 0 6

2 4 7 7
Pooling Layer
• Activation Function:
𝑖𝑚𝑎𝑥 , 𝑗𝑚𝑎𝑥 = arg max 𝑥𝑖𝑗
• Gradient w.r.to input:
𝑑𝑎𝑟𝑐 1 𝑖𝑓 𝑖 = 𝑖𝑚𝑎𝑥 , 𝑗 = 𝑗𝑚𝑎𝑥
=ቊ
𝑑𝑥𝑖𝑗 0 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
LeNet-5
Softmax with 10 neu
LeNet-5

Layers Activation shape Activation size No. parameters

Input (32,32,3) 3072 0
CONV1 (f=5, s=1) (28,28,6) 4704 456
POOL1 (14,14,6) 1176 0
CONV2 (f=5,s=1) (10,10,16) 1600 2416
POOL2 (5,5,16) 400 0
FC3 (120,1) 120 48120
FC4 (84,1) 84 10164
Softmax (10,1) 10 850
Benefits of LeNet-5
• Parameters Sharing :
• a filter that detects specific features (e.g., edges) from a specific part of the
image can be useful in detecting those features from a different part of the
image.
• The same mask will go over the whole image..
• Fewer parameters compared to a fully connected network.
• Sparsity of Connections:
• each output value depends on a small part of the image. So other
• parts will not affect the output.
• Translation Invariance