Professional Documents
Culture Documents
FTMBA – Trim 6
03
Jan-21
ANN
2
Confidential |
ANN
4
Confidential |
ANN
Assumption: a neuron is activated when at least two of its inputs are active
– Identity function: if neuron A is activated, neuron C gets activated as well
• If neuron A is off, neuron C is off as well
– Logical AND: neuron C activated only when
• Both neurons A and B are activated
– A single input signal is not enough to activate neuron C
– Logical OR: neuron C activated if
• Either neuron A or neuron B is activated (or both)
– Neuron C activated only if neuron A is active & neuron B is off
• Neuron A active all the time logical NOT
– Neuron C is active when neuron B is off, and vice versa
5
Confidential |
ANN
6
Confidential |
ANN
7
Confidential |
ANN
8
Confidential |
ANN
9
Confidential |
ANN
𝑤𝑖 , 𝑗: connection weight between the 𝑖𝑡ℎ input neuron and the 𝑗𝑡ℎ
output neuron.
𝑥𝑖 : 𝑖𝑡ℎ input value of the current training instance
ŷ𝑗: output of the 𝑗𝑡ℎ output neuron for the current training
instance
𝑦𝑗: target output of the 𝑗𝑡ℎ output neuron for the current
training instance
𝜂: learning rate
11
Confidential |
Activation function
ReLu: some gradients are fragile during training and can die
– Causes a weight update which will never activate a neuron on any
data point again: dead neurons
– Fix: Leaky ReLu: introduces a small slope to keep the updates alive
• Ranges from -∞ to +∞
13
Confidential |
ANN
14
Confidential |
ANN
Components of ANN
– Input Layer: input variables, bias term
– Hidden Layer
• Neurons where all mathematical calculations are done
• ANN can have more than one neuron in a hidden layer
– Multiple hidden layers also possible
– The Activation Function: mathematical equations
• Transforms output of a given layer
– Before passing on the information to consecutive layers
– Determine the output of an ANN
– Part of each neuron in the hidden layers
» Determines output relevant for prediction
– The Output Layer
• Final "output prediction" of the network
19
Confidential |
ANN Architecture
20
Confidential |
ANN
Learning
– Cost Function: One half of the squared difference between
actual and output value
• For each layer of the network, cost function is analyzed
– Used to adjust the threshold and weights for the next input
• Aim: minimize the cost function
– Lower the cost function, closer the actual value is to predicted value
» Error keeps becoming marginally lesser in each run
» As the network learns how to analyze values
• Resulting data fed back through the entire neural network
– Weighted synapses connecting input variables to the neuron
» Only thing that can be accessed
• Adjustment of weights: till no disparity between the actual value
and the predicted value
– Tweak values, run the neural network again:
» New Cost function produced
– Repeat process: until cost function reduced to as small as possible
21
Confidential |
ANN
22
Confidential |
ANN
Batch-Gradient Descent
– Iterative optimization algorithm
• Responsibility: to find the minimum cost value (loss)
– In the process of training the model with different weights
– Rather than evaluating every possible weight value, evaluate slope
» Angle of the function line
– If slope → Negative, proceed along (down) the curve: lower cost
» If slope → Positive, Do nothing
• Gradient Descent works fine in case of a convex curve
23
Confidential |
ANN
26
Confidential |
ANN
28
Confidential |
ANN
30
Confidential |
ANN
31
Confidential |
Thank you