Lecture 1

The neuron is the fundamental cell of the central nervous system. It is divided into three main
parts:

• Dendrites: to acquire, continuously, stimuli from several other neurons (connectors) or

the external environment.
• Cell body (Soma): responsible for processing all the information that comes from the
dendrites, to produce an activation potential that indicates if the neuron can trigger an
electric impulse along its axon.
• Axon: composed of a single extension whose mission is to guide the electrical impulses
to other connecting neurons.
The synapses are the connections that enable the transfer of electric axon impulses from a
particular neuron to dendrites of other neurons
Artificial Neuron is the simplest neuron model that includes the main features of a biological
neural network—parallelism and high connectivity—and was proposed by McCulloch and Pitts
(1943).
The artificial neuron is composed of seven basic elements:
1. Input signals (x1, x2, …, xn) are the signals or samples coming from the external
environment and representing the values assumed by the variables of a particular
application.
2. Synaptic weights (w1, w2,…, wn) are the values used to weight each one of the input
variables, which enables the quantification of their relevance concerning the
functionality of the neuron.
3. Linear aggregator (R) gathers all input signals weighted by the synaptic weights to
produce an activation voltage.
4. Activation threshold or bias (h) is a variable used to specify the proper threshold that
the result produced by the linear aggregator should have to generate a trigger value
toward the neuron output
5. Activation potential (u) is the result produced by the difference between the linear
aggregator and the activation threshold. If this value is positive, i.e. if u >= ɵ, then the
neuron produces an excitatory potential; otherwise, it will be inhibitory.
6. Activation function (g) whose goal is limiting the neuron output within a reasonable
range of values, assumed by its functional image
7. Output signal (y) consists of the final value produced by the neuron given a particular
set of input signals, and can also be used as input for other sequentially interconnected
neurons
Activation functions: establish bounds for the output of neurons. Neural networks can use
many different activation functions:
1. Linear Activation Function: it does not change the neuron output.

2. Step Activation Function: it outputs a value of 1.0 for incoming values of 0.5 or higher
and 0 for all other values
3. Sigmoid (logistic) Activation Function: is a very common choice for feedforward neural
networks that need to output only positive numbers. values above or below 0 are
compressed to the approximate range between 0 and 1

4. Hyperbolic Tangent: is also a very common activation function for neural networks that
must output values in the range between -1 and 1

5. Rectified Linear Units (ReLU) Activation Function: Most current research now
recommends the ReLU due to superior training results. As a result, most neural
networks should utilize the ReLU on hidden layers and either softmax or linear on the
output layer
 6. Softmax Activation Function: The softmax function allows us to output a categorical
probability distribution over K classes. We can use the softmax to produce a vector of
probabilities according to the output of that neuron. If we apply the softmax function to
the output of the last layer, we get a probability for each class by assigning a class to
each neuron

Loss functions: The selection of the error function depends on the type of problem being
addressed.

• For a classification problem, we want to predict a probability distribution over a set of

classes.
• In regression problems, however, we want to predict a specific value rather than a
distribution.
Types of loss functions:
1. Mean Squared (L2) Error: Mean squared error (MSE) computes the squared error
between the classification prediction and the target. Training with it minimizes the
difference in magnitude. So far, we have been using the MSE for its simplicity as the loss
for binary classification problems, classifying it as a 0 if ˆ y ≥ 0.5 or 1 if ˆ y < 0.5;
however, it is typically used for regression problems

2. Mean Absolute (L1) Error: Mean absolute error gives a measure of the absolute
difference between the target value and prediction.

3. Log Loss Function (Binary cross entropy): used as an objective function for
classifications that have two outcomes. The variable y-hat is the neural network’s
prediction, and the variable y is the known correct answer. In this case, y will always be
0 or 1
 4. Multi-Class Log Loss: If more than two outcomes are classified, then we must use multi-
class log loss.

Artificial neural network can be divided into three parts, named layers, which are known as:

• Input layer: This layer is responsible for receiving information (data), signals, features,
or measurements from the external environment.
• Hidden, intermediate, or invisible layers: These layers are composed of neurons that
are responsible for extracting patterns associated with the process or system being
analyzed
• Output layer: This layer is also composed of neurons, and thus is responsible for
producing and presenting the final network outputs
Based on how these layers are connected the main architectures of ANN are:
1. Single-Layer Feedforward Architecture: has just one input layer and a single neural
layer, which is also the output layer. The information always flows in a single direction
(thus, unidirectional), which is from the input layer to the output layer.

2. Multiple-Layer Feedforward Architectures: These are composed of one or more hidden

neural layers. These networks are employed to solve diverse problems such as function
approximation, pattern classification, system identification, process control,
optimization, and robotics.
 3. Recurrent or Feedback Architecture: The outputs of the neurons are used as feedback
inputs for other neurons. The feedback feature qualifies these networks for dynamic
information processing, meaning that they can be employed on time-variant systems,
such as time series prediction, system identification and optimization, process control,
and so forth. Among the main feedback networks are the Hopfield and the Perceptron
with feedback between neurons from distinct layers.

4. Mesh Architecture: considering the spatial arrangement of neurons for pattern

extraction purposes, that is, the spatial localization of the neurons is directly related to
the process of adjusting their synaptic. These networks serve a wide range of
applications and are used in problems involving data clustering, pattern recognition,
system optimization, graphs, and so forth.

Training Processes and Properties of Learning:

• Supervised Learning: The learning strategy consists of having available the desired
outputs for a given set of input signals
• Unsupervised Learning: does not require any knowledge of the respective desired
outputs. The learning algorithm adjusts the synaptic weights and thresholds of the
network to reflect these clusters within the network itself
• Reinforcement Learning: The network learning process is usually done by trial and error
because the only available response for a given input is whether it was satisfactory or
unsatisfactory
• Offline Learning: In offline learning, also named batch learning, the adjustments on the
weight vectors and thresholds of the network are performed after all the training set is
presented
• Online Learning: in online learning, the adjustments on the weights and thresholds of
the network are performed after presenting each training sample.
 Lecture 2
Simple single-layer neuron

• Input xi initially pondered by synaptic weights to quantify the importance of the inputs
on the functional goals of the neuron.
• The value resulting from the composition of all inputs pondered by weights, added to
the activation threshold ʘ, is used as an argument for the activation function, whose
value is the output y produced by the Perceptron.
• g(.) is the activation function and u is the activation potential.
• The activation functions usually used on the Perceptron are the step and the bipolar
step functions
• The adjustment of weights and threshold in the Perceptron is made through supervised
training.
Single-layer perceptron, a perceptron of this kind can be used to:

• Compute truth tables, such as AND and OR.

• Another, perhaps more interesting, way of looking at such systems is as a means of
classifying patterns.
• Given the importance of the recognition and classification of patterns in natural
intelligence, this is an idea we should follow up
• A single-layer neural network is limited to the classification of linearly separable
patterns.
Training Process of the Perceptron

• The adjustment of Perceptron’s weights and thresholds, to classify patterns that belong
to one of the two possible classes, is performed by the use of Hebb’s learning rule.
• If the output produced by the Perceptron coincides with the desired output, its synaptic
weights and threshold remain unchanged (inhibitory condition)
• In the case the produced output is different from the desired value, then its synaptic
weights and threshold are adjusted proportionally to its input signals (excitatory
condition).
• This process is repeated sequentially for all training samples until the output produced
by the Perceptron is similar to the desired output of all samples.

Lecture 3
Multilayer perceptron

• The model of each neuron in the network includes a nonlinear activation function that
is differentiable.
• The network contains one or more layers that are hidden from both the input and
output nodes.
• The network exhibits a high degree of connectivity, the extent of which is determined
by the synaptic weights of the network
These characteristics, however, are also responsible for the deficiencies in the network:

• The presence of a distributed form of nonlinearity and the high connectivity of the
network makes the theoretical analysis of a multilayer perceptron difficult to
undertake
• The use of hidden neurons makes the learning process more difficult because the
search must be conducted in a much larger space.
Fully connected means that a neuron in any layer of the network is connected to all the
neurons (nodes) in the previous layer.
Two kinds of signals are identified in this network:
Function Signals: the input that comes at the input of the network and propagates forward
(neuron by neuron) and emerges at the output of the network as an output signal.
Error signals: originate at an output neuron of the network and propagate backward (layer by
layer) through the network.

Each hidden or output neuron of a multilayer perceptron is designed to perform two

computations:

• The hidden neurons act as feature detectors

1. The computation of the function signal appearing at the output of each neuron, which
is expressed as a continuous nonlinear function of the input signal and synaptic weights
associated with that neuron
2. The computation of an estimate of the gradient which is needed for the backward pass
through the network
Training process
A popular method for the training of multilayer perceptron is the back-propagation algorithm,
which includes the LMS (least Mean Square Error) algorithm as a special case
The training proceeds in two phases:

• In The forward phase, the synaptic weights of the network are fixed, and the input
signal is propagated through the network, layer by layer until it reaches the output.
Thus, in this phase, changes are confined to the activation potentials and outputs of the
neurons in the network
• The backward phase, an error signal is produced by comparing the output of the
network with a desired response. The resulting error signal is propagated through the
network, again layer by layer, but this time the propagation is performed in the
backward direction. In this phase, successive adjustments are made to the synaptic
weights of the network.
Batch Learning

• The adjustments for the synaptic weights of the multilayer perceptron are performed
after the presentation of all the N examples in the training sample that constitute one
epoch of training.
• The cost function for batch learning is defined by the average error energy.
• Adjustments to the synaptic weights of the multilayer perceptron are made on an
epoch-by-epoch basis
• Advantages:
Accurate estimation of the gradient vector
Parallelization of the learning process.
Online Learning

• The adjustments to the synaptic weights of the multilayer perceptron are performed on
an example-by-example basis.
• The cost function to be minimized is therefore the total instantaneous error energy.
• Consider an epoch of N training examples arranged in the order {x(1), d(1)}, {x(2), o
d(2)}, {x(N), d(N)}. The first example pair {x(1), d(1)} in the epoch is presented to the
network, and the weight adjustments are performed using the method of gradient
descent. Then the second example {x(2), d(2)} in the epoch is presented to the network,
which leads to further adjustments to weights in the network.
Logistic Function: This form of sigmoidal nonlinearity, in its general form, is defined by

Differentiating this equation with respect to vj(n), we get

Hyperbolic tangent function: where a and b are positive constants.

Differentiating this equation with respect to vj(n), we get

To get the “best” implementation of our model, we can use an optimization algorithm to
identify the set of inputs (such as synaptic weights) that maximizes – or minimizes the objective
function.
Generally, in machine learning, we want to minimize the objective function to lower the error
of our model.
This is why the objective function is called the loss function amongst practitioners, but it can
also be called the cost function.
There are tons of popular optimization algorithms:

• Fibonacci Search
• Bisection Method
• Line Search
• Gradient Descent
Gradient Descent: is a first-order optimization algorithm. It involves taking steps in the opposite
direction of the gradient to find the global minimum (or local minimum in non-convex
functions) of the objective function.
The image below provides a great illustration of how Gradient Descent takes steps towards the
global minimum of a convex function.

Consider N to be the number of observations, Y_hat to be the predicted values for the
instances, and Y the actual values of the instances.

For our optimization algorithm to determine how big of a step (the magnitude) to take, and in
what direction, we compute:

Where η is the learning rate between 0 and 1

The learning rate determines the step size at each iteration while moving toward a minimum of
a loss function.
Backpropagation algorithm
The training process of MLP networks using the backpropagation algorithm, also known as the
generalized Delta rule, is usually done by the successive application of two specific stages:
1. Forward propagation:
• The signals {x1, x2, …, xn} of a given sample from the training set are inserted
into the network inputs and are propagated layer-by-layer until the production
of the corresponding outputs
• The current values of the synaptic weights and thresholds of its neurons, which
will remain unmodified during the execution of this stage
• The respective deviations (errors) between the desired responses and those
produced by the output neurons are calculated
2. Backward propagation:
• The backward phase, an error signal is produced by comparing the output of the
network with a desired response.
• The resulting error signal is propagated through the network, again layer by
layer, but this time the propagation is performed in the backward direction.
• In this phase, successive adjustments are made to the synaptic weights of the
network.
The squared error function is employed to measure the local performance associated with the
results produced by the output neurons concerning the given sample, that is:
Consequently, assuming a training set composed of p samples, the measurement of the global
performance of the backpropagation algorithm can be calculated through the “mean squared
error” defined by

Case 1: Adjusting the Synaptic Weights of the Output Layer

Case 2: Adjusting the Synaptic Weights of the Intermediate Layers

(A) Adjusting the synaptic weights of the second hidden layer
(B) Adjusting the synaptic weights of the first hidden layer
The MLP will be considered completely trained (adjusted) when the mean squared error {EM}
between two successive epochs is smaller than the precision {e}, which is required for the
problem being mapped

Where is the precision required for the convergence process, and is specified by the type of
application that the network is mapping

Lecture 4
Overfitting: when the model tries to learn from the details along with the noise in the data and
tries to fit each data point on the curve. In the figure, the model fits every data point. If given
new data (cannot predict very well)
Underfitting: When the model has not been allowed to look at data enough, the model won’t
be able to find patterns. The model neither learns the relationship between variables of testing
data nor predict new data point. In the figure, the model does not fit properly into the data
given, ignoring a large part of the data

Simple models trained on different samples of the data do not differ much from each other;
however, they are very far from the true sinusoidal curve (underfitting). Simple model: high
bias, low variance
Complex models trained on different samples of the data are very different from each other
(high variance). Complex model: low bias, high variance
There is always a trade-off between the bias and variance, both contribute to the mean square
error.
Deep neural networks are highly complex models (many parameters and many nonlinearities)
→ easy to overfit; hence, we need some form of regularization. Using various regularization
techniques, we can try to achieve low training and testing error
Different forms of regularization:

• l2 regularization
• Dataset augmentation
• Parameter Sharing and tying
• Adding Noise to the inputs
• Adding Noise to the outputs
• Early stopping
• Ensemble methods
• Dropout
Regularization: techniques used to calibrate models to minimize the adjusted loss function and
prevent overfitting or underfitting.
Types of Regularization:
1-Connections with Noise Injection

• One approach to improving generalization error and improving the structure of the
mapping problem is to add random noise.
• The most common type of noise used during training is the addition of Gaussian noise to
the input
• Gaussian / White noise has a mean of zero and a standard deviation of one.
2. L1- Regularization:

• Called Lasso Regression. It modifies the over-fitted or underfitted models by adding the
penalty equivalent to the sum of the absolute values of coefficients.
3. L2- Regularization:
Called Ridge Regression, it modifies the over-fitted or under-
fitted models by adding the penalty equivalent to the sum of
the squares of the magnitude of coefficients. The
regularization objective function is defined by:

Lecture 5

Optimization

• a technique used to find the best parameters for a given model to minimize a loss function and
improve performance.

• The goal is to find the best weights and biases for the model to make accurate predictions.

• it is used because models have many parameters, finding the best values for the parameters is
hard

• With optimization techniques, the model can automatically search for the best parameters,
rather than relying on manual tunning by the user

Cost Function

• a function which measures the error between predictions and their actual values across the dataset

• Minimizing the cost function help the learning algorithm find the optimal set of parameters, like
weights and biases to produce the best predictions

• Cost function is a measure of how wrong the model is estimating the relationship between the
X(input) and Y(output) parameter

Loss function / objective function

• is a measure of how well a model is able to make predication that match the true values.

• A loss function measures the error between a single prediction and the corresponding actual value.

• Loss and cost functions are methods of measuring the error in machine learning predictions.

*Loss functions measure the error per observation, whilst cost functions measure the error over all
observations*
• Mean Square Error (MSE): measure the average squared difference between the predicted values and
the true values, Where Y is the actual value, Y (hat) is the predicted value, and n is the number of
observations

• Mean Absolute Error (MAE): This loss function measures the average absolute difference between the
predicted values and the true values.

learning rate

• Learning rate is a hyperparameter in ML that determines the step size at which the optimization
algorithm updates the model’s parameters. It controls the speed of model learning.

• Smaller learning rate requires many updates → very slow.

• Optimal learning rate smoothly reaches the minima

• Large learning rate may lead to big updates → GD may diverge from minima

Gradient Descent

• Gradient = slope of a surface. → reach the lowest point on that surface.

• GD enables a model to learn the direction that the model should take to reduce error

• GD tries to minimize the loss function of the model.

Limitations:

• Local minima: Gradient Descent can get stuck in a local minimum, which may not be the global
minimum→ not the best result.

• Vanishing gradient: when gradient becomes very small, which can slow down or prevent
convergence.

• selecting the best learning rate

• Gradient descent is not efficient when data is very small.

Types of Gradient Descent

1. BATCH GRADIENT DESCENT (vanilla)

• calculates the error for each example within the training dataset, after all training examples have been
evaluated the model gets updated.
• This whole process is like a cycle and it’s called a training epoch.

- advantages:

• computational efficiency: a stable error and convergence

- Disadvantage:

• It requires the entire training dataset to be in memory and available to use

2. STOCHASTIC GRADIENT DESCENT (random):

• a variant of Gradient Descent optimization algorithm, that is used to update the parameters of a
model in a more efficient and faster way

• In SGD, at each step, the algorithm calculates the gradient for one observation instead of calculating
the gradient for the entire data sets.

• a dataset contains 1000 row, it updates the model parameters 1000 times in one cycle of a dataset

- Advantages

• easy fit into memory due to a single training sample being processed

• For larger datasets it can converge faster as it causes updates to the parameters more
frequently.

• Due to frequent updates the steps taken towards the minima of the lost function has
oscillations which can help getting out of local minimums of the loss function

3-MINI-BATCH GRADIENT DESCENT (mix between batch and stochastic)

• it splits the dataset into small subsets(batches) and compute the gradients for each batch

- Steps

1- Pick a mini-batch, Feed it to Neural network

2- Calculate the mean gradient of the mini-batch

3- Use the mean gradient we calculated in step 3 to update the weights

4- Repeat steps 1-3 for the mini-batches we created

-Advantages

• Reduces variance of the parameter update → stable convergence.

 • Speed up the learning

• Helpful to estimate the approximate location of the actual minimum

• Let us adapt the learning rate of each parameter, performing larger updates for infrequent
and smaller updates for frequent parameters

- Disadvantages

• Loss is computed for each mini-batch → total loss needs to be accumulated across all mini
batches

Adaptive Gradient Methods

• AdaGrad (Adaptive Gradient Method)

•AdaDelta

•RMSProp (Root Mean Square Propagation)

•Adam (Adaptive Moment Estimation

Adam Optimizer

• The Adam optimization algorithm is an extension to stochastic gradient descent

• efficient when working with large problem involving a lot of data or parameters.

• Adam is an adaptive learning rate method, which means it computes the individual learning rates for
different parameters.

• Adam computes individual adaptive learning rates for different parameters from estimation of the first
and second moments of the gradients.

- Adam optimizer involves a combination of two GD methods

1. Momentum: this algorithm is used to accelerate the gradient descent.

2. Root Mean Square Propagation (RMSP)

- Advantages of Adam Optimizer

1. easy implementation, Computationally efficient ,Less memory required

2. Well suited for problems that are large in terms of data/and or parameters

3. Appropriate for problems with very noisy/or spare gradients