Professional Documents
Culture Documents
5 Algorithms To Train A Neural Network - Machine Learning PDF
5 Algorithms To Train A Neural Network - Machine Learning PDF
The
Problem formulation
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 1/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 2/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
w
∗
. At any point A , we can calculate the first and
C O M PA N Y LOGIN FREE TRIAL
second derivatives of the loss function. The first
derivatives are grouped in the gradient vector,
whose elements
B Lcan
O G beSwritten
O L U T I O as
NS PRODUCTS LEARN
∂f
∇i f (w) = ,
∂wi
for i = 1, … , n .
2
∂ f
Hi,j f (w) = ,
∂wi ∂wj
for i, j = 0, 1, … .
One-dimensional optimization
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 3/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
Multidimensional optimization
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 4/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
Contents:
1. Gradient descent
2. Newton method
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 5/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
3. Conjugate gradient
C O M PA N Y LOGIN FREE TRIAL
4. Quasi-Newton method
5. Levenberg-Marquardt algorithm
BLOG SOLUTIONS PRODUCTS LEARN
1. Gradient descent
Let denote f (w
(i)
) = f
(i)
and ∇f (w
(i)
) = g
(i)
.
The method begins at a point w
(0)
and, until a
stopping criterion is satisfied, moves from w
(i)
for i = 0, 1, … .
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 6/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 7/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
2. Newton's method
Let denote f (w
(i)
) = f
(i)
, ∇f (w
(i)
) = g
(i)
and
. Consider the quadratic
(i) (i)
Hf (w ) = H
approximation of f at w
(0)
using the Taylor's
series expansion
(0) 2 (0)
+ 0.5 ⋅ (w − w ) ⋅ H
point w
(0)
. By setting g equal to 0 for the
minimum of f (w) , we obtain the next equation
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 8/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
for i = 0, 1, … .
C O M PA N Y LOGIN FREE TRIAL
for i = 0, 1, … .
training direction.
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 9/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
3. Conjugate gradient
for i = 0, 1, … .
4. Quasi-Newton method
for i = 0, 1, … .
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 13/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
5. Levenberg-Marquardt algorithm
The Levenberg-Marquardt algorithm, also known
as the damped least-squares method, has been
designed to work specifically with loss functions
which take the form of a sum of squared errors.
It works without computing the exact Hessian
matrix. Instead, it works with the gradient vector
and the Jacobian matrix.
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 14/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
m
2
f = ∑e
C O M P A Ni Y LOGIN FREE TRIAL
i=1
∂ei
Ji,j = ,
∂wj
for i = 1, … , m and j = 1, … , n .
T
∇f = 2J ⋅ e
T
Hf ≈ 2J ⋅ J + λI
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 15/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
Marquardt algorithm
B L O G (i+1)
S O L U T(i)
IONS PRODUCTS LEARN
w = w
for i = 0, 1, … .
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 16/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
Neural Designer
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 18/19
28/10/2019 5 algorithms to train a neural network | Machine learning blog
Related posts:
https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network 19/19