Neural Network

A multi-layer neural network.

Each layer takes in input activations of the

precedent layer (except the first layer that takes X).
Forward Propagation
First layer:

Subsequent layers:

With Activations:

Finally the Loss for each layer:

Back-Propagation
Gradient of the Loss with respect to the
activations of the last layer:

Gradient of the Loss with respect to Z in any

layer:

Update Rule for Weights and bias:

Explanations in the post.

Thanks for
Reading
I hope this helps!