CHEAT SHEET

Neural Network Cheat Sheet

Algorithm Name Neural networks

Description Neural networks are versatile models. They use piecewise linear functions to
approximate decision boundaries (classification) or the label function (regression).

Applicability Any supervised learning problem (classification or regression)

Assumptions Input features are homogeneous.

Label function is smooth.

Underlying Transition function

Mathematical Loss function
Principles
Forward propagation
Backward propagation
Stochastic gradient descent

Transition Functions ReLU =

Sigmoid =
tanh =

Loss Functions Regression: Squared loss

Classification: Cross entropy loss

Additional Details • Training is usually done using SGD — a variant of GD that uses gradient computed
from a small batch of training examples (sampled in each iteration).
• Gradient is computed efficiently using backwards propagation.
• Weight decay and dropout are used to regularize the model.

CIS537: Deep Learning

and Neural Networks
1
© 2019 eCornell. All rights reserved. All other copyrights, trademarks,
trade names, and logos are the sole property of their respective owners.
Computing and
Information Science
Pseudocode Forward Propagation (x):
z0 x;
for `=1:L do
a` = w`> z`−1 + b` ;
z` = σe ll(a` ) ;
end
return (zL )

Backward Propagation
({W1 . . . WL } , {b1 . . . bL } , {z1 . . . zL } , {a1 . . . aL } , L)
~δL @L 0
@zL � σ` (aL ) ;
for `=1:L do
W` = W` − ↵~δ` zT`−1 ;
b` = b` − ↵δ` ; ⇣ ⌘
~δ`−1 = σ 0 (a`−1 ) � WT ~δ` ;
`−1 `
end

CIS537: Deep Learning

and Neural Networks
2
© 2019 eCornell. All rights reserved. All other copyrights, trademarks,
trade names, and logos are the sole property of their respective owners.
Computing and
Information Science