CS1AC16 DEEP LEARNING

Neural networks:

They multiply input by a weight and produce output.

You can increase the number of layers for more complicated computations:

They can be trained using back propagation.

Deep learning is the methodology to train deep neural networks. Training them with traditional
methods becomes harder with the increasing number of layers.
A net is deep if there is more than 1 hidden layer. Each layer of nodes trains on distinct set of features
based on the previous layer's output. The further into the net, the more complex the features are – they
aggregate and recombine features from the previous layers.
Example:

Deep nets can discover hidden structures within unlabeled, unstructured data.

Automatic feature extraction

 Unlabeled data – each layer learns features automatically by repeatedly trying to reconstruct
input from which it draws its samples. The network learn to recognize correlations between
certain features
 Labeled data – a deep network trained on labeled data can then be applied to unstructured
data.
 The more data a network can train on, the more accurate is it likely to be.

Training methodologies
Dropout – prevents overfitting – performing good on training data and poor on test data
At each training iteration some nodes are randomly dropped and the ones dropped previously are
added back with their original weights
Autoencoders – used for unsupervised learning
Aim is to reduce input data to get rid of redundant data – it trains a network to reduce inputs into a
meaningful encoded form
The way it works:
Create a network with multiple layers – the number of neurons must be less than number of inputs
except in the final layer where it must be equal to the number of inputs – the network is trained to
reproduce the original input:

Once trained: remove output neurons from the network:

Attach the network to a new set of layers and train to your labels:

Convolutional layers – used a lot for computer vision

Every neuron in a layer takes all inputs from previous layer
Convolutional layer has a bank of filters/kernels that convolve with the inputs to the layer to produce
and output
 Filter/kernel - a small set of weights that can only process a subset of the input
 Convolve – the filter/kernel is multiplied over the input at a specific stride – moving a small
square across a larger one and multiplying everything where it stops

Pooling neurons together reduces the amount of computation for later layers.

Rectified Linear Units (ReLU) - alternative activation function for a neuron:

 𝑓(𝑥)= max(0,𝑥)
Simplifies training and doesn't affect overall performance

The data problem

Training deep networks requires a huge amount of training data
Ways around it:
o Create fake data from real data – transpose data or blur them
o Pre-train with other data then train on your own data

Application of deep learning

Google DeepMind: Alpha Go – the first program to defeat Go World Champions
Semantic Style Transfer
Converts between different art styles
Algorithm based on combination of Markov Random Fields and Convolutional Neural Net
Zoom and enhance – uses deeply recursive convolutional network to perfom super resolution
Tesla-automated driving – the processor uses deep neural networks