Foundations of Deep Learning

Dr. Panashe Chiurunge

 Agenda
Session 1: Foundations of Deep Learning
 What is Deep Learning?
 What is behind the Deep Learning revolution?
 Deep Learning terminology
 Deep Neural Networks
 How Deep Neural Networks Learn
 The power of Deep Neural Networks
 Agenda
Session 2: Deep learning coding labs
 Introduction Deep Learning with TensorFlow
 Regression with Deep Learning
 Deep Learning for Classification
 TensorFlow Keras API
What is AI?
 What is AI?

• Intelligence:
– “the capacity to learn and solve problems” (Websters dictionary)
– in particular,
• the ability to solve novel problems
• the ability to act rationally
• the ability to act like humans

• Artificial Intelligence
– build and understand intelligent entities or agents
– 2 main approaches: “engineering” versus “cognitive modeling”
 What is AI?

• What is artificial intelligence?

It is the science and engineering of making intelligent machines, especially intelligent
computer programs. It is related to the similar task of using computers to understand
human intelligence, but AI does not have to confine itself to methods that are
biologically observable.

• Yes, but what is intelligence?

Intelligence is the computational part of the ability to achieve goals in the world.
Varying kinds and degrees of intelligence occur in people, many animals and some
machines.

• Isn't there a solid definition of intelligence that doesn't depend on relating it

to human intelligence?
Not yet. The problem is that we cannot yet characterize in general what kinds of
computational procedures we want to call intelligent. We understand some of the
mechanisms of intelligence and not others.

• More in: http://www-formal.stanford.edu/jmc/whatisai/node1.html

What is AI?
What is Deep Learning ?

Deep learning and AI used interchangeably

What is behind the DL Revolution?
 Deep learning requires lots of data
What is behind the DL Revolution?
How happens online in 60 secs
How happens online in 60 secs
How much data is generated per day ?

 According to IBM 25 Zettabytes of data are generated every

day
How much data is generated per day ?
DL requires computational power
Deep Learning foundations
 Biological inspiration from neurons
Deep Learning foundations
Deep Learning foundations
Deep Learning foundations
 Artificial Neuron (Perceptron)
 Deep Learning foundations
 Artificial Neuronal Networks

Deep Neural Network has at least 2 hidden layers

Deep Learning foundations
 Feed Forward Multilayer Perceptron (MLP)

Feed Forward Neural Network

Deep Learning foundations
Deep Learning foundations
 Sigmoid
Sigmoid Activation function
 Squashes the neurons pre-activation between 0 and 1
 Always positive
 Bounded
 Strictly increasing
 Tanh
Hyperbolic tangent (tanh) Activation function
 Squashes the neurons pre-activation between -1 and 1
 Can be positive or negative
 Bounded
 Strictly increasing
 ReLu
Rectified Linear Unit (ReLu) Activation function
 Bounded below by 0 (always non-negative)
 Can be positive or negative
 Not upper Bounded
 Strictly increasing
Deep Learning foundations
Inputs x1, x2, ..Xn are known as Features
 Universal Approximation theorem

Universality: For any arbitrary function f(x), there exists a neural network tha
closely approximate it for any input x

Universality is an incredible property!* And it holds for just 1

hidden layer
 Universal Approximation theorem

Universality: For any arbitrary functions f1(x) and f2(x), there exists a neural
network that closely approximate it for many inputs x1, x2, x3

Universality also holds for multiple inputs, multiple outputs and

multiple hidden layers
This is the incredible power of
deep learning
Deep Learning foundations
Deep Learning foundations
Deep Learning foundations
Supervised learning: Classification
Deep Learning foundations
Deep Learning foundations
Clustering
Deep Learning foundations
Reinforcement Learning
Self Driving Cars
Deep Learning foundations
Reinforcement Learning
Deep Learning foundations
Reinforcement Learning
Deep Learning foundations
Training and Testing
Deep Learning foundations
How Neural Networks learn: Backpropagation
Deep Learning foundations
Backpropagation
Deep Learning foundations
Backpropagation
Backpropagation
Deep Learning foundations
Deep Learning foundations
Deep Learning foundations
 Deep Learning foundations
Learning is an optimization problem
Deep Learning foundations
Neural Network Playground
Deep learning foundations

Q&A