Recurrent Neural Networks

(RNN)
FFNN & CNN
one to one

• Input size is fixed

• Current input is always independent of previous inputs
• 28*28
Problems
• Input size – variable
• Current input – dependent on previous inputs
Sequence learning problems
• Input is based on time step
• Current output depends on current and previous input(s)
• Each steps performs same function
• Type of inputs
• Text (Words)
• Video
• Audio
• Time series (Medicine)
• Output
• One at the end of sequence of input
• One at each time-step
Recurrent Neural Networks

a) Sequence of input
b) Sequence of output
c) Sequence of input and output
 Recurrent Neural Networks
Sequences:
one to many many to one many to many many to many

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
 Recurrent Neural Networks
Sequences:
one to many

One girl looking at the other girl

b. Sequence output
Image captioning

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
 Recurrent Neural Networks
Sequences:
many to one
P/N

b. Sequence input
Sentiment analysis:
Movie review

The movie was intense

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
 Recurrent Neural Networks
Sequences:
many to many

b. Sequence input &

Sequence output
Machine translation
INPUT: how are you?
OUTPUT: ನೀವು ಹೇಗಿದ್ದ ೀರಿ?

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
 Recurrent Neural Networks
Sequences:
many to many

b. Synced equence input & output

Video classification
Each framelabel

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
 Recurrent Neural Networks
y1 y2 y3

W hy W hy W hy
W hh W hh W hh W hh
...

W xh W xh W xh
Si is the state of the network at timestamp i
Parameters: W xh, W hh, W hy, b1and b2 (all weights &biases)

x1 x2 x3
Recurrent Neural Networks (RNN)
• Compact representation
yi

W hy

Si W hh

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

W hy

Si W hh

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

n • Xi∊R

W hy

Si W hh

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

n • Xi∊R
• Si∊Rd
W hy

Si W hh

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

n • Xi∊R
• Si∊Rd
W hy
• yi∊Rk

Si W hh

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

n • Xi∊R
• Si∊Rd
W hy
• yi∊Rk
• Wxh∊ Rd×n
Si W hh

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

n • Xi∊R
• Si∊Rd
W hy
• yi∊Rk
• Wxh∊ Rd×n
Si W hh • Whh∊Rd×d

W xh

xi
Recurrent Neural Networks
y
(RNN)-
Dimension i

n • Xi∊R
• Si∊Rd
W hy
• yi∊Rk
• Wxh∊ Rd×n
Si W hh • Whh∊Rd×d
• Why∊Rk×d
W xh

xi
 Backpropagation through time
y1 y2 y3
• Randomly initialize all the parameters (weights and
biases)
• Predict the probability
W hy W hy W hy
• Compute the total loss
W hh W hh W hh W hh
• Backpropagate this loss into the network and update
. . . the parameters

W xh W xh W xh

x1 x2 x3
https://nptel.ac.in/courses/106/106/106106184/
 y1
Backpropagation
y y
through time
2 3

W hy W hy W hy
W W W W
...

W xh W xh W xh

x1 x2 x3
https://nptel.ac.in/courses/106/106/106106184/
Backpropagation through time

https://nptel.ac.in/courses/106/106/106106184/
Backpropagation through time

https://nptel.ac.in/courses/106/106/106106184/
Backpropagation through time

https://nptel.ac.in/courses/106/106/106106184/
References
1. http://karpathy.github.io/2015/05/21/rnn-effectiveness/
2. https://nptel.ac.in/courses/106/106/106106184/