Program: B.

Tech, CSE, 6th sem, 3rd year

CS 601: Machine Learning

Unit-4
Machine Learning: RNN in ML
Layer
Lecture No. 1

Dr. Deepak Gupta

Associate Professor, Computer Science & Engineering
1
 Outlines
• Prerequisite
• Objective
• Concept of Recurrent Neural Network
• Machine Learning: RNN
• Exercise
• Learning Outcome
• Student Effective Learning Outcome
• Text & Reference Books
2
CS 601
 Prerequisite

•Basics understanding of RNN

• Basics understanding of Recurrent Network

3
CS 601
 Objective

• To acquire Basic understanding of Neural Network.

• To acquire Basic understanding and working of RNN.

4
CS 601
 Recurrent neural network
Recurrent Neural Network (RNN) are a type of Neural Network
where the output from previous step are fed as input to the
current step.
In traditional neural networks, all the inputs and outputs are
independent of each other, but in cases like when it is required to
predict the next word of a sentence, the previous words are
required and hence there is a need to remember the previous
words.
Thus, RNN came into existence, which solved this issue with the
help of a Hidden Layer. The main and most important feature of
RNN is Hidden state, which remembers some information about
a sequence.
5
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Use cases of RNN
 Recurrent neural network

7
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Recurrent neural network
parameters for each input as it performs the same task on all the
inputs or hidden layers to produce the output.
This reduces the complexity of parameters, unlike other neural
networks.
RNNs can take one or more input vectors and produce one or more
output vectors and the output(s) are influenced not just by weights
applied on inputs like a regular NN, but also by a “hidden” state
vector representing the context based on prior input(s)/output(s).
So, the same input could produce a different output depending on
previous inputs in the series.

8
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Recurrent neural network

Figure :A Recurrent Neural Network, with a hidden state that is meant to

carry pertinent information from one input item in the series to others.
9
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Recurrent neural network
The formula for the current state can be written as –

Here, Ht is the new state, ht-1 is the previous state while xt is the
current input.
We now have a state of the previous input instead of the input
itself, because the input neuron would have applied the
transformations on our previous input.
So each successive input is called as a time step.

10
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Recurrent neural network
Taking the simplest form of a recurrent neural network, the
activation function is tanh, the weight at the recurrent neuron is
Whh and the weight at the input neuron is Wxh, we can write the
equation for the state at time t as –

The Recurrent neuron in this case is just taking the immediate

previous state into consideration. For longer sequences the
equation can involve multiple such states. Once the final state is
calculated we can go on to produce the output.
Now, once the current state is calculated we can calculate the
output state as-

11
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Training through RNN
1. A single time step of the input is supplied to the network i.e. xt is supplied to
the network
2. We then calculate its current state using a combination of the current input and
the previous state i.e. we calculate ht
3. The current ht becomes ht-1 for the next time step
4. We can go as many time steps as the problem demands and combine the
information from all the previous states
5. Once all the time steps are completed the final current state is used to calculate
the output yt
6. The output is then compared to the actual output and the error is generated
7. The error is then backpropagated to the network to update the weights and the
network is trained

12
CS 601 SELO: 1,6,12 Reference No.: T1, R1
Types of RNN
 One-to-one:
• This is also called Plain Neural networks. It deals with a
fixed size of the input to the fixed size of output, where
they are independent of previous information/output.

• Example: Image classification.

 One-to-Many:
• It deals with a fixed size of information as input that gives
a sequence of data as output.

• Example: Image Captioning takes the image as input and

outputs a sentence of words.
 Many-to-One:
• It takes a sequence of information as input and outputs a
fixed size of the output.

• Example: sentiment analysis where any sentence is

classified as expressing the positive or negative sentiment.
 Many-to-Many:
• It takes a Sequence of information as input and processes
the recurrently outputs as a Sequence of data.

• Example: Machine Translation, where the RNN reads any

sentence in English and then outputs the sentence in
French.
 Advantages of Recurrent Neural Network
1. An RNN remembers each and every information through
time. It is useful in time series prediction only because of
the feature to remember previous inputs as well. This is
called Long Short-Term Memory.

18
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Disadvantages of Recurrent Neural Network
1. Gradient vanishing and exploding problems.
2. Training an RNN is a very difficult task.
3. It cannot process very long sequences if using tanh or ReLU
as an activation function.

19
CS 601 SELO: 1,6,12 Reference No.: T1, R1
 Exercises
1. What Is the Difference Between a Feedforward Neural
Network and Recurrent Neural Network?

2. What Are Vanishing and Exploding Gradients?

3. What Is the Difference Between Epoch, Batch, and

Iteration in Deep Learning?

20
CS 601
 Learning Outcomes

• Understand the basic concept of Recurrent

• Be able to understand Recurrent Neural Network with

example.

21
CS 601
 Student Effective Learning Outcomes

1. Ability to solve problems through application of

theoretical & practical concept.

6. Ability to observe and develop sense making, logical

skills for abstract concepts.

12. Computational thinking ability (including to translate

vast data into concepts and to understand database
reasoning.

22
CS 601
 Text & Reference Books
Text Book:
1. Tom M. Mitchell, “Machine Learning”, McGraw Hill Education, First
edition, 2017
2. Ian Goodfellow and Yoshua Bengio and Aaron Courville, “Deep
Learning”, MIT Press, 2016
Reference Book:
1. Andreas Muller, "Introduction to Machine Learning with Python: A
Guide for Data Scientists", Shroff/O'Reilly; First edition (2016).
2. Aurelien Geon, “Hands-On Machine Learning with Scikit-Learn and
Tensorflow: Concepts, Tools, and Techniques to Build Intelligent
Systems”, Shroff/O'Reilly; First edition (2017).
23
CS 601
 24
CS 601