1

2
3
4
 Recurrent Neural Network(RNN) 

In traditional neural networks, all the inputs and outputs are

independent of each other, but in cases like when it is required
to predict the next word of a sentence, the previous words are
required and hence there is a need to remember the previous
words. Thus RNN came into existence, which solved this
issue with the help of a Hidden Layer. The main and most
important feature of RNN is Hidden state, which remembers
some information about a sequence.
5
Recurrent Neural Network(RNN) 

6
7
8
9
10
11
12
13
14
15
16
In epochs training for recurrent networks, the term “epoch” is used in a sense different
from that for an ordinary multilayer perceptron.

Although an epoch in the training of a multilayer perceptron involves the entire training
Sample of input–target response pairs, an epoch in the training of a recurrent neural
the network involves a single string of temporally consecutive input–target response
pairs.

The number of epochs is a hyperparameter that defines the number times
that the learning algorithm will work through the entire training dataset.
One epoch means that each sample in the training dataset has had an 17
opportunity to update the internal model parameters
 Back Propagation Through Time

Recurrent Neural Networks are those networks that deal with

sequential data. They predict outputs using not only the current
inputs but also by taking into consideration those that occurred
before it. In other words, the current output depends on current
output and a memory element (which considers the past inputs).
For training such networks, we use good old backpropagation
but with a slight twist. We don’t independently train the system
at a specific time “t”. We train it at a specific time “t” as well
as all that has happened before time “t” like t-1, t-2, t-3.

18
Back Propagation Through Time

19
Back Propagation Through Time

20
Back Propagation Through Time

21
Back Propagation Through Time

22
Back Propagation Through Time

23
Back Propagation Through Time

24
Back Propagation Through Time

25
 Hopfield Network

The Hopfield Neural Networks, invented by Dr John J. Hopfield

consists of one layer of ‘n’ fully connected recurrent neurons. It is
generally used in performing auto association and optimization
tasks. It is calculated using a converging interactive process and it
generates a different response than our normal neural nets.

Discrete Hopfield Network: It is a fully interconnected neural

network where each unit is connected to every other unit. It
behaves in a discrete manner, i.e. it gives finite distinct output,
generally of two types: 
•Binary (0/1)
•Bipolar (-1/1)
The weights associated with this network is symmetric in nature
and has the following properties. 

26
Hopfield Network

27
Hopfield Network

28
Hopfield Network

29
Hopfield Network

30
Hopfield Network

31
Associative Memories

32
Associative Memories

33
Associative Memories

34
Linear Associative Model

35
36
37