Ministry of Higher Education and Scientific Research

University of Technology
Computer Engineering Department

recurrent neural networks

‫احمد قاسم علي‬

4th Stage\NW

9\9\2020:‫تاريخ التسليم‬
Introduction to Recurrent Neural Network

Recurrent Neural Network(RNN) are a type of Neural Network where the output
from previous step are fed as input to the current step. In traditional neural
networks, all the inputs and outputs are independent of each other, but in cases
like when it is required to predict the next word of a sentence, the previous words
are required and hence there is a need to remember the previous words. Thus
RNN came into existence, which solved this issue with the help of a Hidden
Layer. The main and most important feature of RNN is Hidden state, which
remembers some information about a sequence.

RNN have a “memory” which remembers all information about what has been
calculated. It uses the same parameters for each input as it performs the same task
on all the inputs or hidden layers to produce the output. This reduces the
complexity of parameters, unlike other neural networks.

1
A recurrent neural network (RNN) is a class of artificial neural networks where
connections between nodes form a directed graph along a temporal sequence. This
allows it to exhibit temporal dynamic behavior. Derived from feedforward neural
networks, RNNs can use their internal state (memory) to process variable length
sequences of inputs. This makes them applicable to tasks such as unsegmented,
connected handwriting recognition or speech recognition.

The term “recurrent neural network” is used indiscriminately to refer to two broad
classes of networks with a similar general structure, where one is finite impulse
and the other is infinite impulse. Both classes of networks exhibit temporal
dynamic behavior. A finite impulse recurrent network is a directed acyclic graph
that can be unrolled and replaced with a strictly feedforward neural network,
while an infinite impulse recurrent network is a directed cyclic graph that can not
be unrolled.

Both finite impulse and infinite impulse recurrent networks can have additional
stored states, and the storage can be under direct control by the neural network.
The storage can also be replaced by another network or graph, if that incorporates
time delays or has feedback loops. Such controlled states are referred to as gated
state or gated memory, and are part of long short-term memory networks
(LSTMs) and gated recurrent units. This is also called Feedback Neural Network.

2
Architectures

RNNs come in many variants.

Fully recurrent

Basic RNNs are a network of neuron-like nodes organized into successive layers.
Each node in a given layer is connected with a directed (one-way) connection to
every other node in the next successive layer.[citation needed] Each node (neuron)
has a time-varying real-valued activation. Each connection (synapse) has a
modifiable real-valued weight. Nodes are either input nodes (receiving data from
outside of the network), output nodes (yielding results), or hidden nodes (that
modify the data en route from input to output).

For supervised learning in discrete time settings, sequences of real-valued input

vectors arrive at the input nodes, one vector at a time. At any given time step, each
non-input unit computes its current activation (result) as a nonlinear function of
the weighted sum of the activations of all units that connect to it. Supervisor-given
target activations can be supplied for some output units at certain time steps. For
example, if the input sequence is a speech signal corresponding to a spoken digit,
the final target output at the end of the sequence may be a label classifying the
digit.

In reinforcement learning settings, no teacher provides target signals. Instead a

fitness function or reward function is occasionally used to evaluate the RNN's
performance, which influences its input stream through output units connected to

3
actuators that affect the environment. This might be used to play a game in which
progress is measured with the number of points won.

Each sequence produces an error as the sum of the deviations of all target signals
from the corresponding activations computed by the network. For a training set of
numerous sequences, the total error is the sum of the errors of all individual
sequences.

Recurrent neural networks (RNNs) are capable of learning features and long term
dependencies from sequential and time-series data. The RNNs have a stack of
non-linear units where at least one connection between units forms a directed
cycle. A well-trained RNN can model any dynamical system; however, training
RNNs is mostly plagued by issues in learning long-term dependencies.

Advantages

• Possibility of processing input of any length

• Model size not increasing with size of input
• Computation takes into account historical information
• Weights are shared across time

Drawbacks

• Computation being slow

• Difficulty of accessing information from a long time ago

• Cannot consider any future input for the current state

4
References

1- Dupond, Samuel (2019). "A thorough review on the current advance of neural
network structures". Annual Reviews in Control. 14: 200–230.

2- Graves, Alex; Liwicki, Marcus; Fernandez, Santiago; Bertolami, Roman;

Bunke, Horst; Schmidhuber, Jürgen (2009). "A Novel Connectionist System for
Improved Unconstrained Handwriting Recognition" (PDF). IEEE Transactions on
Pattern Analysis and Machine Intelligence. 31 (5): 855–868. CiteSeerX
10.1.1.139.4502. doi:10.1109/tpami.2008.137. PMID 19299860. S2CID
14635907.

3- Sak, Haşim; Senior, Andrew; Beaufays, Françoise (2014). "Long Short-Term

Memory recurrent neural network architectures for large scale acoustic modeling"
(PDF).

4-Li, Xiangang; Wu, Xihong (2014-10-15). "Constructing Long Short-Term

Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech
Recognition".