ASSIGNMENT -4

6. What is a recurrent neural network (RNN), and how does it differ from other
types of neural networks? Explain the fundamental architecture of an RNN and its
applications.

RNNs are neural networks that model sequence data for

predictions. Sequence data could be audio, text, or time
series, among others. RNNs are neural networks that model
sequence data for predictions. Sequence data could be audio,
text, or time series, among others.
The main difference between a CNN and an RNN is the ability
to process temporal information — data that comes in
sequences, such as a sentence. Recurrent neural networks
are designed for this very purpose, while convolutional neural
networks are incapable of effectively interpreting temporal
information.
Suppose there is a deeper network with one input layer, three hidden
layers, and one output layer. Then like other neural networks, each
hidden layer will have its own set of weights and biases, let’s say, for
hidden layer 1 the weights and biases are (w1, b1), (w2, b2) for the
second hidden layer, and (w3, b3) for the third hidden layer. This
means that each of these layers is independent of the other, i.e. they
do not memorize the previous outputs.

 RNN converts the independent activations into dependent

activations by providing the same weights and biases to all the
layers, thus reducing the complexity of increasing parameters and
memorizing each previous output by giving each output as input to
the next hidden layer.
 Hence these three layers can be joined together such that the
weights and bias of all the hidden layers are the same, in a single
recurrent layer.
Applications of Recurrent Neural Networks:
 Prediction problems
 Machine Translation
 Speech Recognition
 Language Modelling and Generating Text
 Video Tagging
 Generating Image Descriptions
 Text Summarization
 Call Center Analysis
 Face detection, 
 OCR Applications as Image Recognition

7. How does the Long Short-Term Memory (LSTM) architecture address the
problem of vanishing gradients in RNNs? Explain the key components of an
LSTM cell and their roles in capturing long-term dependencies.

The LSTM model can be used to solve the vanishing gradient problem; the
weight update of the activation function converges to zero when long-term
data are learned in an RNN algorithm, which is useful for data with a long
learning period.

A common LSTM unit is composed of a cell, an input gate, an output gate and
a forget gate. The cell remembers values over arbitrary time intervals and the
three gates regulate the flow of information into and out of the cell.

There are four main components to an LSTM network: the forget gate, the
input gate, the output gate, and the cell state. The forget gate controls how
much information from the previous time step is retained in the current time
step. The input gate controls how much new information from the current
time step is added to the cell state. The output gate controls how much
information from the cell state is used to produce an output at the current
time step. And finally, the cell state is a vector that represents the “memory”
of the LSTM network; it contains information from both the previous time
step and the current time step. 
8. Explain the concept of attention mechanisms in RNNs and their applications in
machine translation and image captioning. How do attention mechanisms help in
focusing on relevant parts of the input sequence while processing it?

Attention is a mechanism combined in the RNN allowing it to focus on certain

parts of the input sequence when predicting a certain part of the output
sequence, enabling easier learning and of higher quality.

The Attention Module takes the encoded image from the Encoder, and the
hidden state from the Sequence Decoder and computes the weighted
Attention Score. The input sequence is passed through the Embedding layer
and then combined with the Attention Score.

Attention is proposed as a method to both align and translate. Alignment is the
problem in machine translation that identifies which parts of the input
sequence are relevant to each word in the output, whereas translation is the
process of using the relevant information to select the appropriate output.

9. Explain the concept of autoencoders in deep learning and their applications.

Discuss the fundamental architecture of an autoencoder and the different types of
autoencoders used.

An autoencoder is a type of artificial neural network used to learn data

encodings in an unsupervised manner.

The aim of an autoencoder is to learn a lower-dimensional representation

(encoding) for a higher-dimensional data, typically for dimensionality
reduction, by training the network to capture the most important parts of
the input image.

Autoencoders consist of 3 parts:

1. Encoder: A module that compresses the train-validate-test set input
data into an encoded representation that is typically several orders of
magnitude smaller than the input data.

2. Bottleneck: A module that contains the compressed knowledge

representations and is therefore the most important part of the network.

3. Decoder: A module that helps the network“decompress” the

knowledge representations and reconstructs the data back from its
encoded form. The output is then compared with a ground truth.

Architecture:

Applications of autoencoders
Dimensionality reduction

Image denoising

Generation of image and time series data

Anomaly Detection

10. What is a convolutional autoencoder (CAE), and how does it differ from a
regular autoencoder? Explain the application of CAEs in image compression and
denoising.

Convolutional autoencoders (CAEs) are unsupervised

dimensionality reduction models composed by convolutional
layers capable of creating compressed image
representations.
The main difference between AutoEncoder and Convolutional Network is the
level of network hardwiring. Convolutional Nets are pretty much hardwired.
Convolution operation is pretty much local in image domain, meaning much
more sparsity in the number of connections in neural network view.