Tensorflow

TensorFlow is a free and open-source software library for machine learning and artificial intelligence. It
can be used across a range of tasks but has a particular focus on training and inference of deep neural
networks.

machine learning

Machine learning is a branch of artificial intelligence (AI) and computer science which focuses on the use
of data and algorithms to imitate the way that humans learn, gradually improving its accuracy.

Cost Function

A Cost Function is used to measure just how wrong the model is in finding a relation between the input
and output.

Gradient descent

Gradient descent is an optimization algorithm which is commonly-used to train machine learning models
and neural networks.

Variations of gradient descent

There are three types of gradient descent learning algorithms: batch gradient descent, stochastic
gradient descent and mini-batch gradient descent.

Batch gradient descent sums the error for each point in a training set, updating the model only after all
training examples have been evaluated. This process referred to as a training epoch.

Stochastic gradient descent (SGD) runs a training epoch for each example within the dataset and it
updates each training example's parameters one at a time.

Mini-batch gradient descent combines concepts from both batch gradient descent and stochastic
gradient descent. It splits the training dataset into small batch sizes and performs updates on each of
those batches.

Model Selection

Model evaluation is the process of checking the model performance to see how much our model is able
to explain the data whereas model selection is the process of seeing the level of flexibility we need for
describing the data.

Tensor

The word tensor informally refers to two different concepts that organize and represent data.
Building data pipelines for Tensorflow

So we can build data pipelines in Tensorflow basically in two ways :

Data that’s not in that library of dataset

On Builtin datasets like MNIST

Data that’s not in that library of dataset :

So rather than writing tons of lines of code to manage comma separated files, space separated files,
binary file formats. tf.data helps to set the data into a standardized format makes it much easier to
manage the data and also to share the data with others.

Following the basic steps of Data Pipelines , here is the code :

Using version 2.0 of Tensorflow

import tensorflow as tf

Extracting the data and reading files in TFRecord format or creating dataset from from_tensor_slices
module

dataset = tf.data.TFRecordDataset(file_name)

or

dataset = tf.data.Dataset.from_tensor_slices((x_t , y_t))

Transforming the dataset, once created

dataset = dataset.shuffle(buffer_size = X)

dataset = dataset.batch(batch_size = Y)

Load the data

model.fit(dataset , epochs = 10)

Text processing with Tensorflow

TensorFlow provides two libraries for text and natural language processing: KerasNLP and TensorFlow
Text. KerasNLP is a high-level natural language processing (NLP) library that includes modern
transformer-based models as well as lower-level tokenization utilities. It's the recommended solution for
most NLP use cases. Built on TensorFlow Text, KerasNLP abstracts low-level text processing operations
into an API that's designed for ease of use. But if you prefer not to work with the Keras API, or you need
access to the lower-level text processing ops, you can use TensorFlow Text directly.

Classify images

This tutorial follows a basic machine learning workflow:

1. Examine and understand data

2. Build an input pipeline

3. Build the model

4. Train the model

5. Test the model

6. Improve the model and repeat the process

Underfitting and overfitting.

Underfitting means that your model makes accurate, but initially incorrect predictions. In this case,
train error is large and val/test error is large too. Overfitting means that your model makes not accurate
predictions. In this case, train error is very small and val/test error is large.
Convolution neural network

Convolutional neural networks are often used for image classification. By recognizing valuable features,
CNN can identify different objects on images. This ability makes them useful in medicine, for example,
for MRI diagnostics. CNN can be also used in agriculture.

Transfer learning with pertained CNN

The basic premise of transfer learning is simple: take a model trained on a large dataset and transfer its
knowledge to a smaller dataset. For object recognition with a CNN, we freeze the early convolutional
layers of the network and only train the last few layers which make a prediction.

Recurrent Neural Network

Recurrent Neural Networks are used in several domains. For instance, in Natural Language Processing
(NLP), they've been used to generate handwritten text, perform machine translation and speech
recognition. But their applications are not restricted to processing language.

Text Generation with RNN

When working with text data tokens are words or characters and any network that can model the
probability of the next token is called language model. A language model captures the statistical
structure of the text. If we are training the neural network to predict the next character, it is
called Character Level Model. Similarly, we can train the model to predict the next word, given a
sequence of words called Word Level Models. We are implementing character level model.

Tensorflow customization

A Tensor is a multi-dimensional array. Similar to NumPy ndarray objects, tf.Tensor objects have a data
type and a shape. Additionally, tf.Tensors can reside in accelerator memory (like a GPU). TensorFlow
offers a rich library of operations (for example, tf.math.add, tf.linalg.matmul, and tf.linalg.inv) that
consume and produce tf.Tensors. These operations automatically convert built-in Python types.

Tensorflow distributed training

tf.distribute.Strategy is a TensorFlow API to distribute training across multiple GPUs, multiple machines,
or TPUs. Using this API, you can distribute your existing models and training code with minimal code
changes.

tf.distribute.Strategy has been designed with these key goals in mind:

 Easy to use and support multiple user segments, including researchers, machine learning
engineers, etc.

 Provide good performance out of the box.

 Easy switching between strategies.

Visual computing

Visual computing is a generic term for all computer science disciplines dealing with images and 3D
models, such as computer graphics, image processing, visualization, computer vision, virtual and
augmented reality and video processing.

Feature extraction for visual computing

Feature extraction is a part of the dimensionality reduction process, in which, an initial set of the raw
data is divided and reduced to more manageable groups.

Neural network

A neural network is a series of algorithms that endeavors to recognize underlying relationships in a set
of data through a process that mimics the way the human brain operates.

classification with perceptron model

The Perceptron algorithm is a two-class (binary) classification machine learning algorithm. It is a type of
neural network model, perhaps the simplest type of neural network model. It consists of a single node
or neuron that takes a row of data as input and predicts a class label.

Deep learning with neural network

A neural network is a method in artificial intelligence that teaches computers to process data in a way
that is inspired by the human brain. It is a type of machine learning process, called deep learning,
that uses interconnected nodes or neurons in a layered structure that resembles the human brain.

classification with multilayer perceptron

Multilayer perceptron (MLP) is a technique of feed-forward artificial neural networks using a back
propagation learning method to classify the target variable used for supervised learning.

Autoencoder

An Autoencoder is a type of neural network that can learn to reconstruct images, text, and other data
from compressed versions of themselves. An Autoencoder consists of three layers: Encoder. Code.
Decoder.

MNIST handwritten digits classification using autoencoders

The idea of doing this is to generate more handwritten digits dataset which we can use for a variety of
situations like:

 Train a model better by covering a larger possibility of handwritten digits

As there will be multiple features of handwritten digits, until our autoencoder is over trained, we will
generate a different set of handwritten digits than MNIST which is expected to differ by a small amount
and will be beneficial in expanding the dataset.

https://iq.opengenus.org/mnist-digit-generation-using-autoencoder/

Stacked autoencoders

A stacked autoencoder is a neural network consist several layers of sparse autoencoders where output
of each hidden layer is connected to the input of the successive hidden layer.

Denoising and sparse autoencoders

Sparse Autoencoder (SAE) — uses sparsity to create an information bottleneck. Denoising Autoencoder
(DAE) — designed to remove noise from data or images. Variational Autoencoder (VAE) — encodes
information onto a distribution, enabling us to use it for new data generation.

classification cost function

The cost function is the technique of evaluating “the performance of our algorithm/model”. It takes
both predicted outputs by the model and actual outputs and calculates how much wrong the model was
in its prediction. It outputs a higher number if our predictions differ a lot from the actual values.

SGD and ADAM learning rules

The default learning rate is 0.01 and no momentum is used by default. The learning rate can be
specified via the “lr” argument and the momentum can be specified via the “momentum” argument.

Adam is defined as “a method for efficient stochastic optimization that only requires first-order
gradients with little memory requirement” [2]. Okay, let's breakdown this definition into two parts.
First, stochastic optimization is the process of optimizing an objective function in the presence of
randomness.

Convolutional neural network building block

The convolutional layer is the core building block of a CNN, and it is where the majority of computation
occurs. It requires a few components, which are input data, a filter, and a feature map.

How to build a convolutional neural network model?

Convolutional Neural Network (CNN)

1. On this page.

2. Import TensorFlow.

3. Download and prepare the CIFAR10 dataset.

 4. Verify the data.

5. Create the convolutional base.

6. Add Dense layers on top.

7. Compile and train the model.

8. Evaluate the model.

LeNET

In general, LeNet refers to LeNet-5 and is a simple convolutional neural network. Convolutional neural
networks are a kind of feed-forward neural network whose artificial neurons can respond to a part of
the surrounding cells in the coverage range and perform well in large-scale image processing.

AlexNet model

AlexNet was the first convolutional network which used GPU to boost performance. 1. AlexNet
architecture consists of 5 convolutional layers, 3 max-pooling layers, 2 normalization layers, 2 fully
connected layers, and 1 softmax layer.

VGGNet model

VGG- Network is a convolutional neural network model proposed by K. Simonyan and A. Zisserman in
the paper “Very Deep Convolutional Networks for Large-Scale Image Recognition” [1]. This
architecture achieved top-5 test accuracy of 92.7% in ImageNet, which has over 14 million images
belonging to 1000 classes.

Revisiting AlexNet computational complexity

It's a convolutional network. These limit parameters to make the calculations manageable while still
giving good results.

This particular network is described in many places and papers, so it's not difficult to get the figures for
the number of parameters and conv nets involved.

But you need to start with an understanding of how a convolutional network works. I find this is a good
place to start: http://cs231n.github.io/convolutional-networks/

Revisiting VGGNet for computational complexity

ResNet

Residual Network (ResNet) is a deep learning model used for computer vision applications. It is a
Convolutional Neural Network (CNN) architecture designed to support hundreds or thousands of
convolutional layers.

DenseNet

A DenseNet is a type of convolutional neural network that utilises dense connections between layers,
through Dense Blocks, where we connect all layers (with matching feature-map sizes) directly with each
other.

GoogleNet

GoogLeNet is a convolutional neural network that is 22 layers deep. You can load a pretrained version of
the network trained on either the ImageNet [1] or Places365 [2] [3] data sets. The network trained on
ImageNet classifies images into 1000 object categories, such as keyboard, mouse, pencil, and many
animals.

Space and computational complexity in DNN

The space complexity, like the time complexity, is typically expressed as a function of the size of the
input (or the number of inputs that you have) and, usually, in big-O notation, i.e. in the limiting case. So,
n is not the same thing as O(n), Ω(n) or Θ(n).

The Computational Complexity of Machine Learning is a mathematical study of the possibilities for
efficient learning by computers.

Transfer learning a GoogleNet

Transfer learning is the process of transfer the knowledge that the network learned in a dataset to new
similar problem. When using pretrained network what we are doing is: downloading the netwrok
architacture and the weights and use then in the new problem.

Transfer learning a ResNet

ResNet is originally trained on the ImageNet dataset and using transfer learning[7], it is possible to load
pretrained convolutional weights and train a classifier on top of it. First, needed libraries are imported.
Then, the data is loaded as in the LeNet implementation.

Activation pooling for object detection.

The main purpose of pooling is to reduce the size of feature maps, which in turn makes computation
faster because the number of training parameters is reduced. The pooling operation summarizes the
features present in a region, the size of which is determined by the pooling filter.
Region Proposal Networks

A Region Proposal Network, or RPN, is a fully convolutional network that simultaneously predicts
object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-
quality region proposals.

Semantic segmentation with CNN

Semantic segmentation classifies image pixels into one or more classes which are semantically
interpretable, rather, real-world objects. Categorizing the pixel values into distinct groups using CNN is
known as region proposal and annotation.

Adversarial autoencoder for classification

Adversarial Autoencoder (AAE) is a clever idea of blending the autoencoder architecture with the
adversarial loss concept introduced by GAN. It uses a similar concept with Variational Autoencoder
(VAE) except that it uses adversarial loss to regularize the latent code instead of the KL-divergence that
VAE uses. Adversarial machine learning (AML) is the process of extracting information about the
behavior and characteristics of an ML system and/or learning how to manipulate the inputs into an ML
system in order to obtain a preferred outcome.