Xavier Institute of Engineering

Department of Computer Engineering

(Academic Year: 2020-21 Semester: VII)

Class: BE

Subject: AISC

Experiment No. 09
Title Implement object detection using convolutional Neural Network
learning

Course Outcome LO4

Achieved
Date of
Performance
Date of submission
Roll No. 66
Name of the Aman Sharma
Student

Rubrics used for Laboratory Evaluation:

Below
Average Good
Expectation
Knowledge (4) 2 3 4

Performance (5) 2 3 5

Content & Neatness of

1 2 3
Documentation (3)

Punctuality & Submission on Time

1 2 3
(3)

Marks Obtained:

Signature of Faculty:
 EXPERIMENT NO 9

AIM: Implement object detection using convolutional Neural Network learning.

THEORY:

OBJECT DETECTION

Object recognition is to describe a collection of related computer vision tasks that

involve activities like identifying objects in digital photographs. Image classification involves
activities such as predicting the class of one object in an image. Object localization is refers to
dentifying the location of one or more objects in an image and drawing an abounding box
around their extent. Object detection does the work of combines these two tasks and localizes
and classifies one or more objects in an image. When a user or practitioner refers to the term
“object recognition“, they often mean “object detection“. It may be challenging for beginners to
distinguish between different related computer vision tasks.
So, we can distinguish between these three computer vision tasks with this example:
Image Classification: This is done by Predict the type or class of an object in an image. Input:
An image which consists of a single object, such as a photograph. Output: A class label (e.g.
one or more integers that are mapped to class labels). Object Localization: This is done
through, Locate the presence of objects in an image and indicate their location with a bounding
box. Input: An image which consists of one or more objects, such as a photograph. Output: One
or more bounding boxes (e.g. defined by a point, width, and height). Object Detection: This is
done through, Locate the presence of objects with a bounding box and types or classes of the
located objects in an image.
Input: An image which consists of one or more objects, such as a photograph. Output:
One or more bounding boxes (e.g. defined by a point, width, and height), and a class label for
each bounding box
Humans can detect and identify objects present in an image. The human visual system is
fast and accurate and can also perform complex tasks like identifying multiple objects and
detect obstacles with little conscious thought. The availability of large sets of data, faster GPUs,
and better algorithms, we can now easily train computers to detect and classify multiple objects
within an image with high accuracy. We need to understand terms such as object detection,
object localization, loss function for object detection and localization, and finally explore an
object detection algorithm known as “You only look once” (YOLO).
Image classification also involves assigning a class label to an image, whereas object
localization involves drawing a bounding box around one or more objects in an image. Object
detection is always more challenging and combines these two tasks and draws a bounding box
around each object of interest in the image and assigns them a class label. Together, all these
problems are referred to as object recognition.

CONVOLUTIONAL NEURAL NETWORK

In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep

neural networks, most commonly applied to analyzing visual imagery. They are also known as
shift invariant or space invariant artificial neural networks (SIANN), based on their shared-
weights architecture and translation invariance characteristics. They have applications in image
and video recognition, recommender systems, image classification, medical image analysis,
natural language processing, brain-computer interfaces, and financial time series.

CNNs are regularized versions of multilayer perceptrons. Multilayer perceptrons usually

mean fully connected networks, that is, each neuron in one layer is connected to all neurons in the
next layer. The "fully-connectedness" of these networks makes them prone to overfitting data.
Typical ways of regularization include adding some form of magnitude measurement of weights
to the loss function. CNNs take a different approach towards regularization: they take advantage
of the hierarchical pattern in data and assemble more complex patterns using smaller and simpler
patterns. Therefore, on the scale of connectedness and complexity, CNNs are on the lower
extreme.

Convolutional networks were inspired by biological processes in that the connectivity

pattern between neurons resembles the organization of the animal visual cortex. Individual
cortical neurons respond to stimuli only in a restricted region of the visual field known as the
receptive field. The receptive fields of different neurons partially overlap such that they cover the
entire visual field.

CNNs use relatively little pre-processing compared to other image classification

algorithms. This means that the network learns the filters that in traditional algorithms were hand-
engineered. This independence from prior knowledge and human effort in feature design is a
major advantage.

CODE:

# import libraries
import pandas as pd
import numpy as np

from zipfile import ZipFile

from sklearn.metrics import precision_score, recall_score, accuracy_score

import os,shutil

from tensorflow import keras

import tensorflow as tf
from tensorflow.keras.models import Sequential
from keras.layers import Conv2D, Dense, MaxPooling2D,Flatten,Dropout
from keras import optimizers
from keras.preprocessing.image import ImageDataGenerator

import matplotlib.pyplot as plt

# unzip given input files

import zipfile

with zipfile.ZipFile('/content/drive/MyDrive/Dog vs Cat/train.zip') as z:

z.extractall()

with zipfile.ZipFile('/content/drive/MyDrive/Dog vs Cat/test.zip') as z:

z.extractall()

# load sample submission file

sub = pd.read_csv("/content/drive/MyDrive/Dog vs Cat/sample_submission.csv")
sub.shape

sub.head()

# creating base directory directories

base_dir = './cats_and_dogs_small'
os.mkdir(base_dir)

# creat sub directories(train,validation and test) under base directory

train_dir = os.path.join(base_dir, 'train')
os.mkdir(train_dir)

validation_dir = os.path.join(base_dir, 'validation')

os.mkdir(validation_dir)

test_dir = os.path.join(base_dir, 'test')

os.mkdir(test_dir)

# create sub-directories(cats and dogs) under each sub directories(train,validation

and test) created above
train_cats_dir = os.path.join(train_dir, 'cats')
os.mkdir(train_cats_dir)

train_dogs_dir = os.path.join(train_dir, 'dogs')

os.mkdir(train_dogs_dir)

validation_cats_dir = os.path.join(validation_dir, 'cats')

os.mkdir(validation_cats_dir)

validation_dogs_dir = os.path.join(validation_dir, 'dogs')

os.mkdir(validation_dogs_dir)

test_cats_dir = os.path.join(test_dir, 'cats')

os.mkdir(test_cats_dir)

test_dogs_dir = os.path.join(test_dir, 'dogs')

os.mkdir(test_dogs_dir)

# save the path of train and test directories where we saved cats and dogs images
after unzipping the input files
original_dataset_train_dir = './train'
original_dataset_test_dir = './test'

# copy 1000 cats images to "./cats_and_dogs_small/train/cats" sub directory we

created above, these images will be used for training the model
fnames = ['cat.{}.jpg'.format(i) for i in range(1000)]
for fname in fnames:
src = os.path.join(original_dataset_train_dir, fname) # source folder
dst = os.path.join(train_cats_dir, fname) # destination folder
shutil.copyfile(src, dst) # copying cats images from source to destination
folders

# copy 500 cats images to "./cats_and_dogs_small/validation/cats" sub directory we

created above, these images will be used for validing the model
fnames = ['cat.{}.jpg'.format(i) for i in range(1000, 1500)]
for fname in fnames:
src = os.path.join(original_dataset_train_dir, fname)
dst = os.path.join(validation_cats_dir, fname)
shutil.copyfile(src, dst)

# copy 500 cats images to "./cats_and_dogs_small/test/cats" sub directory we

created above, these images will be used for testing the model
fnames = ['cat.{}.jpg'.format(i) for i in range(1500, 2000)]
for fname in fnames:
src = os.path.join(original_dataset_train_dir, fname)
dst = os.path.join(test_cats_dir, fname)
shutil.copyfile(src, dst)

# copy 1000 dogs images to "./cats_and_dogs_small/train/dogs" sub directory we

created above, these images will be used for training the model
fnames = ['dog.{}.jpg'.format(i) for i in range(1000)]
for fname in fnames:
src = os.path.join(original_dataset_train_dir, fname)
dst = os.path.join(train_dogs_dir, fname)
shutil.copyfile(src, dst)

# copy 500 dogs images to "./cats_and_dogs_small/validation/dogs" sub directory we

created above, these images will be used for validating the model
fnames = ['dog.{}.jpg'.format(i) for i in range(1000,1500)]
for fname in fnames:
src = os.path.join(original_dataset_train_dir, fname)
dst = os.path.join(validation_dogs_dir, fname)
shutil.copyfile(src, dst)

# copy 500 dogs images to "./cats_and_dogs_small/test/dogs" sub directory we

created above, these images will be used for testing the model
fnames = ['dog.{}.jpg'.format(i) for i in range(1500,2000)]
for fname in fnames:
src = os.path.join(original_dataset_train_dir, fname)
dst = os.path.join(test_dogs_dir, fname)
shutil.copyfile(src, dst)

# sanity check - lets check the folder where we copied the data has the data as
expected
print("train_cats_dir:",len(os.listdir(train_cats_dir)))
print("validation_cats_dir:",len(os.listdir(validation_cats_dir)))
print("test_cats_dir:",len(os.listdir(test_cats_dir)))

print("train_dogs_dir:",len(os.listdir(train_dogs_dir)))
print("validation_dogs_dir:",len(os.listdir(validation_dogs_dir)))
print("test_dogs_dir:",len(os.listdir(test_dogs_dir)))

# import ImageDataGenerator for pre pro-processing JPEG images as explained above

from keras.preprocessing.image import ImageDataGenerator

# scale pixes between 0 and 1 by dividing the pixes by 255

train_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(train_dir,target_size=(150,
150),batch_size=20,class_mode='binary')

validation_generator =
test_datagen.flow_from_directory(validation_dir,target_size=(150,
150),batch_size=20,class_mode='binary')

# since image generators yields bathches indefinitely, we need to break the loop
for data_batch, labels_batch in train_generator:
print('data batch shape:', data_batch.shape)
print('labels batch shape:', labels_batch.shape)
break

EPOCH_1 = 1
EPOCH_100 = 100
EPOCH_30 = 30

# Basic CNN archietechture without Regularization

input_shape = [150,150,3]

model = Sequential()

model.add(Conv2D(32, kernel_size=(3, 3), activation='relu',

input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dense(512, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# compile the model

model.compile(loss='binary_crossentropy',
optimizer=optimizers.RMSprop(lr=1e-4),
metrics=['accuracy'])

# fitting the model as discussed above

history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=EPOCH_30,
validation_data=validation_generator,
validation_steps=50)

# Let’s plot the loss and accuracy of the model over the training and validation
data during training
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')

plt.title('Training and validation accuracy')

plt.legend()

plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()

datagen = ImageDataGenerator(
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')

from keras.preprocessing import image

fnames = [os.path.join(train_cats_dir, fname) for fname in

os.listdir(train_cats_dir)]

img_path = fnames[3]
img = image.load_img(img_path, target_size=(150, 150))

x = image.img_to_array(img)
x = x.reshape((1,) + x.shape)
plt.figure(figsize = (12,8))

i=0

for batch in datagen.flow(x, batch_size=1):

plt.subplot(2,2,i+1)
imgplot = plt.imshow(image.array_to_img(batch[0]))
i += 1
if i % 4 == 0:
break

plt.show()

# Basic CNN archietechture with Regularization

input_shape = [150,150,3]

model_1 = Sequential()

model_1.add(Conv2D(32, kernel_size=(3, 3), activation='relu',

input_shape=input_shape))
model_1.add(MaxPooling2D(pool_size=(2, 2)))
model_1.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
model_1.add(MaxPooling2D(pool_size=(2, 2)))

model_1.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))

model_1.add(MaxPooling2D(pool_size=(2, 2)))

model_1.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))

model_1.add(MaxPooling2D(pool_size=(2, 2)))

model_1.add(Flatten())
model_1.add(Dense(512, activation='relu'))
model_1.add(Dropout(0.3)) # to overcome overfitting
model_1.add(Dense(512, activation='relu'))
model_1.add(Dropout(0.3)) # to overcome overfitting
model_1.add(Dense(1, activation='sigmoid'))

train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,)

# note, we are not applying these augmentations to the test data

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(train_dir,target_size=(150,
150),batch_size=32,class_mode='binary')

validation_generator =
test_datagen.flow_from_directory(validation_dir,target_size=(150,
150),batch_size=32,class_mode='binary')

# compile the model

model_1.compile(loss='binary_crossentropy',
optimizer=optimizers.RMSprop(lr=1e-4),
metrics=['accuracy'])

history = model_1.fit_generator(
train_generator,
steps_per_epoch=63,
epochs = EPOCH_100,
validation_data=validation_generator,
validation_steps=32)

# Let’s plot the loss and accuracy of the model over the training and validation
data during training
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')

plt.title('Training and validation accuracy')

plt.legend()

plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()

# lets import VGG16 pre-trained model

from keras.applications import VGG16

conv_base = VGG16(weights='imagenet',include_top=False,input_shape=(150, 150, 3))

conv_base.summary()

#base_dir = '/Users/fchollet/Downloads/cats_and_dogs_small'
#train_dir = os.path.join(base_dir, 'train')
#validation_dir = os.path.join(base_dir, 'validation')
#test_dir = os.path.join(base_dir, 'test')

datagen = ImageDataGenerator(rescale=1./255)
batch_size = 20

def extract_features(directory, sample_count):

features = np.zeros(shape=(sample_count, 4, 4, 512))
labels = np.zeros(shape=(sample_count))
generator = datagen.flow_from_directory(directory,target_size=(150,
150),batch_size=batch_size,class_mode='binary')

i=0
for inputs_batch, labels_batch in generator:
features_batch = conv_base.predict(inputs_batch)
features[i * batch_size : (i + 1) * batch_size] = features_batch
labels[i * batch_size : (i + 1) * batch_size] = labels_batch
i += 1
if i * batch_size >= sample_count:
break
return features, labels

train_features, train_labels = extract_features(train_dir, 2000)

validation_features, validation_labels = extract_features(validation_dir, 1000)

train_features = np.reshape(train_features, (2000, 4*4* 512))

validation_features = np.reshape(validation_features, ( 1000, 4*4* 512))

# Densly Connected Classifier

model = keras.models.Sequential()
model.add(keras.layers.Dense(256, activation='relu', input_dim=4 * 4 * 512))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(1, activation='sigmoid'))

# compile the model

model.compile(optimizer=optimizers.RMSprop(lr=2e-5),
loss='binary_crossentropy',
metrics=['acc'])

# fit the model on the features extracted

history = model.fit(train_features,
train_labels,epochs=EPOCH_30,batch_size=20,validation_data=(validation_features,
validation_labels))

# plotting the results

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

# Adding a densely connected classifier on top of the convolutional base

model = keras.models.Sequential()
model.add(conv_base) # adding pre-trained conv base model
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(256, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))

model.summary()

print('This is the number of trainable weights before freezing the conv base:' ,
len(model.trainable_weights))

# freezing the layers of pre-trained model

conv_base.trainable = False

print('This is the number of trainable weights after freezing the conv base:' ,
len(model.trainable_weights))

# Training the model end to end with a frozen convolutional base

train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(150, 150),
batch_size=20,
class_mode='binary')

validation_generator =
test_datagen.flow_from_directory(validation_dir,target_size=(150,
150),batch_size=20,class_mode='binary')

model.compile(loss='binary_crossentropy',optimizer=optimizers.RMSprop(lr=2e-
5),metrics=['acc'])

history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=EPOCH_30,
validation_data=validation_generator,
validation_steps=50)

# plotting the results

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

conv_base.trainable = True
set_trainable = False

for layer in conv_base.layers:

if layer.name == 'block5_conv1':
set_trainable = True

if set_trainable:
layer.trainable = True
else:
layer.trainable = False

model.compile(loss='binary_crossentropy',optimizer=optimizers.RMSprop(lr=1e-
5),metrics=['acc'])

history =
model.fit_generator(train_generator,steps_per_epoch=100,epochs=EPOCH_100,validation
_data=validation_generator,validation_steps=50)

# plotting the results

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()
plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

OUTPUT: