Sign In

dinushasan / courseproject04 0 0
2 months ago

Notebook Files Records Collaborators Version 9 (2 months ago)

Getting the dataset from kaggle to build image classification model

In [2]:

!pip install opendatasets --upgrade --quiet

In [3]:

import opendatasets as od

In [4]:

from google.colab import files

files.upload()

Choose Files No file chosen Upload widget is only available when the cell has been executed in the
current browser session. Please rerun this cell to enable.

Saving kaggle.json to kaggle.json

Out[4]:
{'kaggle.json': b'{"username":"angiyoangiyo","key":"271740ece31b6f1eee134e300435e279"}'}

In [5]:

od.download('https://www.kaggle.com/chetankv/dogs-cats-images')

6%|▌ | 24.0M/435M [00:00<00:01, 247MB/s]

Downloading dogs-cats-images.zip to ./dogs-cats-images
100%|██████████| 435M/435M [00:01<00:00, 243MB/s]

In [6]:

import os
import torch
import torchvision
import tarfile
from torchvision.datasets.utils import download_url
f t h til d t i t d lit
 from torch.utils.data import random_split

In [7]:

project_name='courseproject04'

Exploring and preprocessing of the data of dataset

In [8]:

data_dir = './dogs-cats-images/dataset'

print(os.listdir(data_dir))
classes = os.listdir(data_dir + "/training_set")
print(classes)

['training_set', 'test_set']
['dogs', 'cats']

In [9]:

cat_files = os.listdir(data_dir + "/training_set/cats")

print('No. of training examples for cats:', len(cat_files))
print(cat_files[:5])

No. of training examples for cats: 4000

['cat.149.jpg', 'cat.1472.jpg', 'cat.506.jpg', 'cat.2792.jpg', 'cat.2489.jpg']

In [10]:

dogs_test_files = os.listdir(data_dir + "/test_set/dogs")

print("No. of test examples for dogs:", len(dogs_test_files))
print(dogs_test_files[:5])

No. of test examples for dogs: 1000

['dog.4057.jpg', 'dog.4201.jpg', 'dog.4077.jpg', 'dog.4440.jpg', 'dog.4406.jpg']

In [11]:

trainset = './dogs-cats-images/dataset/training_set'

In [12]:

for root, dirs, files in os.walk(trainset):

if files == []:
continue
else:
print(len(files))
4000
4000

In [13]:

len(trainset)

Out[13]:
39

In [14]:

testset = './dogs-cats-images/dataset/test_set'

In [15]:

for root, dirs, files in os.walk(testset):

if files == []:
continue
else:
print(len(files))

1000
1000

In [16]:

len(testset)

Out[16]:
35

In [18]:

from torchvision.datasets import ImageFolder

from torchvision.transforms import ToTensor

In [19]:

dataset = ImageFolder(data_dir+'/training_set', transform=ToTensor())

Looping through the dataset and getting the sizes of first 10 images

In [20]:

im_count = 0
for image,_ in dataset:
im_count += 1
print(image.shape)
if im count == 10:
 if im_count == 10:
break

torch.Size([3, 280, 300])

torch.Size([3, 499, 489])
torch.Size([3, 499, 403])
torch.Size([3, 149, 150])
torch.Size([3, 499, 336])
torch.Size([3, 407, 379])
torch.Size([3, 269, 259])
torch.Size([3, 375, 499])
torch.Size([3, 333, 500])
torch.Size([3, 375, 499])

In [21]:

height = []
width = []
for image,_ in dataset:
height.append(image.shape[1])
width.append(image.shape[2])
print("minimum height: {}, maximum height: {}, minimum width: {}, maximum width: {}".format(m

minimum height: 41, maximum height: 768, minimum width: 57, maximum width: 1050

As the minimum height of an image of dataset is 41, I transform them to size of 40 x 40 to avoid problems of the shapes
of images

In [22]:

import torchvision.transforms as transforms

transform = transforms.Compose([transforms.Resize((40, 40)), transforms.ToTensor()])

Transforming all images of train and test set to tensors

In [23]:

train_ds = ImageFolder(data_dir+'/training_set', transform= transform)

valid_ds = ImageFolder(data_dir+'/test_set', transform = transform)

Checking the amount of images of train set and validation set

In [24]:

len(train_ds)

Out[24]:
8000
In [25]:

len(valid_ds)

Out[25]:
2000

Looping again through images to see the shapes of images

In [26]:

im_count = 0
for image,_ in train_ds:
im_count += 1
print(image.shape)
if im_count == 10:
break

torch.Size([3, 40, 40])

torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])
torch.Size([3, 40, 40])

In [27]:

height = []
width = []
for image,_ in train_ds:
height.append(image.shape[1])
width.append(image.shape[2])
print("minimum height: {}, maximum height: {}, minimum width: {}, maximum width: {}".format(m

minimum height: 40, maximum height: 40, minimum width: 40, maximum width: 40

print the shape of an image of train dataset

In [28]:

img, label = train_ds[0]

print(img.shape, label)
img

torch.Size([3, 40, 40]) 0

Out[28]:
tensor([[[0.1529, 0.1647, 0.1608, ..., 0.3137, 0.3529, 0.6627],
[0.1451, 0.1647, 0.1765, ..., 0.2902, 0.3569, 0.6784],
[0.1294, 0.1451, 0.1451, ..., 0.2824, 0.3490, 0.6784],
...,
[0.0784, 0.0824, 0.0667, ..., 0.2667, 0.2588, 0.3020],
[0.0863, 0.0902, 0.0863, ..., 0.2588, 0.2706, 0.2980],
[0.0902, 0.0902, 0.0941, ..., 0.2824, 0.2706, 0.2353]],

[[0.1686, 0.1804, 0.1922, ..., 0.3412, 0.3804, 0.6549],

[0.1569, 0.1843, 0.2078, ..., 0.3216, 0.3882, 0.6549],
[0.1569, 0.1765, 0.1804, ..., 0.3059, 0.3725, 0.6431],
...,
[0.0784, 0.0824, 0.0667, ..., 0.2196, 0.2039, 0.2431],
[0.0863, 0.0863, 0.0863, ..., 0.2196, 0.2157, 0.2392],
[0.0824, 0.0824, 0.0902, ..., 0.2196, 0.2039, 0.1765]],

[[0.1765, 0.1922, 0.1882, ..., 0.4392, 0.4549, 0.5961],

[0.1647, 0.1922, 0.2039, ..., 0.4039, 0.4471, 0.5922],
[0.1569, 0.1765, 0.1725, ..., 0.3804, 0.4157, 0.5608],
...,
[0.0784, 0.0824, 0.0667, ..., 0.1961, 0.1843, 0.2235],
[0.0863, 0.0863, 0.0863, ..., 0.1804, 0.2000, 0.2235],
[0.0863, 0.0863, 0.0863, ..., 0.1843, 0.1882, 0.1569]]])

Getting the classes of train dataset

In [29]:

print(train_ds.classes)

['cats', 'dogs']

In [30]:

print(len(train_ds.classes))

Importing releveant libraries to visualize the images

In [31]:

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

matplotlib.rcParams['figure.facecolor'] = '#ffffff'

In [32]:
def show_example(img, label):
print('Label: ', train_ds.classes[label], "("+str(label)+")")

plt.imshow(img.permute(1, 2, 0))

In [33]:

show_example(*train_ds[0])

Label: cats (0)

In [34]:

show_example(*train_ds[6000])

Label: dogs (1)

In [ ]:

!pip install jovian --upgrade -q

In [ ]:

import jovian

In [ ]:
 [ ]:

jovian.commit(project=project_name)

In [35]:

batch_size = 128

Pytorch data loaders

In [36]:

from torch.utils.data import DataLoader

train_dl = DataLoader(train_ds, batch_size, shuffle=True, num_workers=4, pin_memory=True)
valid_dl = DataLoader(valid_ds, batch_size*2, num_workers=4, pin_memory=True)

In [37]:

from torchvision.utils import make_grid

def show_batch(dl):
for images, labels in dl:
fig, ax = plt.subplots(figsize=(14, 14))
ax.set_xticks([]); ax.set_yticks([])
ax.imshow(make_grid(images[:32], nrow=8).permute(1, 2, 0).clamp(0,1))
break

In [38]:

show_batch(train_dl)

In [39]:
In [39]:

import torch.nn as nn
import torch.nn.functional as F

In [40]:

conv = nn.Conv2d(3, 8, kernel_size=3, stride=1, padding=1)

In [41]:

simple_model = nn.Sequential(
nn.Conv2d(3, 8, kernel_size=3, stride=1, padding=1),
nn.MaxPool2d(2, 2)
)

In [42]:

for images, labels in train_dl:

print('images.shape:', images.shape)
out = simple_model(images)
print('out.shape:', out.shape)
break

images.shape: torch.Size([128, 3, 40, 40])

out.shape: torch.Size([128, 8, 20, 20])

Extending an ImageClassificationBase class which contains helper methods for training & validation

In [43]:

class ImageClassificationBase(nn.Module):
def training_step(self, batch):
images, labels = batch
out = self(images) # Generate predictions
loss = F.cross_entropy(out, labels) # Calculate loss
return loss

def validation_step(self, batch):

images, labels = batch
out = self(images) # Generate predictions
loss = F.cross_entropy(out, labels) # Calculate loss
acc = accuracy(out, labels) # Calculate accuracy
return {'val_loss': loss.detach(), 'val_acc': acc}

def validation_epoch_end(self, outputs):

batch_losses = [x['val_loss'] for x in outputs]
epoch_loss = torch.stack(batch_losses).mean() # Combine losses
batch_accs = [x['val_acc'] for x in outputs]
epoch acc = torch.stack(batch accs).mean() # Combine accuracies
 epoch_acc torch.stack(batch_accs).mean() # Combine accuracies
return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

def epoch_end(self, epoch, result):

print("Epoch [{}], train_loss: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}".format(
epoch, result['train_loss'], result['val_loss'], result['val_acc']))

def accuracy(outputs, labels):

_, preds = torch.max(outputs, dim=1)
return torch.tensor(torch.sum(preds == labels).item() / len(preds))

Using nn.Sequential to chain the layers and activations functions into a single network architecture.

In [44]:

class PetsModel(ImageClassificationBase):
def __init__(self):
super().__init__()
self.network = nn.Sequential(
nn.Conv2d(3, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool2d(2, 2), # output: 64 x 20 x 20

nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),

nn.ReLU(),
nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool2d(2, 2), # output: 128 x 10 x 10

nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),

nn.ReLU(),
nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool2d(2, 2), # output: 256 x 5 x 5

nn.Flatten(),
nn.Linear(256*5*5, 2048),
nn.ReLU(),
nn.Linear(2048, 1024),
nn.ReLU(),
nn.Linear(1024, 512),
nn.ReLU(),
nn.Linear(512, 2))

def forward(self, xb):

return self.network(xb)

In [45]:
 model = PetsModel()
model

Out[45]:
PetsModel(
(network): Sequential(
(0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): ReLU()
(2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(3): ReLU()
(4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(6): ReLU()
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(8): ReLU()
(9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(11): ReLU()
(12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(13): ReLU()
(14): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(15): Flatten(start_dim=1, end_dim=-1)
(16): Linear(in_features=6400, out_features=2048, bias=True)
(17): ReLU()
(18): Linear(in_features=2048, out_features=1024, bias=True)
(19): ReLU()
(20): Linear(in_features=1024, out_features=512, bias=True)
(21): ReLU()
(22): Linear(in_features=512, out_features=2, bias=True)
)
)

In [46]:

for images, labels in train_dl:

print('images.shape:', images.shape)
out = model(images)
print('out.shape:', out.shape)
print('out[0]:', out[0])
break

images.shape: torch.Size([128, 3, 40, 40])

out.shape: torch.Size([128, 2])
out[0]: tensor([-0.0010, 0.0071], grad_fn=<SelectBackward>)

Using GPU

In [47]:

def get_default_device():
""" i k if il bl l """
 """Pick GPU if available, else CPU"""
if torch.cuda.is_available():
return torch.device('cuda')
else:
return torch.device('cpu')

def to_device(data, device):

"""Move tensor(s) to chosen device"""
if isinstance(data, (list,tuple)):
return [to_device(x, device) for x in data]
return data.to(device, non_blocking=True)

class DeviceDataLoader():
"""Wrap a dataloader to move data to a device"""
def __init__(self, dl, device):
self.dl = dl
self.device = device

def __iter__(self):
"""Yield a batch of data after moving it to device"""
for b in self.dl:
yield to_device(b, self.device)

def __len__(self):
"""Number of batches"""
return len(self.dl)

In [48]:

device = get_default_device()
device

Out[48]:
device(type='cuda')

In [49]:

train_dl = DeviceDataLoader(train_dl, device)

val_dl = DeviceDataLoader(valid_dl, device)
to_device(model, device);

Training the model

In [50]:

@torch.no_grad()
def evaluate(model, val_loader):
model.eval()
outputs = [model.validation_step(batch) for batch in val_loader]
return model.validation_epoch_end(outputs)

def fit(epochs lr model train loader val loader opt func=torch optim SGD):
def fit(epochs, lr, model, train_loader, val_loader, opt_func torch.optim.SGD):
history = []
optimizer = opt_func(model.parameters(), lr)
for epoch in range(epochs):
# Training Phase
model.train()
train_losses = []
for batch in train_loader:
loss = model.training_step(batch)
train_losses.append(loss)
loss.backward()
optimizer.step()
optimizer.zero_grad()
# Validation phase
result = evaluate(model, val_loader)
result['train_loss'] = torch.stack(train_losses).mean().item()
model.epoch_end(epoch, result)
history.append(result)
return history

In [51]:

model = to_device(PetsModel(), device)

In [52]:

evaluate(model, val_dl)

Out[52]:
{'val_acc': 0.48828125, 'val_loss': 0.6933683156967163}

In [53]:

num_epochs = 10
opt_func = torch.optim.Adam
lr = 0.0001

In [54]:

jovian.reset()
jovian.log_hyperparams({
'num_epochs': num_epochs,
'opt_func': opt_func.__name__,
'batch_size': batch_size,
'lr': lr,
})

[jovian] Please enter your API key ( from https://jovian.ai/ ):

API KEY: ··········
[jovian] Hyperparams logged.
In [55]:

history = fit(num_epochs, lr, model, train_dl, val_dl, opt_func)

Epoch [0], train_loss: 0.6930, val_loss: 0.6900, val_acc: 0.5305

Epoch [1], train_loss: 0.6740, val_loss: 0.6563, val_acc: 0.6166
Epoch [2], train_loss: 0.6416, val_loss: 0.6115, val_acc: 0.6725
Epoch [3], train_loss: 0.6097, val_loss: 0.5878, val_acc: 0.7038
Epoch [4], train_loss: 0.5749, val_loss: 0.5710, val_acc: 0.7019
Epoch [5], train_loss: 0.5476, val_loss: 0.5637, val_acc: 0.7185
Epoch [6], train_loss: 0.5301, val_loss: 0.5660, val_acc: 0.7050
Epoch [7], train_loss: 0.5182, val_loss: 0.5146, val_acc: 0.7455
Epoch [8], train_loss: 0.4830, val_loss: 0.5228, val_acc: 0.7356
Epoch [9], train_loss: 0.4656, val_loss: 0.4771, val_acc: 0.7713

In [56]:

history = fit(num_epochs, 0.0005, model, train_dl, val_dl, opt_func)

Epoch [0], train_loss: 0.7072, val_loss: 0.6251, val_acc: 0.6519

Epoch [1], train_loss: 0.5921, val_loss: 0.5750, val_acc: 0.7026
Epoch [2], train_loss: 0.5701, val_loss: 0.5766, val_acc: 0.6916
Epoch [3], train_loss: 0.5303, val_loss: 0.5263, val_acc: 0.7216
Epoch [4], train_loss: 0.4919, val_loss: 0.4874, val_acc: 0.7811
Epoch [5], train_loss: 0.4607, val_loss: 0.4564, val_acc: 0.7893
Epoch [6], train_loss: 0.4275, val_loss: 0.5278, val_acc: 0.7552
Epoch [7], train_loss: 0.3889, val_loss: 0.4171, val_acc: 0.8105
Epoch [8], train_loss: 0.3651, val_loss: 0.4174, val_acc: 0.8184
Epoch [9], train_loss: 0.3299, val_loss: 0.4219, val_acc: 0.8261

In [ ]:

In [57]:

evaluate(model, val_dl)

Out[57]:
{'val_acc': 0.8261343240737915, 'val_loss': 0.42191097140312195}

In [58]:

jovian.log_metrics(train_loss=history[-1]['train_loss'],
val_loss=history[-1]['val_loss'],
val_acc=history[-1]['val_acc'])
[jovian] Metrics logged.

Plotting accuracies and losses of the model

In [59]:

def plot_accuracies(history):
accuracies = [x['val_acc'] for x in history]
plt.plot(accuracies, '-x')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.title('Accuracy vs. No. of epochs');

In [60]:

plot_accuracies(history)

In [61]:

def plot_losses(history):
train_losses = [x.get('train_loss') for x in history]
val_losses = [x['val_loss'] for x in history]
plt.plot(train_losses, '-bx')
plt.plot(val_losses, '-rx')
plt.xlabel('epoch')
plt.ylabel('loss')
plt.legend(['Training', 'Validation'])
plt.title('Loss vs. No. of epochs');

In [62]:

plot_losses(history)
In [64]:

jovian.commit(project=project_name)

[jovian] Detected Colab notebook...

[jovian] Uploading colab notebook to Jovian...
[jovian] Capturing environment..
[jovian] Attaching records (metrics, hyperparameters, dataset etc.)
[jovian] Committed successfully! https://jovian.ai/dinushasan/courseproject04
Out[64]:
'https://jovian.ai/dinushasan/courseproject04'

Testing the model with individual images

In [68]:

def predict_image(img, model):

# Convert to a batch of 1
xb = to_device(img.unsqueeze(0), device)
# Get predictions from model
yb = model(xb)
# Pick index with highest probability
_, preds = torch.max(yb, dim=1)
# Retrieve the class label
return train_ds.classes[preds[0].item()]

In [69]:

img, label = valid_ds[0]

plt.imshow(img.permute(1, 2, 0).clamp(0, 1))
print('Label:', train_ds.classes[label], ', Predicted:', predict_image(img, model))

Label: cats , Predicted: cats

In [81]:

img, label = valid_ds[500]

plt.imshow(img.permute(1, 2, 0).clamp(0, 1))
print('Label:', valid_ds.classes[label], ', Predicted:', predict_image(img, model))

Label: cats , Predicted: cats

In [72]:

img, label = valid_ds[1100]

plt.imshow(img.permute(1, 2, 0).clamp(0, 1))
print('Label:', train_ds.classes[label], ', Predicted:', predict_image(img, model))

Label: dogs , Predicted: dogs

In [73]:

img, label = valid_ds[1500]

plt.imshow(img.permute(1, 2, 0).clamp(0, 1))
 ( ( ) ( ))
print('Label:', train_ds.classes[label], ', Predicted:', predict_image(img, model))

Label: dogs , Predicted: dogs

Saving and committing the final model

In [74]:

jovian.commit(project=project_name)

[jovian] Detected Colab notebook...

[jovian] Uploading colab notebook to Jovian...
[jovian] Capturing environment..
[jovian] Attaching records (metrics, hyperparameters, dataset etc.)
[jovian] Committed successfully! https://jovian.ai/dinushasan/courseproject04
Out[74]:
'https://jovian.ai/dinushasan/courseproject04'

In [75]:

jovian.reset()
jovian.log_hyperparams({
'num_epochs': num_epochs,
'opt_func': opt_func.__name__,
'batch_size': batch_size,
'lr': lr,
})

[jovian] Hyperparams logged.

In [77]:

jovian.log_metrics(train_loss=history[-1]['train_loss'],
val_loss=history[-1]['val_loss'],
val_acc=history[-1]['val_acc'])

[jovian] Metrics logged.

In [82]:

jovian.commit(project=project_name)

[jovian] Detected Colab notebook...

[jovian] Uploading colab notebook to Jovian...
[jovian] Capturing environment..
[jovian] Attaching records (metrics, hyperparameters, dataset etc.)
[jovian] Committed successfully! https://jovian.ai/dinushasan/courseproject04
Out[82]:
'https://jovian.ai/dinushasan/courseproject04'

Summary and conclusion

As the final project of The course I chose to build an image classification model for a dataset that divides cats and
dogs. In this dataset there were two directories as training set and test set. As test set and validation test, I have used
the test set. As I learned throughout this course I used convolutional neural network model as the model and defined
the model by extending imageclassificationbase class.As the final accuracy, I have got about 82% . And as future work, I
hope to try improving the accuracy further to get a better model for identify cats and dogs correctly. And also, I'm
working on another project of identifying gem stones correctly using an image classification model. I hope to improve
both of these models to get a better accuracy.

Docs Forum Help hello@jovian.ai