TMA01 Question 1 (45 marks)

Name: Parth Shah

PI: E395923X
In this question, you will create some neural network models that use a dataset of natural
scenes. The dataset is currently hosted on Kaggle and has been divided into separate
datasets for this TMA.
In this question we will only use two classes of the dataset (in question 2 you will use all six
classes).
In this question, you are not being asked to select a "best" model. Therefore, for the
purposes of the question, you'll be using the test dataset to explore how models behave
differently on seen and unseen data. Don't do this on models you want to deploy, or in
subsequent TMAs!

Completing the TMA

The tasks in this notebook can be addressed using the techniques discussed in the
Foundation and Block 1 of the module materials, and the associated notebooks.
You should be able to complete this question when you have completed the
practical activities in Block 1
You should look at the notebooks for Block 1 while working through this
question. You will find many useful examples in those notebooks which will help
you in this assignment.
Record all your activity and observations in this notebook. Insert additional notebook cells
as required. Remember to run each cell in sequence and to rerun cells if you make any
changes in earlier cells.
Include Markdown cells (like this one) liberally in your solutions, to describe what you are
doing. This will help your tutor give full credit for all you have done, and is invaluable in
reminding you what you were doing when you return to the TMA after a few days away.
Before you submit your notebook make sure you run all cells in order and check that you
get the results you expect. (It is not unknown to receive notebooks which don't work when
the cells are run in order.)
See the VLE for details of how to submit your completed notebook. You should submit only
this notebook file for this question.

Marks are based on process, not results

In this notebook, you will be asked to create, train, and evaluate several neural networks.
Training neural networks is inherently a stochastic process, based on the random
allocation of initial weights and the shuffled order of training examples. Therefore, your
results will differ from results generated by other students, and those generated by the
module team and presented in the tutor's marking guide.
The marks in this question are awarded solely on your ability to carry out the steps of
training and evaluation, not on any particular results you may achieve. There are no
thresholds for accuracy (or any other metric) you must achieve. You will gain credit
for carrying out the tasks specified in this question, including honest evaluations of how the
models perform.

Setup
This imports the required libraries.
import tensorflow as tf
from tensorflow.keras import layers, optimizers, metrics, Sequential,
utils

import os
import json

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Loading and preparing the dataset

This section of the notebook loads the dataset and makes it available for training.
First, we define some constants we will use later, such as the image size, and define some
metrics to use for model evaluation.
BATCH_SIZE = 64

IMAGE_SIZE = (150, 150, 3)

IMAGE_RESCALE = (IMAGE_SIZE[0], IMAGE_SIZE[1])

METRICS = [
lambda : tf.keras.metrics.TruePositives(name='tp'),
lambda : tf.keras.metrics.FalsePositives(name='fp'),
lambda : tf.keras.metrics.TrueNegatives(name='tn'),
lambda : tf.keras.metrics.FalseNegatives(name='fn'),

lambda : tf.keras.metrics.BinaryAccuracy(name='accuracy'),
lambda : tf.keras.metrics.Precision(name='precision'),
lambda : tf.keras.metrics.Recall(name='recall'),
lambda : tf.keras.metrics.AUC(name='auc'),
]
def fresh_metrics():
return [metric() for metric in METRICS]

The dataset contains several classes of image. For this notebook, we'll use just two then
convert them to a tensor of strings.
# Where to find the test data
base_dir = '/datasets/intel-multiclass/'
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'validation')
test_dir = os.path.join(base_dir, 'test')

label_list = ['buildings', 'sea']

desired_labels_tensor = tf.constant([l.encode('utf-8') for l in

label_list])
desired_labels_tensor

<tf.Tensor: shape=(2,), dtype=string, numpy=array([b'buildings',

b'sea'], dtype=object)>

Some dicts to convert between numbers and text labels.

# Human-sensible labels for the classification
class_names = {i: l for i, l in enumerate(sorted(label_list))}
class_numbers = {l: i for i, l in enumerate(sorted(label_list))}
num_classes = len(label_list)
class_names, class_numbers, num_classes

({0: 'buildings', 1: 'sea'}, {'buildings': 0, 'sea': 1}, 2)

Some Tensorflow functions for use in defining the datasets we'll use.
• A predicate that determines if a given image is in one of our desired classes.
• A function that finds the class number from a label (in the image file's directory
name)
• A function that loads an image, given a path.
def desired_class(image_path):
label_text = tf.strings.split(image_path, os.path.sep)[-2]
return tf.math.reduce_any(label_text == desired_labels_tensor)

def lookup_class_label(label_text):
return class_numbers[label_text.numpy().decode('utf-8')]

def load_image(image_path):
# read the image from disk, decode it, resize it, and scale the
# pixels intensities to the range [0, 1]
image = tf.io.read_file(image_path)
image = tf.io.decode_jpeg(image, channels=3)
image = tf.image.resize(image, IMAGE_RESCALE)
image /= 255.0
# grab the label and encode it
 label_text = tf.strings.split(image_path, os.path.sep)[-2]
label = tf.py_function(lookup_class_label, inp=[label_text],
Tout=tf.int32)
label = tf.ensure_shape(label, [])

# return the image and the integer encoded label

return (image, label)

These are all the image filenames we will use. Note the use of the filter to retain only the
two classes we're interested in.
train_dataset_files = tf.data.Dataset.list_files(
os.path.join(train_dir, '*', '*.jpg'),
shuffle=True).filter(desired_class)

train_data = train_dataset_files.map(load_image,
num_parallel_calls=tf.data.AUTOTUNE)
train_data = train_data.cache()
train_data = train_data.shuffle(20000)
train_data = train_data.batch(BATCH_SIZE)
train_data = train_data.prefetch(tf.data.AUTOTUNE)

validation_dataset_files = tf.data.Dataset.list_files(
os.path.join(validation_dir, '*', '*.jpg'),
shuffle=True).filter(desired_class)

validation_data = validation_dataset_files.map(load_image,
num_parallel_calls=tf.data.AUTOTUNE)
validation_data = validation_data.cache()
validation_data = validation_data.batch(BATCH_SIZE)
validation_data = validation_data.prefetch(tf.data.AUTOTUNE)

test_dataset_files = tf.data.Dataset.list_files(
os.path.join(test_dir, '*', '*.jpg'),
shuffle=True
).filter(desired_class)

test_data = test_dataset_files.map(load_image,
num_parallel_calls=tf.data.AUTOTUNE)
test_data = test_data.cache()
test_data = test_data.batch(BATCH_SIZE)
test_data = test_data.prefetch(tf.data.AUTOTUNE)

A check that we have what we're expecting: a dataset of batches of 150×150×3 images,
each paired with an integer label.
train_data

<PrefetchDataset element_spec=(TensorSpec(shape=(None, 150, 150, 3),

dtype=tf.float32, name=None), TensorSpec(shape=(None,),
dtype=tf.int32, name=None))>
Examining the data
Now we've loaded the data, we can look at some example images. We treat the dataset as
an iterator of Numpy arrays (each element is a batch of images and labels), and use that
interface to load a batch of images into memory. We then display them as a grid.
sample_imgs, sample_labels = train_data.as_numpy_iterator().next()

sample_imgs.shape, sample_labels.shape

((64, 150, 150, 3), (64,))

A batch of 64 images (each 150×150 pixels, 3 colour channels) and a batch of 64 labels.
plt.figure(figsize=(10,10))
for i in range(25):
plt.subplot(5,5,i+1)
plt.imshow(sample_imgs[i])
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.title(class_names[sample_labels[i]])
plt.show()
Jittered labels
The labels of the validation set, jittered. These may be useful for charts similar to those in
the Foundations notebooks.
test_labels = np.array(list(test_data.unbatch().map(lambda x, y:
y).as_numpy_iterator()))
test_labels.shape

(1353,)

jittered_labels = test_labels + (np.random.random(test_labels.shape) *

0.8)
jittered_labels.shape

(1353,)
Define and train a sample model
We now create and train a simple model using these datasets.
You should use this example as a basis for the models of your own that you create in this
question.
model = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=IMAGE_SIZE),
tf.keras.layers.Dense(1024, activation='sigmoid'),
tf.keras.layers.Dense(1, activation='sigmoid')
])

Note that we're using binary cross entropy as the loss function (as there are two classes).
Categorical cross-entropy is used when there are multiple classes, one-hot encoded.
opt = tf.keras.optimizers.SGD()
model.compile(optimizer=opt,
loss='binary_crossentropy',
metrics=['accuracy'])

history = model.fit(train_data,
validation_data=validation_data,
epochs=5)

Epoch 1/5
58/58 [==============================] - 2s 25ms/step - loss: 0.8806 -
accuracy: 0.5477 - val_loss: 0.6577 - val_accuracy: 0.5667
Epoch 2/5
58/58 [==============================] - 1s 14ms/step - loss: 0.6805 -
accuracy: 0.5796 - val_loss: 0.6932 - val_accuracy: 0.5556
Epoch 3/5
58/58 [==============================] - 1s 14ms/step - loss: 0.6518 -
accuracy: 0.6223 - val_loss: 0.6419 - val_accuracy: 0.6528
Epoch 4/5
58/58 [==============================] - 1s 14ms/step - loss: 0.6300 -
accuracy: 0.6361 - val_loss: 0.6766 - val_accuracy: 0.5611
Epoch 5/5
58/58 [==============================] - 1s 14ms/step - loss: 0.6191 -
accuracy: 0.6602 - val_loss: 0.6225 - val_accuracy: 0.6639

Save and reload the model and the training history.

model.save('q1_sample.h5')

with open('q1_sample_history.json', 'w') as f:

json.dump(history.history, f)

model = tf.keras.models.load_model('q1_sample.h5')
with open('q1_sample_history.json') as f:
sample_history = json.load(f)

Plot the training history.

acc = sample_history['accuracy']
val_acc = sample_history['val_accuracy']
loss = sample_history['loss']
val_loss = sample_history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'ro', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'ro', label='Training loss')

plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
Update the metrics used on the model and evaluate them on the validation data.
model.compile(metrics=fresh_metrics())
model.evaluate(validation_data, return_dict=True)

6/6 [==============================] - 0s 13ms/step - loss: 0.0000e+00

- tp: 136.0000 - fp: 72.0000 - tn: 103.0000 - fn: 49.0000 - accuracy:
0.6639 - precision: 0.6538 - recall: 0.7351 - auc: 0.7169

{'loss': 0.0,
'tp': 136.0,
'fp': 72.0,
'tn': 103.0,
'fn': 49.0,
'accuracy': 0.6638888716697693,
'precision': 0.6538461446762085,
'recall': 0.7351351380348206,
'auc': 0.7169111967086792}

You are now able to work on the tasks in this TMA question.

(a) (5 marks)
Referring to the sample model above:
• show how many trainable parameters it has
• show the shape of inputs to the model
• describe in words the shape of the inputs.
Give your answer below
Insert additional code and markdown cells as needed.
1a.)
• The input layer consists of 150 ×150 ×3=67 , 500 inputs.
• The second layer has 1 , 024 inputs, so there must be 67 , 500 ×1 , 024 weights,
resulting in 69 , 120 ,000 parameters so far.
• Each input in the second layer also has a bias, resulting in
69 , 120 ,000+ 1, 024=69 , 121 ,024 parameters so far.
• The third layer only has 1 output, which results in 1 , 024 weights and 1 bias, so the
total number of trainable parameters are 69 , 121, 024 +1 ,024 +1=69 , 122 ,049 .
The shape of the input is:
model.layers[0].input.shape

TensorShape([None, 150, 150, 3])

This is a 150 x 150 pixel image with 3 color channels. It can be represented by an array of
150 arrays, each of which has 150 arrays too, which contain 3 pixel values, each for red,
green and blue:
# [
# [[r1_1, g1_1, b1_1], ..., [r1_150, g1_150, b1_150]]
# ...
# [[r150_1, g150_1, b150_1], ..., [r_150_150, g_150_150,
b150_150]]
# ]

(b) (10 marks)

Using the sample model defined above, create and train a new classifier model of this
dataset. Your new model should be different from the sample one in the following way:
• Insert an additional Dense layer of 256 neurons between the two existing Dense
layers
• All Dense layers, except the last, should use ReLU activitation
• Training should use the SGD optimiser with a learning rate of 0.001
• Remember to use binary_crossentropy as the loss function
Train your modified model for 40 epochs. Show plots of how the accuracy and loss changed
over training, for both the training and validation datasets.
(You may wish to save your model and the training history.)
Give your answer below
Insert additional code and markdown cells as needed.
model2 = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=IMAGE_SIZE),
tf.keras.layers.Dense(1024, activation='relu'),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])

opt = tf.keras.optimizers.SGD(learning_rate=0.001)
model2.compile(
optimizer=opt,
loss='binary_crossentropy',
metrics=['accuracy']
)

history = model2.fit(
train_data,
validation_data=validation_data,
epochs=40
)

Epoch 1/40
58/58 [==============================] - 1s 17ms/step - loss: 0.6661 -
accuracy: 0.5994 - val_loss: 0.6241 - val_accuracy: 0.6139
Epoch 2/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5883 -
accuracy: 0.6913 - val_loss: 0.7870 - val_accuracy: 0.5417
Epoch 3/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5891 -
accuracy: 0.6959 - val_loss: 0.5806 - val_accuracy: 0.7111
Epoch 4/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5569 -
accuracy: 0.7264 - val_loss: 0.5696 - val_accuracy: 0.7083
Epoch 5/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5475 -
accuracy: 0.7267 - val_loss: 0.5653 - val_accuracy: 0.6972
Epoch 6/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5368 -
accuracy: 0.7372 - val_loss: 0.5521 - val_accuracy: 0.7250
Epoch 7/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5190 -
accuracy: 0.7580 - val_loss: 0.5488 - val_accuracy: 0.7222
Epoch 8/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5162 -
accuracy: 0.7586 - val_loss: 0.5722 - val_accuracy: 0.7333
Epoch 9/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5054 -
accuracy: 0.7678 - val_loss: 0.5411 - val_accuracy: 0.7500
Epoch 10/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5030 -
accuracy: 0.7686 - val_loss: 0.6288 - val_accuracy: 0.6194
Epoch 11/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4885 -
accuracy: 0.7870 - val_loss: 0.5286 - val_accuracy: 0.7444
Epoch 12/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4790 -
accuracy: 0.7910 - val_loss: 0.5218 - val_accuracy: 0.7528
Epoch 13/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4798 -
accuracy: 0.7862 - val_loss: 0.5209 - val_accuracy: 0.7444
Epoch 14/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4616 -
accuracy: 0.8081 - val_loss: 0.5493 - val_accuracy: 0.7694
Epoch 15/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4646 -
accuracy: 0.7989 - val_loss: 0.5086 - val_accuracy: 0.7806
Epoch 16/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4518 -
accuracy: 0.8054 - val_loss: 0.5053 - val_accuracy: 0.7750
Epoch 17/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4527 -
accuracy: 0.8035 - val_loss: 0.5029 - val_accuracy: 0.7750
Epoch 18/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4476 -
accuracy: 0.8129 - val_loss: 0.4999 - val_accuracy: 0.7667
Epoch 19/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4307 -
accuracy: 0.8221 - val_loss: 0.5047 - val_accuracy: 0.7417
Epoch 20/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4290 -
accuracy: 0.8227 - val_loss: 0.5169 - val_accuracy: 0.7833
Epoch 21/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4401 -
accuracy: 0.8191 - val_loss: 0.5783 - val_accuracy: 0.6750
Epoch 22/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4221 -
accuracy: 0.8327 - val_loss: 0.5015 - val_accuracy: 0.7528
Epoch 23/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4093 -
accuracy: 0.8413 - val_loss: 0.5170 - val_accuracy: 0.7750
Epoch 24/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4116 -
accuracy: 0.8375 - val_loss: 0.4852 - val_accuracy: 0.7889
Epoch 25/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4036 -
accuracy: 0.8467 - val_loss: 0.5073 - val_accuracy: 0.7306
Epoch 26/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3975 -
accuracy: 0.8500 - val_loss: 0.4848 - val_accuracy: 0.7722
Epoch 27/40
58/58 [==============================] - 1s 15ms/step - loss: 0.3879 -
accuracy: 0.8597 - val_loss: 0.5052 - val_accuracy: 0.7333
Epoch 28/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3920 -
accuracy: 0.8494 - val_loss: 0.5078 - val_accuracy: 0.7222
Epoch 29/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3951 -
accuracy: 0.8451 - val_loss: 0.4806 - val_accuracy: 0.7861
Epoch 30/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3743 -
accuracy: 0.8548 - val_loss: 0.4808 - val_accuracy: 0.7694
Epoch 31/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3818 -
accuracy: 0.8529 - val_loss: 0.4751 - val_accuracy: 0.8028
Epoch 32/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3686 -
accuracy: 0.8627 - val_loss: 0.4825 - val_accuracy: 0.7917
Epoch 33/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3702 -
accuracy: 0.8646 - val_loss: 0.4765 - val_accuracy: 0.7639
Epoch 34/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3675 -
accuracy: 0.8602 - val_loss: 0.4814 - val_accuracy: 0.7694
Epoch 35/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3509 -
accuracy: 0.8765 - val_loss: 0.4929 - val_accuracy: 0.7528
Epoch 36/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3498 -
accuracy: 0.8737 - val_loss: 0.5218 - val_accuracy: 0.7139
Epoch 37/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3547 -
accuracy: 0.8675 - val_loss: 0.5959 - val_accuracy: 0.6639
Epoch 38/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3654 -
accuracy: 0.8570 - val_loss: 0.4716 - val_accuracy: 0.7972
Epoch 39/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3580 -
accuracy: 0.8621 - val_loss: 0.4680 - val_accuracy: 0.7806
Epoch 40/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3583 -
accuracy: 0.8600 - val_loss: 0.4612 - val_accuracy: 0.7944

(c) (5 marks)
Comment on the plots of loss and accuracy, for both training and validation data, during the
training of this model. Do you think this model would benefit from additional training?
acc = history.history["accuracy"]
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'ro', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'ro', label='Training loss')

plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
Give your answer below
Insert additional code and markdown cells as needed.
print("Epoch 1 Training Accuracy: ", str(history.history['accuracy']
[0] * 100) + '%')
print("Epoch 5 Training Accuracy: ", str(history.history['accuracy']
[4] * 100) + '%')
print("Epoch 40 Training Accuracy: ", str(history.history['accuracy']
[39] * 100) + '%')

Epoch 1 Training Accuracy: 59.9351167678833%

Epoch 5 Training Accuracy: 72.66829013824463%
Epoch 40 Training Accuracy: 85.99621653556824%

Some epochs saw a worse training accuracy compared to the previous epoch, but overall,
the training accuracy gradually improved as the epochs increased, reaching 88% in the
final epoch. It is worth mentioning that the rate at which the training accuracy improved
slowed down significantly after 5 epochs — the training accuracy improved by ~13%
between the first and fifth epoch, but took a further 35 epochs to improve by another
~13%:
Although the validation accuracy gradually increased over the 40 epochs, it was much more
volatile. For example:
print("Epoch 37 Validation Accuracy: ",
str(history.history['val_accuracy'][36] * 100) + '%')
print("Epoch 38 Validation Accuracy: ",
str(history.history['val_accuracy'][37] * 100) + '%')
Epoch 37 Validation Accuracy: 66.38888716697693%
Epoch 38 Validation Accuracy: 79.72221970558167%

The validation accuracy between the 37th and 38th epoch decreased by ~13%. Like the
training accuracy, the validation accuracy also saw diminishing returns as the epochs
increased, settling between 75% - 80% from the 10th epoch.
The training loss decreased at a relatively stable pace, reducing from ~65% to ~35% over
the 40 epochs. The validation loss however suffered from diminishing returns, settling
between ~45% to ~50% from the 15th epoch.
To conclude, the training accuracy is already very good, and despite the training loss is
35%, it would most likely decrease significantly if the model is trained further. However,
the validation accuracy and loss have flattened and do not show any signs of improvement.
Therefore, training the model further would just lead to the training loss/accuracy
increasing to perfect-like values and the validation loss/accuracy remaining constant.
There is not much benefit of training the model further, as the validation performance, and
therefore, real-world performance too, may not increase by much (if at all). Training the
model further also increases the risk of over-fitting.

(d) (10 marks)

Recompile the model from part (b) above to use the metrics defined by the
fresh_metrics function defined above.

Evaluate the model, using these metrics, on all three of the train, validation, and test
datasets.
Use that model to generate predicted classes for all elements in the test dataset. Plot a
scatter chart of the predicted results with the actual results (defined above as either
test_labels or jittered_labels.)

Comment on these results.

Give your answer below

Insert additional code and markdown cells as needed.
model2.compile(metrics=fresh_metrics())

train_metric = model2.evaluate(train_data, return_dict=True)

validation_metric = model2.evaluate(validation_data, return_dict=True)
test_metric = model2.evaluate(test_data, return_dict=True)

58/58 [==============================] - 1s 10ms/step - loss:

0.0000e+00 - tp: 1666.0000 - fp: 131.0000 - tn: 1665.0000 - fn:
237.0000 - accuracy: 0.9005 - precision: 0.9271 - recall: 0.8755 -
auc: 0.9655
6/6 [==============================] - 0s 11ms/step - loss: 0.0000e+00
- tp: 141.0000 - fp: 30.0000 - tn: 145.0000 - fn: 44.0000 - accuracy:
0.7944 - precision: 0.8246 - recall: 0.7622 - auc: 0.8670
22/22 [==============================] - 0s 9ms/step - loss:
0.0000e+00 - tp: 549.0000 - fp: 92.0000 - tn: 565.0000 - fn: 147.0000
- accuracy: 0.8234 - precision: 0.8565 - recall: 0.7888 - auc: 0.8947

train_metric

{'loss': 0.0,
'tp': 1666.0,
'fp': 131.0,
'tn': 1665.0,
'fn': 237.0,
'accuracy': 0.9005136489868164,
'precision': 0.9271007180213928,
'recall': 0.8754597902297974,
'auc': 0.9654821753501892}

validation_metric

{'loss': 0.0,
'tp': 141.0,
'fp': 30.0,
'tn': 145.0,
'fn': 44.0,
'accuracy': 0.7944444417953491,
'precision': 0.8245614171028137,
'recall': 0.7621621489524841,
'auc': 0.8669652938842773}

test_metric

{'loss': 0.0,
'tp': 549.0,
'fp': 92.0,
'tn': 565.0,
'fn': 147.0,
'accuracy': 0.823355495929718,
'precision': 0.8564742803573608,
'recall': 0.7887930870056152,
'auc': 0.8947311043739319}

The accuracy, precision, and recall values are the lowest in the validation set and highest in
the training set, with the testing set falling in between, but leaning more closely to the
validation set.
The fact that the accuracy, precision, and recall values in the training set are higher than
the values in the testing/validation set is to be expected. This is because the model is tuned
to optimise the training data. Some over-fitting is inevitably present.
model2_input = np.array(list(test_data.unbatch().map(lambda x, y:
x).as_numpy_iterator()))
expected_output = np.array(list(test_data.unbatch().map(lambda x, y:
y).as_numpy_iterator()))

model2_predictions = model2.predict(model2_input)[:, 0]

is_sea = np.count_nonzero((model2_predictions >= 0.5) &

(expected_output >= 0.5))
is_building = np.count_nonzero((model2_predictions < 0.5) &
(expected_output < 0.5))

plt.figure(figsize=(10, 10))

misclassifications = 0
for i in range(25):
plt.subplot(5, 5, i + 1)
plt.imshow(model2_input[i])
plt.xticks([])
plt.yticks([])
plt.grid(False)
if model2_predictions[i] < 0.5 and expected_output[i] >= 0.5:
print(f"Misclassification at row {(i // 5) + 1}, column {(i %
5) + 1} - expecting {class_names[1]}.")
misclassifications += 1
elif model2_predictions[i] >= 0.5 and expected_output[i] < 0.5:
print(f"Misclassification at row {(i // 5) + 1}, column {(i %
5) + 1} - expecting {class_names[0]}.")
misclassifications += 1
plt.title(class_names[1 if model2_predictions[i] >= 0.5 else 0])
print(f"{(misclassifications/25) * 100}% of data points were
misclassified.")
plt.show()

43/43 [==============================] - 0s 5ms/step

Misclassification at row 1, column 1 - expecting buildings.
Misclassification at row 1, column 5 - expecting sea.
Misclassification at row 2, column 1 - expecting buildings.
Misclassification at row 2, column 2 - expecting sea.
Misclassification at row 2, column 5 - expecting buildings.
Misclassification at row 4, column 3 - expecting buildings.
Misclassification at row 4, column 5 - expecting buildings.
28.000000000000004% of data points were misclassified.
plt.scatter(jittered_labels, model2_predictions)
plt.xlabel('Label')
plt.ylabel('Prediction');
plt.axline((0, 0.5), (2, 0.5), c="tab:orange", label="Threshold")
plt.legend()

<matplotlib.legend.Legend at 0x7f81987d5370>
print(f"False Negatives: {test_metric['fn']}")
print(f"False Positives: {test_metric['fp']}")

False Negatives: 147.0

False Positives: 92.0

As can be seen from the graph above, there are more points in the lower right quadrant
(false negatives) compared to the top left quadrant (false positives).
• points in the scatter chart are clustered at the more extreme values for each class, with few
with predicted values near 0.5. This indicates that the model can classify well.
• for the test data, precision, accuracy and recall are all high and similar, indicating that there
(e) (5 marks) isn’t a systematic bias in the results and that both classes are classified correctly.
• for the training data, the values of all metrics are extremely high, indicating that the model
has learnt most features of the training data.
Referring to the evaluations in part (d), explain why it is important to have separate
datasets for training and testing ML models. What would be the implications if the same
dataset was used for both roles? When should testing data be used? What is the advantage
of having a separate validation dataset?

Give your answer below

Insert additional code and markdown cells as needed.

Implications if same dataset is used for both training and testing

If the same dataset is used for both training and testing, it becomes difficult to evaluate the
performance of the model. Neural Networks are trained to minimise the loss between the
input data and the expected output, and so it is obvious that the evaluation results would
appear to be unnaturally good if the same dataset is used for both training and testing. This
may not be an accurate representation of the actual performance of the model.
When testing data should be used
The model should first be trained using training data. The testing data should only be
touched at the final model evaluation stage. This is to allow for a good approximation of
how the model would perform with real-world data.

Advantage of having a separate validation dataset

Without a validation dataset, a model is improved by looking at the results from the testing
data and using that to fine-tune parameters in a way which provides better results to the
testing data. However, this means that some of the testing data is being converted into
training data, defeating the purpose of splitting the testing and training data in the first
place. When a validation dataset is used, this problem can be avoided, as the validation data
can be used to fine-tune the model parameters, leaving the training data only to be used at
the final step. Validation data is useful during model development, as a source of unseen data for validation
of the model.However, it is still possible for the model development to focus on optimising performance on
the validation data, so testing data is still needed for final evaluation.

(f) (10 marks)

Starting from the model defined in part (b), create and train two new models.
• Model S should have a different number of neurons in its hidden layer (you decide
how many)
• Model L should use a different value for the learning rate with the SGD optimizer
(you decide what it should be).
Apart from these changes, the model creation and training should be identical to what you
did in part (b) above. Comment on any relevant observations you make of the training.
Evaluate your new models and compare their performance with each other and what you
found in part (d) above. Comment on your results.
As a reminder, note that there are no marks in this TMA for whether your
modified models work better or worse then the original. The marks in this
question are purely for the experiment and commenting on the results.

Give your answer below

Insert additional code and markdown cells as needed.
modelS = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=IMAGE_SIZE),
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])

opt = tf.keras.optimizers.SGD(learning_rate=0.001)

modelS.compile(
 optimizer=opt,
loss='binary_crossentropy',
metrics=['accuracy']
)

historyS = modelS.fit(
train_data,
validation_data=validation_data,
epochs=40
)

acc = historyS.history["accuracy"]
val_acc = historyS.history['val_accuracy']
loss = historyS.history['loss']
val_loss = historyS.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'ro', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'ro', label='Training loss')

plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

modelS.compile(metrics=fresh_metrics())
train_metric = modelS.evaluate(train_data, return_dict=True)
validation_metric = modelS.evaluate(validation_data, return_dict=True)
test_metric = modelS.evaluate(test_data, return_dict=True)

Epoch 1/40
58/58 [==============================] - 1s 10ms/step - loss: 0.6722 -
accuracy: 0.6018 - val_loss: 0.6674 - val_accuracy: 0.6306
Epoch 2/40
58/58 [==============================] - 0s 8ms/step - loss: 0.6510 -
accuracy: 0.6418 - val_loss: 0.6903 - val_accuracy: 0.5444
Epoch 3/40
58/58 [==============================] - 0s 8ms/step - loss: 0.6348 -
accuracy: 0.6559 - val_loss: 0.6398 - val_accuracy: 0.6417
Epoch 4/40
58/58 [==============================] - 0s 8ms/step - loss: 0.6189 -
accuracy: 0.6726 - val_loss: 0.6460 - val_accuracy: 0.5889
Epoch 5/40
58/58 [==============================] - 0s 8ms/step - loss: 0.6053 -
accuracy: 0.6899 - val_loss: 0.6245 - val_accuracy: 0.6250
Epoch 6/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5957 -
accuracy: 0.6907 - val_loss: 0.6135 - val_accuracy: 0.6583
Epoch 7/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5852 -
accuracy: 0.7086 - val_loss: 0.6105 - val_accuracy: 0.6667
Epoch 8/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5757 -
accuracy: 0.7142 - val_loss: 0.6021 - val_accuracy: 0.6667
Epoch 9/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5710 -
accuracy: 0.7105 - val_loss: 0.6086 - val_accuracy: 0.6500
Epoch 10/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5587 -
accuracy: 0.7248 - val_loss: 0.5897 - val_accuracy: 0.6944
Epoch 11/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5543 -
accuracy: 0.7305 - val_loss: 0.5893 - val_accuracy: 0.6778
Epoch 12/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5461 -
accuracy: 0.7364 - val_loss: 0.5794 - val_accuracy: 0.6972
Epoch 13/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5413 -
accuracy: 0.7324 - val_loss: 0.5853 - val_accuracy: 0.6667
Epoch 14/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5344 -
accuracy: 0.7364 - val_loss: 0.5725 - val_accuracy: 0.7056
Epoch 15/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5250 -
accuracy: 0.7502 - val_loss: 0.5803 - val_accuracy: 0.6833
Epoch 16/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5254 -
accuracy: 0.7540 - val_loss: 0.5679 - val_accuracy: 0.7056
Epoch 17/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5165 -
accuracy: 0.7632 - val_loss: 0.5689 - val_accuracy: 0.6861
Epoch 18/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5128 -
accuracy: 0.7591 - val_loss: 0.5605 - val_accuracy: 0.7083
Epoch 19/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5103 -
accuracy: 0.7583 - val_loss: 0.5625 - val_accuracy: 0.7056
Epoch 20/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5049 -
accuracy: 0.7597 - val_loss: 0.5684 - val_accuracy: 0.7139
Epoch 21/40
58/58 [==============================] - 0s 8ms/step - loss: 0.5019 -
accuracy: 0.7705 - val_loss: 0.5815 - val_accuracy: 0.6917
Epoch 22/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4986 -
accuracy: 0.7637 - val_loss: 0.5663 - val_accuracy: 0.6778
Epoch 23/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4942 -
accuracy: 0.7710 - val_loss: 0.5566 - val_accuracy: 0.7056
Epoch 24/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4878 -
accuracy: 0.7848 - val_loss: 0.5551 - val_accuracy: 0.7056
Epoch 25/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4868 -
accuracy: 0.7799 - val_loss: 0.5471 - val_accuracy: 0.7167
Epoch 26/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4809 -
accuracy: 0.7805 - val_loss: 0.5468 - val_accuracy: 0.7111
Epoch 27/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4796 -
accuracy: 0.7837 - val_loss: 0.5588 - val_accuracy: 0.7222
Epoch 28/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4743 -
accuracy: 0.7821 - val_loss: 0.5473 - val_accuracy: 0.7194
Epoch 29/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4781 -
accuracy: 0.7818 - val_loss: 0.5427 - val_accuracy: 0.7167
Epoch 30/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4646 -
accuracy: 0.7975 - val_loss: 0.5401 - val_accuracy: 0.7194
Epoch 31/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4650 -
accuracy: 0.7924 - val_loss: 0.5351 - val_accuracy: 0.7306
Epoch 32/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4595 -
accuracy: 0.8070 - val_loss: 0.5419 - val_accuracy: 0.7333
Epoch 33/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4548 -
accuracy: 0.8043 - val_loss: 0.5391 - val_accuracy: 0.7139
Epoch 34/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4551 -
accuracy: 0.8029 - val_loss: 0.5464 - val_accuracy: 0.7194
Epoch 35/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4486 -
accuracy: 0.8072 - val_loss: 0.5309 - val_accuracy: 0.7167
Epoch 36/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4467 -
accuracy: 0.8045 - val_loss: 0.5312 - val_accuracy: 0.7222
Epoch 37/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4424 -
accuracy: 0.8094 - val_loss: 0.5359 - val_accuracy: 0.7250
Epoch 38/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4385 -
accuracy: 0.8151 - val_loss: 0.5268 - val_accuracy: 0.7194
Epoch 39/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4401 -
accuracy: 0.8135 - val_loss: 0.5379 - val_accuracy: 0.7333
Epoch 40/40
58/58 [==============================] - 0s 8ms/step - loss: 0.4334 -
accuracy: 0.8143 - val_loss: 0.5538 - val_accuracy: 0.7000
58/58 [==============================] - 1s 8ms/step - loss:
0.0000e+00 - tp: 1752.0000 - fp: 588.0000 - tn: 1208.0000 - fn:
151.0000 - accuracy: 0.8002 - precision: 0.7487 - recall: 0.9207 -
auc: 0.9003
6/6 [==============================] - 0s 10ms/step - loss: 0.0000e+00
- tp: 160.0000 - fp: 83.0000 - tn: 92.0000 - fn: 25.0000 - accuracy:
0.7000 - precision: 0.6584 - recall: 0.8649 - auc: 0.8135
22/22 [==============================] - 0s 8ms/step - loss:
0.0000e+00 - tp: 616.0000 - fp: 292.0000 - tn: 365.0000 - fn: 80.0000
- accuracy: 0.7251 - precision: 0.6784 - recall: 0.8851 - auc: 0.8408

The number of neurons in both middle layers was changed to 32. This significantly reduced
the training time, from ~1 second per step to ~8 ms per step. This is attributed to there
being exponentially fewer weights and biases to update upon every learning iteration.
Surprisingly, reducing the number of neurons only reduced the test-set accuracy slightly
(77% compared to ~81% seen in 1(d)).
A possible explanation for why good accuracy is retained when reducing the number of
neurons significantly could be the fact that there are fewer features distinguishing an
image of a sea and an image of a building, compared to something more complex like an
image of a cat or a dog. An image of a sea generally has more bluer colors and has less
granular detail (smoothness of water compared to a building facade with many windows).
Therefore, as images of seas and buildings could theoretically be more linearly separable,
good results still may be achieved with fewer neurons.
The compromise in accuracy may be worth the gain in computation time for some non-
critical applications.
modelL = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=IMAGE_SIZE),
tf.keras.layers.Dense(1024, activation='relu'),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])

opt = tf.keras.optimizers.SGD(learning_rate=0.03)

modelL.compile(
optimizer=opt,
loss='binary_crossentropy',
metrics=['accuracy']
)

historyL = modelL.fit(
train_data,
validation_data=validation_data,
epochs=40
)
acc = historyL.history["accuracy"]
val_acc = historyL.history['val_accuracy']
loss = historyL.history['loss']
val_loss = historyL.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'ro', label='Training acc')

plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'ro', label='Training loss')

plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

modelS.compile(metrics=fresh_metrics())
train_metric = modelS.evaluate(train_data, return_dict=True)
validation_metric = modelS.evaluate(validation_data, return_dict=True)
test_metric = modelS.evaluate(test_data, return_dict=True)

Epoch 1/40
58/58 [==============================] - 1s 16ms/step - loss: 1.2910 -
accuracy: 0.5407 - val_loss: 0.6742 - val_accuracy: 0.5306
Epoch 2/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6697 -
accuracy: 0.5783 - val_loss: 0.6830 - val_accuracy: 0.5361
Epoch 3/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6594 -
accuracy: 0.5983 - val_loss: 0.6521 - val_accuracy: 0.5917
Epoch 4/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6441 -
accuracy: 0.6150 - val_loss: 0.6331 - val_accuracy: 0.6167
Epoch 5/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6634 -
accuracy: 0.6010 - val_loss: 0.6518 - val_accuracy: 0.6083
Epoch 6/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6326 -
accuracy: 0.6459 - val_loss: 0.6161 - val_accuracy: 0.6806
Epoch 7/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6072 -
accuracy: 0.6718 - val_loss: 0.6524 - val_accuracy: 0.5972
Epoch 8/40
58/58 [==============================] - 1s 14ms/step - loss: 0.6009 -
accuracy: 0.6734 - val_loss: 0.6196 - val_accuracy: 0.6167
Epoch 9/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5946 -
accuracy: 0.6799 - val_loss: 0.6670 - val_accuracy: 0.6222
Epoch 10/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5633 -
accuracy: 0.7045 - val_loss: 0.5932 - val_accuracy: 0.6528
Epoch 11/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5585 -
accuracy: 0.7094 - val_loss: 0.6176 - val_accuracy: 0.6500
Epoch 12/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5401 -
accuracy: 0.7313 - val_loss: 0.5841 - val_accuracy: 0.6722
Epoch 13/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5368 -
accuracy: 0.7242 - val_loss: 0.5855 - val_accuracy: 0.6833
Epoch 14/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5351 -
accuracy: 0.7326 - val_loss: 0.5699 - val_accuracy: 0.7028
Epoch 15/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5292 -
accuracy: 0.7399 - val_loss: 0.5937 - val_accuracy: 0.7222
Epoch 16/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5097 -
accuracy: 0.7494 - val_loss: 0.5774 - val_accuracy: 0.7028
Epoch 17/40
58/58 [==============================] - 1s 14ms/step - loss: 0.5106 -
accuracy: 0.7410 - val_loss: 0.7333 - val_accuracy: 0.5944
Epoch 18/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4934 -
accuracy: 0.7591 - val_loss: 0.6937 - val_accuracy: 0.6611
Epoch 19/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4866 -
accuracy: 0.7648 - val_loss: 0.6245 - val_accuracy: 0.6889
Epoch 20/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4896 -
accuracy: 0.7610 - val_loss: 0.7110 - val_accuracy: 0.6472
Epoch 21/40
58/58 [==============================] - 1s 15ms/step - loss: 0.4754 -
accuracy: 0.7753 - val_loss: 0.5777 - val_accuracy: 0.7028
Epoch 22/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4450 -
accuracy: 0.7978 - val_loss: 0.5889 - val_accuracy: 0.7083
Epoch 23/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4402 -
accuracy: 0.7940 - val_loss: 0.6355 - val_accuracy: 0.6611
Epoch 24/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4657 -
accuracy: 0.7791 - val_loss: 0.5875 - val_accuracy: 0.7306
Epoch 25/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4589 -
accuracy: 0.7853 - val_loss: 0.5737 - val_accuracy: 0.6944
Epoch 26/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4413 -
accuracy: 0.7924 - val_loss: 0.5628 - val_accuracy: 0.7306
Epoch 27/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4338 -
accuracy: 0.7983 - val_loss: 0.5584 - val_accuracy: 0.7250
Epoch 28/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4484 -
accuracy: 0.7940 - val_loss: 0.7197 - val_accuracy: 0.6583
Epoch 29/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4451 -
accuracy: 0.7929 - val_loss: 0.5591 - val_accuracy: 0.7278
Epoch 30/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4229 -
accuracy: 0.8102 - val_loss: 0.7303 - val_accuracy: 0.6000
Epoch 31/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4278 -
accuracy: 0.8010 - val_loss: 0.6060 - val_accuracy: 0.7139
Epoch 32/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4228 -
accuracy: 0.8156 - val_loss: 0.5679 - val_accuracy: 0.7278
Epoch 33/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3744 -
accuracy: 0.8378 - val_loss: 0.6726 - val_accuracy: 0.7139
Epoch 34/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4053 -
accuracy: 0.8275 - val_loss: 0.5585 - val_accuracy: 0.7389
Epoch 35/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4169 -
accuracy: 0.8064 - val_loss: 0.5464 - val_accuracy: 0.7222
Epoch 36/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4946 -
accuracy: 0.7716 - val_loss: 0.5534 - val_accuracy: 0.7056
Epoch 37/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4370 -
accuracy: 0.7945 - val_loss: 0.5461 - val_accuracy: 0.7167
Epoch 38/40
58/58 [==============================] - 1s 14ms/step - loss: 0.4342 -
accuracy: 0.8097 - val_loss: 0.5664 - val_accuracy: 0.7250
Epoch 39/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3885 -
accuracy: 0.8216 - val_loss: 0.5505 - val_accuracy: 0.7500
Epoch 40/40
58/58 [==============================] - 1s 14ms/step - loss: 0.3784 -
accuracy: 0.8245 - val_loss: 0.5495 - val_accuracy: 0.7417
58/58 [==============================] - 1s 8ms/step - loss:
0.0000e+00 - tp: 1752.0000 - fp: 588.0000 - tn: 1208.0000 - fn:
151.0000 - accuracy: 0.8002 - precision: 0.7487 - recall: 0.9207 -
auc: 0.9003
6/6 [==============================] - 0s 10ms/step - loss: 0.0000e+00
- tp: 160.0000 - fp: 83.0000 - tn: 92.0000 - fn: 25.0000 - accuracy:
0.7000 - precision: 0.6584 - recall: 0.8649 - auc: 0.8135
22/22 [==============================] - 0s 8ms/step - loss:
0.0000e+00 - tp: 616.0000 - fp: 292.0000 - tn: 365.0000 - fn: 80.0000
- accuracy: 0.7251 - precision: 0.6784 - recall: 0.8851 - auc: 0.8408

The learning rate was changed from 0.001 to 0.03. This led to a decrease in accuracy and
more volatile jumps in training loss and accuracy.
A key motivation behind increasing the learning rate by such a high margin was to discover
if the results from 1(d) were constrained by the Stochastic Gradient Descent algorithm
getting stuck in a local minimum.
A higher learning rate allows the algorithm to be less concerned about deeply honing into
anything that appears to lead to a minimum, and more freely explore the nearby space,
which could potentially lead to a lower minimum. This is why the training loss and
accuracy is more volatile than previously.
However, since worse results for accuracy and precision were achieved on the testing set
compared to previously, it appears that the learning rate is perhaps too high. It could be
interesting to experiment with a learning rate lower than 0.001 to see whether better
accuracy can be achieved. This would require more epochs though, which would increase
computation time.

I am expecting comments on training and validation data sets and any evidence of over fitting.
I am also expecting comments on accuracy, precision and recall scores as well.