Milestone Project 1

🍔👁 Food Vision Big™

 Where can you get help?
“If in doubt, run the code”

• Follow along with the code

• Try it for yourself
• Press SHIFT + CMD + SPACE to read the docstring
• Search for it
• Try again
• Ask (don’t forget the Discord chat!)
(yes, including the “dumb”
questions)
A new way of approaching things
 What we’re going to cover(broadly)
• Using TensorFlow Datasets to download and explore data (all of Food101)

• Creating a preprocessing function for our data

• Batching & preparing datasets for modelling (making them run fast)

• Setting up mixed precision training (faster model training)

What you’re going to cover…

• Building and training a feature extraction model

• Fine-tuning your feature extraction model to the beat the DeepFood paper 📈

• Evaluating your model results on TensorBoard

• Evaluating your model results by making and plotting predictions

👩🍳 👩🔬
(cook up lots of code!)

How:
 Serial experimentation
Everything you got

Xtra-Large dataset,
xtra-large model

Larger dataset, Larger dataset,

🍔👁 Food Vision Big™
small model larger model

Small dataset,
small model

Start small All in

 🍔👁 Food Vision: Dataset(s) we’re using
Note: For randomly selected data, the Food101 dataset was downloaded and modi ed using the Image Data Modi cation Notebook

Dataset Name Source Classes Training data Testing data

pizza_steak Food101 Pizza, steak (2)
Chicken curry, chicken wings,
fried rice, grilled salmon,
10_food_classes_1_percent Same as above
hamburger, ice cream, pizza,
ramen, steak, sushi (10)
750 images of pizza and steak 250 images of pizza and steak
(same as original Food101 (same as original Food101
dataset) dataset)
7 randomly selected images 250 images of each class
of each class (1% of original (same as original Food101
training data) dataset)

75 randomly selected images

10_food_classes_10_percent Same as above Same as above of each class (10% of original Same as above
training data)

750 images of each class

10_food_classes_100_percent Same as above Same as above (100% of original training Same as above
data)

75 images of each class (10% 250 images of each class

101_food_classes_10_percent Same as above All classes from Food101 (101) of the original Food101 (same as original Food101
training dataset) dataset)

250 images of each class

Same as above (from 75,750 images (750 images
“food101” All classes from Food101 (101) (same as original Food101
TensorFlow Datasets) per class)
dataset)

🍔👁 Food Vision Big™

fi
fi
Let’s code!
 What is TensorFlow Datasets (TFDS)?
TensorFlow Datasets is a place for prepared and ready-to-use machine learning datasets.

• Why use TensorFlow Datasets?

• Load data already in tensor format

• Practice on well established datasets (for many di erent problem types)

• Experiment with di erent modelling techniques on a consistent dataset

• Why not use TensorFlow Datasets?

• Datasets are static (do not change like real-world datasets)

ff
ff
 Batching & preparing datasets

Being prepared

n s o r 1
Te CPU
[[0, 13, 125…], [[0., 13., 125.…], Tensor 3
[1, 14, 200…], [1., 14., 200.…],
[2, 20, 210…], [2., 20., 210.…], Tensor 2
GPU
…, …, Tensor 1
Being computed on

Data Map Shu le Batch Prefetch

ff
 Prefetching

Source: Page 422 of Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow Book by Aurélien Géron
 What are callbacks?
• Callbacks are a tool which can add helpful functionality to your models during training,
evaluation or inference

• Some popular callbacks include:

Callback name Use case Code

Log the performance of multiple models and then view and compare
these models in a visual way on TensorBoard (a dashboard for
TensorBoard tf.keras.callbacks.TensorBoard()
inspecting neural network parameters). Helpful to compare the results
of di erent models on your data.
Save your model as it trains so you can stop training if needed and
Model checkpointing come back to continue o where you left. Helpful if training takes a tf.keras.callbacks.ModelCheckpoint()
long time and can't be done in one sitting.

Leave your model training for an arbitrary amount of time and have it
Early stopping stop training automatically when it ceases to improve. Helpful when tf.keras.callbacks.EarlyStopping()
you've got a large dataset and don't know how long training will take.
ff
ff
 What is mixed precision training?
Mixed precision training uses the combination of float32 and float16 data types to speed up
model training (can result in a 3x speed up on modern GPUs).

No mixed precision With mixed precision*

*3x speedup= maybe
Before and after mixed precision
1
2

No mixed precision
With mixed precision
 Feature extraction vs. Fine-tuning
Custom final layer
Stays same
gets trained to custom 10 10
data
Top layers get

…
unfrozen and fine-
Layer 235 Layer 235 tuned on custom data
Changes
(unfrozen)
Layer 234 Layer 234
Working architecture

…
(e.g. E cientNet) pre-
trained on ImageNet
Layer 2 Layer 2
Stays same
Bottom layers (ma
(frozen) y)
stay frozen
Input Layer Input Layer
…

…
Fine-t
Might change usual uning
ly req
more d uires
ata th
featur an
Custom dataset (e.g. 10 classes of food) extrac e
tion

Feature extraction Fine-tuning

ffi
 Feature extraction -> Fine-tuning
Custom final layer
gets trained to custom 101 101
data
Top layers get

…
unfrozen and fine-
Layer 235 Layer 235 tuned on custom data

Layer 234 Layer 234

Working architecture

…
(e.g. E cientNet) pre-
trained on ImageNet
Layer 2 Layer 2
Bottom layers (ma
y)
stay frozen
Input Layer Input Layer
…

…
Fine-t
usual uning
ly req
more d uires
ata th
featur an
extrac e
Custom dataset (e.g. 101 classes of food) tion

1. Feature extraction 2. Fine-tuning

ffi
 Performance of fast data fetching vs non-
data fetching

Prefetch and mixed precision No prefetch and no mixed precision

Time per epoch ~280-300s ~1127-1397s

Results from ne-tuning 🍔👁 Food Vision Big™ on Food101 dataset using an E cienetNetB0
backbone using a Google Colab Tesla T4 GPU.
fi
ffi
 Model we’ve created (for m i x e d p re c i s i on
training)
Input data Model Output
a x a c t iv a t i on
Softm
p e = f l oa t 3 2
with dty

GlobalAvgPool2D

Softmax
101
E cientNetB0 architecture.
Source: https://ai.googleblog.com/2019/05/e cientnet-improving-accuracy-and.html

- c o n n e c t e d
Fully
a s s if ie r l a y e r
(dense) cl
ffi
ffi
 (some common)

Classification evaluation methods

Key: tp = True Positive, tn = True Negative, fp = False Positive, fn = False Negative

Metric Name Metric Forumla Code When to use

tp + tn tf.keras.metrics.Accuracy() Default metric for classi cation

Accuracy Accuracy = or problems. Not the best for
tp + tn + fp + fn sklearn.metrics.accuracy_score() imbalanced classes.

tp tf.keras.metrics.Precision() Higher precision leads to less false

Precision Precision = or
tp + fp sklearn.metrics.precision_score() positives.

tp tf.keras.metrics.Recall() Higher recall leads to less false

Recall Recall = or
tp + fn sklearn.metrics.recall_score() negatives.

precision ⋅ recall Combination of precision and recall,

F1-score F1-score = 2 ⋅ sklearn.metrics.f1_score() usually a good overall metric for a
precision + recall classi cation model.

When comparing predictions to truth

Custom function labels to see where model gets
Confusion matrix NA or
sklearn.metrics.confusion_matrix() confused. Can be hard to use with
large numbers of classes.
fi
fi
 Finding the most wrong predictions
• A good way to inspect your model’s performance is to view the wrong predictions with the
highest prediction probability (or highest loss)
• Can reveal insights such as:
• Data issues (wrong labels, e.g. model is right, label is wrong)
• Confusing classes (get better/more diverse data)

Wrong label?

i n g c la s s
Co nf u s
 Anatomy of a confusion matrix
Confusion Matrix
False positives

0 99 (98.0%) 2 (2.0%) • True positive = model predicts 1 when truth is 1

• True negative = model predicts 0 when truth is 0
True Label

• False positive = model predicts 1 when truth is 0

• False negative = model predicts 0 when truth is 1
1 0 (0.0%) 99 (100.0%)

Correct predictions
t i v e s 0 1 (true positives,
e g a
False n Predicted Label true negatives)