91242019 ‘Tne Quest of Higher Accuracy for CNN Models -Towards Data Sclence The Quest of Higher Accuracy for CNN Models In this post, we will learn techniques to improve accuracy using data redesigning, hyper-parameter tuning and model optimization & ‘Swenand Mbalagi May 23 + 8 min reac Performance is key when it comes to deep learning models and it becomes an arduous task when you have limited resources. One of the important parameter to measure performance is ‘Accuracy’. This article is all about achieving higher accuracy with various hitpstowardsdatascience.convthe-quest-ot-higher-accuracy-for-enn-madels-4261567°1fat atospainor9 “Tne Quest of Higher Accuracy for CNN Models - Towards Data Science techniques related to data, algorithm tuning and model optimization. We will also see effects on accuracy after varying parameters like learning rate, batch size, number of training images, images size and number of trainable model parameters. After reading this article, you will know 1. Several Performance improvement techniques. 2. Various ways of loading image data, data pre-processing with easy to use data pipeline. 3. Customized CNN model to classify complex images. 4, Saving the model for future use and predicting the new images using the saved model. To start with, I will implement a code which | recently posted. (Link below) This is a TE/Keras implementation for Diabetic Retinopathy detection. This is a very basic version of CNN model with very less accuracy. I have made several changes to this code in order to achieve high accuracy as well as fast training/data load time. | will walk you through the best performing code later in this article. Convolutional Neural Network (CNN) implementation for Diabetic Retinopathy Detection with TF The aim of this tutorial is to develop automated detection system for diabetic retinopathy using CNN. This wes one of... medium.com The best performing code implemented using ImageDataGenerator, flow_from_dataframe, various data augmentation techniques, L2. regularization, batch normalization, one hot encoding on a subset of data considering the class weights in Keras. Lets take a look at all the performance improvement techniques. Techniques for performance improvement with hyper-parameter tuning 1. L2 regularization >> Penalizing complexity of the model, penalizing big weights. L2 regularization encourage weights to be small but doesn’t force them to exactly 0. hitpsowardsdatascience.convthe-quest-ot-highor-accuracy-for-enn.madels-<261567°1fat 20spainor9 “Tne Quest of Higher Accuracy for CNN Models - Towards Data Science 2. Learning Rate (LR) optimization > > Starting with a base LR and subsequently decreasing it for next epoch. 3. Batch Size >> Usually try with max batch size your GPU can handle, depends on the memory of the GPU, 4, Increase model capacity >> Increasing model depth (more number of layers) and width (number of filters in each convolution layer). Techniques for performance improvement with data redesigning 1. Increase image resolution (progressive resizing) > > From 128 x 128 x3 to 256x 256 x3 or to even higher size, 2. Random image rotations > > Change orientation of the image. 3. Random image shifts >> Useful when object is not at the center of the image. 4, Vertical and horizontal shift >> Randomly flip the image vertically or horizontally. (Algorithm should identify a glass whether its face up or face down) Techniques for performance improvement with model optimization 1, Fine tuning the model with subset data > > Dropping few data samples for some of the overly sampled data classes. 2. Class weights >> Used to train highly imbalanced (biased) database, class weights will give equal importance to all the classes during training. 3. Fine tuning the model with train data > > Use the model to predict on training data, retrain the model for the wrongly predicted images. Below table depicts the performance summary for various models created hitpsowardsdatascience.convthe-quest-t-higher-accuracy-for-enn.madele-<261567°1fat91242019 ‘Tne Quest of Higher Accuracy for CNN Models -Towards Data Science The last 4 columns show the highlights of all the performance improvement techniques we are applying while defining the model as well as during data processing. In general while doing performance measurement its recommended to change one variable at a time, the table shows only such runs where we see some performance improvement or if we have applied any new tuning technique. All the measurements are done on NVIDIA Quadro M400 GPU. This GPU has a 8 GB RAM. GPU is configured with CUDA 9.0, cuDNN 7.4, TF for GPU 1.8.0 on Ubuntu 16.04 instance. Some of the limitations of these measurements 1. Since the GPU had only 8 GB of memory, you need to carefully design the data pipeline and the model otherwise you will frequently get “Resource Exhausted Error”. You can easily circumvent this issue by reducing batch size, reducing image size while loading the data and changing number of trainable parameters in the model by removing few CNN layers or introducing a MAX pooling layer. During my experiments I have tried all possible max values. 2. Since majority of the codes use flow_from_dataframe functionality to load the images. It restricts us from doing feature-wise normalization and setting image mean to 0 by using feature-wise center (Keras functionality provided by hitpsowardsdatascience.convthe-quest-ot-highor-accuracy-for-enn.madels-<261567°1fat 4n0spainor9 “Tne Quest of Higher Accuracy for CNN Models - Towards Data Science ImageDataGenerator). In order to use these 2 features we need to fit the data which is in NumPy array (image.load_img) format. Unfortunately the VM instance which I am using takes long time to read the data into NumPy array. Now lets take a look at the code where we got 80% accuracy This code uses Linux packages such as mat-plot, NumPy, Keras, Pillow, HSpy and Tensor Flow. Ihave already downloaded the database from Kaggle. The database has several zip files, we need to unzip them into train/test folders containing respective images. All the labels for train images are provided in a separate CSV file, In the code below we will read a CSV file containing labels and name of the images. We need to do some sanity checks (adding .jpeg extension, delete all the images of size 0 KB, remove entries of deleted images from the data frame) before we move forward. Lets split the database into train and validation data and then convert the labels into one hot encoded labels. We are purposefully doing one hot encoding after splitting the data to retain the original label format to compare the labels to the predicted once later. (This will be useful when we retrain on wrongly predicted images) We will also calculate class weights since this is a highly imbalanced database. As I said before we will use ImageDataGenerator (to apply all image augmentation techniques, image rotations, width/height shift and re-scale) and flow_from_dataframe (to read all the images, set batch size and image size) Lets define the model. This model consist of 2 dimensional convolution layers with Relu as activation function, zero padding layers and max pooling layers. We will be also implementing batch normalization and L2 regularization. This is a Keras functional API implementation. Initialize the optimizer and loss function. We will be using stochastic gradient descent (SGD) with a learning rate of 0.01 which will be decaying every epoch. In practice SGD is 2 dimensional function but the neural network might have millions of parameters hitpsowardsdatascience.convthe-quest-ot-highor-accuracy-for-enn.madels-<261567°1fat sitospainor9 “Tne Quest of Higher Accuracy for CNN Models - Towards Data Science which means the millions of directions the loss function can move. If the loss function has a local minimum point/saddle point, there will be “zero gradient” and the gradient descent will get stuck. If we add momentum to SGD, the saddle points will get velocity to move forward. Adding nesterov momentum will take shorter path towards convergence. Callbacks will help us retain the best weights and biases we trained so far, since we will be training with multiple epochs and each epoch will go through entire training set. There is always a possibility that the model is over-fitting after certain number of epoch or accuracy may stop improving, the already saved weights will be helpful. The model we defined looks like below Tayer (type) Param # input_3 (InputLayer) convad_3 nv2D) (None, 254, 254, 16) 448 conv2d_34 (Conv2b) (None, 252, 252, 16) 2320 batch_normalization_17 (Bats (None, 252, 252, 1€) zero_padding2d_5 (ZeroPaddin (None, c convad_3& (Conv2D) (one, Teac convad_36 (ConvaDy Wone, 3248 batch _normalization_16 (Bate (None, 128 conv2d_37 (Conv2D) (one, 18436 conv2d_3@ (Cenv2D) (None, 3e928 batch_normalization_1S (Bate (None, 246, 246, 64) 256 max_pooling2d_5 (MaxFoclingz (None, Cc zAno_padding2d_6 (Zeraraddin (Nana, c convad_3 nv2D) (Nene, conv2d_4C (Conv2D) (None, 122, batch_normalization_20 (Bate (None, 122, hitpsowardsdatascience.convthe-quest-ot-highor-accuracy-for-enn.madels-<261567°1fat e1091242019 ‘The Quest of Higher Accuracy for CNN Models ~ “Towards Data Science conv2d_41_ (Conv2Dy (one, 120, 120, 64) T3792 conv2d_42 (Conv2D) 118, €4) 3e928 batch_normalization_2i (Bate 118, 64) 256 conv2d_43 (Conv2D) 116, 64) 3€928 convad_44_(ConvaDy Tia, €4) 3e928 batch_normalization_22 (Bate 114, 64) 256 mmax_pooling2d_6 (MaxFoclingz C convad_48 (ConvaDy 3e928 convid_46 (ConvaDy 3e928 batch_normalization, 256 convad_47 (ConvaDy convad_4@ (ConvaDy batch_normalization_2z4 (Batc (None, 50, £0, 128) £12 flatten_? (Flatten) Cc dense_7 (Dense) Gropout_é (Drepout) C dense_8 (Dense) 1056 pout_6 (Dropout) C (Dense) Trainable params: 10,851,925 Non-trainable params: 1,120 Its time to train the model on a GPU with tf.device( /device:GPu:0"): m= model. #it_generator( train_generator, epochs=5, hitpsowardsdatascience.convhe-quest-l-highor-accuracy-forenn-madel-4261567°1fat m091242019 ‘Tne Quest of Higher Accuracy for CNN Models -Towards Data Science vassuessvn_savenvaasu einer ater class_weight = class_weights, callbacks= Callbacks, verbose=1) train_model py hosted with © by GitHub View raw This is how the training will progress and you can see we achieved 80% training as well as validation accuracy. 16.25CE - Epoch 4/5 370/379 20347,.928 = a Epoch 5/5 370. We can also see how accuracy and losses are changing over time using below code This is how the plots look hitps:towardsdatascience.convthe-quest-ot-highor-accuracy-for-enn-madels-4261567°1fat ano91242019 ‘Tne Quest of Higher Accuracy for CNN Models -Towards Data Science In order to further improve the accuracy, we will be retraining on wrongly predicted training images. For this we will load the model that we just saved, later we will use the predict_generator to predict on the same training images. In this way we will be fine tuning model to specific set of images for which previous model miss predicted. Now we have a new database in df_filtered. we can use same old data pipeline and model definition we used earlier. This retraining allowed me to reach up to 82% accuracy, GitHub Repository All the iPython notebooks are available at https://github.com/swanandM/Diabetic- Retinopathy-Detection-with-TF What else we can do to improve the accuracy? Sometime in real world scenario its difficult to relay on single model. Different models can have different properties depending on number of layers it had, number of parameters it is trained on, etc. A particular model may not be able to predict best on particular set/type of images. In such cases we can combine multiple models to make a best prediction. In my next article, we will implement model ensembles by using some of the best performing models which we already created. Model ensembles can be used to take a vote from those models to make final prediction. hitpsowardsdatascience.convthe-quest-ot-highor-accuracy-for-enn.madels-<261567°1fat anno91242019 ‘Tne Quest of Higher Accuracy for CNN Models -Towards Data Science References 1. “Different ways of improving training accuracy” >> https://towardsdatascience.com/different-ways-of-improving-training-accuracy- 526db15a5b2 2, “Image augmentation for deep learning with Keras” >> https://machinelearningmastery.com/image-augmentation-deep-learning-keras/ 3. “Diagnosing diabetic retinopathy with deep learning” >> https://deepsense.ai/diagnosing-diabetic-retinopathy-with-deep-learning/ Deep Leerring Keres Imagedatagenerator Accuracy Cnn hitpsowardsdatascience.convthe-quest-ot-highor-accuracy-for-enn.madels-<261567°1fat 010