Deep Learning

and
Pytorch Tutorial
Outline

 Environment Settings on Ubuntu 16.04

 Why Pytorch
 Installation
 Code explanation
 Convolutional neural networks
 Recurrent neural networks
 Variational autoencoder
 Generative adversarial networks
 Conclusion
Environment Settings on Ubuntu 16.04

 GPU drivers
 CUDA
 cuDNN
 Virtual Environment
 GPU drivers

 軟體目錄下新增ppa地址，然後更新資料庫資訊
 利用ppa來管理GPU driver’s version

$ sudo add-apt-repository ppa:graphics-drivers/ppa

$ sudo apt-get update
安裝 dirver 410版

 安裝前必須先關閉桌面環境 lightdm

$ sudo service lightdm stop

$ sudo apt-get install nvidia-410 nvidia-settings
 重開機後，輸入 nvidia-smi，安裝成功後會顯示顯卡資訊
 查看 driver 版本

$ cat /proc/driver/nvidia/version
nvidia-smi
下載 CUDA 10.0

 https://developer.nvidia.com/cuda-10.0-download-archive
 安裝前必須先關閉桌面環境 lightdm
$ sudo service lightdm stop
 接著安裝CUDA
$ sudo sh cuda_10.0.130_410.48_linux.run
下載 CUDA 10.0

 步驟中有選擇安裝 CUDA Sample，可以選擇安裝在哪

 將以下加入 ~/.bashrc : (Global setting, 設定在 /etc/profile 下)

export CUDA_HOME=/usr/local/cuda
export CUDA_ROOT=$CUDA_HOME
export CUDA_INC_DIR=$CUDA_HOME/include/
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64
PATH=${CUDA_HOME}/bin:${PATH}
export PATH
下載 CUDA 10.0

設定完後，記得 $ source ~/.bashrc 可以透過 $ nvcc --version 查看 cuda 是否安裝成功

驗證 CUDA 是否安裝成功

 安裝 CUDA Samples 目的主要是驗證 CUDA 安裝是否成功

 如果編譯 CUDA Samples 沒有 Errors (Warning 忽略), 就表示沒問題

# 切換到 NVIDIA CUDA-10.0 Samples

$ cd /usr/local/cuda/samples
# 編譯 CUDA Sample
$ make –j8
# 編譯完後，切換到 release 目錄
$ cd /usr/local/cuda/samples/bin/x86_64/linux/release
# 驗證是否成功, 會顯示 NVIDIA 顯卡資訊
$ ./deviceQuery
ldconfig

進入 /etc/ld.so.conf.d 下, 建立新文件 cuda.conf：

/usr/local/cuda/lib64
/lib
/usr/lib
/usr/lib32
Install cuDNN

 https://developer.nvidia.com/rdp/cudnn-download
 Install cuDNN

1.Install the runtime library, for example:

sudo dpkg -i libcudnn7_7.5.0.56-1+cuda10.0_amd64.deb
2.Install the developer library, for example:
sudo dpkg -i libcudnn7-dev_7.5.0.56-1+cuda10.0_amd64.deb
3.Install the code samples and the cuDNN Library User Guide, for example:
sudo dpkg -i libcudnn7-doc_7.5.0.56-1+cuda10.0_amd64.deb
測試 cuDNN

1.Copy the cuDNN sample to a writable path.

$ cp -r /usr/src/cudnn_samples_v7/ $HOME
2.Go to the writable path.
$ cd $HOME/cudnn_samples_v7/mnistCUDNN
3.Compile the mnistCUDNN sample.
$ make clean && make
4.Run the mnistCUDNN sample.
$ ./mnistCUDNN
Deep learning platforms

 There are many open source deep learning platforms.

 Theano
 Tensorflow
 Mxnet
 Torch
 Caffe
 Pytorch
Theano

 Python framework developed by the MILA Lab, University of Montreal

 Provide computational graph
 Pros
 Computational graph is a nice abstraction
 High level wrappers
 Cons
 Long compile time
 Fewer support
Tensorflow

 Supported by Google Brain

 Pros
 Strong support
 Distributed computation
 Implement on edge devices
 Cons
 Hard to learn
 A bit different from Numpy and Python
 Intstall tensorflow
 $ pip3 install tensorflow-gpu
Mxnet

 Adopted by Amazon Web Services, AWS

 Pros
 Less memory usage
 Variety of interfaces
 Cons
 Upgrade slowly
Torch

 Has its API in Lua

 Python version of Torch was open-sourced by Facebook in January, 2017
 Pros
 Flexibility and fast
 Faster compile times
Caffe

 Caffe was created by Yangqing Jia during his PhD at UC Berkeley

 Used by computer vision people because of the fast convolutional nets
 Pros
 Good for feedforward networks and image processing
 Good for fintuning existing networks
 Cons
 Not good for recurrent networks
Pytorch

 Supported by Facebook AI Research(FAIR)

 Has user coming from Torch and Caffe
 More and more projects are written in Pytorch
 More Python style compared to Tensorflow
 Easier to learn
Installation

 Quick start in https://pytorch.org/

 The stable version is Pytorch 1.0
 Can be installed in Linux, Mac and Windows systems
 Can be implemented in different python versions
 Supported by different version of Cuda
Prerequisites

 Operating systems, especially Linux distributions

 Python versions
 python/python3
 virtualenv
 Package manager
 Numpy if installed via pip
 CUDA if using GPU
Tensors

 Tensors are multi-dimensional arrays

 1-D tensors are vectors
 2-D tensors are matrices
 Similar to ndarrays in Numpy
 Can be accelerated by GPU
Uninitialized matrix
Randomly initialized matrix
Zero matrix
Tensor directly from data
Tensor based on existing tensor
Get the size
Operations

 Operations do calculations on tensors

 May have different syntaces
Addition syntax 1
Addition syntax 2
Addition with output tensor
Addition in-place
Indexing
Resizing
Retrieve one element tensor
Torch tensor to Numpy array
Numpy array to Torch tensor
CUDA Tensors
CNN experiment

 CIFAR 10 classification
 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck
 The images are 3x32x32
 Train an image classifier
Implementation steps

1. Load and normalizing the CIFAR 10 training and test datasets using
torchvision
2. Define a Convolutional Neural Network
3. Define a loss function
4. Train the network on the training data
5. Test the network on the test data
Loading data

 load with torchvision

output of torchvision are PILImage of range [0, 1],
transform to [-1, 1]
CNN
Loss function and optimizer
Train the network
Testing
Accuracy
Accuracy by classes
Training on GPU