Deep Learning for Image Classification

© All Rights Reserved

45 views

Deep Learning for Image Classification

© All Rights Reserved

- Deep Learning Networks for New Computer Vision Technologies
- MachineLearningResourceGuide.pdf
- Neural Networks and Deep Learning
- Deep Learning eBook
- 1706.01040v1
- hart
- Specification and Schedule(1)
- Deep Learning Challenges in Embedded Platforms __ Radio-Electronics
- HALO15 Keynote
- ANN - wiki
- CSE deep learning seminar Report
- A COGNITIVE APPROACH FOR FLOWER IMAGE CLASSIFICATION BASED ON CONVOLUTION NEURAL NETWORKS.
- Wang_W_D_2018
- The 9 Deep Learning Papers You Need to Know About (Understanding CNNs Part 3) – Deshpande
- Deep learning and computer vision
- 1812.04056v2
- 1702.07392
- Deep Learning Notes
- [11]
- Transform Video Into a Comics

You are on page 1of 75

GEOINT Training

larryb@nvidia.com

June 2015

1 What is Deep Learning?

2 GPUs and Deep Learning

3 cuDNN and DiGiTS

AGENDA 4 Neural Network Motivation

5 Working with Deep Neural Networks

6 Using Caffe for Deep Learning

7 Summary DL For GEOINT

2

What is Deep Learning?

3

DATA SCIENCE LANDSCAPE

Data Analytics

SQL Query

Learning

Methods Networks

Regression

SVM

Recommender systems

4

DEEP LEARNING & AI

CUDA for

Deep Learning

Machine Learning is in some sense a

rebranding of AI.

perceptual tasks, and there are many

successes.

companies, as well as the foremost research

institutions, are using GPUs for machine

learning.

5

INDUSTRIAL USE CASES

machine learning is pervasive

6

TRADITIONAL ML HAND TUNED FEATURES

Images/video

Audio

Text translation, Information

retrieval, ....

Text Text features

7

Slide courtesy of Andrew Ng, Stanford University

WHAT IS DEEP LEARNING?

Systems that learn to recognize objects that are important, without

us telling the system explicitly what that object is ahead of time

Key components

Task

Features

Model

Learning Algorithm

8

THE PROMISE OF MACHINE LEARNING

ML Systems Extract Value From Big Data

minute

9

WHAT MAKES DEEP LEARNING DEEP?

Todays Largest

Networks

~10 layers

1B parameters

10M images

~30 Exaflops

~30 GPU days

of parameters only

1,000 more.

Input Result

10

IMAGE CLASSIFICATION WITH DNNS

Training Inference

truck

11

IMAGE CLASSIFICATION WITH DNNS

Training Typical training run

Pick a DNN design

cars buses trucks motorcycles

Input 100 million training images

spanning 1,000 categories

One week of computation

Test accuracy

If bad: modify DNN, fix training

set or update training parameters

12

DEEP LEARNING ADVANTAGES

Deep Learning

Dont have to figure out the features ahead of time.

Use same neural net approach for many different problems.

Fault tolerant.

Scales well.

Linear classifier

Clustering

Regression

Association Rules

Decision Trees

13

CONVOLUTIONAL NEURAL NETWORKS

Biologically inspired.

called the receptive field.

Each filter has the same weights across the whole layer.

Supervised

Unsupervised

Reinforcement ideal system automatically switches modes

14

CONVOLUTIONAL NETWORKS BREAKTHROUGH

15

CNNS DOMINATE IN PERCEPTUAL TASKS

16

Slide credit: Yann Lecun, Facebook & NYU

RECURRENT NEURAL NETWORK - RNN

AKA: LSTM

Good for sequences.

Translate sentence between languages.

Generate a caption for an image.

17

SENSOR/PLATFORM CONTROL

Reinforcement learning

(predicted future reward, actual reward)

Data sequence

Control

policy

Applications

Sensor tasking

Autonomous vehicle

navigation

[11] Google DeepMind in Nature

18

WHY IS DEEP LEARNING HOT NOW?

Three Driving Factors

350 millions

images uploaded

per day

customer data

hourly

uploaded every

minute

19

GEOINT ANALYSIS WORKFLOW

TODAY

Big Data BOTTLENECK

NUMBERS

Metadata Human Mission focused

filters perception analysis

IMAGES VIDEOS

Near perfect

SOUNDS TEXT Noisy content

perception

VISION

Big Data

NUMBERS DL based Mission focused

Content

machine analysis

based filters

perception

IMAGES VIDEOS Near

SOUNDS TEXT Mission relevant

human level

content

perception

20

GPUs and Deep Learning

21

GPUs THE PLATFORM FOR DEEP LEARNING

GPU Entries

Image Recognition Challenge 120

100 110

1.2M training images 1000 object categories 80

60

Hosted by 60

40

20

4

0

2010 2011 2012 2013 2014

person

car bird

Classification Error Rates

helmet frog

30%

motorcycle 28%

25% 26%

20%

person 15% 16%

person 12%

hammer 10%

dog

flower pot 7%

5%

chair

power drill 0%

2010 2011 2012 2013 2014

22

GPUS MAKE DEEP LEARNING ACCESSIBLE

GOOGLE DATACENTER STANFORD AI LAB

Deep learning with COTS HPC

systems

A. Coates, B. Huval, T. Wang, D. Wu,

A. Ng, B. Catanzaro

ICML 2013

$1M Artificial Brain on the Cheap

1,000 CPU Servers 600 kWatts 3 GPU-Accelerated Servers 4 kWatts

2,000 CPUs 16,000 cores 12 GPUs 18,432 cores

$5,000,000 $33,000

23

Deep Image: Scaling up Image Recognition

IMAGENET

CHALLENGE Baidu: 5.98%, Jan. 13, 2015

Accuracy %

DNN

Delving Deep into Rectifiers: Surpassing Human-Level

Performance on ImageNet Classification

84%

Microsoft: 4.94%, Feb. 6, 2015

CV

74%

72% Batch Normalization: Accelerating Deep Network

Training by Reducing Internal Covariant Shift

2010 2011 2012 2013 2014

Google: 4.82%, Feb. 11, 2015

24

GOOGLE KEYNOTE AT GTC 2015

25

GOOGLE USES DEEP LEARNING FOR UNDERSTANDING

What are all these numbers? What are all these words?

Large-Scale Deep Learning For Building Intelligent Computer Systems, Jeff Dean (Google), http://www.ustream.tv/recorded/60071572

26

WHY ARE GPUs GOOD FOR DEEP LEARNING?

ImageNet Challenge Accuracy

Neural 93%

GPUs

Networks

Inherently

Parallel 88%

Matrix

Operations

84%

FLOPS

GPUs deliver --

- same or better prediction accuracy

- faster results

- smaller footprint 74%

- lower power 72%

27

GPU ACCELERATION

Training A Deep, Convolutional Neural Network

Training Time Training Time GPU

Batch Size

CPU GPU Speed Up

7 layers 1 Tesla K40 GPU

5 convolutional layers + 2 fully-connected CPU times utilized Intel MKL BLAS library

ReLU, pooling, drop-out, response normalization GPU acceleration from CUDA matrix libraries

(cuBLAS)

Implemented with Caffe

28

DEEP LEARNING EXAMPLES

Action Recognition, Scene Understanding Natural Language Processing

Pedestrian Detection, Traffic Sign Recognition

Volumetric Brain Image Segmentation

29

GPU-ACCELERATED

DEEP LEARNING FRAMEWORKS

CUDA-

CAFFE TORCH THEANO KALDI

CONVNET2

Deep Learning Scientific Computing Math Expression Deep Learning Speech Recognition

Domain

Framework Framework Compiler Application Toolkit

Multi-CPU (nnet2)

Text-based

Python, Lua,

Interface(s) definition files, Python C++ C++, Shell scripts

MATLAB

Python, MATLAB

Embedded (TK1)

http://developer.nvidia.com/deeplearning

30

cuDNN

31

HOW GPU ACCELERATION WORKS

Application Code

Compute-Intensive Functions

Rest of Sequential

5% of Code CPU Code

GPU ~ 80% of run-time CPU

+ 32

WHAT IS CUDNN?

cuDNN is a library of primitives for deep learning

Applications

Languages Directives

Flexibility Acceleration Applications

33

ANALOGY TO HPC

cuDNN is a library of primitives for deep learning

Application

Fluid Dynamics

Computational Physics

cuBLAS/NVBLAS

implementations

Tesla Titan

Intel CPUs IBM Power

TK1 TX1

34

DEEP LEARNING WITH CUDNN

cuDNN is a library of primitives for deep learning

Applications

Frameworks

cuDNN

35

CUDNN ROUTINES

Pooling - Spatial smoothing

36

CONVOLUTIONS THE MAIN WORKLOAD

Very compute intensive, but with a large parameter space

2 Input feature maps 7 Kernel Width

3 Image Height 8 Top zero padding

4 Image Width 9 Side zero padding

5 Output feature maps 10 Vertical stride

11 Horizontal stride

37

EXAMPLE OVERFEAT LAYER 1

/* Allocate memory for Filter and ImageBatch, fill with data */

cudaMalloc( &ImageInBatch , ... );

cudaMalloc( &Filter , ... );

...

/* Set descriptors */

cudnnSetTensor4dDescriptor( InputDesc, CUDNN_TENSOR_NCHW, 128, 96, 221, 221);

cudnnSetFilterDescriptor( FilterDesc, 256, 96, 7, 7 );

cudnnSetConvolutionDescriptor( convDesc, InputDesc, FilterDesc,

pad_x, pad_y, 2, 2, 1, 1, CUDNN_CONVOLUTION);

cudnnGetOutputTensor4dDim(convDesc, CUDNN_CONVOLUTION_FWD, &n_out, &c_out, &h_out, &w_out);

cudnnSetTensor4dDescriptor( &OutputDesc, CUDNN_TENSOR_NCHW, n_out, c_out, h_out, w_out);

cudaMalloc(&ImageBatchOut, n_out * c_out * h_out * w_out * sizeof(float));

cudnnConvolutionForward( handle, InputDesc, ImageInBatch, FilterDesc, Filter, convDesc,

OutputDesc, ImageBatchOut, CUDNN_RESULT_NO_ACCUMULATE); 38

CUDNN V2 - PERFORMANCE

CPU is 16 core Haswell E5-2698 at 2.3 GHz, with 3.6 GHz Turbo

GPU is NVIDIA Titan X

39

CUDNN EASY TO ENABLE

uncomment USE_CUDNN := 1

Use cudnn module in Torch instead

Install CAFFE as usual of regular nn module.

with standard nn module.

Replace nn with cudnn

DiGiTS

Deep Learning GPU Training System

41

DIGITS

Interactive Deep Learning GPU Training System

Quickly design the best deep

neural network (DNN) for your

data

Visually monitor DNN training

quality in real-time

Manage training of many DNNs in

parallel on multi-GPU systems

developer.nvidia.com/digits

42

DIGITS

Deep Learning GPU Training System

Available at

developer.nvidia.com/digits

Free to use

v1.0 supports classification on images

Future versions: More problem types

and data formats (video, speech)

advanced developers)

43

HOW DO YOU GET DIGITS

Two options

www.github.com/nvidia/digits

44

Main Console

DIGITS Workflow

Configure your Network

Create your database

Start training

one, or create your own 45

CREATE THE DATABASE

DIGITS can automatically create your

training and validation set

OR

and validation set

Image parameter

options

46

NETWORK

CONFIGURATION

OR add it here

Start training

47

DIGITS

Visualize DNN performance in real time

Compare networks

Download

network files

Accuracy and

loss values

during training

Learning rate

Classification on

the with the

network

snapshots

48

Neural Network Motivation

49

NEURAL NETWORK MOTIVATION

One learning algorithm hypothesis

We can connect any sensor to any part of the brain, and the brain figures it out.

Adding sense of direction

50

NEURAL NETS SCALE EASIER

Why use neural nets? Consider computer vision

When the decision space is non-linear, and the number of features is very large.

X

X

Pixel 1 Pixel 2

Quadratic features (x1 * x2) - over 4 billion!

51

WHATS IN A NEURON?

Artificial neuron is modeled as a Logistic Unit.

x1 z = x1 w1 + x2 w2 + x3 w3

w1

w2 1

x2 Activation =

1 + e-z

w3

x3 1

Neuron Sigmoid function

0

52

NEURONS CAN COMPUTE

Artificial neuron can compute logical operations like AND OR

1

x2 x3 Activation

-30

0 0 0

20

x2 1 0 0

20 0 1 0

x3 1

1 1

Neuron

53

DEEP VERSUS TWO-LAYER NETWORKS

Theory says two fully-connected layers can solve any problem.

G. Cybenko - Approximation by Superpositions of a Sigmoidal Function, Mathematics of Control, Signals and Systems, 1989

In practice, there is.

More memory versus more time.

Few functions can be computed in two layers without an exponentially large look-up table.

Using more than 2 steps can reduce memory by an exponential factor.

CUDA for

Machine Learning

54

Working with Deep Neural Networks

55

OVERFITTING & UNDERFITTING

Important terminology

High Bias

High Variance

Underfitting Overfitting

Just right

56

LEARNING CURVE

Underfitting example

High Bias

validation

training

Actions

Increase size of neural network.

57

LEARNING CURVE

Overfitting example

training

Actions

Get more data / examples Augmentation

DATA AUGMENTATION

Augmentation expands your dataset

Mirror images

Distorted / blurred

Rotations

Color changes

59

NEURAL NETWORK GUIDANCE

60

NEURAL NETWORK STRENGTH

Good

Neural Networks + Big Data =

Stuff

61

Using Caffe for Deep Learning

62

LEARNING A BIT MORE WITH CAFFE

Lets learn a bit more about DNNs by learning a bit about Caffe.

Well learn about layer types, and how to think about neural network architecture.

Though well use Caffe as our working example, these concepts are useful in general.

63

NETWORKS, LAYERS & BLOBS

Network Layer2

blob

Neural Layer2

Blob describes data

Layer1 - batch of images

blob - model parameters

Data

blob

Data Layer

64

OVERALL NETWORK STRUCTURE

Ignoring blobs here

Loss Layer

Neural Layer

Neural Layer

Data Layer

65

CAFFE MODELS DEFINED IN PLAINTEXT

66

CAFFE NEURAL LAYERS

Convolution

Neural Layer Inner Product = Fully Connected

1 or more Neural Layer

Pooling

Neural Layer

Local Response Normalization

67

CAFFE LOSS LAYERS

Softmax (Logistic)

Accuracy

Neural Layer

Neural Layer

Data Layer

68

Summary Deep Learning for GEOINT

69

DEEP LEARNING AS GEOINT FORCE MULTIPLIER

Managing Big Data

Real-time near-human level perception at web-scale

Data exploration and discovery

Semantic and similarity based search

Dimensionality reduction

Transfer learning

Model sharing

Compact model representations

Models can be fine-tuned based on multiple analysts feedback

70

SUMMARY - DL FOR GEOINT

Deep Learning

Adaptable to many varied GEOINT workflows and

deployments scenarios

Available to apply in production and R&D today

Approachable using open-source tools and libraries

71

Machine Learning and Data Analytics

72

TRADITIONAL MACHINE LEARNING

For your many non-DL applications

and deploying ML systems.

Holds records for performance on many

common ML tasks, on single nodes or

clusters.

Uses Scala. Feels like SciPy or Matlab.

73

GPU ACCELERATION FOR GRAPH ANALYTICS

Comms & Social networks 1 GPU vs 60 Nodes

Cyber pattern recognition

280x vs optimized Spark

Shortest path finding

1440x vs Spark

3420x vs Hadoop

PageRank : 19x Speedup

1.2

Time per iteration (s)

1 0.967 3

2

1

1

0.8 2

Lower 2

0.6 is 2

0

Better 1 1

0.4 2

2 2

0.2 1

0.051 2 3

0 2

3

Intel Xeon E5-2690 v2

74

Thank you!

- Deep Learning Networks for New Computer Vision TechnologiesUploaded byIndrasen Poola
- MachineLearningResourceGuide.pdfUploaded byHari Atharsh
- Neural Networks and Deep LearningUploaded byperl.freak
- Deep Learning eBookUploaded byyeisongarces
- 1706.01040v1Uploaded byHardik
- hartUploaded byD
- Specification and Schedule(1)Uploaded byHlynur Davíð Hlynsson
- Deep Learning Challenges in Embedded Platforms __ Radio-ElectronicsUploaded bylcm3766l
- HALO15 KeynoteUploaded byFastAnalytix
- ANN - wikiUploaded byrudy_tanaga
- CSE deep learning seminar ReportUploaded byRaj Jha
- A COGNITIVE APPROACH FOR FLOWER IMAGE CLASSIFICATION BASED ON CONVOLUTION NEURAL NETWORKS.Uploaded byIJAR Journal
- Wang_W_D_2018Uploaded byAnkur Pandey
- The 9 Deep Learning Papers You Need to Know About (Understanding CNNs Part 3) – DeshpandeUploaded byAnton Van Wyk
- Deep learning and computer visionUploaded bypipers10
- 1812.04056v2Uploaded byAskar Myktybekov
- 1702.07392Uploaded byWafa Benzaoui
- Deep Learning NotesUploaded byAc Ebirim
- [11]Uploaded bySAI CHAKRADHAR G
- Transform Video Into a ComicsUploaded byGustavo Schimpp
- Animal ClassificationUploaded byAadarsh Ramakrishnan
- ERISNet Deep Neural Network (1)Uploaded byMarcos Pérez
- TRANSFER LEARNING BASED IMAGE VISUALIZATION USING CNNUploaded byAdam Hansen
- ProposalUploaded byAbhishek Jha
- TR2019-084Uploaded byJuan Daniel Garcia Veiga
- Satellite Imagery for Automatic TaggingUploaded bydbu bca
- Deep Neural Networks in Machine TranslationUploaded byDiego Cortés Rodriguez
- 1705.04448.pdfUploaded byQaiser Muhammad
- 1Deep Learning for Entity Matching_ a Design Space ExplorationUploaded byRM IDEAL SOLUTIONS
- Sohai ArifUploaded byShashank Dahagama

- Learning to Generate Reviews and Discovering SentimentUploaded byFurio Ruggiero
- List of Deep Learning and NLP ResourcesUploaded byBartoszSowul
- Porting c to RustUploaded byBartoszSowul
- tut_sbrnUploaded byBartoszSowul
- Jeff Heaton-Artificial Intelligence for Humans, Volume 3_ Deep Learning and Neural Networks-CreateSpace Independent Publishing Platform (2015)Uploaded byAdriano Carafini
- Defeating Encrypted and Deniable File Systems.pdfUploaded byPrivate Infringer
- v56i10Uploaded byBartoszSowul
- Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500 (2016) - Krauss, Do, Huck.pdfUploaded byBartoszSowul
- TrueSkill Ranking AlgorithmUploaded byArshed Nabeel
- Levitt Gambling 2004Uploaded bymjherman
- Risk, Return, and Gambling Market Efficiency (2006) - Dare.pdfUploaded byBartoszSowul
- Analyzing the Characteristics of Shared Playlists for Music Recommendation (2014) - Jannach Et AlUploaded byBartoszSowul
- 15 YEARS LATER: ON THE PHYSICS OF HIGH-RISE BUILDING COLLAPSESUploaded byrawo
- Successful Prediction of Horse Racing Results using a Neural Network.pdfUploaded byBartoszSowul
- IELTS Prep Writing1 1Uploaded byBartoszSowul
- Saariluoma - Chess and Content-Oriented Psychology of ThinkingUploaded byanon-355439
- Computer Chess and Search [Marsland 1991].pdfUploaded byPrakashRoxx
- Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks (2017) - Rajpurkar et al.pdfUploaded byBartoszSowul
- Spotify-ed: Music Recommendation and Discovery in Spotify (2014) - BateiraUploaded byBartoszSowul
- Designing a Career in Biomedical EngineeringUploaded bymedjac
- Manage Frame ProtUploaded byBartoszSowul
- Microsoft Machine Learning Algorithm Cheat Sheet v6 (1)Uploaded byrevcarcass
- Xyz - hjk + 2dc.pdfUploaded byDavor Predavec
- A Neural Parametric Singing Synthesizer (2017) - Blaauwa, BonadaUploaded byBartoszSowul
- 1503.02531Uploaded byPrasad Seemakurthi
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017) - Chen Et AlUploaded byBartoszSowul
- 1704.01568Uploaded byPeter Pan
- Dance Dance Convolution (2017) - Donahue, Lipton, McAuleyUploaded byBartoszSowul
- Numbers of ClassifierUploaded byDurgesh Kumar
- A Large Self-Annotated Corpus for Sarcasm (2017) - Khodak, Saunshi, VodrahalliUploaded byBartoszSowul

- neural networkUploaded byfarhan danish
- Classification of Intrusion Detection SystemsUploaded byFaruk Mohamed
- Artificial Intelligence for Humans, Volume 3 - Jeff HeatonUploaded byHemant Chaudhari
- Survey on Brain Tumour Detection and Segmentation Techniques on MRI ImagesUploaded byInternational Journal of Innovative Science and Research Technology
- NeuralNetworksUploaded byRijas Rasheed
- Neuralnetworksanddeeplearning ComUploaded byharshit
- Machine Learning With MatlabUploaded byAjit Kumar
- Jasper Busschers - Thesis FinalUploaded byJasperBusschers
- Afrooz Purarjomandlangrudi ThesisUploaded bySerag El-Deen
- hmUploaded bytata
- Development of Prediction Model for Mechanical Properties of Batch Annealed Thin SteelUploaded byvaalgatamilram
- cs229-notes1Uploaded byfatihy73
- Application of Artificial Neural Network for Height ModellingUploaded byEditor IJRITCC
- List of Seminar TopicsUploaded bykartikkumar461990
- Movidius Neural Computer StickUploaded bymaximin
- MELJUN CORTES RESEARCH PAPERS Decision Support for University Enrollement Management Implementation ExperienceUploaded byMELJUN CORTES, MBA,MPA
- Mathematical Modelling and Dynamic SimulationUploaded byJojo Kaway
- neuro-fuzzy systemUploaded byGaurav Kumar
- Optimising the Topology of CNNUploaded byNguyen Truong Giang
- Manual Neuraltools6 EnUploaded byJohn Kennedy Fernandes
- The Use of Kalman Filter and Neural Network Methodologies in Gas Turbine Performance Diagnostics: A Comparative StudyUploaded byAdha Montpelierina
- Management Information System Chapter 11 GTU MBAUploaded byRushabh Vora
- Neural Network Time Series Prediction SP500 2Uploaded bymotaheri
- SPHandler: Multi-criteria based intelligent stakeholder quantification system for value-based softwareUploaded byDr Muhamma Imran Babar
- IRM.Press.Managing.Multimedia.Semantics.eBook-kB.pdfUploaded byMilena Stefanova
- Human Computer Interaction Using Hand GesturesUploaded byEditor IJRITCC
- Neural Network in Financial AnalysisUploaded bynp_nikhil
- Importance of Computer Ethics and Software PiracyUploaded bymonee143
- Deep Learning the Indus Script (Satish Palaniappan & Ronojoy Adhikari, 2017)Uploaded bySrini Kalyanaraman
- An Artificial Neural Network for Pollution EvaluationUploaded byJ.92Flom

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.