Unit 1: Generating New Images Using Gans: Week 6: Advanced Deep Learning Topics

Week 6: Advanced Deep Learning Topics
Unit 1: Generating New Images Using GANs

Generating New Images Using GANs
Intro: The what and why of GANs
What are GANs?

▪ A method for generating new images based on
an existing dataset!
▪ Consist of two competing networks:
– A generative network (G) that generates fake Before GANs:
images Images tended to be blurry or unclear
– A discriminative network (D) that can pick
apart fakes from your real dataset
Why GANs for image generation?

▪ Past methods (such as VAEs) have a visually
distinct ‘blurriness’ compared to GANs
▪ Discriminative (classification) models would be
able to tell the difference easily!
GANs: Images look similar to real images – in
this case a real face
© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2
The two networks: the generator Noise:
The generator is the most important part of the

network, as it is what we use in the end!
Why do we want to generate images?

▪ Generating images is a difficult task
▪ GANs yield visually convincing results
Generator
▪ Can be used for data inspection or for dataset
augmentation when samples are lacking
How do we generate images?

▪ Learn to transform Gaussian noise into a realistic
looking image!
▪ Input: a vector of Gaussian noise
▪ Output: a flattened image – essentially just a vector
of pixels
© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Generated Fake 3
Generator Image:
Pixel
Noise:
Pixel
Pixel
Pixel
Generator
Noise: Fake:
Unflatten
Generator
Generator: Turns noise into convincing fakes

The two networks: the discriminator Real or Fake Image:
Why do we want to discriminate images?

▪ Discriminative (classification) models have
superhuman performance
▪ If generated images look ‘off’, discriminative models
should be able to tell the difference
▪ Key: Even though we call the networks adversarial,
the discriminator actually helps the generator
improve!
How do we discriminate images? Discriminator

▪ Classic binary classification task – real image from
dataset, or fake image from generator?
▪ No different than building e.g. a dog vs cat classifier,
architectures can be identical
Classification:
© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Real/Fake? 6
Discriminator
Pixel
Pixel
0/1 Real/Fake?
ö
Pixel
Pixel

Discriminator
Fake or Real
Image
Unflatten
Discriminator Real/Fake?
Discriminator: Is image a generated fake, or real?

Overview: the adversarial training process Generator:
Networks take turns training!

Turn 0:
Discriminator:
▪ Takes the first turn
▪ Get several batches of images from training dataset,
and several from generator
▪ Try to tell real dataset images vs generated fakes!
Generator: Turn X:
▪ Key network we care about – it will do the final
image generation when training is done
▪ Learns from discriminator how to generate better
fake images based on a ‘combined network’
▪ Never directly sees a training image!
Final Turn:

Training the discriminator
Fake Image
(from Generator)
Discriminator Fake!
Real Image
(from Dataset)
Discriminator Real!
Straightforward Classification Task!

Training the generator
Noise:
(Frozen:)
Generator Discriminator Real!
Train generator: Combine both networks!

Goal: G learns to get D to predict ‘Real’
Note: D is not updated – it is G‘s turn!

Discriminator ‘helps’ generator
Generator: Frozen Discriminator:

Pixel Pixel
Noise:
Pixel Pixel
0/
1
Pixel Pixel
Pixel Pixel
Key: Discriminator tells G how to improve!!

G+D training network (to train generator)
Generator: Frozen Discriminator:

Pixel
Noise:
Pixel
0/1 Real!
Pixel
Pixel
Note the Label! G is learning to fool D.

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Noise normally would be labeled ‘Fake’! 13
GANs: some key challenges
Getting Generator to Converge:

▪ No single metric to measure how well network
does, so harder to optimize
▪ One important trick: Since D is the only source
of info for G, a well-trained D is vital
Working with Discrete Data:

▪ GANs today are applied primarily to images,
which are easy to represent as continuous
▪ Due to the G+D network training step, data in
typical GAN structures must be continuous
▪ Some extensions of GANs attempt to address
this, but in many situations VAEs and other
approaches can be better for discrete data
Pictured: The losses of a GAN while it trains. Note the loss is
continuous – it would not be defined for discrete data!
Extensions of ‘classic GAN’ architecture
Deep Convolutional GANs: (DC-GANs)

▪ One natural extension: instead of dense layers,
use convolutional layers end to end!
▪ For image tasks, this yields significantly better
results when properly tuned
Conditional GANs:
▪ GANs can also be used in a more supervised
setting, where G and D are also fed labels
▪ This allows you to feed the network’s labels at
runtime to generate specific outputs!
Pictured: A conditional DC-GAN morphing a

generated female face into a generated male face
with similar attributes

Summary Pictured: Another interesting conditional DC-GAN
application: transforming an image by adding or
subtracting labels
GANs are a useful way of generating visually
convincing images based on a dataset Original Generated Image
GANs consist of two competing adversarial
networks
▪ The generator network creates convincing fake Original + Add Young
images
▪ The discriminator learns to tell faked images vs real
images from the dataset
Original + Subtract Blond
Networks take turns improving at their jobs!
Generating images with adversarial back-and-
forth training of two networks! Original + Subtract Smile
Original + Add Bald, Add Male

Thank you.
Contact information:
open@sap.com
© 2017 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.
The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components
of other software vendors. National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated
companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are
set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release
any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products,
and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The
information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various
risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements,
and they should not be relied upon in making purchasing decisions.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company)
in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.
See http://global.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.
Unit 2: Reinforcement Learning
Reinforcement Learning
Applications of machine learning
ML applications fall into three broad contexts

▪ Supervised learning
– Dataset + labels/annotations
UNSUPERVISED SUPERVISED
▪ Unsupervised learning
– Dataset (without labels/annotations)
▪ Reinforcement learning ML
Applications
– No initial dataset
▫ Dataset accumulated with experience
▫ ML agents interact with environment (trial and error)
REINFORCEMENT

Reinforcement Learning 2
Machine learning – Supervised learning
▪ ~100,000s of examples
sample #1
– Each example consists of
▫ Data [ Vector ]
3
▫ Label(s)
…
.
N
.
.
label #1: 8, ‘eight’, [ 0, 0, 0, 0, 0, 0, 0, 0, 1, 0 ]

Reinforcement Learning Time Series Data
Machine learning – Unsupervised learning
▪ In unsupervised learning, no labels are available

– Typically larger datasets (millions)
Gene Sequence Data
▫ Data [ Vector ] A G A T A A G A A G T G T T G T G G A A A A G T A T G T T G
T A T T T A G A A G T A A G T G A T G G T T A T T T G G A G
▫ No Labels T A A G T A G G T A G A A T T A A G A G G T T A T A G G G A
A A G G G A A T T G G T G G A A G G T G A T T T A A T G A A
G A A T G T T T G T G A T T A G G A T A G T A T G T A A G A
A T A G A G A T T G G A A T G G G G G G A T G A T G A T T A
A T G A A G T T T A G G A G G T G T A T G A A T G A G A A G
T T A G G G A A G G A T T A A T G T G T A A G G G T A T T A
G G A A A A T G G A A A A A T T A T G G G T T T T T G T G G
A G G G A G T G T A A A G T G G T G T G G A A A A G A T T G
A G G T G A G A T A T T A A A T A T G G T G G A A T A A T G
G G T T A T A T A G A T T A T T T A T G T T G T G G T A T G
G A G G G T G G T A G A G A A T T G A T A A G G T A G G A T
A A T A T T A G A A G A A G G A A T G A T T T A A G T T T G
G G T T G G G A T T A T T T T A G G A T A G A G G G T A T A
T T G G A T G A G G G A G G G T A T T T G G A A G A T G A A
T A G G G G T G T A G T G A G G T T G G G T A T A T G G T G
A T T G G T T T A A A A A T G A A T G G G G A T T A G A A G
G T A A G T A A A T T A G A A G T T G T A A G G A A T T T T
A G A G T A T G G G G T T T T T A G T G G A A A A A G A A A
A G T G G T T G G G A G G G G G G G T G T T A T T G T T G G
T A A T A A T G G G G T G A G A A A T A G A T T G A T A A T
G T A A T T A T T G A A A T T T G G A T A G T A A G T T T A
G G G T A A G G G G A G T G G G T T T G T T G T T A G G T T
A T A A A G T T G T T T A G G G G A T A A T A G A T A G A G
G G G T G A T T T T T T T T T A T A T A T T A A T A G A G G
G T A A A A A A G G T G T G T A A A T A T T A A G T T T T T
T T A A T G A A A T T G T A T T T G A T A A T A A A G A T T
A G A G A A T T A T T G G A A G T G A T A T T G G T G T G G
© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC G G G T A T T A T A G T G G A T T A G G A T A T A A A T T G 4
Machine Learning – Reinforcement Learning Agent
RL: Dataset built with experience

▪ Experience = Env. State + Action + Next Env. State + Reward
Environment
Machine Learning – Reinforcement Learning Agent
RL: Dataset built with experience

▪ Experience = Env. State + Action + Next Env. State + Reward
RL Feedback Loop
▪ At each step the agent
– Executes action: 𝐴𝑡
– Receives observation: 𝑂𝑡
– Receives reward: 𝑅𝑡
▪ The environment
– Receives action: 𝐴𝑡
– Emits observation: 𝑂𝑡+1
– Emits reward: 𝑅𝑡+1
Environment
Applying RL to games
Atari Example
▪ Agent is a DL network
– Interprets screen pixels
– Outputs game action
– Possible to use CNN+RNN
▪ Environment is Atari Emulator
– Game AI
▪ Reward is game score

Application to Atari Boxing
Multiple frames make a single percept

▪ Enables network to learn dynamics

Application to bin picking
Industrial robot reaches human level performance in 8hrs

▪ Input is RGB+depth
▪ Output is probability map
▪ Reward is success of pick action

Thank you.
open@sap.com
Unit 3: Unsupervised Learning
Unsupervised Learning
Applications of machine learning
ML applications fall into three broad contexts

▪ Supervised learning
– Dataset + labels/annotations
UNSUPERVISED SUPERVISED
▪ Unsupervised learning
– Dataset (without labels/annotations)
▪ Reinforcement learning ML
Applications
– No initial dataset
▫ Dataset accumulated with experience
▫ ML agents interact with environment (trial and error)
REINFORCEMENT

Unsupervised Learning 2
Machine learning – Supervised learning
▪ ~100,000s of examples
sample #1
▫ Data [ Vector ] 3
▫ Label(s)
…
.
N
.
.
label #1: 8, ‘eight’, [ 0, 0, 0, 0, 0, 0, 0, 0, 1, 0 ]

Unsupervised Learning Time Series Data
Machine learning – Unsupervised learning
▪ In unsupervised learning, no labels are available

– Typically larger datasets (millions)
▫ Data [ Vector ] Gene Sequence Data
▫ No Labels A
T
G
A
A
T
T
T
A
T
A
A
G
G
A
A
A
A
G
G
T
T
G
A
T
A
T
G
G
T
T
G
G
A
G
T
A
G
A
G
A
T
A
T
G
A
T
T
A
T
T
T
G
G
T
G
T
A
G
G
T A A G T A G G T A G A A T T A A G A G G T T A T A G G G A
A A G G G A A T T G G T G G A A G G T G A T T T A A T G A A
G A A T G T T T G T G A T T A G G A T A G T A T G T A A G A
A T A G A G A T T G G A A T G G G G G G A T G A T G A T T A
A T G A A G T T T A G G A G G T G T A T G A A T G A G A A G
T T A G G G A A G G A T T A A T G T G T A A G G G T A T T A
G G A A A A T G G A A A A A T T A T G G G T T T T T G T G G
A G G G A G T G T A A A G T G G T G T G G A A A A G A T T G
A G G T G A G A T A T T A A A T A T G G T G G A A T A A T G
G G T T A T A T A G A T T A T T T A T G T T G T G G T A T G
G A G G G T G G T A G A G A A T T G A T A A G G T A G G A T
A A T A T T A G A A G A A G G A A T G A T T T A A G T T T G
G G T T G G G A T T A T T T T A G G A T A G A G G G T A T A
T T G G A T G A G G G A G G G T A T T T G G A A G A T G A A
T A G G G G T G T A G T G A G G T T G G G T A T A T G G T G
A T T G G T T T A A A A A T G A A T G G G G A T T A G A A G
G T A A G T A A A T T A G A A G T T G T A A G G A A T T T T
A G A G T A T G G G G T T T T T A G T G G A A A A A G A A A
A G T G G T T G G G A G G G G G G G T G T T A T T G T T G G
T A A T A A T G G G G T G A G A A A T A G A T T G A T A A T
G T A A T T A T T G A A A T T T G G A T A G T A A G T T T A
G G G T A A G G G G A G T G G G T T T G T T G T T A G G T T
A T A A A G T T G T T T A G G G G A T A A T A G A T A G A G
G G G T G A T T T T T T T T T A T A T A T T A A T A G A G G
G T A A A A A A G G T G T G T A A A T A T T A A G T T T T T
T T A A T G A A A T T G T A T T T G A T A A T A A A G A T T
A G A G A A T T A T T G G A A G T G A T A T T G G T G T G G
© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC G G G T A T T A T A G T G G A T T A G G A T A T A A A T T G 4
Anomaly detection using deep learning
Deep Autoencoder Network

▪ Input layer
– Size of data vector
▪ Bottleneck layer
– Summarized representation ෩
Input: X Output: X
▫ ‘embedding’
▪ Output layer
– Same dimensionality as input
▪ Reconstruction error
– High errors indicate potential anomaly
෩
X−X
© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Reconstruction Error 5
DL anomaly detection in time series
Time Series Signals

▪ Split into sliding windows [ size is hyperparameter ]
– Normalization and preprocessing
w
1
w
2
w
3
… … w
N

Detecting anomalies via reconstruction error
Output (Reconstruction)
Reconstruction error (RE) as a proxy to outliers
▪ Whenever RE is high, it indicates something
– Threshold can be set using statistical bounds
Reconstruction vs Input
Input
Interpreting anomalies
Projecting Bottleneck Activations to 2D/3D

▪ Maximize domain expert interpretation
– Only reason about cluster centers in embedding space
▪ Future examples can be classified
– Semi-supervised model

Thank you.
open@sap.com
Unit 4: Deep Learning on Mobile
Deep Learning on Mobile
Getting to a trained model
Typical Deep Learning Workflow:

Candidate Neural Networks
▪ Prepare dataset
– Normalize, augment, handle missing data
▪ Search for best model architecture
– Iteratively train and evaluate model variants
– [ High compute requirement – e.g. cloud + GPUs ]
▪ Deploy
– Introduce model into production system
– [ High throughput requirement – e.g. cloud + Hadoop cluster ]
▪ Re-train with incoming data

Deploying trained model on a mobile device
Mobile platforms present a unique challenge

▪ Although mobile CPUs are fast…
– e.g. Quad core 1.9Ghz
▪ And have large memory… 09:13
– e.g. 64GBs
▪ Battery efficiency is a bottleneck!
– 20-44 hrs depending on mode & consumption

Deep Learning on Mobile Get Layer Subset 1, Compute
Deploying trained model on a mobile device Get Layer Subset 2, Compute
.
.
Large DL models require lots of data movement
Get Layer Subset N, Compute
▪ CPU can only compute a subset of the network at a time
– # of weights & input/outputs overwhelms available
registers
▪ Each network subset has to be loaded from memory
– Large networks may require many subsets
5
▪ Each read from memory and transfer to CPU is costly in
terms of energy 3 4
– Many fetches may be needed for a single inference
[ 1 subset = CPU register

pass
Layer Subsets
– Especially critical for applications that process 2
capacity ]
streaming data
1

Optimization approach – Pruning
Weight Pruning
▪ Opportunistically remove weights
– Can lead to significant parameter reduction
– Requires a [full] model to be initially trained
– Inspired by synaptic pruning in biological neurons
▪ Continue streaming training data
– Monitor performance loss due to reduced weights
▪ Target layers with highest battery impact
– Requires energy estimation model
▫ Track load, multiply, and accumulate calls

Optimization approach – Quantization
Weights & activations can be expressed using fewer bits

▪ Quantization leads to a significant compression in memory
– Up to 4x reduction for each parameter
– Especially critical for very large networks
▪ Typically, quantization leads to minimal accuracy loss
4,294,967,296
– Note: weights & activations still require full precision in training FP.32
▪ Finding best quantization mapping is hard (and lossy) 256 INT.8
– Tools like TensorRT automate this process
▪ Preprocessing and postprocessing steps need to be added
– Inputs need to be quantized
– Outputs need to be de-quantized

Model Definition
[ high-level language ]
Steps to mobile deployment
DL Model Mobile Optimization Workflow

▪ High level model definition
[ intermediate representation ]
– e.g. Python code
Compute Graph
▪ Intermediate representation
– Compilation and optimization
– Target-independent
▪ Target-dependent optimizations
– Code generation
[ pruning + quantization ]
▪ Runtime
Optimizations
– Packaged binary +
Runtime
Available tools
Mobile Optimization Platforms

▪ TensorRT [ Real Time ] – NVIDIA
– https://developer.nvidia.com/tensorrt
▪ TensorFlow XLA – Google
– https://www.tensorflow.org/performance/xla/
▪ Core ML – Apple
– https://developer.apple.com/machine-learning/
▪ NVML – UW & Amazon
– http://www.tvmlang.org/2017/10/06/nnvm-compiler-announcement.html

Thank you.
open@sap.com
Unit 5: Summary, Recap, and Further Resources
Summary, Recap, and Further Resources
Summary and recap
▪ Getting Started with Deep Learning

▪ Building TensorFlow Applications
▪ Deep Networks and Sequence Models
▪ Convolutional Networks
▪ Industry Applications of Deep Learning
▪ Advanced Deep Learning Topics

Key Messages
▪ Deep learning is real-world relevant today

▪ Deep learning is an engineering skill for practical problems
▪ You have first hands-on experience with industry use cases
▪ Practice makes perfect!

Selected resources
deeplearning.ai specialization at Coursera (MooC)
CS231n, CS224n at Stanford University

(image and text processing)
www.deeplearningbook.org (theoretical foundations)
SAP Leonardo Machine Learning microsite:

sap.com/ml
openSAP Leonardo Machine Learning course

(forthcoming)

Thank you.
open@sap.com

Unit 1: Generating New Images Using Gans: Week 6: Advanced Deep Learning Topics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Unit 1: Generating New Images Using Gans: Week 6: Advanced Deep Learning Topics

Uploaded by

Copyright:

Available Formats

Week 6: Advanced Deep Learning Topics

Unit 1: Generating New Images Using GANs

What are GANs?

Why GANs for image generation?

The generator is the most important part of the

Why do we want to generate images?

How do we generate images?

Generator: Turns noise into convincing fakes

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 5

Why do we want to discriminate images?

How do we discriminate images? Discriminator

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 7

Discriminator: Is image a generated fake, or real?

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 8

Networks take turns training!

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 9

Straightforward Classification Task!

Generator Discriminator Real!

Train generator: Combine both networks!

Note: D is not updated – it is G‘s turn!

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 11

Generator: Frozen Discriminator:

Key: Discriminator tells G how to improve!!

Generator: Frozen Discriminator:

Note the Label! G is learning to fool D.

Getting Generator to Converge:

Working with Discrete Data:

Deep Convolutional GANs: (DC-GANs)

Pictured: A conditional DC-GAN morphing a

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 15

Original + Add Bald, Add Male

ML applications fall into three broad contexts

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2

Machine learning – Supervised learning

label #1: 8, ‘eight’, [ 0, 0, 0, 0, 0, 0, 0, 0, 1, 0 ]

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 3

▪ In unsupervised learning, no labels are available

RL: Dataset built with experience

RL: Dataset built with experience

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 7

Multiple frames make a single percept

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 8

Industrial robot reaches human level performance in 8hrs

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 9

ML applications fall into three broad contexts

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2

Machine learning – Supervised learning

label #1: 8, ‘eight’, [ 0, 0, 0, 0, 0, 0, 0, 0, 1, 0 ]

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 3

▪ In unsupervised learning, no labels are available

Deep Autoencoder Network

Time Series Signals

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 6

Projecting Bottleneck Activations to 2D/3D

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 8

Typical Deep Learning Workflow:

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2

Mobile platforms present a unique challenge

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 3

[ 1 subset = CPU register

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 4

© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 5

Weights & activations can be expressed using fewer bits