Deep Learning and Bayesian Methods

Lavanya Tekumalla, Aifonic Labs

Copyright: AiFonic Labs

 PhD Machine Learning Lab,
Indian Institute of Science

MS Computer Graphics,
University of Utah

Industry: Amazon^2
Myntra, InMobi, Kenome

Founder: AiFonic Labs,

MachineLearningInterview.com

Developer -> Data Scientist -> Product Manager

Copyright: AiFonic Labs

Motivation: Bayesian Modeling

Copyright: AiFonic Labs

Motivation: Bayesian Modeling

Copyright: AiFonic Labs

Motivation: Bayesian Modeling

Copyright: AiFonic Labs

 Motivation: Bayesian Modeling

Epistemic uncertainty : More data makes us more confidant

Copyright: AiFonic Labs

 Motivation: Bayesian Modeling

Aleatoric uncertainty :

Inherant uncertainty in the function we are trying to learn...

Image Credit : https://blog.tensorflow.org/2019/03/regression-with-probabilistic-layers-in.html

Copyright: AiFonic Labs

Bayesian : How confident are our predictions ?

● Learn distribution instead of a

point estimate to capture
uncertainty
● Start with a prior, refine with
data to learn Posterior

Copyright: AiFonic Labs

Product Size Recommendation

Copyright: AiFonic Labs

Bayesian Modeling : Product Size Recommendation
Bayesian Models For Product Size Recommendation, WWW
2018, Vivek Sembium, Rajeev Rastogi, Lavanya Tekumalla,
Atul Saroop.

Copyright: AiFonic Labs

Bayesian Approach: Size Recommendation

Copyright: AiFonic Labs

Bayesian PSIMS : Generative Process

Copyright: AiFonic Labs

Deep learning Motivation

● Capture Complex non-linear relationships

● Avoid feature engineering

● Improve predictive capability

● Availability of flexible scalable platforms

Copyright: AiFonic Labs

Bayesian methods & Deep Learning

Copyright: AiFonic Labs

Deep Bayesian Learning & Bayesian Deep Learning

Deep Bayesian Learning Bayesian Deep Learning

Use principles of deep Use principles of Bayesian

learning to build deeper learning to model
Bayesian networks and uncertainty in deep neural
model more complex networks
relationships

Copyright: AiFonic Labs

Deep Bayesian Learning

Copyright: AiFonic Labs

Deep Bayesian Learning

"Nested Hierarchical Dirichlet Process for Nonparametric Entity-Topic

Analysis", Priyanka Agrawal*, Lavanya Tekumalla*, and Indrajit
Bhattacharya. ECML 2013, Prague.

● Document : Distribution over

entities
● Entities : Distribution over topics
● Topics : Distribution over words

Copyright: AiFonic Labs

Deep Bayesian Learning
Deep Infinite Admixture Models, NIPS 2018, Bayesian Deep Learning
Workshop, Lavanya Sita Tekumalla, Priyanka Agarwal, Indrajit
Bhattacharya.

Copyright: AiFonic Labs

Deep Bayesian Learning : Platforms

Earlier : Bayesian models required complex inference

Now : Black box inference

→ MCMC : Now in PL (HMC, NUTS)

→ Black-box Variational Inference: Now in PL (Autograd)

Platform Support : Probabilistic Programming

● PymC3
● Stan
● Edward
● Pyro

Copyright: AiFonic Labs

Bayesian Deep Learning

Copyright: AiFonic Labs

Bayesian Deep Learning - Many aspects

Copyright: AiFonic Labs

Bayesian Deep Learning : Platforms

Tensorflow Probability :
# Build model.
model = tfk.Sequential([
tf.keras.layers.Dense(1 + 1),
tfp.layers.DistributionLambda(
lambda t: tfd.Normal(loc=t[..., :1],
scale=1e-3 + tf.math.softplus(0.05 *
t[..., 1:]))),
])

# Do inference.
model.compile(optimizer=tf.optimizers.Adam(learning_rate=0.05),
loss=negloglik)
model.fit(x, y, epochs=500, verbose=False)

# Make predictions.
yhat = model(x_tst)

Copyright: AiFonic Labs

Why Bayesian Models ?

Copyright: AiFonic Labs

Product Size Recommendation

Copyright: AiFonic Labs

Product Size Recommendation

Copyright: AiFonic Labs

Why is Size Recommendation Hard ?

Copyright: AiFonic Labs

Approach

Copyright: AiFonic Labs

Machine Learning Models

Copyright: AiFonic Labs

Data Sparsity

Copyright: AiFonic Labs

Bayesian: How confident are our predictions ?

Copyright: AiFonic Labs

Bayesian Approach: Size Recommendation

Copyright: AiFonic Labs

Bayesian PSIMS: Generative Process

Copyright: AiFonic Labs

The Dangers of Overfitting :
The need for priors & confidance estimates

Copyright: AiFonic Labs

Implementation

Copyright: AiFonic Labs

Experimental Results: ML Metrics

Copyright: AiFonic Labs

Summary

Copyright: AiFonic Labs