Generative neural networks

Romain Tavenard (Université de Rennes)

 Generative models in a nutshell

• Goal: model p(x) or p(x | y)

• explicitly
• eg. Gaussian Mixture Models
• implicitly
• at least allow for sampling (i.e. generate new data)
• e.g. Variational Auto Encoders,
Generative Adversarial Networks

 Auto-encoders
[Hinton & Salakhutdinov, 2006]

• Encode information in a latent space

• typically lower-dimensional
• similar in spirit to a non-linear PCA
• not a generative model per se!

3 Source: lilianweng.github.io



 Variational auto-encoders
[Kingma & Welling, 2014]

• Goal: turn auto-encoders into generative models

• Idea: set a prior on the distribution of z
(through penalization of the loss function)
• Generative process: draw z from the prior, and decode it

4
Source: lilianweng.github.io





 Generative Adversarial Networks

[Goodfellow et al., 2014]

• Model:
<latexit sha1_base64="OA4n3u133Y4GxtwF5LGmhok6QYc=">AAADP3icjVFNb9QwEJ0EKGX5aIAjlxErEKdVgkDlglSVC8cisW2lpqoc72RrrfMhx6lYov1x/RWIGzfggrhyY2y8ErRC4CjJ85v3nj120WrV2TT9GMVXrl7buL55Y3Tz1u07W8nde/td0xtJU9noxhwWoiOtappaZTUdtoZEVWg6KBavXP3gjEynmvqtXbZ0XIl5rUolhWXqJHm/xJeYayptPowQ84Lmqh6EMWK5GqReMYeY4WPMLb2zQ9kYFMgLaOxE1WpaYZ57TXpBU4oFrTUumOpZiB3lRs1P7eQkGaeT1A+8DLIAxhDGXpN8gBxm0ICEHiogqMEy1iCg4+cIMkihZe4YBuYMI+XrBCsYsbdnFbFCMLvg75xnR4Gtee4yO++WvIrm17AT4RF7GtYZxm419PXeJzv2b9mDz3R7W/K/CFkVsxZOmf2Xb638X5/rxUIJL3wPintqPeO6kyGl96fido6/dWU5oWXO4RnXDWPpnetzRu/pfO/ubIWvf/FKx7q5DNoevrpd8gVnF6/zMth/OsmeT9I3z8Y7u+GqN+EBPIQnfJ/bsAOvYQ+mnP0p2oi2oiQ+jz/H3+Lvv6RxFDz34Y8R//gJa8W5lA==</latexit>

⇢
1 for a real sample
y=
0 for a fake sample

Source: lilianweng.github.io

• Loss function:
min max L(D, G) = Ex⇠pr (x) [log D(x)] + Ez⇠pz (z) [log(1
<latexit sha1_base64="MBuH7NgxPbkHhENuUVl6lFJKrNg=">AAADsnicnVFdb9MwFL1Z+BjhYx173ItFBUoFVAnaBC+TJlZUHngYgm4TTRU5rpdZcz6UOKhtlBf4kYh/AA/8B65NygoTAuEoyfG55xz72lEuRak877O1Zl+5eu36+g3n5q3bdzY6m3ePyqwqGB+xTGbFSURLLkXKR0ooyU/ygtMkkvw4Oj/Q9eP3vChFlr5V85xPEhqn4lQwqpAKO9+cIOKxSGsqRZzyaeMEiUhJWA8bEiR0hmjQkFfu4BEZ9siDPeTUWRTVL5qwnpGgFAnJw7po3FmvGQcyi8kA4eThqm7xU7do3MVS5/qPB+4Q573ehASB83/ZF7p4RWeyZxjsBDydXvQWdrpe3zODXAZ+C7rQjsOs8wkCmEIGDCpIgEMKCrEECiU+Y/DBgxy5CdTIFYiEqXNowEFvhSqOCorsOX5jnI1bNsW5ziyNm+EqEt8CnQTuoydDXYFYr0ZMvTLJmv1Tdm0y9d7m+I/arARZBWfI/s23VP6rT/ei4BSemR4E9pQbRnfH2pTKnIreOVnpSmFCjpzGU6wXiJlxLs+ZGE9petdnS039i1FqVs9Zq63gq94lXrD/+3VeBkdP+v5u33u9091/3l71OmzDPXDxPp/CPryEQxgBs95Yc+uD9dHesd/Z1GY/pGtW69mCX4YtvwPhjeC1</latexit>

D(G(z)))]
G D
= Ex⇠pr (x) [log D(x)] + Ex⇠pg (x) [log(1 D(x))]
5



 Generative Adversarial Networks
[Goodfellow et al., 2014]

• Generative process
• Draw z from pz
• Pass it to the generator to compute G(z)
• Optimization
• Alternate between generator and discriminator
• Very unstable process in practice

min max L(D, G) = Ex⇠pr (x) [log D(x)] + Ez⇠pz (z) [log(1
<latexit sha1_base64="MBuH7NgxPbkHhENuUVl6lFJKrNg=">AAADsnicnVFdb9MwFL1Z+BjhYx173ItFBUoFVAnaBC+TJlZUHngYgm4TTRU5rpdZcz6UOKhtlBf4kYh/AA/8B65NygoTAuEoyfG55xz72lEuRak877O1Zl+5eu36+g3n5q3bdzY6m3ePyqwqGB+xTGbFSURLLkXKR0ooyU/ygtMkkvw4Oj/Q9eP3vChFlr5V85xPEhqn4lQwqpAKO9+cIOKxSGsqRZzyaeMEiUhJWA8bEiR0hmjQkFfu4BEZ9siDPeTUWRTVL5qwnpGgFAnJw7po3FmvGQcyi8kA4eThqm7xU7do3MVS5/qPB+4Q573ehASB83/ZF7p4RWeyZxjsBDydXvQWdrpe3zODXAZ+C7rQjsOs8wkCmEIGDCpIgEMKCrEECiU+Y/DBgxy5CdTIFYiEqXNowEFvhSqOCorsOX5jnI1bNsW5ziyNm+EqEt8CnQTuoydDXYFYr0ZMvTLJmv1Tdm0y9d7m+I/arARZBWfI/s23VP6rT/ei4BSemR4E9pQbRnfH2pTKnIreOVnpSmFCjpzGU6wXiJlxLs+ZGE9petdnS039i1FqVs9Zq63gq94lXrD/+3VeBkdP+v5u33u9091/3l71OmzDPXDxPp/CPryEQxgBs95Yc+uD9dHesd/Z1GY/pGtW69mCX4YtvwPhjeC1</latexit>

D(G(z)))]
G D
= Ex⇠pr (x) [log D(x)] + Ex⇠pg (x) [log(1 D(x))]
6

 Generative Adversarial Networks

• Many variants to the original model

• Class-conditional variants
• Different losses
• Different structures
• Very realistic samples generated (BigGAN, StyleGAN)