Professional Documents
Culture Documents
3 GANs
3 GANs
3 GANs
Spring 2023
In this Lecture
›
Reminder: 𝑓-divergence
D.. Derkach Generative Modeling To optimize 𝑟𝐾𝐿 properly we need access to true PDF, p(x).
Spring 2023 6
Reminder: Variational Lower Bound
P(x) Q(x)
x x
2𝑝 𝑥 2𝑞 𝑥
JS(𝑃| 𝑄 ≥ 𝔼#~' log + 𝔼#~( log
𝑝 𝑥 +𝑞 𝑥 𝑝 𝑥 +𝑞 𝑥
›
Rationale
▶ Need to optimize the model 𝑞, without the direct access to the 𝑝 𝑥 .
𝐶 𝐺 = max 𝑉 𝐺, 𝐷 =
7
= 𝔼#~"(#) log(𝐷/∗ 𝑥 ) + 𝔼#~$! (#) log(1 − 𝐷/∗ 𝑥 ) =
𝑝 𝑥 𝑞 𝑥
= 𝔼#~' log + 𝔼#~$! (#) log
𝑝 𝑥 +𝑞 𝑥 𝑝 𝑥 +𝑞 𝑥
𝑝 𝑥 𝑞 𝑥
𝐶 𝐺 = 𝔼#~' log + 𝔼#~( log
𝑝 𝑥 +𝑞 𝑥 𝑝 𝑥 +𝑞 𝑥
›
Unsupervised Feature Learning
E Denton et al. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
D.. Derkach Generative Modeling Spring 2023 22
LAPGAN: results
TV Window
›
Game Approach Problems
Martin Arjovsky, Towards Principled Methods for Training Generative Adversarial Networks , ICLR17
D.. Derkach Generative Modeling Spring 2023 29
Mode Collapse
▶ When D restarts:
– Easily finds this x ∗ .
– Pushes G to the next point 𝑥 ∗∗ .
▶ 𝐿! = − log 𝐷(𝐺 𝑧 )
"#(%)
▶ ,"% ≈ 0 for generated 𝑥
"'! (%)
▶ ,"% ≈ 0 for generated 𝑥
▶ Generator can’t train!
▶ Need to start closer (how?)
›
Diminishing Gradients
▶ We have seen already that signal data is located on manifold.
▶ GAN case is in fact more complicated, as we need a discriminator that
distinguishes two supports.
▶ This is way too easy, if supports are disjoint.
Martin Arjovsky, Towards Principled Methods for Training Generative Adversarial Networks , ICLR17
D.. Derkach Generative Modeling Spring 2023 37
Feature Matching
›
Reminder: Variational Lower Bound
▶ This is bounded:
𝐷! (𝑃| 𝑄 ≥ max 𝔼#~' 𝑇 𝑥 − 𝔼#~( 𝑓 ∗ 𝑇 𝑥 ,
% #
S. Nowozin et al. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization