Generative Adversarial Networks

GANs (mainly in image synthesis)

PreviousSemantic Segmentation NextStyle Transfer

Last updated 3 years ago

Was this helpful?

Generative Adversarial Networks

GANs (mainly in image synthesis)

Survey Papers / Repos

Are GANs Created Equal? A Large-Scale Study
Which Training Methods for GANs do actually Converge?
A Large-Scale Study on Regularization and Normalization in GANs

Resources

: TensorFlow-GAN
: IS, FID implementation in TF, PyTorch

Models

Loss functions

Vanilla GAN
EBGAN
LSGAN
WGAN
BEGAN
Hinge Loss

Regularization

Architecture

Conditional GANs

Others

Tricks

- Assumption
  - MEANINGFUL: The generated image should be clear, the output probability of a classifier network should be [0.9, 0.05, ...] (largely skewed to a class). $p(y|\mathbf{x})$ is of low entropy.
  - DIVERSITY: If we have 10 classes, the generated image should be averagely distributed. So that the marginal distribution $p(y) = \frac{1}{N} \sum_{i=1}^{N} p(y|\mathbf{x}^{(i)})$ __is of high entropy.
  - Better models: KL Divergence of $p(y|\mathbf{x})$ and $p(y)$ should be high.
- Formulation
  - $\text{IS} = \exp (\mathbb{E}_{\mathbf{x} \sim p_g} D_{KL} [p(y|\mathbf{x}) || p(y)] )$
  - where
    $\mathbf{x}$ is sampled from generated data
    $p(y|\mathbf{x})$ is the output probability of Inception v3 when input is $\mathbf{x}$
    $p(y) = \frac{1}{N} \sum_{i=1}^{N} p(y|\mathbf{x}^{(i)})$ is the average output probability of all generated data (from InceptionV3, 1000-dim vector)
    $D_{KL} (\mathbf{p}||\mathbf{q}) = \sum_{j} p_{j} \log \frac{p_j}{q_j}$ , where $j$ is the dimension of the output probability.
- Reference
- Formulation
  - $\text{FID} = ||\mu_r - \mu_g||^2 + Tr(\Sigma_{r} + \Sigma_{g} - 2(\Sigma_r \Sigma_g)^{1/2})$
  - where
    $X_r \sim \mathcal{N}(\mu_r, \Sigma_r)$ and $X_g \sim \mathcal{N}(\mu_g, \Sigma_g)$ are the 2048-dim activations the Inception v3 pool3 layer
    $\mu_r$ is the mean of real photo's feature
    $\mu_g$ is the mean of generated photo's feature
    $\Sigma_r$ is the covariance matrix of real photo's feature
    $\Sigma_g$ is the covariance matrix of generated photo's feature
- Reference

PreviousSemantic Segmentation NextStyle Transfer

Last updated 3 years ago

Was this helpful?

Survey Papers / Repos

Are GANs Created Equal? A Large-Scale Study
Which Training Methods for GANs do actually Converge?
A Large-Scale Study on Regularization and Normalization in GANs

Resources

: TensorFlow-GAN
: IS, FID implementation in TF, PyTorch

Models

Loss functions

Vanilla GAN
EBGAN
LSGAN
WGAN
BEGAN
Hinge Loss

Regularization

Gradient Penalty
DRAGAN
SNGAN
Consistency Regularization

Architecture

Deep Convolution GAN (DCGAN)
Progressive Growing of GANs (PGGAN)
Self Attention GAN (SAGAN)
BigGAN
Style based Generator (StyleGAN)
Mapping Network (StyleGAN)
LOGAN: Latent Optimisation for Generative Adversarial Networks

Conditional GANs

Vanilla Conditional GANs
Auxiliary Classifer GAN (ACGAN)

Others

Tricks

Two time-scale update rule (TTUR)
Self-Supervised GANs via Auxiliary Rotation Loss (SS-GAN)

Metrics (my implementation: )

Inception Score
- Assumption
  - MEANINGFUL: The generated image should be clear, the output probability of a classifier network should be [0.9, 0.05, ...] (largely skewed to a class). $p(y|\mathbf{x})$ is of low entropy.
  - DIVERSITY: If we have 10 classes, the generated image should be averagely distributed. So that the marginal distribution $p(y) = \frac{1}{N} \sum_{i=1}^{N} p(y|\mathbf{x}^{(i)})$ __is of high entropy.
  - Better models: KL Divergence of $p(y|\mathbf{x})$ and $p(y)$ should be high.
- Formulation
  - $\text{IS} = \exp (\mathbb{E}_{\mathbf{x} \sim p_g} D_{KL} [p(y|\mathbf{x}) || p(y)] )$
  - where
    $\mathbf{x}$ is sampled from generated data
    $p(y|\mathbf{x})$ is the output probability of Inception v3 when input is $\mathbf{x}$
    $p(y) = \frac{1}{N} \sum_{i=1}^{N} p(y|\mathbf{x}^{(i)})$ is the average output probability of all generated data (from InceptionV3, 1000-dim vector)
    $D_{KL} (\mathbf{p}||\mathbf{q}) = \sum_{j} p_{j} \log \frac{p_j}{q_j}$ , where $j$ is the dimension of the output probability.
- Reference
  - Official TF implementation is in
  - Pytorch Implementation:
  - TF seemed to provide a
FID Score
- Formulation
  - $\text{FID} = ||\mu_r - \mu_g||^2 + Tr(\Sigma_{r} + \Sigma_{g} - 2(\Sigma_r \Sigma_g)^{1/2})$
  - where
    $Tr$ is
    $X_r \sim \mathcal{N}(\mu_r, \Sigma_r)$ and $X_g \sim \mathcal{N}(\mu_g, \Sigma_g)$ are the 2048-dim activations the Inception v3 pool3 layer
    $\mu_r$ is the mean of real photo's feature
    $\mu_g$ is the mean of generated photo's feature
    $\Sigma_r$ is the covariance matrix of real photo's feature
    $\Sigma_g$ is the covariance matrix of generated photo's feature
- Reference
  - Official TF implementation:
  - Pytorch Implementation:
  - TF seemed to provide a

$Tr$ is