Unifying Regularization Term in Auto-encoder

 

Generative Adversarial Networks (GAN)

\[\underset{G}{argmin}\text{ }\underset{D}{argmax}\mathcal{L}(G, D)=\mathbb{E}_{Z\sim P(Z)}[\log{D(Z)}] + \mathbb{E}_{X\sim P(X)}{[\log{(1-D(G(X)))}]}\]

Variational Auto-Encoder (VAE)

Minimize loss.

\[\mathcal{L}(\phi, \theta)=-\mathbb{E}_{X\sim P(X)}[\mathbb{E}_{Z\sim q_\phi(Z|X)}[\log{p_\theta(X|Z)}] - KL(q_\phi(Z|X)||p(Z))]\]

Adversarial Auto-Encoder (AAE)

Minimize both losses.

\[\begin{aligned} \mathcal{L}_G&=-\mathbb{E}_{X\sim P(X)}[\mathbb{E}_{Z\sim q_\phi(Z|X)}[\log{p_\theta(X|Z) - D_\psi(Z)}]] \\ \mathcal{L}_D&=-\mathbb{E}_{X\sim P(X)}[\mathbb{E}_{Z\sim q_\phi(Z|X)}[D_\psi(Z)] + \mathbb{E}_{Z\sim P(Z)}[\log{(1-D_\psi(Z))}]] \end{aligned}\]

Adversarial Variational Bayes (AVB)

Minimize both losses.

\[\begin{aligned} \mathcal{L}_G&=-\mathbb{E}_{X\sim P(X)}[\mathbb{E}_{Z\sim q_\phi(Z|X)}[\log{p_\theta(X|Z) - D_\psi(X, Z)}]] \\ \mathcal{L}_D&=-\mathbb{E}_{X\sim P(X)}[\mathbb{E}_{Z\sim q_\phi(Z|X)}[D_\psi(X,Z)] + \mathbb{E}_{Z\sim P(Z)}[\log{(1-D_\psi(X,Z))}]] \end{aligned}\]