What I cannot create, I do not understand.
— Richard P. Feynman

Calculus of Variations

The cornerstone of this thesis is the Euler-Lagrange (EL) framework, which is at the heart of Calculus of Variations. The EL conditions are of fundamental importance in solving several problems in physics and serve as a functional equivalent to the variable optimization performed in univariate or multivariate Calculus.

Consider the 1-D integral cost

to be optimized with respect to \(y(x)\), \(x \in [a,b] \subset \mathbb{R}\), which is assumed to be continuously differentiable or at least continuous with a piecewise-smooth derivative \(y^{\prime} = \frac{dy}{dx}\), with finite Dirichlet boundary conditions (i.e., the function vanishes on the boundary at \(x=a\) and \(x=b\)). Let \(y^*(x)\) denote the optimizer of \(\mathcal{L}\). The first variation of \(\mathcal{L}\), evaluated at \(y^*\), is defined as the derivative \(\delta\mathcal{L}(y^*;\eta) = \frac{\partial\mathcal{L}_{y,\epsilon}(\epsilon)}{\partial\epsilon}\) evaluated at \(\epsilon=0\), where \(\mathcal{L}_{y,\epsilon}(\epsilon)\) denotes an \(\epsilon\)-perturbation of the argument \(y\) about the optimum \(y^*\), given by

where, in turn, \(\eta(x)\) is a family of compactly supported, infinitely differentiable functions that are identically zero at the boundary \(x = a\) and \(x=b\). The key idea in variational Calculus is the reformulation of a functional optimization over \(y(x)\) into a variable one, over \(\epsilon\).

Another key concept, the fundamental lemma of Calculus of Variations states that if a function \(f(x)\) satisfies the condition

for all compactly supported, infinitely differentiable functions \(\eta(x)\), then \(f\) must be identically zero almost everywhere in \([a,b]\). Setting the first variation to zero and invoking the fundamental lemma of Calculus of Variations gives rise to the Euler-Lagrange condition.

The Euler-Lagrange condition that the optimizer \(y^*(x)\) must satisfy is given as follows:

In the special case where the cost \(\mathcal{L}\) does not involve the derivative of \(y\), the EL condition reduces to the simplified form:

which corresponds to a point-wise optimization with respect to \(y\), over the interval \([y(a),y(b)]\).

The Multivariate Case

In the multivariate case, that is, \(\mathbf{x} \in \mathbb{R}^n\), the cost is of the type

where \(\mathcal{X}\) is the domain of integration and \(y^{\prime}_{i}\) denotes the partial derivative of \(y(\mathbf{x})\) w.r.t. the \(i^{\mathrm{th}}\) entry of \(\mathbf{x}\), that is, \(x_i\), and the boundary is defined as \(\partial\mathcal{X}\). The corresponding EL condition is

The EL condition is a first-order condition and enforcing it yields the optimum. Whether the optimum corresponds to a minimizer or maximizer of the cost must be checked by invoking the second-order condition, more specifically the Legendre-Clebsch necessary condition for a minimizer. In the 1-D case, the condition is given by \(\frac{\partial^2 \mathcal{F}}{\partial y^{\prime 2}} \geq 0.\)

In the multivariate setting, this condition translates to the positive-semi-definiteness (p.s.d.) of the Hessian matrix \(\mathbb{H}\) of the Hamiltonian \(\mathcal{H}\), computed with respect to \(y^{\prime}_i(\mathbf{x})\) for \(i =1,\ldots,n\) and evaluated at \(y(\mathbf{x}) = y^*(\mathbf{x})\): \(\mathbb{H}_{y,\mathcal{H}}\vert_{y=y^*} \geq 0\), where \(\geq\) is used to denote the p.s.d. property. The Hamiltonian is given by

and the entries of the Hessian are given by

As in the 1-D case, when the integral cost \(\mathcal{L}\) (and the integrand \(\mathcal{F}\)) do not depend on the derivatives \(\{y_i^{\prime}\}\), we have the simplified form of the EL condition:

which is identical to the point-wise optimization seen in the scalar case. Similarly, when \(\frac{\partial\mathcal{F}}{\partial x} = 0\), we have the Beltrami identity \(\mathcal{F} - y^{\prime}\frac{\partial\mathcal{F}}{\partial y^{\prime}} = 0\).

The Brachistochrone Problem

The Brachistochrone or “shortest time” problem is considered to be one of the earliest known formulations of a variational problem, the solution to which supposedly gave rise to the field of Variational Calculus. The problem was posed by Johann Bernoulli in 1696 as follows: “Given two points on a plane at different heights, what is the shape of the wire down which a bead will slide (without friction) under the influence of gravity so as to pass from the upper point to the lower point in the shortest amount of time?” Isaac Newton and Jakob Bernoulli showed that the solution is a cycloid. The approach adopted by them was eventually formalized into the Calculus of Variations [1].

The Solution

Given a curve \(y(x)\) joining the points \(\mathrm{A} = (0,a)\) and \(\mathrm{B} = (b,0)\), we can define the boundary conditions as \(y(0) = a\) and \(y(b) = 0\). The speed of a particle on the curve is given by \(v = \frac{ds}{dt} = \sqrt{\left(\frac{dx}{dt}\right)^2 + \left(\frac{dy}{dt}\right)^2}\), where \(s\) is the arc length. Then, we have \(ds = \sqrt{1 + (y^{\prime})^2}\,dx\). From the law of conservation of energy, under the action of gravity, we have \(v = \sqrt{2gy}\), where \(g\) is the acceleration due to gravity. Let the curve between \(\mathrm{A}\) and \(\mathrm{B}\) be of length \(s_y\), traversable in time \(t_y\). The functional optimization problem is given by:

Enforcing the Euler-Lagrange condition (and in particular, the Beltrami identity) yields:

for some constant \(k\). The above differential equation can be solved in polar coordinates, to result in the family of solutions

which are parametric equations of a cycloid.

The Higher-Order Euler-Lagrange Condition

We now recall the generalization to the EL condition, considering higher-order gradient penalties in \(n\)-D.

Consider a vector \(\mathbf{x} = [x_1, x_2,\,\ldots\,,x_n]^{\mathsf{T}} \in \mathbb{R}^n\) and a function \(y:\mathbb{R}^n \rightarrow \mathbb{R}\). The notation \(\nabla_{\mathbf{x}}^m y(\mathbf{x})\) denotes the vector of \(m^{\mathrm{th}}\)-order partial derivatives of \(y\) with respect to the entries of \(\mathbf{x}\). We drop the subscript \(\mathbf{x}\) for convenience. \(\nabla^0\) is the identity operator. The elements of \(\nabla^m y\) are represented using the multi-index \(\boldsymbol{\alpha} = [\alpha_1, \alpha_2,\,\ldots\,,\alpha_n]^{\mathsf{T}}\), as:

where in turn \(\mathbb{Z}_{*}^n\) is the set of \(n\)-dimensional vectors with non-negative integer entries. For example, with \(n=4,m=3\), the index \(\boldsymbol{\alpha} = [2,0,0,1]^{\mathsf{T}}\) yields the element \(\frac{\partial^3}{\partial x_1^2 \partial x_4}y(\mathbf{x})\). Within this formulation, the higher-order gradient vector \(\nabla^m y\) can be operated on similar to the classical first-order gradient vector \(\nabla y\). For example, the square of the \(\ell_2\)-norm of \(\nabla^m y\) is given by a multidimensional sum:

where \(\boldsymbol{\alpha}! = \alpha_1!\alpha_2!\,\ldots\,\alpha_n!\).

Higher-Order EL Condition

Consider an integral cost \(\mathcal{L}\) with the integrand \(\mathcal{F}\) dependent on \(y\) and all its partial derivatives up to and including order \(\ell\), given by

defined on a suitable domain \(\mathcal{X}\) over which \(y\) and its partial derivatives up to and including order \(\ell\) are continuously differentiable.

Consider the perturbations \(y(\mathbf{x})= y^*(\mathbf{x}) + \epsilon \eta(\mathbf{x})\), characterized by \(\eta(\mathbf{x})\) drawn from the family of compactly supported and infinitely differentiable functions. Then, the second-order Taylor-series approximation of \(g(\epsilon) = \mathcal{L}\left(y(\mathbf{x})\right) = \mathcal{L}\left(y^*(\mathbf{x}) + \epsilon \eta(\mathbf{x})\right)\) is given by

where \(\delta\mathcal{L}(\cdot)\) and \(\delta^2\mathcal{L}(\cdot)\) denote the first variation and second variation of \(\mathcal{L}\), respectively, and can be evaluated through the scalar optimization problems [2]:

and

Evaluating the first variation \(\delta\mathcal{L}(y^*;\eta) = \frac{\partial\mathcal{L}_{y,\epsilon}(\epsilon)}{\partial\epsilon}\) at \(\epsilon=0\) yields the Euler-Lagrange condition that the optimizer \(y^*\) must satisfy:

As before, the EL condition merely provides us the extrema, and the second-order conditions must be checked to ascertain if the extrema is a minimizer or maximizer. The second-order Legendre condition for \(y^*(\mathbf{x})\) to be a minimizer is \(g^{\prime\prime}(0) > 0\) [2].

This general higher-order form reduces to the standard first-order condition when \(\ell=1\), and allows us to handle regularized optimization problems involving higher-order derivatives, which will be relevant in the analysis of gradient-penalized GANs and regularized discriminators.

Connection to Generative Adversarial Networks

The Euler-Lagrange framework provides a powerful mathematical foundation for analyzing GANs from a variational perspective. In the context of GANs:

• The discriminator optimization can be viewed as a functional optimization problem where we seek the optimal function \(D^*(\mathbf{x})\) that maximizes a divergence or IPM between distributions. Regularized GANs that incorporate gradient penalties (such as WGAN-GP [3], Sobolev GAN [4]) naturally fit into the higher-order Euler-Lagrange framework, where the regularizers constrain the class of admissible discriminator functions. The simplified EL condition (when derivatives are not involved) can be used to analyze unregularized GAN formulations, while the full EL condition with gradient terms will be essential for understanding regularized GANs. This variational perspective allows us to:
  1. Derive closed-form expressions for optimal discriminators
  2. Analyze the effect of various regularization schemes
  3. Connect GAN optimization to classical problems in signal processing and interpolation theory
• The generator optimization seeks the optimal transformation \(G^*(\mathbf{z})\) that minimizes a loss given the optimal discriminator. What will also be key in this setting, in the time-evolving nature of the generator, as the discriminator changes between iterations. We can further also use the EL setting to understand how GANs link to flow- and score-based diffusion models!!

Navigation

• Previous: Chapter 1.3: Flow-based and Diffusion Models
• Next: Chapter 2: Divergence-Minimizing GANs
• Back to: Thesis Project

References

1. Goldstine, H. H. (1980). A History of the Calculus of Variations from the 17th through the 19th Century. Springer-Verlag.

2. Gelfand, I. M., & Fomin, S. V. (1963). Calculus of Variations. Prentice-Hall.

3. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems 30 (NIPS 2017) (pp. 5767-5777).

4. Mroueh, Y., Sercu, T., & Goel, V. (2018). Sobolev GAN. In International Conference on Learning Representations (ICLR).

5. Jost, J., & Li-Jost, X. (1998). Calculus of Variations. Cambridge University Press.

6. Dacorogna, B. (2007). Introduction to the Calculus of Variations (2nd ed.). Imperial College Press.

7. Evans, L. C. (2010). Partial Differential Equations (2nd ed.). American Mathematical Society.

What I cannot create, I do not understand.
— Richard P. Feynman

Flow-based Models

Flow-based models such as normalizing flows (NFs) [1] leverage the change-of-variables formula to learn a transformation from a parametric prior distribution to the target. Unlike in GANs and AEs, in NF models, the input to model \(\mathbf{z}\) is assumed to be of the same dimensionality as the output, i.e., \(\mathbf{z}\in\mathbb{R}^{n}\). Normalizing flows model a forward process as the gradual corruption of a target distribution \(p_d\) to the noise distribution \(p_z\). The forward process is modeled as a series of invertible transformations \(f_i\), such that the composite function yields the desired push-forward distribution:

Then, by the change-of-variables formula, we have

where \(\left\vert \mathrm{det}\, \mathbf{J}_{f}(\cdot)\right\vert\) denotes the determinant of the Jacobian of \(f\). Consider a vector \(\mathbf{x} = [x_1, x_2,\,\ldots\,,x_n]^{\mathsf{T}} \in \mathbb{R}^n\) and the function \(f:\mathbb{R}^n \rightarrow \mathbb{R}^n\), i.e., \(f(\mathbf{x}) = [f^1(\mathbf{x}), f^2(\mathbf{x}),\ldots,f^n(\mathbf{x})]\). The notation \(\nabla_{\mathbf{x}} f(\mathbf{x})\) represents the gradient matrix associated with the generator, with entries consisting of the partial derivatives of the entries of \(f\) with respect to the entries of \(\mathbf{x}\):

The Jacobian \(\mathbf{J}\) can be thought of as measuring the transformation that the function imposes locally at the point of evaluation, and is defined to be the transpose of the gradient, i.e., \(\nabla_{\mathbf{z}}f(\mathbf{z}) = \mathbf{J}_f^{\mathsf{T}}(\mathbf{z})\). The generative model is then defined as the reverse process, given by:

where \(\mathbf{z}_i = f_i(\mathbf{z}_{i-1})\) with \(\mathbf{z}_0 = \mathbf{z}\) and \(\mathbf{z}_N = \mathbf{x}\). An in-depth analysis of normalizing flows is presented by recent comprehensive surveys [2]. The \(f_i\) network architecture is constrained to facilitate easy computation of the Jacobian. Recent approaches design flows based on autoregressive models [3,4,5,6,7], or architectures motivated by the Sobolev GAN loss [9,10]. Flows have also been explored in the context of improving the quality of the noise input in GANs [8,11,12].

Score-based Diffusion Models

Diffusion models are a recent class of generative models that implement discretized Langevin flow [13], and can be viewed as a random walk in a high-dimensional space. The random walks correspond to a Markov chain, whose stationary distribution converges to the desired distribution. As in the case of flow-based approaches, a forward process is assumed, wherein the data is corrupted with noise, while the generative model is an approximation of the reverse process. Langevin Monte Carlo (LMC) is the canonical algorithm for sampling from a given distribution in this framework, where we have access to the score function, \(\nabla_{\mathbf{x}}\ln p_d(\mathbf{x})\), which is the gradient of the logarithm of the target distribution. LMC is an instance of Markov chain Monte Carlo (MCMC) [16], and is a discretization of the Langevin diffusion stochastic differential equation given by

where \(\mathbf{B}(t)\) is the standard Brownian motion. The associated discretized update is given by

where \(\mathbf{z}_{t}\) is an instance of the noise distribution drawn at time instant \(t\) and is independent of the sample \(\mathbf{x}_{t}\) generated at the corresponding time instant. As in normalizing flows, we have \(\mathbf{z}\in\mathbb{R}^{n}\). The evolution is initialized with parametric noise, i.e., \(\mathbf{x}_0 \sim p_z\). The choice of the mapping function \(f\) gives rise to various diffusion models. For example, the popular family of score-based models [14,15] consider \(f(\mathbf{x}) = \nabla_{\mathbf{x}}\ln p_d(\mathbf{x})\), or in inverse-heat diffusion models [24], we have a frequency-domain transformation corresponding to Gaussian deblurring. In this thesis, we primarily focus on score-based diffusion, and their relation to GAN optimization.

Existing score-based approaches train a neural network \(s_{\theta}(\mathbf{x})\) to approximate the score, by means of a score-matching loss [17], originally considered in the context of independent component analysis:

The output of the trained network is used to generate samples through the annealed Langevin dynamics in noise-conditioned score networks (NCSN) [14]. However, a major limitation is that the predicted score is weak in regions far away from the target distribution, which is typically the case at the start of the Markov chain, around \(\mathbf{x}_0\sim p_z\). Various approaches such as noise scaling [14], sliced SM [18], and denoising SM [13,15] have been proposed to improve the strength of the gradients. Works considering improved discretization of the underlying differential equation [15,20,21,22] have been developed to accelerate the sampling process. Recently, denoising diffusion GANs (DDGANs) [23] were introduced, wherein a GAN is trained to model the diffusion process, with the generator and discriminator networks conditioned on the time index.

Navigation

• Previous: Chapter 1.2: Generative Adversarial Networks
• Next: Chapter 1.4: An Introduction to Variational Calculus
• Back to: Thesis Project

References

1. Rezende, D., & Mohamed, S. (2015). Variational inference with normalizing flows. In International Conference on Machine Learning (ICML) (pp. 1530-1538).

2. Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., & Lakshminarayanan, B. (2021). Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57), 1-64.

3. Dinh, L., Krueger, D., & Bengio, Y. (2015). NICE: Non-linear independent components estimation. In International Conference on Learning Representations (ICLR) Workshop.

4. Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2017). Density estimation using Real NVP. In International Conference on Learning Representations (ICLR).

5. Kingma, D. P., & Dhariwal, P. (2018). Glow: Generative flow using invertible 1×1 convolutions. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018) (pp. 10215-10224).

6. Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016). Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems 29 (NIPS 2016) (pp. 4743-4751).

7. Papamakarios, G., Pavlakou, T., & Murray, I. (2017). Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems 30 (NIPS 2017) (pp. 2338-2347).

8. Chen, X., Kingma, D. P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., & Abbeel, P. (2018). Variational lossy autoencoder. In International Conference on Learning Representations (ICLR).

9. Mroueh, Y., Sercu, T., & Goel, V. (2019). Sobolev descent. In International Conference on Artificial Intelligence and Statistics (AISTATS) (pp. 2976-2984).

10. Gong, C., Wang, D., & Liu, Q. (2020). Unbiased Sobolev descent. In International Conference on Machine Learning (ICML) (pp. 3558-3567).

11. Kumar, M., Weissenborn, D., & Kalchbrenner, N. (2021). Regularizing normalizing flows with Kallenberg-Leibler divergence. In Conference on Uncertainty in Artificial Intelligence (UAI).

12. Dinh, L., Sohl-Dickstein, J., Pascanu, R., & Larochelle, H. (2021). A RAD approach to deep mixture models. In International Conference on Learning Representations (ICLR).

13. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (pp. 6840-6851).

14. Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 11895-11907).

15. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR).

16. Sinclair, A. (1992). Improved bounds for mixing rates of Markov chains and multicommodity flow. Combinatorics, Probability and Computing, 1(4), 351-370.

17. Hyvärinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6, 695-709.

18. Song, Y., & Ermon, S. (2020). Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (pp. 12438-12448).

19. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR).

20. Song, Y., Durkan, C., Murray, I., & Ermon, S. (2020). Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (pp. 1415-1428).

21. Jolicoeur-Martineau, A., Li, K., Piché-Taillefer, R., Kachman, T., & Mitliagkas, I. (2021). Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080.

22. Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (pp. 26565-26577).

23. Xiao, Z., Kreis, K., & Vahdat, A. (2022). Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations (ICLR).

24. Bansal, A., Borgnia, E., Chu, H.-M., Li, J., Kazemi, H., Huang, F., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Cold diffusion: Inverting arbitrary image transforms without noise. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).

What I cannot create, I do not understand.
— Richard P. Feynman

Generative Adversarial Networks (GANs)

The original GAN formulation presented by Goodfellow et al. (2014) [1], (hereafter referred to as the Standard GAN, or SGAN) is a min-max game between two players — a generator (\(G\)), and a discriminator (\(D\)), both of which are typically implemented as neural networks. The role of the generator, akin to the decoder in AEs and VAEs, is to create fake samples that mimic the ones coming from the training data distribution. However, the generated samples are not trained with a one-to-one correspondence with the real/target data samples. Instead, the generator is trained to create samples that appear realistic to a critic, which is the discriminator network \(D\) tasked with telling apart the real samples from the fake ones. The optimal \(G\) is the one that outsmarts \(D\) into confusing the fake samples for real.

We briefly introduce the GAN optimization framework here. An in-depth discussion of various GAN losses and their regularization schemes is presented in Section 1.3, while a comprehensive study of GAN algorithms and training practices for image generation, video synthesis, and neural rendering is provided in recent surveys [2]. The SGAN optimization comprises the generator and the discriminator with respective loss functions. The generator \(G\) accepts high-dimensional noise \(\mathbf{z} \sim p_z\) as input and generates fake samples \(\mathbf{x} = G(\mathbf{z}) \sim p_g\). Unlike in VAEs, the latent space in GANs is not learnt, but assumed to be a standard parametric distribution, typically Gaussian or uniformly distributed in \(\mathbb{R}^{100}\) or \(\mathbb{R}^{128}\). The discriminator \(D\) accepts an input \(\mathbf{x}\), which could come from either the data distribution \(p_d\) as a sample from the dataset \(\mathcal{D}\), or the generator distribution \(p_g\) as the output of the generator. The discriminator outputs a value \(D(\mathbf{x})\), which serves as a measure of how realistic the input sample \(\mathbf{x}\) is. Effectively, the generator must learn a mapping from the noise distribution to the data distribution, whereas the discriminator must learn the optimal two-class classifier.

The SGAN discriminator \(D_{\phi}\) and generator \(G_{\theta}\) are implemented as neural networks and their parameters \((\phi,\theta)\) are optimized as follows:

Ideally, the min-max problem can be solved simultaneously to attain the Nash equilibrium of the two-player game. In practice, however, alternating gradient ascent-descent algorithms are used to train the networks until a suitable convergence criterion is met.

Over the past decade, numerous variants of GANs have been proposed with several successful applications. From an analytical standpoint, the discriminator in these variants can be viewed as approximating a measure of dissimilarity between \(p_d\) and \(p_g\). On the other hand, the generator can be viewed as minimizing the measure that the discriminator approximates. Almost all known GAN flavors consider either a divergence or an integral probability metric. We review important GANs under each category.

Divergence-minimizing GANs

GANs were originally designed to minimize the divergence between the target data distribution \(p_d\) and the generator distribution \(p_g\). For instance, the SGAN formulation corresponds to optimizing the Jensen-Shannon divergence, whereas the least-squares GAN (LSGAN) [3] has been shown to optimize the Pearson-\(\chi^2\) divergence under certain assumptions. The \(f\)-GAN formulation [4] is a generalization to any choice of \(f\)-divergence, while other works [5] proposed extensions to various Bregman divergences. In the original SGAN, the discriminator output is constrained to the interval \([0,1]\), representing the probability of the input sample being real or fake, whereas LSGAN requires the discriminator output to match a set of chosen class-labels. In \(f\)-GANs, the discriminator output is real-valued, but an appropriately chosen activation function maps it to a desired interval.

Integral Probability Metric (IPM) GANs

The divergence metric approaches fail if \(p_d\) and \(p_g\) are of disjoint support [6], which shifted focus to integral probability metrics (IPMs). In these GAN flavors, the discriminator is replaced with a real-valued critic \(C\) that differentiates between the generator and data distributions in terms of an integral probability metric (IPM) defined over the class of critics to choose from. The choice of the class of critics gives rise to variants such as the Wasserstein GAN (WGAN) [7] with a Lipschitz-1 critic, the Fisher GAN [8] in which the second-order moments of the critic are bounded or Sobolev GANs [9], which favor critics with a finite energy in the gradient. The critic is a neural network similar to the discriminator, where the constraints are enforced appropriately, either by means of an adjustment of the network weights [7,10,11], or through a suitable penalty incorporated into the loss function [12,13,14]. Throughout this thesis, we use the term discriminator \(D(\mathbf{x})\) to refer to either a divergence-based discriminator or the IPM-based critic with the context and GAN loss function resolving any ambiguity.

Wasserstein Autoencoders

Wasserstein autoencoders (WAEs) [15] and adversarial autoencoders (AAE) [16] lie at the intersection of AEs and GANs. In these models, a vanilla autoencoder’s [17,18] latent space representation is required to conform to a given prior distribution, usually a Gaussian or a mixture of Gaussians, through an auxiliary network that minimizes the distance between the two distributions. While VAEs achieve this by means of a loss defined via an evidence lower bound, in AAEs and WAEs, the GAN formulation is leveraged.

The encoder-decoder pair is trained by minimizing an appropriate distance measured between the input data distribution and that of the reconstructed samples. In the WAE-GAN setting, the encoder of the WAE also plays the role of the GAN generator, which is trained using a discriminator network to force the latent space distribution to match the prior using a suitable GAN loss. Considering the Euclidean distance metric between a data sample \(\mathbf{x}\) and the corresponding reconstruction \(\tilde{\mathbf{x}}\), and the SGAN loss, we obtain the AAE formulation. The vanilla WAE-GAN formulation employs the Euclidean loss for the reconstruction in combination with the KL divergence for the GAN loss. Sliced WAE [19] extended the framework to accommodate the sliced Wasserstein loss [20]. As an alternative to the adversarial formulation, maximum mean discrepancy (MMD) based variations of the WAE have also gained popularity, the most recent of which, the Cramér-Wold autoencoder (CWAE) [21] presents a characteristic kernel that allows for closed-form computation of the distance between the latent distribution and a standard Gaussian. Optimal transport based approaches either approximate the semi-discrete latent-space transportation map with the continuous Bernier potential by drawing links between the latent-space matching in WAEs and the Monge-Ampère partial differential equation [22,23] or determine the Kantorovich potential in a two-step manner to learn a mapping from the source to the target using a WGAN-GP discriminator [24]. Adding regularizers to the discriminator cost in AAEs has been shown to improve the interpolation capabilities of the AE [25].

A Deep Dive into GANs

Divergence-minimizing GANs consider a discriminator loss that approximates a divergence between \(p_d\) and \(p_g\). Table 1 presents the discriminator and generator loss functions of various divergence-minimizing GANs. The standard GAN (SGAN) proposed by Goodfellow et al. (2014) [1] considers both saturating and non-saturating losses. The term saturation refers to the generator gradients vanishing during training. The vanilla SGAN (employing the saturating loss) results in a min-max zero-sum game where the optimal generator minimizes the Jensen-Shannon divergence between \(p_d\) and \(p_g\). On the contrary, the SGAN with a non-saturating loss (SGAN-NS) does not readily map to a divergence, but results in a superior performance when training the generator in practice. In the least-squares GAN (LSGAN) [3], one minimizes the squared distance between the discriminator prediction and the class labels \((a,b,c)\) for fake, real, and generated samples, respectively. The loss is not symmetric – the discriminator is trained to assign a class label \(a\) to the fakes and a class label \(b\) to the reals, while the generator loss is designed to promote confusion by assigning a label \(c\) to both the real and fake samples. The optimization objective can be seen as minimizing the Pearson-\(\chi^2\) divergence when \(b-c=1\) and \(b-a=2\).

Nowozin et al. (2016) [4] considered \(f\)-divergences of the form:

where the divergence function \(f: \mathbb{R}_+ \rightarrow \mathbb{R}\) is convex, lower-semicontinuous and satisfies \(f(1) = 0\), and \(\mathcal{X}\) is a suitable domain of integration. They demonstrated that, in a GAN setting, the minimization of \(\mathfrak{D}_f(p_g\|p_d)\) is equivalent to the sequential minimization of the discriminator and generator losses:

with respect to \(D\) and \(p_g\), respectively, where \(T(\mathbf{x})\) is the output of the discriminator subjected to activation \(g\), that is, \(T(\mathbf{x}) = g(D(\mathbf{x}))\) and \(T^*(\mathbf{x}) = g(D^*(\mathbf{x}))\), where \(D^*(\mathbf{x})\) is the optimal discriminator, and \(f^{c}\) is the Fenchel conjugate of \(f\). The choice of the divergence \(f\) and the activation \(g\) gives rise to GANs that minimize divergences such as Kullback-Leibler (KL), reverse KL, Pearson-\(\chi^2\), squared Hellinger, Jensen-Shannon, etc.

Table 1: Divergence-Minimizing GAN Losses

Table 1: A summary of various divergence minimizing GAN losses. The SGAN and \(f\)-GAN losses are symmetric and lead to a min-max optimization problem, whereas the LSGAN and SGAN-NS are not symmetric. The LSGAN loss is parametrized by class labels \((a,b,c)\), wherein the discriminator learns a classifier with the class label \(b\) for the reals and a class label \(a\) for the fakes. The LSGAN generator is trained to confuse the discriminator by enforcing the same class label \(c\) to both the reals and the fakes. Various choices of the Csiszár divergence function \(f\) (and consequently, its conjugate \(f^c\)), and activation function \(g\) give rise to the family of \(f\)-GANs.

IPM-Based GANs and Regularization

The divergence metric approaches fail if \(p_d\) and \(p_g\) are of disjoint support [6], which shifted focus to integral probability metrics (IPMs), where a critic function is chosen to approximate a chosen IPM between the distributions [7,8]. Choosing the IPM is equivalent to constraining the class of functions from which the critic/discriminator is drawn. The most popular variant, inspired by optimal transport, is the Wasserstein GAN (WGAN) [7], in which the objective is to minimize the Wasserstein-1 or earth mover’s distance between \(p_d\) and \(p_g\), and the critic is constrained to be Lipschitz-1. Gulrajani et al. (2017) [12] enforced a first-order gradient penalty on the discriminator network to approximate the Lipschitz constraint. Here, the loss is defined via the Kantorovich–Rubinstein duality, subjected to a regularizer \(\Omega_D\):

where \(\lambda_d\) is the Lagrange multiplier associated with the regularizer. Table 2 summarizes the IPM-GAN losses, the corresponding constraint space of the discriminator, and the ideal form of the regularizer \(\Omega_D\). However, these regularizers are usually replaced with an implementable counterpart, typically involving first-order gradient-based terms [12,14,31,32]. On the other hand, several works [13,27,28,33] showed the empirical success of the first-order gradient penalty in divergence-minimizing GANs. Cramér GANs [34] and Sobolev GANs [9,29] bound the energy in the gradients of \(D\), enforcing Sobolev-space constraints.

Table 2: IPM-GAN Losses

Table 2: A summary of various IPM-GAN losses considered in the literature, compared against an \(f\)-GAN loss. Various choices of the \(f\)-divergence function give rise to the family of \(f\)-GANs. Other GAN variants consider the standard Wasserstein-1 IPM, with varying choices for the discriminator regularizer (the closed-form metric), which in turn induces a constraint space for the discriminator. WGAN enforces a Lipschitz-1 discriminator by means of weight-clipping on the network, while Fisher GANs draw discriminators with finite \(L_2\) norms. While Cramér GANs enforce a gradient-norm penalty in 1-D with a metric defined over the cumulative density functions (CDF), Sobolev GANs consider a generalization involving the sum of all marginal CDFs (indexed by \(i\)). Maximum mean discrepancy GANs (MMD-GANs) derived a kernel-mean statistic associated with the RKHS \(\mathscr{H}_{\phi_{k}}\).

Navigation

• Previous: Chapter 1.1: Introduction to Generative Modeling
• Next: Chapter 1.3: Flow-based and Diffusion Models
• Back to: Thesis Project

References

1. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems 27 (NIPS 2014) (pp. 2672-2680).

2. Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021). Generative adversarial networks for image and video synthesis: Algorithms and applications. Proceedings of the IEEE, 109(5), 839-862.

3. Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., & Smolley, S. P. (2017). Least squares generative adversarial networks. In International Conference on Computer Vision (ICCV) (pp. 2813-2821).

4. Nowozin, S., Cseke, B., & Tomioka, R. (2016). f-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems 29 (NIPS 2016) (pp. 271-279).

5. Tao, C., Chen, L., Henao, R., Feng, J., & Carin, L. (2018). Chi-square generative adversarial network. In International Conference on Machine Learning (ICML) (pp. 4843-4852).

6. Arjovsky, M., & Bottou, L. (2017). Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations (ICLR).

7. Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein generative adversarial networks. In International Conference on Machine Learning (ICML) (pp. 214-223).

8. Mroueh, Y., & Sercu, T. (2017). Fisher GAN. In Advances in Neural Information Processing Systems 30 (NIPS 2017) (pp. 2513-2523).

9. Mroueh, Y., Sercu, T., & Goel, V. (2018). Sobolev GAN. In International Conference on Learning Representations (ICLR).

10. Li, Z., Hu, Z., Luo, S., Chen, S., Gao, K., Chen, C., Yang, J., & Xu, B. (2019). Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR).

11. Wang, D., Gong, Z., & Liu, Q. (2016). Stein variational gradient descent as gradient flow. In Advances in Neural Information Processing Systems 29 (NIPS 2016).

12. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems 30 (NIPS 2017) (pp. 5767-5777).

13. Kodali, N., Abernethy, J., Hays, J., & Kira, Z. (2017). On convergence and stability of GANs. arXiv preprint arXiv:1705.07215.

14. Mescheder, L., Geiger, A., & Nowozin, S. (2018). Which training methods for GANs do actually converge? In International Conference on Machine Learning (ICML) (pp. 3481-3490).

15. Tolstikhin, I., Bousquet, O., Gelly, S., & Schoelkopf, B. (2018). Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR).

16. Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B. (2015). Adversarial autoencoders. arXiv preprint arXiv:1511.05644.

17. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length and Helmholtz free energy. In Advances in Neural Information Processing Systems 6 (NIPS 1993) (pp. 3-10).

18. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.

19. Kolouri, S., Pope, P. E., Martin, C. E., & Rohde, G. K. (2019). Sliced Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR).

20. Kolouri, S., Zou, Y., & Rohde, G. K. (2018). Sliced Wasserstein kernels for probability distributions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5258-5267).

21. Patrini, G., van den Berg, R., Forré, P., Carioni, M., Bhargav, S., Welling, M., Genewein, T., & Roth, F. (2020). Sinkhorn autoencoders. In Conference on Uncertainty in Artificial Intelligence (UAI) (pp. 733-743).

22. Fan, L., Dai, W., Lee, D.-J., & Liu, W. (2019). On the effectiveness of optimal transport for generative modeling. arXiv preprint arXiv:1901.11373.

23. Patrini, G., van den Berg, R., Forré, P., Carioni, M., Bhargav, S., Welling, M., Genewein, T., & Roth, F. (2020). Optimal transport autoencoders. In International Conference on Learning Representations (ICLR).

24. Rubenstein, P., Schoelkopf, B., & Tolstikhin, I. (2018). On the latent space of Wasserstein auto-encoders. arXiv preprint arXiv:1802.03761.

25. Ghosh, P., Sajjadi, M. S. M., Vergari, A., Black, M., & Schölkopf, B. (2019). From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12436.

26. Peyré, G., & Cuturi, M. (2019). Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6), 355-607.

27. Kodali, N., Abernethy, J., Hays, J., & Kira, Z. (2017). On convergence and stability of GANs. arXiv preprint arXiv:1705.07215.

28. Kodali, N., Hays, J., Abernethy, J., & Kira, Z. (2017). How to train your DRAGAN. arXiv preprint arXiv:1705.07215.

29. Adler, J., & Lunz, S. (2018). Banach Wasserstein GAN. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018) (pp. 6754-6763).

30. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems 30 (NIPS 2017) (pp. 5767-5777).

31. Petzka, H., Fischer, A., & Lukovnikov, D. (2018). On the regularization of Wasserstein GANs. In International Conference on Learning Representations (ICLR).

32. Patel, A. B., Nguyen, T., & Baraniuk, R. G. (2020). A probabilistic framework for deep learning. In Advances in Neural Information Processing Systems 29 (NIPS 2016).

33. Mescheder, L., Nowozin, S., & Geiger, A. (2018). The numerics of GANs. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018) (pp. 1825-1835).

34. Bellemare, M. G., Danihelka, I., Dabney, W., Mohamed, S., Lakshminarayanan, B., Hoyer, S., & Munos, R. (2017). The Cramer distance as a solution to biased Wasserstein gradients. arXiv preprint arXiv:1705.10743.

What I cannot create, I do not understand.
— Richard P. Feynman

Generative Models

Generative modeling is a branch of machine learning that addresses the problem of learning the distribution of given data samples, which can subsequently be used to create more such samples. Generative models have gained immense popularity in the generation of images [1], text [2], and audio [3] and learning multi-modal transformations [4]. In this thesis, we focus on image datasets, and consider generative modelling for images. Consider the dataset \(\mathcal{D} = \big\{\mathbf{x}_i~\|~\mathbf{x}_i\in\mathbb{R}^n,\,i=1,2,\ldots,N\big\}\), with samples drawn from an unknown distribution \(p_d\) (the data distribution). We are considering \(\mathbf{x}_i\) to be the vectorized representation of an image, with \(n = h\times w\times c\), where \((h,w,c)\) denote the height, width, and the number of channels in the image. This representation also allows one to consider multichannel images, such as color images, multispectral images, and hyperspectral images.

Neural-network-based generative models output samples \(\tilde{\mathbf{x}}\) that are governed by a parametric distribution \(p_{\text{model}}(\tilde{\mathbf{x}};\theta)\). The goal in optimizing/training the neural network model is to learn the optimal parameters \(\theta^*\) such that the generated samples appear realistic, and \(p_{\text{model}}(\tilde{\mathbf{x}};\theta^*) = p_d\). Early neural-network-based generative models, going back to the 1980s, are energy-based models (EBMs) that define an energy functional on data samples. However, these models relied on Markov chain Monte Carlo (MCMC) for sampling [5], and could not be scaled to high-dimensional objects such as images.

Modern approaches to generative modelling rely on the Manifold Hypothesis [6], which posits that although the ambient dimension of images is large \((\mathbf{x}\in\mathbb{R}^n)\), they actually reside in a much lower-dimensional manifold \((\mathbf{z}\in\mathbb{R}^d)\) with \(d\ll n\), often several orders lower. Leveraging this property, generative models have targeted nearly invertible low-dimensional latent representations \(\{\mathbf{z}_i\}\) of images \(\{\mathbf{x}_i\}\). Autoencoders (AEs) [7], variational autoencoders (VAEs) [8], Wasserstein autoencoders (WAEs) [9], generative adversarial networks (GANs) [10], and variants thereof, are all examples of such generative models. By contrast, advanced frameworks such as normalizing flows (NFs) [11] and diffusion models (DMs) [13] operate on the ambient dimensionality of the data and are far more challenging to train and intensive in terms of memory and compute. Our focus in this thesis is predominantly on GANs, with occasional connections to WAEs, NFs, and DMs.

We briefly introduce the various generative models considered in this thesis.

Autoencoders

Autoencoders comprise a pair of neural networks, namely the encoder \((\text{Enc}_{\phi})\) and decoder \((\text{Dec}_{\theta})\), trained to learn a compressed (low-dimensional) representation of the data \(\mathcal{D}\). Given a sample \(\mathbf{x}_i\), the encoder learns a latent representation of the data \(\mathbf{z}_i = \text{Enc}_{\phi}(\mathbf{x}_i)\), while the decoder learns to invert the encoder transformation, \(\tilde{\mathbf{x}}_i = \text{Dec}_{\theta}(\mathbf{z}_i)\) such that \(\tilde{\mathbf{x}}_i \approx \mathbf{x}_i\). While the encoder and decoder networks are typically fully-connected multi-layer perceptron (MLP) networks, convolutional AEs (CAEs) consider encoders with convolutional layers, while the decoders consist of transposed convolution layers. While autoencoders were originally proposed as the nonlinear counterpart of principal component analysis (PCA), in the recent literature, AEs have been leveraged to implement flow-based and generative pre-training architectures.

The standard AE optimization involves training the encoder-decoder pair to generate accurate reconstructions. Given the dataset \(\mathcal{D}\), the \(N\)-sample estimate of the AE loss is given by

where in turn, \(\phi^*\) and \(\theta^*\) denote the optimal encoder and decoder parameters:

Typical choices for the distance measure include the \(\ell_1\) or \(\ell_2\) loss:

where \(p = 1\) or \(2\), respectively, or a weighted combination of these. Regularized autoencoders have also been considered, wherein the encoder and decoder networks are regularized to either be smooth transformations [12] or learn sparse representations [15]. Although autoencoders are generative models in the sense that the decoders learn to output realistic images given a latent representation, a major limitation is their inability to generate new/unseen images. This can be attributed to the lack of a closed-form or parametric structure to the latent-space distribution, which results in inferior reconstructions of latent-space interpolations. The variational and Wasserstein autoencoder schemes alleviate this issue, by enforcing constraints on the distribution of the latent representations.

Variational Autoencoders

While AEs consider the encoder output as a deterministic representation of the input image, in variational autoencoders (VAEs) [8], the latent representations are assumed to be random, with the objective of modelling the joint distribution \(p_{\text{model}}(\mathbf{x},\mathbf{z})\), from which the data distribution can be computed as \(p_d(\mathbf{x}) = \int_{\mathcal{Z}} p_{\text{model}}(\mathbf{x},\mathbf{z})\,d\mathbf{z}\), where \(\mathcal{Z}\) denotes the support of the latent-space distribution. Invoking Bayes’ rule, we have \(p_{\text{model}}(\mathbf{x},\mathbf{z}) = \frac{p_{\text{model}}(\mathbf{x}\|\mathbf{z}) p_z(\mathbf{z})}{p_{\text{model}}(\mathbf{z}\|\mathbf{x})}\). The decoder can be interpreted as approximating the posterior, i.e., \(\text{Dec}_{\theta}(\mathbf{z}) \sim p_{\text{model}}(\mathbf{x}\|\mathbf{z})\). As the underlying data distribution is inaccessible, the encoder models an approximation of the likelihood, \(\text{Enc}_{\phi}(\mathbf{x}) \sim q(\mathbf{z}\|\mathbf{x}) \approx p_{\text{model}}(\mathbf{z}\|\mathbf{x})\). In VAEs, one optimizes the so called evidence lower bound (ELBO), given by

where \(D_{\text{KL}}\) denotes the Kullback-Leibler (KL) divergence between two distributions. The reconstruction error, as in the case of AEs, can be approximated in terms of the \(\ell_2\) or \(\ell_1\) loss. The KL divergence can be further simplified by assuming a Gaussian model \(p_z(\mathbf{z})\), with the latent sample defined by means of a change-of-variable formula as \(\mathbf{z}_i = \mu_{\phi}(\mathbf{x}_i)+ \sigma_{\phi}(\mathbf{x}_i)\odot\boldsymbol{\epsilon}\), where \((\mu_{\phi},\sigma_{\phi})= \text{Enc}_{\phi}(\mathbf{x}_i)\) and \(\boldsymbol{\epsilon}\) is drawn from the standard normal distribution, i.e., \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0}_d,\mathbb{I}_d)\), where in turn, \(\mathbf{0}_d\) denotes the \(d\)-dimensional null vector, \(\mathbb{I}_d\) is the \(d\)-dimensional identity matrix, and \(\odot\) stands for pointwise multiplication. Although VAEs are superior to AEs in terms of enforcing structure on the latent space, generating novel unseen samples remains challenging. Further, the use of the \(\ell_2\) cost results in over-smoothing of the reconstructed images, owing to the implicit Gaussian prior that this choice of loss introduces [16]. Generative adversarial networks (GANs) [10] were proposed as an alternative to VAEs, to alleviate its limitations.

Navigation

• Next: Chapter 1.2: Generative Adversarial Networks
• Back to: Thesis Project

References

1. Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023). Scaling up GANs for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

2. Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., & Wang, J. (2018). Long text generation via adversarial training with leaked information. Proceedings of the AAAI Conference on Artificial Intelligence, 32.

3. Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., & Roberts, A. (2019). GANSynth: Adversarial neural audio synthesis. In International Conference on Learning Representations (ICLR).

4. Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., Hutchinson, B., Han, W., Parekh, A., Li, X., Zhang, H., Baldridge, J., & Wu, Y. (2022). Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789.

5. Kalos, M. H., & Whitlock, P. A. (1986). Monte Carlo Methods. Wiley Publications.

6. Carlsson, G., Ishkhanov, T., de Silva, V., & Zomorodian, A. (2008). On the local behavior of spaces of natural images. International Journal of Computer Vision, 76(1), 1-12.

7. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length and Helmholtz free energy. In Advances in Neural Information Processing Systems 6 (NIPS 1993) (pp. 3-10).

8. Kingma, D. P., & Welling, M. (2013). Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114.

9. Tolstikhin, I., Bousquet, O., Gelly, S., & Schoelkopf, B. (2018). Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR).

10. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems 27 (NIPS 2014) (pp. 2672-2680).

11. Rezende, D., & Mohamed, S. (2015). Variational inference with normalizing flows. In International Conference on Machine Learning (ICML) (pp. 1530-1538).

12. Liang, K., Chang, H., Cui, Z., Shan, S., & Chen, X. (2015). Representation learning with smooth autoencoder. In 12th Asian Conference on Computer Vision (pp. 72-86).

13. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (pp. 6840-6851).

14. Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., & Glorot, X. (2011). Higher order contractive auto-encoder. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 645-660). Springer.

15. Ng, A. (2011). Sparse autoencoder. CS294A Lecture Notes, 72, 1-19.

16. Figueiredo, M. (2001). Adaptive sparseness using Jeffreys prior. In Advances in Neural Information Processing Systems 2001, 14.