From Probabilistic Autoencoders to VAEs David I. Inouye
Dynamic Content · Last Updated: November 9, 2026
Arc 7 · Image generation and evaluation
Thirty seconds. Continue from autoencoders and denoising autoencoders. First replace deterministic point mappings with conditional distributions, following the instructor’s original slides. Then add the generative model and develop VAE inference and training.
Probability primer
Distributions, expectations, likelihood, and KL
Twenty seconds. Before making the encoder and decoder probabilistic, review the probability tools we will use.
A distribution describes possibilities; a sample is one outcome
For a latent vector, \(\mathbf{z}\sim p\) describes a random draw; \(\boldsymbol{z}\) denotes one realized value.
Discrete: \(p(\boldsymbol{x})\geq0\) and \(\sum_{\boldsymbol{x}}p(\boldsymbol{x})=1\) .Continuous: \(p(\boldsymbol{x})\geq0\) and \(\int p(\boldsymbol{x})\,d\boldsymbol{x}=1\) ; probability is area , not density height.
Three minutes. Introduce the probability primer before probabilistic autoencoders. A random variable maps an uncertain outcome to a number; a random vector collects multiple such numbers. A distribution supplies probabilities. Upright bold denotes a random vector; bold italic a realized vector. The plot uses one scalar feature for readability. Discrete values 0 and 1 have probabilities .25 and .75. Uniform density .5 over [0,2] integrates to one; [.5,1.5] has probability .5. For an absolutely continuous variable the probability of one exact value is zero, yet its density can be positive and even exceed one. The same distinction applies in many-dimensional image space. Modeling images continuously is a choice even though stored pixel values are quantized. [Sources] source:goodfellow-probability-2016, sections 3.2–3.3. Adapted in scope from the instructor’s Spring 2026 probability review; examples and plots are original.
Gaussian distributions specify a center and a spread
For one feature, \(\textnormal{z}\sim\mathcal{N}(\mu,\sigma^2)\) : mean \(\mu\) , variance \(\sigma^2\) , standard deviation \(\sigma\) .
For vectors, \(\mathbf{z}\sim\mathcal{N}(\boldsymbol{\mu},\Sigma)\) uses a covariance matrix .
Diagonal entries are feature variances; off-diagonal entries are covariances.
\(\Sigma=I\) : independent Gaussian features, each with variance one.
Three minutes. The second Gaussian argument is variance for a scalar and covariance for a vector, never standard deviation. The narrower curve is taller to preserve area one. Covariance records pairwise linear co-variation; diagonal covariance implies independence here because the distribution is jointly Gaussian, not for arbitrary distributions. The standard normal vector also has zero mean. N(mu,diag(sigma²)) lets each feature have its own spread. We need this notation for the probabilistic AE, VAE encoder and diffusion noise, not the full Gaussian normalization or covariance derivations. [Sources] source:goodfellow-probability-2016, sections 3.8 and 3.9.3. Adapted in scope from the instructor’s Spring 2026 probability review; examples and plots are original.
Marginalization sums or integrates out a variable
The joint distribution \(p(\boldsymbol{x},\boldsymbol{z})\) describes images and codes together. To describe images alone, remove the code variable:
Discrete codes: \[p(\boldsymbol{x})=\sum_{\boldsymbol{z}}p(\boldsymbol{x},\boldsymbol{z}).\]
Continuous codes: \[p(\boldsymbol{x})=\int p(\boldsymbol{x},\boldsymbol{z})\,d\boldsymbol{z}.\]
Similarly, \(p(\boldsymbol{z})=\int p(\boldsymbol{x},\boldsymbol{z})\,d\boldsymbol{x}\) for continuous images.
Three minutes. Define the joint before using it, then introduce the marginal before the conditional. Sum or integrate over the variable being removed, across its full support; this does not hold it fixed. For discrete images use a sum over x in the final expression. The joint itself is nonnegative and sums/integrates to one over both variables. Marginals inherit those properties. [Sources] source:goodfellow-probability-2016, sections3.3–3.4.
A conditional distribution holds one quantity fixed
Divide the joint by the marginal of the quantity being held fixed:
\[p(\boldsymbol{x}\mid\boldsymbol{z})=\frac{p(\boldsymbol{x},\boldsymbol{z})}{p(\boldsymbol{z})}
\quad\text{for }p(\boldsymbol{z})>0.\]
\[p(\boldsymbol{z}\mid\boldsymbol{x})=\frac{p(\boldsymbol{x},\boldsymbol{z})}{p(\boldsymbol{x})}
\quad\text{for }p(\boldsymbol{x})>0.\]
\(p(\boldsymbol{x}\mid\boldsymbol{z})\) : possible images for a fixed code .\(p(\boldsymbol{z}\mid\boldsymbol{x})\) : possible codes for a fixed image .
For fixed \(\boldsymbol{z}\) , \(\int p(\boldsymbol{x}\mid\boldsymbol{z})\,d\boldsymbol{x}=1\) ; use a sum for discrete images.
Three minutes. The vertical bar reads given. Its notation is not a division symbol, but the conditional is defined by this joint-to-marginal ratio wherever the denominator is positive. Continuous conditioning uses densities, not positive point-event probabilities. The conditional is nonnegative and normalized over the variable before the bar. No independence or causal story is required. [Sources] source:goodfellow-probability-2016, sections3.5–3.6.
The chain rule and Bayes’ rule use the same joint in two directions
Multiply each conditional definition by its denominator:
\[p(\boldsymbol{x},\boldsymbol{z})
=p(\boldsymbol{z})p(\boldsymbol{x}\mid\boldsymbol{z})
=p(\boldsymbol{x})p(\boldsymbol{z}\mid\boldsymbol{x}).\]
Rearrange to get Bayes’ rule , for \(p(\boldsymbol{x})>0\) :
\[p(\boldsymbol{z}\mid\boldsymbol{x})=
\frac{p(\boldsymbol{x}\mid\boldsymbol{z})p(\boldsymbol{z})}{p(\boldsymbol{x})}.\]
\(p(\boldsymbol{z})\) : code distribution before seeing the image (prior ).\(p(\boldsymbol{z}\mid\boldsymbol{x})\) : code distribution after seeing it (posterior ).
The denominator comes from marginalization: \(p(\boldsymbol{x})=\int p(\boldsymbol{z})p(\boldsymbol{x}\mid\boldsymbol{z})\,d\boldsymbol{z}\) .
Three minutes. Preserve the meanings of x and z in both factorizations; these are not variable renamings. The product rule does not assume independence. Bayes reverses the conditioning direction and normalizes the result. For discrete codes, the denominator uses a sum. This is one joint model; a later generative model chooses its prior and decoder factors, and its learned encoder approximates the resulting posterior. [Sources] source:goodfellow-probability-2016, sections3.4–3.6 and3.11.
An expectation is a probability-weighted average
If a loss is \(1\) with probability \(0.75\) and \(5\) with probability \(0.25\) , its expectation is \(0.75(1)+0.25(5)=2\) .
For a function of a continuous code,
\[\mathbb{E}_{\mathbf{z}\sim q}[h(\mathbf{z})]
=\int h(\boldsymbol{z})q(\boldsymbol{z})\,d\boldsymbol{z}
\ \approx\ \frac{1}{m}\sum_{j=1}^{m}h(\boldsymbol{z}^{(j)}),
\qquad \boldsymbol{z}^{(j)}\text{ sampled from }q.\]
The integral is the expectation; the sample average estimates it.
One sample gives a noisy estimate, not the exact average.
For reconstruction, fix the image and average its loss over codes drawn from its encoder.
Three minutes. A discrete expectation uses a sum weighted by probabilities; the opening arithmetic demonstrates why an unweighted average of the distinct loss values would be wrong. The sample average has equal weights because the samples themselves occur according to q. Use independent draws for the elementary law-of-large-numbers interpretation and assume integrability. More draws generally reduce estimation noise; do not promise monotonic accuracy for each sample batch. VAE training also averages over images, a separate average. h is a placeholder for a scalar function such as loss, not necessarily the identity. Linearity gives E[ah+bg]=aE[h]+bE[g] when finite, but E[g(z)] generally differs from g(E[z]); leave these in notes unless needed. [Sources] source:goodfellow-probability-2016, section 3.8. Adapted in scope from the instructor’s Spring 2026 probability review; examples and plots are original.
Likelihood evaluates an observation we already have
Sampling: hold the model fixed and draw a new observation.Likelihood: hold the observed \(\boldsymbol{x}\) fixed and evaluate \(p_\theta(\boldsymbol{x})\) as the model changes.A common loss is \(-\log p_\theta(\boldsymbol{x})\) : higher density gives lower loss.
For a decoder, evaluate \(p_\theta(\boldsymbol{x}\mid\boldsymbol{z})\) at the original image .
Three minutes. The illustration compares N(0,1) and N(2,1) at the same scalar observed value .3; all values are analytic, not model results. The scalar picture stands in for an image-density evaluation. A likelihood is the probability mass or density of fixed observed data considered as a function of model parameters, not a probability distribution over parameters. For a continuous model, density at a point is not the probability of that exact point. Negative log density may be negative if density exceeds one; only the ordering matters here. In reconstruction we sample latent codes, then evaluate the original image under the decoder; no new output image draw is needed to compute likelihood. [Sources] source:goodfellow-probability-2016, sections 3.3, 3.13; source:kingma-welling-vae-2013, section 2. Adapted in scope from the instructor’s Spring 2026 probability review; examples and plots are original.
Probabilistic autoencoders
Replace point outputs with distributions
Twenty seconds. Use the probability primer to change both encoder and decoder mappings, then train them to reconstruct the original image.
A deterministic AE maps one point to one point
\[\boldsymbol{z}=f_\phi(\boldsymbol{x}),\qquad
\widehat{\boldsymbol{x}}=g_\theta(\boldsymbol{z}).\]
Each + is one example or code. The outlines show the spaces where they live.
Two minutes. Reuse the visual grammar of the instructor’s probabilistic-autoencoder drawing: observed space, latent space, individual plus marks, then distributions in later slides. Use one observed space on the left and one latent space on the right. The encoder goes out; the decoder returns to the original space so reconstruction is compared with the original point. Each plus in observed space stands for a whole image, not one pixel. The space boundaries are schematic, not support estimates or learned manifolds. For a fixed input, both deterministic functions produce fixed outputs. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
A probabilistic encoder maps an image to a distribution of codes
Replace \(\boldsymbol{z}=f_\phi(\boldsymbol{x})\) with a conditional distribution \(q_\phi(\boldsymbol{z}\mid\boldsymbol{x})\) .
The yellow circle is a code distribution; its + marks are sampled codes . The decoder still returns one point per code to the original observed space.
Three minutes. Start with the observed plus on the left. The yellow circle is the encoder’s output distribution, not one large point. Point to each latent plus as a possible sampled code for this same input. The model predicts distribution parameters; a sampling operation produces a code. Shaded regions are schematic probability contours, not hard boundaries: Gaussian samples can lie outside them. At this intermediate stage the decoder is still deterministic: its green plus marks are outputs, not samples from an image distribution. We have not introduced an unconditional latent prior. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
A probabilistic decoder maps each code to a distribution of images
Replace \(\widehat{\boldsymbol{x}}=g_\theta(\boldsymbol{z})\) with \(p_\theta(\boldsymbol{x}\mid\boldsymbol{z})\) .
During training, measure the likelihood of the original input image under the distributions represented by the green circles. We sample latent codes, not new images from the decoder .
Four minutes. Follow one latent plus and its green arrow to one green output circle, then the other. A green circle represents the decoder distribution for that code, returning to the same observed space as the original input. There are no green sampled-image marks. Training evaluates the original observed point’s density under each green distribution; it does not draw an image from that distribution. Decoder distribution means need not equal the original input exactly. The original observed plus remains the reconstruction target, not a fresh output draw. This is a conceptual redrawing of the instructor’s circles-and-plus illustration; no model was trained. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
Gaussian mappings let the networks predict the distribution means
For a first example, hold both covariance matrices fixed at the identity:
\[q_\phi(\boldsymbol{z}\mid\boldsymbol{x})
=\mathcal{N}\!\left(\boldsymbol{z};f_\phi(\boldsymbol{x}),I\right).\]
\[p_\theta(\boldsymbol{x}\mid\boldsymbol{z})
=\mathcal{N}\!\left(\boldsymbol{x};g_\theta(\boldsymbol{z}),I\right).\]
\(f_\phi\) predicts the center of the latent distribution.\(g_\theta\) predicts the center of the image distribution.We can also learn the spreads; first keep them fixed to see the mechanism.
Three minutes. This matches the instructor’s original fixed-covariance probabilistic-AE bridge. I has size k by k in the encoder and d by d in the decoder. The neural networks remain deterministic parameter-producing functions; the conditional mappings are probabilistic. Fixed unit observation variance is a simplifying likelihood assumption, not a good default for every normalized image dataset. Diagonal learned encoder covariance comes later. Do not yet call q an approximate posterior of a joint model, because we have not specified the prior. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
A probabilistic AE can train with expected reconstruction loss
For one observed image, average over sampled codes:
\[\mathcal{J}_{\mathrm{PAE}}(\boldsymbol{x})=
\mathbb{E}_{\mathbf{z}\sim q_\phi(\cdot\mid\boldsymbol{x})}
[-\log p_\theta(\boldsymbol{x}\mid\mathbf{z})].\]
With the fixed Gaussian mappings, one noise draw gives the estimate
\[\tfrac12\left\|\boldsymbol{x}-g_\theta\!\left(f_\phi(\boldsymbol{x})+
\boldsymbol{\epsilon}\right)\right\|_2^2+C,
\qquad \boldsymbol{\epsilon}\sim\mathcal{N}(\boldsymbol{0},I).\]
Evaluate the original \(\boldsymbol{x}\) under the green decoder distribution. Sample codes, not output images.
Four minutes. Follow the original slides’ expected conditional negative-log-likelihood derivation. For unit-covariance Gaussian p, negative log density is half squared error plus d log(2pi)/2. Replace the expectation over codes with z=f(x)+epsilon, then use one sampled epsilon for a Monte Carlo estimate. This is an estimate, not equality to the full expectation. Average this per-image loss over training examples. This is conditional reconstruction likelihood, not the marginal likelihood of a fully specified generative model. No KL term or unconditional latent prior has been added. Connect directly to the returning green circles: evaluate p_theta at the original x for each sampled z. No output image sample is needed to compute this loss. Formal reparameterization gradients come later. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
Where does the noise enter these two autoencoders ?
Both models reconstruct the original clean image.
Which description matches the models we just defined?
A. Denoising AE: noisy input; probabilistic AE: sampled latent code.
B. Denoising AE: sampled latent code; probabilistic AE: noisy input.
C. Both models must use the noisy input as their target.
A. We changed the input for denoising, then changed the mappings for probabilistic autoencoding.
Three minutes including vote and reveal. A is the best answer for the concrete constructions presented. B swaps the location of the noise; C confuses input with target. A probabilistic decoder also defines output randomness, but its likelihood loss evaluates the original target rather than requiring an output draw. The distinctions are pedagogical choices, not mutually exclusive model families: probabilistic encoders can also be used for denoising. This question checks the transition from denoising inputs to probabilistic mappings. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
VAEs: a generative model
Pair a latent prior with a probabilistic decoder
Twenty seconds. Move from reconstructing an observed image to assumptions that define how to generate a new image.
Fitting the generative model means averaging over possible codes
For an observed image, the model assigns density
\[p_\theta(\boldsymbol{x})=
\int p(\boldsymbol{z})\,p_\theta(\boldsymbol{x}\mid\boldsymbol{z})\,d\boldsymbol{z}.\]
We would like to maximize \(\sum_i\log p_\theta(\boldsymbol{x}_i)\) .
The code that produced an observed image is unknown.
With a nonlinear decoder, this integral is generally difficult to compute exactly.
The encoder helps identify codes that could explain each observed image.
Three minutes. Explain marginalization as combining contributions of different possible latent codes under the prior. Do not imply all codes receive equal weight. Intractability is general for nonlinear neural decoders, not a claim about every latent model. Direct sampling from the prior is possible but can be inefficient for explaining a particular observation. This motivates learned inference and a tractable lower bound, as in the instructor’s old generative-model slide. No derivation of the bound on this slide. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
The VAE encoder approximates the codes that explain an image
The prior \(p(\boldsymbol{z})\) is shared; the posterior \(p_\theta(\boldsymbol{z}\mid\boldsymbol{x})\) depends on this image.
\[q_\phi(\boldsymbol{z}\mid\boldsymbol{x})=
\mathcal{N}\!\left(\boldsymbol{z};\boldsymbol{\mu}_\phi(\boldsymbol{x}),
\operatorname{diag}(\boldsymbol{\sigma}_\phi^2(\boldsymbol{x}))\right)
\quad\text{approximates }p_\theta(\boldsymbol{z}\mid\boldsymbol{x}).\]
Four minutes. The figure is an original schematic, not an inferred posterior from the trained demo. Several separated code regions can explain the same image under a nonlinear decoder; a diagonal Gaussian can miss modes and correlations. Conditioning uses the image pixels, not a supplied class label. Connect deterministic and probabilistic AE mappings to the generative model and variational inference role. The exact posterior follows from the prior and decoder by Bayes’ rule but is generally difficult to evaluate because of the marginal likelihood. q is a tractable learned approximation. Mu and positive sigma have k entries; diag squares each standard deviation into a covariance diagonal. q is approximating p_theta(z|x), not the unconditional prior. This distinction explains the ELBO identity in the continuation. The former fixed-I encoder is a restricted special case. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
Training a VAE
Use the encoder to fit the generative model
Twenty seconds. The encoder approximates the posterior. Now use it to build a tractable objective and compute training gradients.
Encoder samples let us estimate a usable training objective
For a fixed image, draw codes from \(q_\phi(\boldsymbol{z}\mid\boldsymbol{x})\) and define the evidence lower bound (ELBO)
\[\mathcal{L}(\theta,\phi;\boldsymbol{x})=
\mathbb{E}_{\mathbf{z}\sim q_\phi(\cdot\mid\boldsymbol{x})}
\left[\log p_\theta(\boldsymbol{x},\mathbf{z})
-\log q_\phi(\mathbf{z}\mid\boldsymbol{x})\right].\]
Evaluate the joint with \(p_\theta(\boldsymbol{x},\boldsymbol{z})=p(\boldsymbol{z})p_\theta(\boldsymbol{x}\mid\boldsymbol{z})\) .
These densities are available; the difficult marginal \(p_\theta(\boldsymbol{x})\) is absent.
Average sampled values to estimate \(\mathcal{L}\) .
Training objective: maximize \(\mathcal{L}\) over $ heta,$ (equivalently, minimize \(-\mathcal{L}\) ).
Next, derive why it is a lower bound on image log likelihood.
Three minutes. Introduce the actual quantity before claiming it is computable. The expectation may still require Monte Carlo estimation; tractability means we can evaluate its integrand without the intractable marginal or true posterior. Gaussian KL will later remove sampling from part of this estimate. The next slide explains the lower-bound guarantee. Keep x fixed and average over encoder codes. [Sources] source:kingma-welling-vae-2013, section 2.2, equations 1–3.
Multiplying and dividing by the encoder separates the ELBO and its gap
Fix \(\boldsymbol{x}\) ; abbreviate \(q=q_\phi(\boldsymbol{z}\mid\boldsymbol{x})\) . Every expectation below is under \(q\) .
\(\log p_\theta(\boldsymbol{x})\)
\(=\)
\(\mathbb{E}_q[\log p_\theta(\boldsymbol{x})]\)
(fixed image)
\(\,\)
\(=\)
\(\mathbb{E}_q\!\left[\log\frac{p_\theta(\boldsymbol{x},\mathbf{z})}{p_\theta(\mathbf{z}\mid\boldsymbol{x})}\right]\)
(chain rule)
\(\,\)
\(=\)
\(\mathbb{E}_q\!\left[\log\left(\frac{p_\theta(\boldsymbol{x},\mathbf{z})}{q_\phi(\mathbf{z}\mid\boldsymbol{x})}\frac{q_\phi(\mathbf{z}\mid\boldsymbol{x})}{p_\theta(\mathbf{z}\mid\boldsymbol{x})}\right)\right]\)
(multiply by \(q/q\) )
\(\,\)
\(=\)
\(\mathbb{E}_q\!\left[\log\frac{p_\theta(\boldsymbol{x},\mathbf{z})}{q_\phi(\mathbf{z}\mid\boldsymbol{x})}\right]+\mathbb{E}_q\!\left[\log\frac{q_\phi(\mathbf{z}\mid\boldsymbol{x})}{p_\theta(\mathbf{z}\mid\boldsymbol{x})}\right]\)
(split the log)
\(\,\)
\(=\)
\({\color{teal}{\mathcal{L}(\theta,\phi;\boldsymbol{x})}}+D_{\mathrm{KL}}\!\left(q_\phi(\cdot\mid\boldsymbol{x})\,\|\,p_\theta(\cdot\mid\boldsymbol{x})\right)\)
(definitions)
Five minutes. Follow the instructor’s Spring 2026 derivation: put the fixed log evidence inside an expectation, use the posterior factorization, multiply and divide by q, then split the logarithm into two expectations. The first is exactly the previously defined ELBO; the second is the KL to the true posterior. Assume positive densities on the support and finite expectations, as in the Gaussian examples. Keep the first equality visible; reveal one justified row at a time. [Sources] source:kingma-welling-vae-2013, section2.2, equations1–3. Adapted from lectures/ece473-spring-2026/14-autoencoders.qmd, Derivation of VAE Objective.
The ELBO gives a lower bound on image log likelihood
For one image, the following identity connects inference and generation:
\[\log p_\theta(\boldsymbol{x})=
\underbrace{\mathcal{L}(\theta,\phi;\boldsymbol{x})}_{\text{ELBO}}
+\underbrace{D_{\mathrm{KL}}\!\left(q_\phi(\cdot\mid\boldsymbol{x})\,\|\,
p_\theta(\cdot\mid\boldsymbol{x})\right)}_{\text{gap from approximate inference}}.\]
Because KL is nonnegative, \(\mathcal{L}(\theta,\phi;\boldsymbol{x})\leq\log p_\theta(\boldsymbol{x})\) .
Fit \(\theta\) and \(\phi\) by maximizing this evidence lower bound .
The bound is tight when the encoder equals the true posterior.
Two minutes. Read the conclusion of the preceding revealed derivation. Define L=E_q[log p_theta(x,z)-log q(z|x)]. Substitute p_theta(x,z)=p_theta(z|x)p_theta(x), separate the expectation, and rearrange to obtain the displayed identity. This KL compares the encoder with the true posterior, not with the prior. The next slide rewrites the same ELBO using the joint factorization to obtain a different, tractable KL against the prior. Joint optimization need not monotonically tighten the gap because theta changes the true posterior. [Sources] source:kingma-welling-vae-2013, sections 2–3; source:goodfellow-autoencoders-2016, chapter 14. Adapted from the instructor’s Spring 2026 autoencoder lecture.
The VAE loss balances reconstruction and agreement with the prior
Factor \(p_\theta(\boldsymbol{x},\boldsymbol{z})=p(\boldsymbol{z})p_\theta(\boldsymbol{x}\mid\boldsymbol{z})\) inside the ELBO:
\[\mathcal{L}=\mathbb{E}_q[\log p_\theta(\boldsymbol{x}\mid\mathbf{z})]
+\mathbb{E}_q\!\left[\log\frac{p(\mathbf{z})}{q_\phi(\mathbf{z}\mid\boldsymbol{x})}\right].\]
Minimize the negative ELBO:
\[\mathcal{J}=-\mathcal{L}=
\underbrace{\mathbb{E}_q[-\log p_\theta(\boldsymbol{x}\mid\mathbf{z})]}_{\text{reconstruction penalty}}
+\underbrace{D_{\mathrm{KL}}\!\left(q_\phi(\cdot\mid\boldsymbol{x})\,\|\,p(\cdot)\right)}_{\text{prior regularization}}.\]
\(D_{\mathrm{KL}}(q_\phi(\cdot\mid\boldsymbol{x})\|p_\theta(\cdot\mid\boldsymbol{x}))\) True posterior: inference error for this image
\(D_{\mathrm{KL}}(q_\phi(\cdot\mid\boldsymbol{x})\|p(\cdot))\) Prior: regularization toward shared sampling codes
Five minutes. Define negative log likelihood as a penalty for assigning low probability density to the observed image. The expectation averages over codes from this image’s encoder distribution, not over prior samples. The ELBO identity just introduced compares q with the true posterior. Factoring p_theta(x,z)=p(z)p_theta(x|z) instead rewrites negative ELBO as reconstruction plus KL to the prior. These two KL terms are different. Both theta and phi are learned; in this example p is fixed. This is the negative evidence lower bound, or negative ELBO. Minimizing it maximizes a lower bound on log p_theta(x); do not claim it is exactly maximum likelihood or that the approximate posterior must equal the prior. The preceding identity motivates the bound; its factorization gives this computable form. For continuous densities negative log likelihood need not be nonnegative, though its optimization role is unchanged. [Sources] source:kingma-welling-vae-2013, sections 2–3 and appendix B.
\(\beta\) controls how strongly codes must agree with the prior \[\mathcal{J}_\beta=
\underbrace{\mathbb{E}_q[-\log p_\theta(\boldsymbol{x}\mid\mathbf{z})]}_{\text{reconstruction}}
+\beta\,\underbrace{D_{\mathrm{KL}}(q_\phi(\cdot\mid\boldsymbol{x})\|p(\cdot))}_{\text{prior regularization}}.\]
\(\beta=0\) Probabilistic autoencoder: reconstruction alone
\(0<\beta<1\) Weaker prior regularization
\(\beta=1\) Standard VAE: exactly the negative ELBO
\(\beta>1\) Stronger prior regularization (β-VAE )
More prior pressure can reduce information about the image and worsen reconstruction.
Side point: as \(\beta\to\infty\) , \(q_\phi(\boldsymbol{z}\mid\boldsymbol{x})=p(\boldsymbol{z})\) gives zero KL. A sufficiently expressive decoder could still fit images with \(p_\theta(\boldsymbol{x}\mid\boldsymbol{z})=p_\theta(\boldsymbol{x})\) , although \(\mathbf{x}\) and \(\mathbf{z}\) are independent.
Four minutes. Beta is the conventional name for this weighting, verified against Higgins et al. section2 equation4. Hold reconstruction scaling fixed when comparing beta. At beta0 this is the earlier probabilistic AE, with no constraint making prior samples useful. For beta other than1 this is not the ordinary negative ELBO; for beta>=1 its negative remains a possibly looser lower bound because the additional KL penalty is nonnegative, whereas beta<1 has no general lower-bound guarantee. Increased beta may encourage simpler representations but does not guarantee disentanglement. Side point only: with KL zero feasible and reconstruction bounded below, increasing beta strongly favors q=p. A sufficiently expressive decoder could still model data with p_theta(x|z)=p_theta(x), making x and z independent and the true posterior equal to the prior. Then integrating out z recovers p_theta(x); good marginal generation need not require useful codes. A fixed-variance Gaussian decoder independent of z is only one Gaussian, so cannot generally fit a multimodal image distribution. Do not claim finite beta always collapses or a simple decoder can reproduce arbitrary data. [Sources] source:higgins-beta-vae-2017, section2 equation4; source:kingma-welling-vae-2013, section2. The independence example is an explicit mathematical consequence, not an experimental claim.
A Gaussian decoder turns likelihood into scaled squared error
Part 1 of the VAE objective: reconstruction.
Let the decoder predict an image mean \(g_\theta(\boldsymbol{z})\) with fixed variance \(\tau^2\) :
\[p_\theta(\boldsymbol{x}\mid\boldsymbol{z})
=\mathcal{N}\!\left(\boldsymbol{x};g_\theta(\boldsymbol{z}),\tau^2 I\right).\]
Then, for a \(d\) -feature image,
\[-\log p_\theta(\boldsymbol{x}\mid\boldsymbol{z})
=\frac{\|\boldsymbol{x}-g_\theta(\boldsymbol{z})\|_2^2}{2\tau^2}
+\frac{d}{2}\log(2\pi\tau^2).\]
\(\tau\) is the decoder observation spread ; \(\boldsymbol{\sigma}\) describes encoder codes.With fixed \(\tau\) , the last term is constant during training.
Changing the error scale changes its balance with KL.
Three minutes. Reconnect the earlier unit-covariance probabilistic-AE example to the VAE reconstruction term. The norm sums squared errors over features; dividing that sum by d without also rescaling KL changes the objective. Tau is fixed here, so dropping the constant does not change optimization; if tau is learned, the log-variance term must remain. A Gaussian likelihood is an explicit observation-model choice, not a requirement for all VAEs. [Sources] source:kingma-welling-vae-2013, appendix C.2.
For Gaussian codes, we can calculate KL directly
Part 2 of the VAE objective: regularization. Compare the encoder’s approximate posterior with the prior :
\[q_\phi(\cdot\mid\boldsymbol{x})=\mathcal{N}\!\left(\boldsymbol{\mu}_\phi(\boldsymbol{x}),
\operatorname{diag}(\boldsymbol{\sigma}_\phi^2(\boldsymbol{x}))\right),
\qquad p(\cdot)=\mathcal{N}(\boldsymbol{0},I).\]
\[D_{\mathrm{KL}}\!\left(q_\phi(\cdot\mid\boldsymbol{x})\,\|\,p(\cdot)\right)
=\frac{1}{2}\sum_{j=1}^{k}\left(\mu_j^2+\sigma_j^2-1-\log\sigma_j^2\right).\]
Here \(\boldsymbol{\mu}=\boldsymbol{\mu}_\phi(\boldsymbol{x})\) and \(\boldsymbol{\sigma}=\boldsymbol{\sigma}_\phi(\boldsymbol{x})\) are this image’s encoder outputs.
Mean \(\boldsymbol{0}\) and spread \(\boldsymbol{1}\) match the prior and give zero KL.
This term is analytic: no latent-code sampling is needed .
Three minutes. Suppress phi and x in mu and sigma only for readability; these remain encoder outputs for the current image. The sum is over k latent features, not d pixels. Natural logs give units of nats. Both extremely small and extremely large standard deviations cost KL. This is the negative of the sign convention used for the KL contribution to the maximized ELBO in appendix B; our minimized objective has positive KL. [Sources] source:kingma-welling-vae-2013, sections 2–3 and appendix B.
The encoder changes the distribution inside the expectation
To maximize the ELBO, we need \(\nabla_{\theta,\phi}\mathcal{L}\) (or minimize \(\mathcal{J}=-\mathcal{L}\) ). The Gaussian KL is explicit; reconstruction still needs an expectation.
Let \(\ell_\theta(\boldsymbol{z})=-\log p_\theta(\boldsymbol{x}\mid\boldsymbol{z})\) for fixed \(\boldsymbol{x}\) :
\[\nabla_\phi\mathbb{E}_{\mathbf{z}\sim q_\phi}[\ell_\theta(\mathbf{z})]
=\nabla_\phi\int q_\phi(\boldsymbol{z}\mid\boldsymbol{x})\ell_\theta(\boldsymbol{z})\,d\boldsymbol{z}
\ne \mathbb{E}_{q_\phi}[\nabla_\phi\ell_\theta(\mathbf{z})]\quad\text{in general}.\]
At a fixed code, \(\ell_\theta\) has no direct \(\phi\) dependence; the naive right side is zero.
But changing the encoder changes which codes are likely .
We need to move \(\phi\) into a differentiable function of parameter-independent noise .
Four minutes. The issue is not that differentiating an integral is forbidden. Under regularity one can differentiate q times the integrand; the missing derivative of q is precisely the problem with naively moving the gradient into an expectation and holding the sampled z fixed. Score-function estimators are another option, but not today’s method. Theta has an ordinary gradient inside the expectation here because q depends on phi, not theta. [Sources] source:kingma-welling-vae-2013, section2.3.
Fixed noise lets backpropagation follow the encoder-to-loss path
Hold \(\boldsymbol{\epsilon}\) fixed for this gradient calculation ; differentiate the scale and shift, not the random draw.
Three minutes. Orange arrows show reconstruction gradients moving right to left. The decoder gradient works in both diagrams; the encoder reconstruction gradient needs the pathwise transformation in the bottom diagram. A black-box random draw is not an ordinary deterministic differentiable operator; this does not rule out score-function estimators. Once the noise is an input, partial z_j/partial mu_j=1 and partial z_j/partial sigma_j=epsilon_j. The analytic KL adds another encoder gradient path. Resample noise on the next update. [Sources] source:kingma-welling-vae-2013, section2.3. Original computational-graph illustration.
Monte Carlo sampling estimates the expectation and its gradient
Draw \(M\) independent noises and form codes via the reparameterization trick :
\[\boldsymbol{\epsilon}^{(m)}\sim\mathcal{N}(\boldsymbol{0},I),\qquad
\boldsymbol{z}^{(m)}=\boldsymbol{\mu}_\phi(\boldsymbol{x})+
\boldsymbol{\sigma}_\phi(\boldsymbol{x})\odot\boldsymbol{\epsilon}^{(m)}.\]
\[\widehat{\mathcal{J}}_\beta=
\underbrace{\frac{1}{M}\sum_{m=1}^{M}-\log p_\theta(\boldsymbol{x}\mid\boldsymbol{z}^{(m)})}_{\text{Monte Carlo reconstruction estimate}}
+\beta\,\underbrace{D_{\mathrm{KL}}(q_\phi(\cdot\mid\boldsymbol{x})\|p(\cdot))}_{\text{closed-form Gaussian KL}}.\]
Backpropagate \(\nabla_{\theta,\phi}\widehat{\mathcal{J}}_\beta\) ; at \(\beta=1\) , it estimates \(-\nabla_{\theta,\phi}\mathcal{L}\) .
Usually \(M=1\) per image ; average over a minibatch and draw fresh noise each update.Reparameterization supplies the gradient path; Monte Carlo approximates the integral.
Four minutes. The estimator and its pathwise gradient are unbiased for the corresponding fixed-image objective under the usual interchange conditions; a single draw has variance and need not equal the expectation. Averaging across images also estimates the dataset objective. Gaussian KL is evaluated exactly from encoder parameters, not estimated using that single code. The VAE paper reports L=1 with sufficiently large minibatches; use M here to avoid confusing sample count with calligraphic L for ELBO. [Sources] source:kingma-welling-vae-2013, section2.3, equations6–7 and algorithm1.
Each minibatch trains the encoder and decoder together
For a batch \(\{\boldsymbol{x}_i\}_{i=1}^{B}\) , compute all \(B\) examples in parallel:
1. Encode \((\boldsymbol{\mu}_i,\boldsymbol{s}_i)=f_\phi(\boldsymbol{x}_i)\) ; \(\boldsymbol{s}_i\) is log variance
2. Sample codes \(\boldsymbol{\epsilon}_i\sim\mathcal{N}(\boldsymbol{0},I)\) ; \(\boldsymbol{z}_i=\boldsymbol{\mu}_i+\exp(\boldsymbol{s}_i/2)\odot\boldsymbol{\epsilon}_i\)
3. Decode and score \(r_i=-\log p_\theta(\boldsymbol{x}_i\mid\boldsymbol{z}_i)\) ; \(k_i=\tfrac12\sum_j(\mu_{ij}^2+e^{s_{ij}}-1-s_{ij})\)
4. Average the loss \(\widehat{\mathcal{J}}_\beta=\tfrac1B\sum_{i=1}^{B}(r_i+\beta k_i)\)
5. Update both networks \(\theta\leftarrow\theta-\eta\nabla_\theta\widehat{\mathcal{J}}_\beta\) ; \(\phi\leftarrow\phi-\eta\nabla_\phi\widehat{\mathcal{J}}_\beta\)
One fresh code per image. At \(\beta=1\) , the batch loss estimates the average negative ELBO.
Five minutes. The equations use per-image vectors while the implementation stacks their transposes into matrices: X has shape[B,d]; means, log variances, epsilon and Z have shape[B,k]. Both encoder and decoder run on the whole batch. Draw independent noise for every image and latent feature. The decoder computes the conditional distribution parameters for each Z row; r_i evaluates the original X row under that distribution. For the preceding fixed-Gaussian likelihood, r_i is summed squared error divided by2tau^2 plus its constant; for the later Bernoulli demo it is summed binary cross entropy. Sum KL over latent features j and reconstruction over image features, then average over batch index i. Never average over latent features in one term while silently changing the relative scale of the other. Exp and multiplication act elementwise. Backpropagate once through the averaged loss, keeping the sampled epsilon fixed for this gradient, and then update both parameter sets using gradients from the same forward pass. The displayed updates are plain SGD for clarity; Adam can use these same gradients. With beta1 and one code per image this is the minibatch Monte Carlo estimate of negative ELBO. [Sources] source:kingma-welling-vae-2013, algorithm1 and appendixB.
What happens if we train using only the KL term ?
The prior is \(\mathcal{N}(\boldsymbol{0},I)\) and the encoder can represent it exactly.
Which encoder minimizes KL for every image?
A. A different narrow distribution for each image.
B. A point concentrated at zero for every image.
C. The same standard Gaussian for every image.
C. Matching both center and spread gives zero KL; that encoder distribution carries no image-specific information.
Three minutes including vote and reveal. A favors identifying individual images, but that is not the KL-only goal. B matches the mean but not the spread; Gaussian KL grows without bound as variance approaches zero. The full objective adds reconstruction, which rewards informative codes. KL-only training does not train the decoder; no image-quality conclusion follows. Do not claim the full VAE necessarily collapses. This second question tests why the objective needs both terms. [Sources] source:kingma-welling-vae-2013, section2.2 and appendixB.
After training, the encoder provides representations and reconstructions
Represent: \(\boldsymbol{x}\mapsto\boldsymbol{\mu}_\phi(\boldsymbol{x})\) gives a feature vector for downstream tasks.Reconstruct: encode an image, then decode its mean code (or a sampled code).
Selected held-out MNIST examples; this demo uses a Bernoulli decoder . Reconstructions can lose detail.
Three minutes. The encoder is useful after training even though generation itself only needs the decoder. Mu is a convenient deterministic representation; sigma also describes uncertainty in the approximate posterior. Code usefulness requires downstream evaluation. Real reproducible demo, not invented outputs: scripts/vae_usage_demo.py and images/07-02-variational-autoencoders/usage-demo.json. Seed570; MNIST archived training images0:12000 used for fitting,59000:60000 held out internally; 40epochs, two latent features, beta1, Bernoulli observation model. Selected held-out offsets[0,6,2,3,4,5]. Instructor requested replacing offset1, whose3 reconstructs ambiguously as8, with offset6 for mechanism clarity; these selected illustrations are not a quality estimate. Display decoder means. This demo uses a Bernoulli decoder; the earlier Gaussian likelihood was an algebra example. [Sources] source:mnist-train-images; source:kingma-welling-vae-2013, sections3 and5.
Interpolation uses encoded endpoints; generation needs only the decoder
Interpolate: encode two images, then decode \(\boldsymbol{z}(a)=(1-a)\boldsymbol{\mu}_1+a\boldsymbol{\mu}_2\) .
Generate: draw \(\boldsymbol{z}\sim p(\boldsymbol{z})\) and decode; no input image or encoder is needed.
Decoder means shown; smooth interpolation does not guarantee good prior samples.
Three minutes. Same reproducible model as the preceding slide. Selected internally held-out offsets[0,6] provide the interpolation endpoints, with seven equally spaced weights; offset1 was replaced at instructor request as on the preceding slide. These are selected illustrations, not representative-quality evidence; generation uses the first eight standard-normal codes with seed571, without selection. A full image sample also draws from the decoder’s Bernoulli likelihood; plotting its mean makes the model’s expected pixels readable. Interpolation codes are not generally prior draws. Show imperfect samples as well as recognizable ones; this small two-feature model is not a benchmark. [Sources] source:mnist-train-images; source:kingma-welling-vae-2013, sections3 and5.
A trained VAE still needs evidence for its intended use
Reconstruct an observed image
Held-out reconstruction under the stated loss
Generate new images
Fresh prior samples: quality, diversity and failure cases
Use the learned code for another task
Held-out performance on that downstream task
Audit the assumptions: chosen prior, decoder likelihood and approximate-posterior family.
A good training objective gives a learning procedure; it does not guarantee all three uses succeed.
Three minutes. Close the VAE story before moving to diffusion. These are separate evaluations, not interchangeable demonstrations. A diagonal Gaussian encoder can limit posterior approximation, and the chosen likelihood/prior may be a poor fit; optimization and finite data are additional limitations. No new measured results are claimed. This is a concise local audit; the full value-driven evaluation unit still follows diffusion. [Sources] source:kingma-welling-vae-2013, sections2 and5; source:goodfellow-autoencoders-2016, chapter14.
