Generating images by diffusion David I. Inouye
Dynamic Content · Last Updated: November 11, 2026
Arc 7 · Image generation and evaluation
Noisy scores → learned fields → multiple levels → SDE/ODE sampling
Thirty seconds. Start with the familiar denoising task, remove the architectural restriction, and change the prediction target. The circle example then motivates multiple noise levels before introducing an additive Gaussian walk. This teaching parameterization follows the earlier course slides; it is not the variance-preserving DDPM schedule. [Sources] source:goodfellow-autoencoders-2016, section14.5; source:ho-ddpm-2020, sections2–3 for the general forward/reverse framework.
Denoising needs a predictor, not a required bottleneck
\[\widehat{\boldsymbol{x}}=D_\theta(\widetilde{\boldsymbol{x}}_\sigma),\qquad
\min_\theta\;\mathbb E\!\left[\|\boldsymbol{x}-D_\theta(\boldsymbol{x}+\sigma\boldsymbol{\epsilon})\|^2\right].\]
Directly map a noisy image to a clean-image prediction of the same shape .
We can use any suitable neural network; an explicit latent code is optional.
An encoder–decoder or U-Net can still be a useful architecture.
Two minutes. Separate the statistical task from the architecture. We no longer require a separately interpreted code or a compression bottleneck; this does not prohibit downsampling and upsampling inside a U-Net. Both expectations average over clean training data and independent standard Gaussian noise. [Sources] source:goodfellow-autoencoders-2016, section14.5; source:ho-ddpm-2020, section3.3.
Predicting the clean image and predicting its noise are equivalent
For \(\widetilde{\boldsymbol{x}}_\sigma=\boldsymbol{x}+\sigma\boldsymbol{\epsilon}\) at a fixed \(\sigma>0\) :
\[\widehat{\boldsymbol{x}}=\widetilde{\boldsymbol{x}}_\sigma-\sigma\epsilon_{\theta,\sigma}(\widetilde{\boldsymbol{x}}_\sigma),
\qquad \widehat{\boldsymbol{\epsilon}}=\frac{\widetilde{\boldsymbol{x}}_\sigma-D_{\theta,\sigma}(\widetilde{\boldsymbol{x}}_\sigma)}{\sigma}.\]
The squared errors differ by a constant:
\[\|\boldsymbol{x}-\widehat{\boldsymbol{x}}\|^2
=\sigma^2\|\boldsymbol{\epsilon}-\widehat{\boldsymbol{\epsilon}}\|^2.\]
Same prediction problem; a different output parameterization.
Three minutes. Define epsilon_theta as predicting standard noise, not sigma times noise. For any clean predictor the residual gives a noise predictor and conversely. With compatible function classes the fixed-sigma population optima correspond. This does not promise identical optimization trajectories for separately parameterized networks. Across varying sigma the factor changes: unweighted clean MSE and unweighted noise MSE weight levels differently. Show that algebra before changing the objective. Original derivation from the stated corruption equation; compare the noise parameterization in source:ho-ddpm-2020, section3.2.
The noise we draw becomes a supervised target
\[J(\theta)=\mathbb E_{\mathbf{x},\boldsymbol{\epsilon}}
\left[\|\boldsymbol{\epsilon}-\epsilon_{\theta,\sigma}(\boldsymbol{x}+\sigma\boldsymbol{\epsilon})\|^2\right].\]
No noise labels need to be collected: we constructed the corruption .
Two minutes. Sigma is still one fixed positive noise level. Expectation notation denotes draws from the clean-data distribution and independent standard Gaussian noise; bold italic epsilon is the sampled realization inside the computation. The network sees only the corrupted input, while the known noise goes to the loss. It generally cannot recover the exact realized noise for every ambiguous input. [Sources] source:ho-ddpm-2020, section3.2, for noise prediction; this fixed-noise version is an original specialization.
One minibatch update learns noise prediction at one level
Fix \(\sigma>0\) . For each batch of \(B\) clean images:
Draw noise
\(\boldsymbol{\epsilon}_i\sim\mathcal N(\boldsymbol 0,I)\) independently
Corrupt
\(\widetilde{\boldsymbol{x}}_{\sigma,i}=\boldsymbol{x}_i+\sigma\boldsymbol{\epsilon}_i\)
Predict
\(\widehat{\boldsymbol{\epsilon}}_i=\epsilon_{\theta,\sigma}(\widetilde{\boldsymbol{x}}_{\sigma,i})\)
Average loss
\(\widehat J=\frac1B\sum_i\|\boldsymbol{\epsilon}_i-\widehat{\boldsymbol{\epsilon}}_i\|^2\)
Update
\(\theta\leftarrow\theta-\eta\nabla_\theta\widehat J\)
Can this one denoiser also turn random noise into a clean image?
Three minutes. Images and predictions retain batch-first shape [B,C,H,W]; the squared norm sums image entries and the average is over examples. The draw is independent across images. End with the limitation question, which the score and learned-field comparison answer next; this is a rhetorical transition, not a second poll. [Sources] source:ho-ddpm-2020, algorithm1, adapted here to a fixed additive noise level.
From denoising to noisy scores
What the predictor learns—and why one noise level is insufficient
Twenty seconds. Connect the supervised noise-prediction task to the noisy density, prove the score identity, then compare learned fields at small and large noise.
Denoising learns the score of the noisy distribution
\[\widetilde{\boldsymbol{x}}_\sigma=\boldsymbol{x}+\sigma\boldsymbol{\epsilon},
\qquad p_\sigma=p_{\mathrm{data}}*\mathcal N(\boldsymbol0,\sigma^2I).\]
\(*\) denotes convolution : adding independent Gaussian noise smooths the clean-data distribution.
At the noisy input \(\widetilde{\boldsymbol{x}}_\sigma\) , the score points toward increasing noisy-data log density :
\[s_\sigma(\widetilde{\boldsymbol{x}}_\sigma)=\nabla_{\widetilde{\boldsymbol{x}}_\sigma}\log p_\sigma(\widetilde{\boldsymbol{x}}_\sigma)
=-\frac{\epsilon^*_\sigma(\widetilde{\boldsymbol{x}}_\sigma)}{\sigma}
\ \approx\ -\frac{\epsilon_{\theta,\sigma}(\widetilde{\boldsymbol{x}}_\sigma)}{\sigma}.\]
Exact for the ideal squared-error predictor; approximate for the trained network.
Three minutes. The density and network are evaluated at the noisy input tilde x_sigma. The star denotes the population MSE optimum E[epsilon|noisy input tilde x_sigma], not the exact noise realization for every example. Score is a gradient with respect to the noisy input/location, not the network parameters. For fixed sigma, the clean correction D*−tilde x_sigma, the negative noise prediction and the noisy score point in the same direction, with scale factors sigma and sigma squared. The model learns p_sigma, not p_data. [Sources] source:vincent-denoising-score-2011, sections3–4; source:karras-edm-2022, equations2–3.
Small noise approaches the clean score where that score exists
For a smooth positive clean density, under suitable regularity: \(\nabla\log p_\sigma(\widetilde{\boldsymbol{x}}_\sigma)\to\nabla\log p_{\mathrm{data}}(\widetilde{\boldsymbol{x}}_\sigma)\) as \(\sigma\to0\) .
Three minutes. This is local convergence under smoothness/integrability and away from zeros of the clean density, not uniform control over all low-density regions. The smooth mixture from our running illustration supplies an example where the limit exists. A perfect circle is singular in2D and has no ordinary ambient clean log-density gradient; only its sigma>0 noisy scores are used in the next experiments. Even for a smooth density, limiting the ideal target does not guarantee learned accuracy where training inputs are rare. The clean-mixture score often has large magnitude in low-density gaps, but those gaps receive little training weight. [Sources] source:vincent-denoising-score-2011 for denoising score matching; original Gaussian-mixture limit calculation with explicit regularity qualification.
Small-noise training leaves poorly constrained scores away from data
Clean data: uniform on a radius-10 circle. Train a small MLP at \(\sigma=1\) .
Measured failure here: wrong magnitudes off the ring; directions mostly remain radial.
Three minutes. Actual neural training, not an analytic field relabeled learned. MLP2→64tanh→64tanh→2, input divided by10, output normalized noise; Adam lr.001,6000steps, batch512, newly drawn clean circle points and Gaussian noise each batch. Analytic values never enter training. First predeclared seed5720 is shown;5721/5722 also reported. On30000shared spatial points, relative vector-field RMSE is6.4% near ring8<r<12,60.1% interior r<6,52.6% exterior r>18. Mean angular errors, excluding near-zero ideal scores, are5.4°,2.4°,4.4° respectively; do not claim angular failure that this symmetric experiment did not show. All three seeds have approximately59–62% interior and51–53% exterior relative errors. Plot arrows preserve relative magnitudes using a stated common scale within the slide. Held-out noisy-data excess epsilon MSE per entry is.00333. Sparse training coverage leaves extrapolation weakly constrained; it does not force every architecture to point the wrong way. Original experiment; source:vincent-denoising-score-2011 supplies only the score/predictor relation.
Large noise improves coverage but changes the score being learned
Same architecture and training budget, now \(\sigma=10\) .
A good estimate of \(\nabla\log p_{10}\) is still not the clean-data score.
Three minutes. Same three seeds, optimizer, batch and steps; only corruption scale changes. First seed5720: field RMSE11.5% near ring,28.7% interior,4.4% exterior; angles4.3°,8.6°,1.8°. It improves broad-region magnitudes substantially relative to sigma1, not every metric in every region. Other seeds show the same qualitative coverage benefit. The exact radius10 sigma10 noisy density has an inward score for every nonzero radius. A correct fixed-noise Langevin sampler would target p10, not the clean circle; following the score deterministically would instead climb toward a mode, not sample either distribution. No claim that clean generation is impossible by every conceivable deconvolution method—this single-level sampling rule targets the noisy density. The next level must change the target score. Original measured experiment, no image-model evidence.
What would perfect single-level sampling produce?
Suppose the \(\sigma=10\) score is exact and the sampler correctly targets its density.
What would more accurate sampling recover?
A. The clean radius-10 circle
B. The broad noisy distribution \(p_{10}\)
C. One conditional-mean point
B. Better approximation cannot change the distribution the score describes.
Two minutes. Separates learning error, numerical error and target smoothing. A is tempting because training started from clean-circle examples; C confuses sampling with averaging or mode seeking. This motivates learning an entire sequence of noisy scores, reducing sigma during generation. Use p_sigma notation here: p10 means standard deviation10; the next section explicitly changes to p_t with variance-time t, where that same density is p100. Original transfer question.
Learning across noise levels
Broad coverage first; clean-data detail at lower noise
Twenty seconds. Move from the limitations of one level to a time-conditioned family, direct training draws and discrete reverse steps.
Every time slice is a distribution over possible noisy samples
\[p_t(\widetilde{x}_t)=\sum_{j=1}^{3}w_j\,\mathcal N(\widetilde{x}_t;\mu_j,\,0.09+t),\qquad
(\mu_1,\mu_2,\mu_3)=(-3,0,3).\]
Two minutes. Return to the smooth1D mixture used in the clean-score limit. Weights .3,.4,.3; small clean component SD.3. Horizontal time measures accumulated variance; nonlinear spacing preserves visible low-noise structure. Color is density under one fixed cividis shade/hue scale, not time and not a separate normalization per slice. A slice integrates to one over all real x; the plot crops tails. At t100 the single Gaussian overlay has variance105.49, the exact marginal variance, and closely approximates p100. Do not call the exact finite-time mixture Gaussian. This same density and fixed analytic predictor recur in reverse paths, step decomposition and histogram checks. Original analytic construction; source:song-score-sde-2021 for time-indexed marginals.
Independent standard-normal steps make noise variance add
\[\widetilde{\boldsymbol{x}}_t=\widetilde{\boldsymbol{x}}_{t-1}+\boldsymbol{\eta}_t,
\qquad\boldsymbol{\eta}_t\overset{\mathrm{iid}}{\sim}\mathcal N(\boldsymbol0,I).\]
After \(t\) steps: noise variance \(t\) ; noise standard deviation \(\sqrt t\) .
Three minutes. Instructor-requested additive walk, matching the earlier offering. The circle now has radius10, so one standard-normal step (noise SD1) is small relative to the data scale and matches the small-noise example. At t100 the noise SD10 matches the large-noise example. Every panel has the same axis limits; these are intermediate forward states, not a claim that the last panel is already a Gaussian endpoint. Use T much larger than R^2=100 when relying on a noise-dominated initialization. Gaussian increments are independent of the clean point and each other. The residual final-step posterior has finite uncertainty even at this larger radius; a mean prediction is still not an exact clean-data sample.
One network learns noise prediction at every level
Add the noise level to the input: \(\widehat{\boldsymbol{\epsilon}}_i=\epsilon_\theta(\widetilde{\boldsymbol{x}}_{t_i},t_i)\) .
Draw level and noise
\(t_i\sim\operatorname{Uniform}\{1,\ldots,T\}\) , \(\boldsymbol{\epsilon}_i\sim\mathcal N(\boldsymbol0,I)\)
Construct input
\(\widetilde{\boldsymbol{x}}_{t_i}=\boldsymbol{x}_i+\sqrt{t_i}\,\boldsymbol{\epsilon}_i\)
Predict and score
\(\widehat J=\frac1B\sum_i\|\boldsymbol{\epsilon}_i-\epsilon_\theta(\widetilde{\boldsymbol{x}}_{t_i},t_i)\|^2\)
Update shared parameters
\(\theta\leftarrow\theta-\eta\nabla_\theta\widehat J\)
One sampled level per image; the same parameters across levels.
Three minutes. Same fixed-level algorithm as before, now with an independent level for each example and an explicit time input. Tensor shapes remain batch-first. This is a simple noise-MSE objective, not a claim of exact ELBO equality. The clean estimate is tilde x_t-sqrt(t)epsilon_theta. Unweighted noise MSE corresponds to clean squared error weighted by1/t. The radius10 circle makes the t1 noise small relative to the radius while retaining unit-variance increments. A finer time grid is possible but is not needed to explain this example. The final conditional mean still has a small inward bias. [Sources] source:ho-ddpm-2020, algorithm1 for the training pattern, adapted to the explicitly different additive walk.
Reverse generation needs small conditional steps, not one clean guess
A clean-image estimate uses the whole predicted noise:
\[\widehat{\boldsymbol{x}}=\widetilde{\boldsymbol{x}}_t-\sqrt t\,\epsilon_\theta(\widetilde{\boldsymbol{x}}_t,t).\]
For the additive walk, the previous-step conditional mean instead uses
\[\widehat{\boldsymbol{\mu}}_{t-1}=\widetilde{\boldsymbol{x}}_t-
\frac{1}{\sqrt t}\epsilon_\theta(\widetilde{\boldsymbol{x}}_t,t).\]
Generation follows \(T\to T-1\to\cdots\to0\) with fixed learned parameters.
A mean is not a full distribution: a reverse sampler must also handle uncertainty.
Four minutes. This closes the opening and sets up the reverse-sampler continuation, rather than silently reusing the incompatible variance-preserving DDPM formula. For t>1 Gaussian conditioning gives p(x_(t-1)|tilde x_t,clean x)=N(((t-1)/t)tilde x_t+clean x/t,((t-1)/t)I). Averaging over clean x conditional on tilde x_t gives E[x_(t-1)|tilde x_t]=tilde x_t-(tilde x_t-E[clean x|tilde x_t])/t, hence the displayed formula with an ideal normalized-noise predictor. The unconditional reverse conditional is generally a mixture, not exactly Gaussian: its covariance is ((t-1)/t)I+Cov(clean x|tilde x_t)/t^2. Adding only the bridge variance to this averaged mean is not generally exact. At t1 the true posterior on clean circle points remains uncertain, whereas its mean lies inside the circle. The next shared partial develops fine-step stochastic reverse approximations and their endpoint treatment. Do not present these conditional-mean equations alone as a complete exact sampler. Original derivation for the additive process; source:ho-ddpm-2020, section2 provides the related Gaussian forward/reverse modeling framework with a different schedule.
A stochastic reverse step denoises, then adds fresh noise
Move from current noise variance \(t\) to a lower variance \(s\geq0\) . The step size is \(h=t-s>0\) : how much we reduce the noise variance.
A DDPM-style step in our additive units is
\[\widetilde{x}_s=\underbrace{\widetilde{x}_t-\frac{h}{\sqrt t}\epsilon_\theta(\widetilde{x}_t,t)}_{m:\ \text{previous-level mean}}
+\underbrace{\sqrt{\frac{sh}{t}}\,\xi}_{\text{fresh noise}},\qquad \xi\sim\mathcal N(0,1).\]
The correction uses \(h/t\) of the predicted clean-data correction.
For small \(h\) relative to \(t\) , the added variance is approximately \(h\) .
This Gaussian step approximates the reverse conditional distribution.
Three minutes. Conditional on both tilde x_t and clean x, the Gaussian bridge variance is sh/t and its mean is (s/t)tilde x_t+(h/t)clean x. Replacing clean x by its conditional mean yields the displayed mean but omits the extra reverse variance (h/t)^2 Var(clean x|tilde x_t). The exact reverse conditional is generally a mixture. This is the fixed-small-variance DDPM construction expressed in additive units; it is not an exact finite-step sampler even with ideal epsilon. In Ho’s units z_t=sqrt(alpha_bar_t)clean x+sqrt(1-alpha_bar_t)epsilon, divide z_t by sqrt(alpha_bar_t) and use t=(1-alpha_bar_t)/alpha_bar_t. Then the rescaled noise coefficient is sqrt(s(t-s)/t). The paper’s index and our variance-time t differ. At s0 the variance vanishes and the last mean is still an approximation. [Sources] source:ho-ddpm-2020, section3.2; source:song-ddim-2021, equation12; source:song-score-sde-2021, reverse-SDE limit.
Mean correction narrows the distribution; noise spreads it again
“Denoise two, add back one” describes local variance change—not a time reversal.
Two minutes. The middle distribution is an algorithmic intermediate, not p at a hidden earlier time. For one local Gaussian of total variance v, the mean map multiplies deviations by 1-h/v, giving v-2h+h^2/v; independent step noise adds sh/t, approximately h. Net variance is v-h to first order. This gives the requested two-down/one-back intuition, not exact global movement between p_t slices. One random draw may move either direction in x; t still moves only from t to s. Diagram uses the exact push-forward density of the mean map then Gaussian convolution, so its small mismatch from the dashed target is real finite-step error. [Sources] source:song-score-sde-2021, equations6/13; original mixture calculation and first-order Gaussian illustration.
Diffusion as a continuous process
From Gaussian steps to SDEs and continuous-time training
Twenty seconds. The discrete step is already concrete. Now introduce Brownian increments, easy marginal training and the reverse SDE.
Continuous noising is the small-step limit of the Gaussian walk
Discrete diffusion → continuous SDE → probability-flow ODE → solver choices
\[d\widetilde{\boldsymbol{x}}_t=d\mathbf{w}_t,\qquad
\widetilde{\boldsymbol{x}}_0=\boldsymbol{x}.\]
An SDE adds random increments: \(\Delta\mathbf{w}\sim\mathcal N(\boldsymbol0,hI)\) over an interval \(h\) .
Our time label is noise variance: \(\sigma(t)=\sqrt t\) .
Its marginal remains \(\widetilde{\boldsymbol{x}}_t=\boldsymbol{x}+\sqrt t\,\boldsymbol{\epsilon}\) in distribution.
Two minutes. Brownian w is a random vector-valued process; its finite increments are independent Gaussians with covariance hI. SDE means stochastic differential equation. This is the additive/variance-exploding example, not the most general diffusion. A marginal draw formula is not a coupled path obtained by keeping the same epsilon for all times. General forward SDEs have f(x,t)dt+g(t)dw; standard linear Gaussian VP/VE choices have tractable conditional marginals. We retain our additive choice to keep all figures and coefficients consistent. [Sources] source:song-score-sde-2021, section3; source:karras-edm-2022, section2.
Continuous-time training still needs one noise draw per example
Draw \(t\sim\rho\) on \([t_{\min},T]\) , \(t_{\min}>0\) , then form \(\widetilde{\boldsymbol{x}}_t=\boldsymbol{x}+\sqrt t\,\boldsymbol{\epsilon}\) .
\[\min_\theta\ \mathbb E\!\left[\lambda(t)
\|\boldsymbol{\epsilon}-\epsilon_\theta(\widetilde{\boldsymbol{x}}_t,t)\|^2\right],
\qquad s_\theta(\widetilde{\boldsymbol{x}}_t,t)=-\frac{\epsilon_\theta(\widetilde{\boldsymbol{x}}_t,t)}{\sqrt t}.\]
No simulated SDE trajectory and no SDE solver are required for this training loss.
Two minutes. Replace discrete level sampling by a continuous density rho; this is still supervised denoising score matching through known Gaussian corruption. Specify positive loss weights lambda and a training noise range. Uniform noise MSE, uniform score MSE and uniform clean MSE imply different weights across time: epsilon error squared=t times score error squared, and clean error squared=t times epsilon error squared. A finite shared-capacity network therefore depends on weighting and time coverage. At ideal unconstrained population optimum positive reweighting does not change the conditional optimum at a given t. Train on the SDE’s marginals and later choose a compatible ODE or SDE sampler. For non-Gaussian/no-closed-form forward processes, the direct-draw shortcut need not exist. [Sources] source:song-score-sde-2021, equation7; source:karras-edm-2022, section5.
The reverse SDE repeatedly combines score drift and random increments
Let generation time \(\tau=T-t\) increase while noise variance \(t\) decreases:
\[d\widetilde{\boldsymbol{x}}_{T-\tau}
=s_\theta(\widetilde{\boldsymbol{x}}_{T-\tau},T-\tau)\,d\tau+d\overline{\mathbf{w}}_\tau.\]
A small numerical step is
\[\widetilde{\boldsymbol{x}}_{t-h}\approx\widetilde{\boldsymbol{x}}_t
+h\,s_\theta(\widetilde{\boldsymbol{x}}_t,t)+\sqrt h\,\boldsymbol{\xi}.\]
Feed the new state into the next step; keep the network parameters fixed.
Three minutes. Use increasing reverse clock tau to avoid hiding a negative dt in the formula. The Brownian increments here are newly drawn during generation. For the true score and correct initial pT, this SDE reverses the forward density evolution. The discrete DDPM bridge variance s(t-s)/t approaches h for small h/t, giving the earlier sampler’s continuous limit. The network approximation and finite solver introduce error. In general forward f,g, reverse-clock drift is -f+g²s and diffusion g; our f0,g1 simplifies it. Start from exact pT in analytic tests; Gaussian approximations are used in practical generation. [Sources] source:song-score-sde-2021, equation6; source:karras-edm-2022, section4.
What if we delete the fresh noise from this sampler?
Keep the random initialization and the same mean correction.
What should we expect if every \(\xi\) is set to zero?
A. The same distribution, computed faster
B. A generally different output distribution
C. The same output on every run
B. A valid deterministic sampler also changes the correction.
Two minutes. A overlooks the lost variance restoration; C forgets random initialization. The previous density split shows why the full mean drift without noise narrows the distribution too far. The generator records this control’s empirical CDF error alongside the valid samplers. The next slide supplies the needed alternative; deterministic generation is not the same as predicting one universal mean. [Sources] source:song-score-sde-2021, equation13.
One model, different sample paths
ODE and SDE samplers can share the same ideal marginals
Twenty seconds. The noise-deletion question has shown that drift and noise must change together. Introduce probability flow and intermediate stochasticity, then revisit the running mixture.
The probability-flow ODE transports the same density without fresh noise
Using the same score predictor and increasing reverse clock \(\tau\) :
\[\frac{d\widetilde{\boldsymbol{x}}_{T-\tau}}{d\tau}
=\frac12\,s_\theta(\widetilde{\boldsymbol{x}}_{T-\tau},T-\tau).\]
An ODE gives a deterministic trajectory once the initial sample is drawn.
Half the additive-process drift replaces the SDE’s drift-plus-noise balance.
Exact score + exact initial distribution + exact integration ⇒ same marginals \(p_t\) .
Three minutes. This is the probability-flow ordinary differential equation, not a separately trained flow-matching network. A learned shared score does not guarantee exact equality of its approximate ODE/SDE output distributions. In our variance-time parametrization the forward heat equation is partial_t p=.5 Laplacian p; forward ODE velocity is -.5 score, whose continuity equation gives the same PDE. For general forward f,g the forward ODE drift is f-.5g²s. The paper’s general noise schedule gives forward drift -sigma dot_sigma score; sigma=sqrt(t) recovers -.5score. EDM often uses noise standard deviation itself as the integration variable; its coefficients therefore look different. [Sources] source:song-score-sde-2021, equation13; source:karras-edm-2022, equation1.
We can vary stochasticity without retraining the score model
For \(0\leq\beta\leq1\) , one family in our reverse clock is
\[d\widetilde{\boldsymbol{x}}_{T-\tau}
=\frac{1+\beta^2}{2}\,s_\theta(\widetilde{\boldsymbol{x}}_{T-\tau},T-\tau)\,d\tau
+\beta\,d\overline{\mathbf{w}}_\tau.\]
Probability-flow ODE
Intermediate stochastic sampler
Reverse diffusion SDE
Change the drift together with the noise amplitude to preserve the ideal marginals.
Three minutes. For true score s=grad log q, the reverse-clock Fokker–Planck equation becomes -((1+beta²)/2)div(qs)+(beta²/2)Laplacian q=-.5Laplacian q for every beta. This proves the shared target evolution; beta can also be a deterministic function of time. This beta denotes noise amplitude, not EDM’s noise-replacement-rate beta(t), and not a DDPM forward variance schedule. The family is a specialization of the ODE-plus-Langevin freedom in EDM section4. Finite integration and learned-model errors still depend on this choice. We show beta0,.5,1 using the same analytic score and identical starts next. Do not imply arbitrary mixing of two update formulas preserves a density. [Sources] source:karras-edm-2022, equation6 and appendixB.5; source:song-score-sde-2021, section4.3; original additive reverse-clock specialization.
Different sample paths can pass through the same marginal distributions
A vertical slice describes a population of samples, not one trajectory.
Two minutes. Exactly the same mixture and cividis density scale in all three panels. Same fourteen initial states; beta0,.5,1, with one matched initial state highlighted. Paths run from right to left; at any selected t, collect values over independent runs to get a marginal. Noise does not send time backward: it changes the sample value at the destination time. Stochastic trajectories may cross and the highlighted sample need not finish in the same mode as flow. In one dimension the exact smooth deterministic flow preserves sample ordering. Exact continuous-time reverse SDE and probability flow have matching p_t under the exact predictor and initial p_T; the plotted midpoint ODE and Euler SDE algorithms approximate this, checked by the later histograms. [Sources] source:song-score-sde-2021, section4.3; original simulation.
Many sampled paths recover the target density at each time
Compare the distributions at matched times—not matched outputs from one seed.
Two minutes. Twelve thousand samples, fixed seed5713 and1800steps. The same initial sample array is drawn from exact p100, isolating solver error from initialization mismatch. Four visible slices plus t.1 are checked against the analytic CDF. The dashed orange histogram outlines also show deterministic DDIM, overlaying the flow histograms. All three are checked against the exact CDF. Histogram variation is sampling noise; finite step differences remain. Axes widen with noise level and are matched within each column, labeled explicitly. This is a known-density numerical demonstration, not evidence for image-model performance. [Sources] source:song-score-sde-2021, marginal equivalence; original numerical checks.
DDIM reuses the predicted noise direction without drawing fresh noise
First predict clean data, then retain the noise appropriate for \(s<t\) :
\[\widehat x=\widetilde{x}_t-\sqrt t\,\epsilon_\theta(\widetilde{x}_t,t),\qquad
\widetilde{x}_s=\widehat x+\sqrt s\,\epsilon_\theta(\widetilde{x}_t,t).\]
Therefore
\[\widetilde{x}_s=\widetilde{x}_t+(\sqrt s-\sqrt t)\epsilon_\theta(\widetilde{x}_t,t)
\ \approx\ \widetilde{x}_t-\frac{h}{2\sqrt t}\epsilon_\theta(\widetilde{x}_t,t).\]
Deterministic DDIM approaches probability flow as steps become small.
Three minutes. This is eta0 DDIM in our additive units, obtained by dividing the original VP state by sqrt(alpha_bar), with noise variance t=(1-alpha_bar)/alpha_bar. Both displayed terms use the same predicted epsilon, not a new draw and not the unknowable true realized noise. For s=t-h, sqrt(s)-sqrt(t) is approximately -h/(2sqrt(t)). Thus DDIM takes Euler-like steps in noise standard deviation, whereas the displayed probability-flow step uses variance time; they are not identical finite-step solvers. No retraining is required to change the sampler. The general rescaled update is tilde x_s=hat_clean x+sqrt(s-a^2)epsilon_theta+a xi; a^2=sh/t recovers the preceding DDPM mean and noise, a0 gives DDIM. This unifies the two rules without mixing incompatible coefficients. [Sources] source:song-ddim-2021, equations12–15 and proposition1.
An image follows one path through a distribution over whole images
The 1D value \(\widetilde{x}_t\) becomes an image array \(\widetilde{\boldsymbol{x}}_t\) ; \(p_t\) is not a pixel histogram
Two minutes. Actual simulated states, not forward corruption frames relabeled as generation. An exactly known784-dimensional mixture is centered at the first three archived MNIST training images with equal weights and isotropic pixel variance .0025. A fixed analytic conditional-mean noise predictor drives both samplers; no neural network trained. Seed5714,2200steps, initial p100 sample shared by both rows. Clipping to [0,1] is only for viewing, never in sampling. This finite template mixture illustrates how the same equations apply to image arrays; it cannot produce diverse new handwritten structures. A learned image model replaces these three centers with a learned distribution. Intermediate predicted clean images could be displayed separately, but are not the noisy states shown here. [Sources] source:mnist-train-images; original analytic simulation; source:song-score-sde-2021 for image-space sampling.
Choosing a numerical sampler
Solver accuracy, computation and downstream quality
Twenty seconds. Turn the continuous viewpoint into a practical solver choice, then separate training decisions from compatible sampling decisions using EDM.
Numerical integration turns the learned field into a practical sampler
Write the reverse ODE drift as \(b(\widetilde{\boldsymbol{x}}_t,t)=\tfrac12s_\theta(\widetilde{\boldsymbol{x}}_t,t)\) .
Euler proposal: \[\widetilde{\boldsymbol{x}}^{\mathrm E}_{t-h}
=\widetilde{\boldsymbol{x}}_t+h\,b(\widetilde{\boldsymbol{x}}_t,t).\]
Heun correction: \[\widetilde{\boldsymbol{x}}_{t-h}=\widetilde{\boldsymbol{x}}_t+
\frac h2\left[b(\widetilde{\boldsymbol{x}}_t,t)
+b(\widetilde{\boldsymbol{x}}^{\mathrm E}_{t-h},t-h)\right].\]
Evaluate two slopes instead of one; compare cost in network evaluations.
Three minutes. Heun averages the start slope and Euler-proposed endpoint slope; it is a classical second-order explicit Runge–Kutta method. h is positive while t decreases; b is the increasing-reverse-clock velocity, so both plus signs are correct. Euler global error is first order and Heun second order under suitable smoothness; that is not a universal image-quality guarantee. The toy mixture has a smooth positive density even at t0, but real epsilon/sqrt(t) interfaces may require a positive cutoff and a special final step. EDM uses Heun, an appropriate noise grid and endpoint handling; its standard choice integrates in sigma rather than our variance time. SDE solvers must additionally handle random increments and strong/weak error. Physics/scientific-computing solvers provide a useful numerical toolbox, not a way to remove learned score error. [Sources] source:karras-edm-2022, section3, algorithm1 and appendixD.
A better solver can improve sampling with exactly the same predictor
This isolates solver error. Image quality must also be measured downstream.
Two minutes. Reproducible known-density toy comparison, not empirical image-generation results. Use the same initial points, exact analytic predictor and target endpoint per point, determined by preserving each point’s CDF rank from p100 to p0. Euler and Heun use the same grid family and are compared at matched numbers of field calls; each Heun step uses two calls. The adaptive SciPy solve_ivp DOP853 point illustrates an available scientific-computing integrator; its actual function-evaluation count is recorded. Lower error here does not imply it is always the best image sampler, especially for imperfect or stiff learned fields. EDM evaluates pretrained image models and motivates comparing sample quality against denoiser evaluations. [Sources] source:karras-edm-2022, figure2 and section3; original toy numerical experiment.
Training and sampling are separate design choices with shared constraints
Data, architecture and conditioning
ODE, SDE or intermediate stochasticity
Noise-level distribution and loss weighting
Noise grid and integration variable
Prediction target and input/output scaling
Solver order, evaluation budget and endpoint
Hold training fixed to compare samplers; evaluate final quality, diversity and cost.
NVIDIA: Elucidating the Design Space of Diffusion-Based Generative Models
Three minutes. This is the final viewpoint: train a noise-conditioned denoiser/score using easy corruption samples; then solve a compatible continuous generative process with a chosen numerical method. EDM separates sampling, network preconditioning and training choices, and tests new samplers on existing pretrained networks. Independence is modularity, not permission to ignore normalization, prediction parameterization, conditioning, time encoding or the trained noise range. A solver cannot repair an inaccurate score; a well-trained score does not specify a unique practical sampler. Downstream evidence includes fidelity, diversity/mode coverage, condition adherence, failure slices and computation; both training and sampler choices can change these. Noise-only MSE or toy integration error is not sufficient. Modern SDE/ODE viewpoint is central without claiming every contemporary image model uses the identical training objective. [Sources] source:karras-edm-2022, table1 and sections3–5; source:song-score-sde-2021, sections3–4.
Conditioning makes denoising depend on the requested output
For an image paired with a condition \(c\) :
\[\widetilde{\boldsymbol{x}}_t=\boldsymbol{x}+\sqrt t\,\boldsymbol{\epsilon},\qquad
\widehat{\boldsymbol{\epsilon}}=\epsilon_\theta(\widetilde{\boldsymbol{x}}_t,t,c).\]
Train: retain \(c\) with each image while learning noise prediction.Generate: supply the same request at every reverse step.A class label, text embedding, or layout can provide \(c\) ; request adherence needs testing .
Three minutes. A condition changes the target from a marginal image distribution to a conditional one. Use c as an abstract label or structured condition, not as a scalar assumption. Text is encoded into embeddings; cross-attention can expose them to image features using the Arc6 attention mechanism. The construction shown adapts the known additive schedule; the cited latent-diffusion paper uses its own schedule. No classifier-free-guidance formula or guarantee of obeying a prompt is implied. [Sources] source:rombach-latent-diffusion-2022, section3.3 equation3 and figure3.
The codec and denoiser create different sources of error
Real image \(\to\) encode \(\to\) decode
Detail lost by the codec
Noise \(\to\) latent denoising \(\to\) decode
End-to-end generation quality and diversity
Repeat requests across seeds
Consistency and condition adherence
Smaller latent arrays can reduce repeated computation, with a compression tradeoff .
Two minutes. A reconstruction test can reveal missing fine detail even before training the latent generator. Good codec reconstruction is necessary for retaining those details through that codec, but does not prove generated codes will resemble encoded real images. Measure the full system and record codec/denoiser/sampler versions separately. No universal compression-speed or quality claim. [Sources] source:rombach-latent-diffusion-2022, sections3.1 and4.1; original component-level evaluation interpretation.
Generative families differ in how a sample is produced
VAE
Sample a latent code, then decode
Decoder evaluation
Diffusion
Start with noise, then follow reverse steps
Repeated denoiser evaluations
Autoregressive
Sample the next symbol conditioned on the prefix
Sequential symbol-generation steps
A vanilla AE or fixed-level DAE does not specify a generator by itself .
Three minutes. Synthesize the paradigms already taught; avoid declaring a quality ranking. A VAE may sample or report a decoder mean, depending on the output likelihood. The diffusion row describes the taught iterative sampler, not every possible distilled model. Autoregressive image symbols could be pixels or learned codes; Arc6 already explained sequential sampling for text. Runtime depends on architecture, representation, resolution, batch and sampler. This comparison is not empirical CIFAR-10 evidence. [Sources] source:kingma-welling-vae-2013, section2; source:ho-ddpm-2020, algorithms1–2; source:rombach-latent-diffusion-2022, section2.
A plausible image still needs evidence for its intended use
Suppose the generator illustrates specified objects in a teaching worksheet.
Correct requested objects and counts
Independent request set; structured human checks
Useful variety without broken examples
Multiple seeds per request; inspect failures by slice
Practical classroom use
Readability at final size, generation time, and review effort
Next: turn these checks into an evaluation protocol and compare models fairly.
Three minutes. A concrete evaluation bridge, not a replacement for the protected full evaluation unit. Selected pretty samples cannot support a population claim; keep evaluation requests independent of tuning, disclose seed/sample selection and inspect uncertainty. Component noise MSE alone does not answer request adherence or usefulness. Preserve the later comparison protocol, failure slices, human judgments and monitoring. Original teaching use case applying Arc3 evaluation principles; no new measured results or grading requirements.
