Learning to Reconstruct and Restore Images David I. Inouye
Dynamic Content · Last Updated: November 4, 2026
Arc 7 · Image generation and evaluation
Vanilla autoencoders and denoising autoencoders
Thirty seconds. Arc 6 ended with the computation needed to generate sequences. We now return to images: what can we learn from an image when no class label is supplied? Reconstruction and restoration give two learning tasks before we introduce explicit generative sampling. This topic is the first part of Arc 7, not the entire generative-modeling arc.
We will use handwritten digits and small color images as examples
MNIST: trace a recognizable digit through encoding, corruption and reconstruction.CIFAR-10: compare models on 32 × 32 color images with varied objects and backgrounds.
Use the same data, evaluation protocol and declared compute budget for comparisons.
Two minutes. The six displayed images are the first six training records in the archived MNIST IDX file; no cherry-picking for a model. MNIST has 784 grayscale input features. CIFAR-10 has 3072 color input features and 50,000 training/10,000 test images; use a train/validation split for model selection and reserve its test set. CIFAR generator comparisons are planned, not measured yet. Use reconstruction criteria for reconstruction, and distributional quality/diversity plus cost for generation; a vanilla AE has no default sampler. Class labels can aid evaluation without serving as training targets for an unconditional model. [Sources] source:mnist-train-images; source:cifar10-dataset.
An autoencoder learns a code that supports reconstruction
\[\boldsymbol{z}=f_\phi(\boldsymbol{x}),\qquad
\widehat{\boldsymbol{x}}=g_\theta(\boldsymbol{z})\]
Four minutes. AE means autoencoder. The running input is a 28 by 28 MNIST image (784 features), flattened into a vector for this notation; convolutional implementations can retain its spatial structure. Phi denotes encoder parameters and theta decoder parameters. The code is an internal learned representation. The target is the observed image, not a class label and not something created by the decoder. Both functions are learned together through the loss; do not confuse the decoder here with the causal transformer decoder. [Sources] source:goodfellow-autoencoders-2016, chapter 14 opening; source:mnist-train-images.
Reconstruction error trains both halves together
For images \(\boldsymbol{x}_i\in\mathbb{R}^d\) , one choice is mean squared error:
\[\min_{\phi,\theta}\ \frac{1}{nd}\sum_{i=1}^{n}
\left\|\boldsymbol{x}_i-g_\theta\!\left(f_\phi(\boldsymbol{x}_i)\right)\right\|_2^2.\]
The image supplies its own reconstruction target.
Backpropagation updates encoder and decoder parameters through the same loss.
A small pixel error measures reconstruction, not every property we care about.
Four minutes. Here n is the number of training images and d is the number of input features. The squared norm sums over image features, so division by nd gives an elementwise mean squared error. Use the already-taught optimization loop; no new backpropagation derivation. MSE is a modeling choice, not the definition of all AEs. Class labels are unnecessary for this objective, but data collection and choice of task still matter. Different images can have similar pixel losses while differing in usefulness. [Sources] source:goodfellow-autoencoders-2016, section 14.1.
Perfect copying can solve the loss without a useful code
If both functions can be the identity,
\[f(\boldsymbol{x})=\boldsymbol{x},\qquad g(\boldsymbol{z})=\boldsymbol{z}
\quad\Longrightarrow\quad\|\boldsymbol{x}-g(f(\boldsymbol{x}))\|_2^2=0.\]
The reconstruction objective needs a constraint that makes learning useful structure worthwhile.
Our first choice: pass through a smaller code, \(\boldsymbol{z}\in\mathbb{R}^k\) with \(k<d\) .
Three minutes. This is a mathematical counterexample, not an empirical claim that every unconstrained network learns identity. An undercomplete AE uses fewer code features than input features. Other constraints exist; sparse AEs are outside this arc’s core. A small code is an inductive bias, not a guarantee of semantics or generalization: a sufficiently flexible model can memorize a finite dataset. [Sources] source:goodfellow-autoencoders-2016, sections 14.1–14.2.
Remember: PCA encodes an input in fewer features and reconstructs it
We already know a linear encoder and decoder : PCA.
For a centered input \(\boldsymbol{x}\) and principal directions \(U_k\in\mathbb{R}^{d\times k}\) :
\[\underbrace{\boldsymbol{z}=U_k^{\mathsf T}\boldsymbol{x}}_{\text{encode into }k\text{ features}},
\qquad\underbrace{\widehat{\boldsymbol{x}}=U_k\boldsymbol{z}}_{\text{decode back to }d\text{ features}}.\]
This gives an optimal rank-\(k\) linear reconstruction under squared error.
Learned linear AE factors can differ while producing the same reconstruction.
Four minutes. Recall the PCA geometry rather than repeat its proof. Center the data and add the mean back after reconstruction. A linear encoder/decoder with a k-dimensional bottleneck and globally minimized squared error can recover a principal-subspace reconstruction. The learned factors need not equal U transpose and U, nor provide ordered orthogonal code features; invertible changes of basis in the code can leave the product unchanged. Degenerate eigenvalues can make the optimum subspace nonunique. Do not promise gradient descent reaches the global optimum. [Sources] source:goodfellow-autoencoders-2016, section 14.1.
What does a small reconstruction error tell us ?
An AE reconstructs held-out images accurately.
Which conclusion is supported?
A. Its codes separate all useful classes.
B. Its reconstructions preserve the measured image features well.
C. Random Gaussian codes produce realistic images.
B. Classification and generation need their own evidence.
Three minutes including voting. Interpret accuracy here under the stated reconstruction metric, such as normalized pixel MSE. A is tempting because representation learning is a motivation, but reconstructing images need not organize a particular label or preserve every task-critical detail. C assumes an untrained latent sampling rule. B says only what the reconstruction measurement supports, not that perceptual quality or deployment usefulness is guaranteed. Transition: another way to shape what a model learns is to change its input/target task. [Sources] source:goodfellow-autoencoders-2016, sections 14.1–14.2; source:kingma-welling-vae-2013, section 2.
A denoising AE receives corruption but targets the clean image
Copying the corrupted input now disagrees with the clean target.
Four minutes. DAE means denoising autoencoder. Begin with a clean training image, construct a corrupted input, and keep the clean original as the loss target. Examples include randomly masking some input features or adding noise. No externally supplied class label is needed. The corruption is chosen by us and determines what restoration task is learned. The original MNIST image is real; its noisy version is constructed with seed 570, Gaussian noise standard deviation 0.3 and clipping to [0,1]. The reconstruction remains symbolic, not a trained output. A DAE need not have an undercomplete code; the changed objective supplies another constraint. [Sources] source:vincent-denoising-2010, section 3; source:mnist-train-images.
