Testing Architectural Assumptions
ECE 57000 — Fall 2026
Compare architectures under one stated recipe
Same 12,000 training / 3,000 validation images
MLP: 784 → 64 → 10
Same batches, five epochs, Adam at \(10^{-3}\)
CNN: the two-convolution model just shown
Three seeds; use the final epoch of every run
Connections, parameter count, and computation
Equal epochs give equal data passes—not equal computation.
No augmentation, tuning, or test-set access in this comparison.
About three minutes. Original classroom experiment, arc5_architecture_comparison.py and comparison-results.json. The same archived split and pixel scaling are reused; training batch size is 64. Seeds 570, 571, 572 control initialization and an independent batch-order generator; the batch sequence matches between architectures for each seed. Both use standard layer initialization, but different architectures do not receive identical initial functions. Each run starts fresh; no best-seed or best-epoch selection. Equal optimizer settings are a transparent baseline, not individually optimized settings or a guarantee of equal optimization difficulty. Five epochs are not a convergence claim. This comparison estimates performance of these two complete recipes; it does not isolate weight sharing alone. Validation is deliberately used for exploration; the official test set is not loaded. [Sources] source:fashion-mnist; source:pytorch-optimization-tutorial.
This CNN improves validation accuracy under this recipe
MLP
83.4%
83.0–83.7%
CNN
86.0% 85.7–86.3%
Same 3,000 validation images; five training epochs per run.
Supported: this CNN recipe performed better on this split.
Not established: every CNN beats every MLP, or the gap persists under equal compute.
About three minutes. Values are rounded from comparison-results.json: MLP 83.0333, 83.4333, 83.6667%; CNN 86.1667, 86.3, 85.6667%. Means are 83.3778 and 86.0444%, a 2.67 percentage-point difference. Seed ranges describe fitting randomness on one fixed split, not confidence intervals or sampling uncertainty. The two models differ in width, connectivity, sharing, depth, and parameter count; do not attribute the gap exclusively to convolution. These new five-epoch PyTorch runs are separate from both Friday’s demonstration and the earlier 60-epoch scikit-learn experiment. No test accuracy is claimed.
Fewer parameters can still require more computation
MLP
50,890
50,816
CNN
32,618 313,600
A convolution stores a filter once but applies it at many positions .
This CNN has about 36% fewer parameters and 6.2× as many multiply-accumulates .
About three minutes. Exact counts for the authored models. MLP parameters: 78464+64+64 10+10=50,890; dense multiply-accumulates: 78464+64 10=50,816. CNN parameters: (18 9+8)+(816 9+16)+(313610+10)=32,618. CNN multiply-accumulates: 28 288 9 + 1414 168 9 + 3136*10 = 313,600. Counts exclude bias additions, activations, memory traffic, backward computation, and optimizer work. One multiply-accumulate is counted as one; it is sometimes counted as two floating-point operations. These are arithmetic counts, not a measured latency ratio. Hardware kernels and memory access affect runtime; activation memory is also distinct from parameter storage.
A better average can hide different mistakes
The CNN fixes one MLP error; the MLP gets another example right.
Inspect disagreements against the label before deciding which errors matter.
About two minutes. These are selected examples, not random or representative error samples. For seed 570, choose the first validation index where CNN is correct and MLP wrong (index 1), then the first with the opposite outcome (index 308). Selection rule was fixed in the experiment code. Labels and both predictions are generated from actual model outputs. The two examples demonstrate that a mean does not imply pointwise dominance; they do not establish class-specific reliability or error severity. In a real task, inspect labels and quantify errors relevant to the intended use before making a choice. [Sources] source:fashion-mnist.
Stress the assumption about position
Architecture evaluation · 2/3 Stress · Compare → Stress → Decide
30-second transition. Original validation favors one recipe. The convolution motivation also invoked patterns occurring at different positions. Test that claim by changing position while keeping the model fixed.
Predict what happens when the image moves
A CNN uses convolution, then flattening and a linear classifier.
Must shifting an image preserve its predicted class?
A. Yes, filters are shared
B. Only for one-pixel shifts
C. No, the classifier can use position
C. A pattern can move in the feature map; the classifier can weight positions differently.
About three minutes, optional runway question. A confuses shared local detection with invariant classification. B incorrectly assumes small shifts are inherently protected; stride can make one-pixel shifts especially consequential. C is a counterexample claim, not a prediction that every image must flip. Keep the architecture condition visible. Do not reveal this answer in the title or preceding divider. The next slide constructs the smallest example so the distinction is immediately concrete. Equivariance itself is qualified by boundaries and stride alignment. [Sources] source:pytorch-conv2d-2-7.
A shared filter can feed a position-sensitive decision
Use a size-one filter with weight 1 and bias 0. It copies each input value.
Input and feature map
\([0,1,0]\) \([0,0,1]\)
Class A score: middle entry
1
0
Class B score: right entry
0
1
Prediction
A B
Same filter at every position; different class after a shift.
About two minutes. Original exact counterexample, verified with Conv1d and a fixed linear readout in the companion script. This is a schematic classifier, not a trained Fashion-MNIST result. The input content is moved without cropping the nonzero entry. The convolutional feature map moves with the input, but the flattened readout assigns different roles to its positions. This proves convolution alone does not guarantee invariant classification. Pooling/aggregation or augmentation may encourage invariance, but each has conditions and tradeoffs; do not imply they provide universal robustness.
Define the shift before interpreting its score
Move each validation image two pixels right . Fill the left edge with zeros; discard the rightmost two columns. No wrapping.
Keep the original labels. This assumes the remaining image still supports the same category.
About two minutes. The illustrative image is the first stored disagreement example. Apply exactly this transformation to all 3,000 validation images without retraining either model or choosing a favorable seed. This is a separately labeled stress condition, not a replacement validation split or a new independent sample. Zero filling and clipping change pixels as well as position, so the resulting accuracy drop cannot be attributed solely to failure of translation equivariance. Retaining labels is plausible for this limited shift but requires scrutiny if informative parts are clipped. [Sources] source:fashion-mnist.
Both models lose accuracy on shifted images
Mean accuracy: MLP 83.4% → 53.5%; CNN 86.0% → 60.3%.
The CNN does better here, but its original validation accuracy does not establish shift robustness .
About three minutes. Each dot is one fixed seed; the short horizontal mark is its mean. Original and shifted images are paired; no retraining occurs. Means: MLP 83.3778→53.5444%; CNN 86.0444→60.3111%. Drops are about 29.8 and 25.7 percentage points. Three seeds are not three new datasets. Stride, positional readout, clipping, and absence of augmentation can all contribute; this experiment does not isolate their effects. One rightward shift does not characterize all perturbations, domains, or CNNs. Do not tune on this stress set and subsequently call it independent final evidence.
Choose a model for the conditions it must handle
Architecture evaluation · 3/3 Decide
Higher original validation accuracy
Candidate for similarly positioned images
Fewer parameters, more arithmetic
Check storage and runtime budgets separately
Large shifted-image accuracy loss
Investigate position variation before deployment
Compare → Stress → Decide: a single accuracy number cannot settle all three.
About three minutes. This is substantive synthesis, not a 30-second orientation: it asks students to connect results to a decision. If the deployment resembles original validation and these resource requirements fit, CNN is a reasonable candidate among these two recipes. If translations are common, neither original score is sufficient. Actual latency, activation memory, and relevant errors require additional measurement. Do not claim a particular model meets an unstated application threshold.
Architectural bias helps when its assumptions fit
Assumption: local patterns recur; sharing is useful when that is true.Evidence: these two recipes favor the CNN on one split; both degrade under one shift.Next test: fix the recipe and selection rule, then evaluate on held-out data matching the intended conditions.
More expressive features, efficient gradients, and a working loop are necessary steps. They do not replace evidence about where the model will be used.
About three minutes. Close the simple comparison before introducing ResNet and U-Net as complete architectural examples. Limits of this comparison: one dataset/split, two small architectures, three initialization/batch-order seeds, five epochs, one optimizer recipe, and one artificial stress condition. No uncertainty over new dataset draws or converged-model ranking was established. All exploration used validation, not the official test set. The earlier scikit-learn study already reported official test results, so even a later test of this new recipe must disclose that prior dataset-level exposure; do not call the benchmark wholly untouched by the course. New independent application data are needed for a stronger deployment claim. Additional augmentation, tuning, or architectures would need an explicit selection protocol and fresh final evidence.
