Implementing Neural Network Training in PyTorch (continued)

Training separates gradient computation, updates, and evaluation

Forward and backward: record the computation, save needed values, then accumulate parameter gradients.
Update: clear old gradients, compute the new batch loss and gradients, then call optimizer.step().
Evaluate and log: use held-out data with no_grad(); store scalar or detached values without their training graphs.

Today: finish the training recipe, then change the model to use image structure.

Initialization is part of the training recipe

PyTorch provides useful defaults: standard layers initialize their parameters when constructed; nn.Linear scales its initial values with the input width.
Match the method you reproduce: follow a paper’s specified initialization, including its scale, distribution, and bias settings. Defaults may differ.
Initialization can materially change training and the final model. Record the strategy and random seed; use multiple seeds when comparing results.

torch.nn.init provides strategies such as Xavier and Kaiming initialization. Choose for the architecture and activation; do not replace a paper’s recipe blindly.

Training curves help diagnose the complete loop

Both losses remain high	Training falls, validation rises	Loss oscillates sharply
Check code, fitting, and representation	Revisit capacity, regularization, or stopping	Inspect initialization, scaling, and step size

Running training loss is cheap and useful on large datasets, but mixes changing weights. For a clearer divergence diagnosis, evaluate a fixed model on a representative training subset and validation data using the same loss.

Illustrative curves, not experiment results. Keep test data out of these choices.

PyTorch connects a model, its gradients, and repeated updates

Stage	What we control
Define	Dataset, batches, model, initialization, scalar loss
Differentiate	Forward builds the graph; backward adds parameter gradients
Update	Clear old gradients, compute new ones, then call optimizer.step()
Evaluate	Use held-out data and log values without retaining graphs

The training loop stays useful when the architecture changes. Next, replace unrestricted pixel connections with local, shared filters.

Convolutions and Image Structure

Image structure changes how we connect and share weights

Arc 5: representations → gradients → PyTorch → image structure → evidence.

Patterns: why use local, shared rules?
Filters: how does convolution compute features?
Networks: how do shapes and layers fit together?

The training loop is ready. Now change the model’s assumptions.

Nearby pixels reveal local image patterns

A bag image and three enlarged five-by-five patches, with their source locations hidden

Matching numbered boxes locate all three patches along different parts of the bag's left edge

Where do these patches belong?

A. Left edge B. Interior C. Right edge

A. Left edge. Each patch goes from dark to bright. One local detector can find this pattern at many positions.

About three minutes including a short vote. First show the image and the three enlarged crops without source boxes. Ask where they could belong, rather than requiring an exact row match; similar local evidence does not uniquely specify position. Reveal the actual source boxes and their matching numbers together with the answer. These are three distinct 5-by-5 patches at separated heights on the same bag’s left boundary, using identical grayscale limits and no resizing before extraction. They are similar dark-to-light transitions, not identical patches. The bag is the first class-8 example in the existing training split. Crop rows are 4, 10, 16; each crop is centered near the first pixel exceeding 0.5 in its middle row. This deliberate illustrative selection is not an empirical accuracy experiment. Contrast the bright interior and opposite light-to-dark right edge. A local detector can respond wherever a similar pattern occurs; the next slides explain how shared weights implement such a rule. No claim that these hand-selected patches are learned filters. [Sources] source:fashion-mnist.

Image features have meaningful spatial neighbors

General feature vector: in [age, income, height], adjacent entries need not be more related than distant entries. Their order identifies features, not spatial neighbors.

Image: neighboring pixels measure nearby locations. Their arrangement forms edges, textures, and shapes.

Fully connected layer	Convolutional layer
Each hidden unit can use every pixel	Each output uses a local patch
Different units have independent weights	Reuse the same filter across locations
Can learn spatial relationships from data	Builds local reuse into the architecture

Convolution uses this structure: local neighborhoods matter, and a useful pattern may recur at different locations.

About three minutes. A vector is ordered: each feature must keep its identity consistently between training and use. However, arbitrary tabular column adjacency does not supply a spatial neighborhood. Permuting those columns consistently changes their indexing, not the underlying measurements. Image row/column positions have a spatial meaning; arbitrary pixel permutations disrupt local visual structure. Flattening with a known ordering is reversible and does not discard pixel values, but a fully connected layer does not explicitly exploit the original grid adjacency. Some vectors, such as time series, do have meaningful adjacency, so do not claim all vector data lack structure. An MLP can represent local rules, but the architecture does not enforce locality or tie corresponding weights. Convolution makes a particular prior convenient; fewer independent parameters do not guarantee better accuracy or fewer required examples in every task. This corrects the original deck’s overly broad translation-invariance claim: stride-one convolution is translation-equivariant away from boundary effects, not inherently an invariant classifier. [Sources] source:pytorch-conv2d-2-7.

Filters turn local patterns into computed features

Image structure · 2/3 Filters · Patterns → Filters → Networks

A convolution reuses one local weighted sum

Sliding 1D kernel, position 1

Sliding 1D kernel, position 2

Sliding 1D kernel, position 3

At each position: multiply the window entries by the same kernel entries, then add. A learned bias may be added to every result.

A 2D kernel slides across rows and columns

Sliding 2D kernel, position 1

Sliding 2D kernel, position 2

Sliding 2D kernel, position 3

A \(2\times2\) window fits at three row starts and three column starts in a \(4\times4\) image: the output is \(3\times3\).

Padding adds a border; stride controls how quickly the window moves

Numbered kernel windows yield 5, 7, or 4 outputs; dotted zero cells show implicit padding.

Padding \(p\): \(p\) implicit zeros on each side. Stride \(s\): move \(s\) entries per step.

Spatial output size counts how many complete windows fit

Apply this separately to height and width. For either spatial dimension, use input size \(n\), kernel size \(k\), padding \(p\), and stride \(s\):

\[n_{\mathrm{out}}=\left\lfloor\frac{n+2p-k}{s}\right\rfloor+1.\]

\(n+2p-k\): distance from the first valid start to the last possible start.
Divide by \(s\) and round down: number of full moves.
Add one: include the first window.

Example: \(n=7,\ k=3,\ p=1,\ s=2\) gives \(\lfloor(7+2-3)/2\rfloor+1=4\).

Next: channels. Their output count follows a different rule.

One filter spans all input channels and creates one output channel

For RGB input, a \(5\times5\) spatial patch contains values from 3 color channels:

\[\underbrace{3\times5\times5}_{\text{input channels × kernel height × kernel width}} \quad\longrightarrow\quad\text{weighted sum}+b.\]

At each position, the filter combines all input channels into one value. Sliding that same filter across the image creates one output channel (one feature map).

One bias per filter, shared across all spatial positions.

Input channels	One filter contains	Output from one filter
Grayscale: 1	25 weights + 1 bias = 26 parameters	1 output channel
RGB: 3	75 weights + 1 bias = 76 parameters	1 output channel

Multiple filters create multiple output channels

One filter → one output channel; \(c_{\mathrm{out}}\) filters → \(c_{\mathrm{out}}\) output channels.

\[[B,c_{\mathrm{in}},h,w]\quad\longrightarrow\quad[B,c_{\mathrm{out}},h_{\mathrm{out}},w_{\mathrm{out}}].\]

Part of the output	What determines it?
Batch size \(B\)	Unchanged: apply the same filters to each example
Output channels \(c_{\mathrm{out}}\)	Number of learned filters we choose
Height and width	Kernel size, padding, and stride

nn.Conv2d(3, 16, kernel_size=5, padding=2) maps [64, 3, 28, 28] → [64, 16, 28, 28].

The weight tensor has shape [16, 3, 5, 5]: sixteen filters, each spanning three input channels.

Parameter reuse is a core neural-network design principle

Same input/output: \(c\times h\times w\); counts below use \(1\times28\times28\), including biases.

MAC = multiply–accumulate; excludes bias additions. Kernels are \(k\times k\), stride 1.

Layer	Parameters	MACs / image	Parameter complexity	MAC complexity
Fully connected	615,440	614,656	\(O(c^2h^2w^2)\)	\(O(c^2h^2w^2)\)
Conv \(3\times3\), padding 1	10	7,056	\(O(k^2c^2)\)	\(O(k^2c^2hw)\)
Conv \(5\times5\), padding 2	26	19,600	\(O(k^2c^2)\)	\(O(k^2c^2hw)\)

Parameters = weights + biases: convolution \(k^2c^2+c\); fully connected \((chw)^2+chw\).

Efficiency: local connections reduce computation; shared weights reduce parameter memory.
Generalization: reuse can help when the same patterns recur across positions.
Looking ahead: transformers also reuse parameters across positions.

About four minutes. Compare a flattened fully connected layer from chw inputs to chw outputs with an ordinary dense convolution from c input channels to c output channels, reshaping the fully connected output to the same spatial shape. The concrete example is grayscale, c=1, h=w=28. Fully connected weights and MACs are (chw)^2; convolution weights are k^2 c^2 and MACs are hw k^2 c^2. Thus 784^2=614656, 9×784=7056, and 25×784=19600 MACs. Parameter totals include biases: fully connected has (chw)^2+chw=615440, and convolution has k^2 c^2+c, giving 10 or 26. Biases add chw additions per image in either case, excluded from MAC counts. These lower-order bias terms do not change the displayed big-O scaling. Padding (k-1)/2 and stride one preserve spatial size. MACs use the standard dense-kernel count including padded positions; implementations can skip work or use other algorithms, so these counts are not wall-clock timing predictions. Each MAC is approximately two floating-point operations. Keep kernel size explicit as O(k^2 c^2) parameters and O(k^2 c^2 hw) MACs in the table; substitute k=3 or k=5 for the numerical examples. Distinguish local connectivity from sharing: an unshared local layer would perform the same local weighted sums but require separate weights at every position. Parameter reuse saves storage and ties statistical estimates across positions; it does not by itself eliminate those weighted sums. Reuse can improve generalization when the repeated-pattern assumption is appropriate, but may hurt when position-specific patterns matter. Transformer token projections and feed-forward layers will reuse weights across token positions; this does not imply their attention computation has CNN complexity. [Sources] source:pytorch-conv2d-2-7.

Networks connect filters through compatible shapes

Image structure · 3/3 Networks · Patterns → Filters → Networks

Odd kernels with matching padding preserve spatial size

With stride 1, choose \(p=(k-1)/2\) for odd \(k\).

Kernel	Padding	Size 28 → output size
\(1\times1\)	0	\(28-1+1=28\)
\(3\times3\)	1	\(28+2-3+1=28\)
\(5\times5\)	2	\(28+4-5+1=28\)

Apply to height and width: \(28\times28\to28\times28\).

Same spatial size does not mean same channel count. We choose the number of filters.

A 3 by 3 kernel moves one step over a 6 by 6 input padded with one border of zeros, giving a 6 by 6 output.

Stride two can halve each spatial dimension

With stride 2, count starts along each axis:

Kernel	Padding	Size 28 → output size
\(2\times2\)	0	\(\lfloor(28-2)/2\rfloor+1=14\)
\(4\times4\)	1	\(\lfloor(28+2-4)/2\rfloor+1=14\)
\(3\times3\)	1	\(\lfloor(28+2-3)/2\rfloor+1=14\)

\(28\times28\to14\times14\): one quarter as many spatial locations.

Check odd sizes: from 27, the first two cases produce 13; the third produces 14.

A 4 by 4 kernel moves two steps over a 6 by 6 input padded with one border of zeros, giving a 3 by 3 output.

A \(1\times1\) convolution mixes channels at each location

At each pixel location, apply the same linear layer to its vector of channel values.

\[[B,16,28,28]\quad\xrightarrow{\ 1\times1,\ s=1,\ p=0\ }\quad[B,8,28,28].\]

No neighboring pixels are combined.
Sixteen input channels become eight learned output features at each location.
Parameter count with bias: \(8(16+1)=136\).

A one-pixel kernel can still learn a useful transformation.

Stacking local layers combines increasingly broad patterns

Two \(3\times3\) convolutions with stride 1 let an interior output depend on a \(5\times5\) input region.

\[1\times1\ \xrightarrow{\text{first }3\times3}\ 3\times3\ \xrightarrow{\text{second }3\times3}\ 5\times5.\]

This dependency region is the receptive field. Each new local window combines neighboring features from the previous layer.

Insert ReLU between layers to learn nonlinear combinations of those features.

A small CNN changes the model but reuses the training loop

cnn = nn.Sequential(
    nn.Conv2d(1, 8, 3, padding=1),
    nn.ReLU(),
    nn.Conv2d(8, 16, 3, stride=2, padding=1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(16 * 14 * 14, 10))
a = torch.zeros(64, 1, 28, 28)
for layer in cnn:
    a = layer(a)
    print(type(layer).__name__, tuple(a.shape))

Output:

Conv2d (64, 8, 28, 28)
ReLU (64, 8, 28, 28)
Conv2d (64, 16, 14, 14)
ReLU (64, 16, 14, 14)
Flatten (64, 3136)
Linear (64, 10)