Convolutions and Image Structure

ECE 57000 — Fall 2026

David I. Inouye

Image structure changes how we connect and share weights

Arc 5: representations → gradients → PyTorch → CNN fundamentals → evidence → ResNet → U-Net.

  1. Patterns: why use local, shared rules?
  2. Filters: how does convolution compute features?
  3. Networks: how do shapes and layers fit together?

The training loop is ready. Now change the model’s assumptions.

Nearby pixels reveal local image patterns

A bag image and three enlarged five-by-five patches, with their source locations hidden

Matching numbered boxes locate all three patches along different parts of the bag's left edge

Where do these patches belong?

A. Left edge   B. Interior   C. Right edge

A. Left edge. Each patch goes from dark to bright. One local detector can find this pattern at many positions.

Image features have meaningful spatial neighbors

General feature vector: in [age, income, height], adjacent entries need not be more related than distant entries. Their order identifies features, not spatial neighbors.

Image: neighboring pixels measure nearby locations. Their arrangement forms edges, textures, and shapes.

Fully connected layer Convolutional layer
Each hidden unit can use every pixel Each output uses a local patch
Different units have independent weights Reuse the same filter across locations
Can learn spatial relationships from data Builds local reuse into the architecture

Convolution uses this structure: local neighborhoods matter, and a useful pattern may recur at different locations.

Filters turn local patterns into computed features

Image structure · 2/3 Filters · Patterns → Filters → Networks

A convolution reuses one local weighted sum

Sliding 1D kernel, position 1

Sliding 1D kernel, position 2

Sliding 1D kernel, position 3

At each position: multiply the window entries by the same kernel entries, then add. A learned bias may be added to every result.

A 2D kernel slides across rows and columns

Sliding 2D kernel, position 1

Sliding 2D kernel, position 2

Sliding 2D kernel, position 3

A \(2\times2\) window fits at three row starts and three column starts in a \(4\times4\) image: the output is \(3\times3\).

A convolution is a sparse matrix with shared entries

A two-by-three image and two-by-two kernel produce a two-by-six matrix. Zeros mark absent connections; repeated letters are tied kernel weights.

\[\operatorname{vec}(Y)=A_K\operatorname{vec}(X),\qquad A_K=\begin{bmatrix}a&b&0&c&d&0\\0&a&b&0&c&d\end{bmatrix}.\]

Padding adds a border; stride controls how quickly the window moves

Numbered kernel windows yield 5, 7, or 4 outputs; dotted zero cells show implicit padding.

Padding \(p\): \(p\) implicit zeros on each side. Stride \(s\): move \(s\) entries per step.

Spatial output size counts how many complete windows fit

Apply this separately to height and width. For either spatial dimension, use input size \(n\), kernel size \(k\), padding \(p\), and stride \(s\):

\[n_{\mathrm{out}}=\left\lfloor\frac{n+2p-k}{s}\right\rfloor+1.\]

  • \(n+2p-k\): distance from the first valid start to the last possible start.
  • Divide by \(s\) and round down: number of full moves.
  • Add one: include the first window.

Example: \(n=7,\ k=3,\ p=1,\ s=2\) gives \(\lfloor(7+2-3)/2\rfloor+1=4\).

Next: channels. Their output count follows a different rule.

One filter spans all input channels and creates one output channel

For RGB input, a \(5\times5\) spatial patch contains values from 3 color channels:

\[\underbrace{3\times5\times5}_{\text{input channels × kernel height × kernel width}} \quad\longrightarrow\quad\text{weighted sum}+b.\]

At each position, the filter combines all input channels into one value. Sliding that same filter across the image creates one output channel (one feature map).

One bias per filter, shared across all spatial positions.

Input channels One filter contains Output from one filter
Grayscale: 1 25 weights + 1 bias = 26 parameters 1 output channel
RGB: 3 75 weights + 1 bias = 76 parameters 1 output channel

Multiple filters create multiple output channels

One filter → one output channel; \(c_{\mathrm{out}}\) filters → \(c_{\mathrm{out}}\) output channels.

\[[B,c_{\mathrm{in}},h,w]\quad\longrightarrow\quad[B,c_{\mathrm{out}},h_{\mathrm{out}},w_{\mathrm{out}}].\]

Part of the output What determines it?
Batch size \(B\) Unchanged: apply the same filters to each example
Output channels \(c_{\mathrm{out}}\) Number of learned filters we choose
Height and width Kernel size, padding, and stride

nn.Conv2d(3, 16, kernel_size=5, padding=2) maps [64, 3, 28, 28] → [64, 16, 28, 28].

The weight tensor has shape [16, 3, 5, 5]: sixteen filters, each spanning three input channels.

Parameter reuse is a core neural-network design principle

Same input/output: \(c\times h\times w\); counts below use \(1\times28\times28\), including biases.

MAC = multiply–accumulate; excludes bias additions. Kernels are \(k\times k\), stride 1.

Layer Parameters MACs / image Parameter complexity MAC complexity
Fully connected 615,440 614,656 \(O(c^2h^2w^2)\) \(O(c^2h^2w^2)\)
Conv \(3\times3\), padding 1 10 7,056 \(O(k^2c^2)\) \(O(k^2c^2hw)\)
Conv \(5\times5\), padding 2 26 19,600 \(O(k^2c^2)\) \(O(k^2c^2hw)\)

Parameters = weights + biases: convolution \(k^2c^2+c\); fully connected \((chw)^2+chw\).

  • Efficiency: local connections reduce computation; shared weights reduce parameter memory.
  • Generalization: reuse can help when the same patterns recur across positions.
  • Looking ahead: transformers also reuse parameters across positions.

Networks connect filters through compatible shapes

Image structure · 3/3 Networks · Patterns → Filters → Networks

Odd kernels with matching padding preserve spatial size

With stride 1, choose \(p=(k-1)/2\) for odd \(k\).

Kernel Padding Size 28 → output size
\(1\times1\) 0 \(28-1+1=28\)
\(3\times3\) 1 \(28+2-3+1=28\)
\(5\times5\) 2 \(28+4-5+1=28\)

Apply to height and width: \(28\times28\to28\times28\).

Same spatial size does not mean same channel count. We choose the number of filters.

A 3 by 3 kernel moves one step over a 6 by 6 input padded with one border of zeros, giving a 6 by 6 output.

Stride two can halve each spatial dimension

With stride 2, count starts along each axis:

Kernel Padding Size 28 → output size
\(2\times2\) 0 \(\lfloor(28-2)/2\rfloor+1=14\)
\(4\times4\) 1 \(\lfloor(28+2-4)/2\rfloor+1=14\)
\(3\times3\) 1 \(\lfloor(28+2-3)/2\rfloor+1=14\)

\(28\times28\to14\times14\): one quarter as many spatial locations.

Check odd sizes: from 27, the first two cases produce 13; the third produces 14.

A 4 by 4 kernel moves two steps over a 6 by 6 input padded with one border of zeros, giving a 3 by 3 output.

A \(1\times1\) convolution mixes channels at each location

At each pixel location, apply the same linear layer to its vector of channel values.

\[[B,16,28,28]\quad\xrightarrow{\ 1\times1,\ s=1,\ p=0\ }\quad[B,8,28,28].\]

  • No neighboring pixels are combined.
  • Sixteen input channels become eight learned output features at each location.
  • Parameter count with bias: \(8(16+1)=136\).

A one-pixel kernel can still learn a useful transformation.

Stacking local layers combines increasingly broad patterns

Two \(3\times3\) convolutions with stride 1 let an interior output depend on a \(5\times5\) input region.

\[1\times1\ \xrightarrow{\text{first }3\times3}\ 3\times3\ \xrightarrow{\text{second }3\times3}\ 5\times5.\]

This dependency region is the receptive field. Each new local window combines neighboring features from the previous layer.

Insert ReLU between layers to learn nonlinear combinations of those features.

A small CNN changes the model but reuses the training loop

cnn = nn.Sequential(
    nn.Conv2d(1, 8, 3, padding=1),
    nn.ReLU(),
    nn.Conv2d(8, 16, 3, stride=2, padding=1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(16 * 14 * 14, 10))
a = torch.zeros(64, 1, 28, 28)
for layer in cnn:
    a = layer(a)
    print(type(layer).__name__, tuple(a.shape))

Output:

Conv2d (64, 8, 28, 28)
ReLU (64, 8, 28, 28)
Conv2d (64, 16, 14, 14)
ReLU (64, 16, 14, 14)
Flatten (64, 3136)
Linear (64, 10)

Keep the pixels, change their arrangement

Randomly permute pixel positions. Use the same permutation for every image, then retrain both models.

Which model’s architectural advantage would you expect to shrink?

A. MLP

B. CNN

C. Both equally

B. CNN. The information remains, but nearby entries no longer form local image patterns.

Convolution provides structure, not a guarantee of accuracy

  • Locality: each output sees a patch, expanded by stacking layers.
  • Sharing: the same filters are reused across positions.
  • Shape control: filters choose channels; kernel, padding, and stride choose spatial sizes.

Next, compare CNN and MLP using the same data splits and a documented training/selection protocol.

The design is now explicit. Its benefit is an empirical question.