Image Segmentation with U-Net

Segmentation needs a prediction at every image location

Arc 5: CNN fundamentals → evidence → ResNet classification → U-Net segmentation.

Task → Upsampling → Architecture

A classifier summarizes an image. A segmentation model must retain where each class occurs.

A road map needs a label at every pixel

Mnih, Massachusetts Roads (2013), validation tile 10228690_15; aligned crop.

The input can stay the same while the output becomes spatial

	Image classification	Semantic segmentation
RGB input	\([B,3,H,W]\)	\([B,3,H,W]\)
Model logits	\([B,K]\)	\([B,K,H,W]\)
Integer targets	\([B]\)	\([B,H,W]\)

For roads: two logits per pixel, road or background.

Reuse cross-entropy, averaged over labeled pixels. Keep the spatial grid instead of globally pooling it away.

Transposed convolution uses the transpose of the linear operator

A convolution matrix maps six flattened inputs to two outputs. Its transpose reverses the connections and maps two inputs to six outputs, without guaranteeing inversion.

\[\boldsymbol{y}=A_K\boldsymbol{x}\qquad\longrightarrow\qquad\boldsymbol{u}=A_K^{\mathsf T}\boldsymbol{z}\]

Fractional-stride intuition explains learned spatial expansion

A stride-two transposed convolution of inputs two and three with kernel one-two-one scatters overlapping patches, summing to two-four-five-six-three.

In 2D, ConvTranspose2d(16, 8, 2, stride=2) maps \([B,16,16,16]\) to \([B,8,32,32]\).

About four minutes. The 1D example uses kernel [1,2,1], stride 2, padding 0: input [2,3] yields [2,4,5,6,3]. An equivalent implementation inserts one zero between inputs and applies a full filter, with appropriate boundaries and kernel orientation; the symmetric kernel deliberately avoids a distracting flip in this visual. For nonsymmetric PyTorch cross-correlation kernels the equivalent ordinary convolution requires the corresponding flip. Fractional stride describes the finer-grid interpretation, not a fractional stride argument in the API. The 2D doubling example uses kernel 2/stride 2/padding 0, so has no overlapping windows; it is separate from the overlapping 1D example. Learned expansion does not create observed evidence. Transposed convolution with stride 1 need not upsample. Interpolation followed by convolution is a common alternative; uneven overlap can matter for artifacts, beyond today’s scope. [Sources] source:pytorch-convtranspose2d-2-7.

U-Net combines coarse context with fine-resolution features

U-shaped MONAI BasicUNet architecture. Front faces shrink while channel depth grows through the encoder; the decoder restores spatial size. Horizontal skip arrows concatenate features at equal spatial resolutions.

Original illustration of MONAI BasicUNet 1.5.1, 2D configuration; shapes are channels × spatial size.

About four minutes. Use spatial_dims=2, in_channels=3, out_channels=2, features=(32,32,64,128,256,32), upsample=‘deconv’, default instance normalization with affine=True and LeakyReLU slope .1. The image passes through the first two-convolution stage, then four max-pool/two-convolution stages. The decoder upsamples with 2×2 stride-2 transposed convolution, concatenates the matching encoder map, and applies two more Conv–InstanceNorm–LeakyReLU operations. Padding 1 preserves each stage’s spatial shape; 256 is divisible by 16. Default MONAI uses per-instance spatial statistics and learned scale/offset; do not label these blocks BatchNorm/ReLU. This is a contemporary maintained implementation of the U-Net family, not a claim of a universally best architecture. The original U-Net used valid convolutions with cropping; this padded version is easier to read. The input and output heads appear on the next slide. [Sources] source:monai-basicunet-1-5-1; source:ronneberger-unet-2015.

The U-Net head keeps one score vector at every pixel

The task determines what the architecture must preserve

Classification: aggregate spatial features into an image-level decision.
Segmentation: combine broad context with location-specific detail.
Both: architectural choices need evidence on the intended data and task.

We can now read complete architectures from familiar operations. Knowing the pieces does not show that a model solves the application.