Visual representation learning
with convolutional neural networks

Week-05 · ML-2026

Alexei Kornaev,
Dr. Sc., Prof.

Innopolis University

Researcher at the RC for AI,
National RC for Oncology n.a. N.N. Blokhin

The sketches suggest progressively composed visual structure; they are a metaphor, not measured CNN activations. Connect the title to the question: what should an image model reuse across locations?

Existing instructor-supplied artwork: creation provenance and public reuse terms remain TODO(instructor).

Lecture map

1. Images and local visual evidence
2. Spatial control and feature hierarchies
3. Deep convolutional architectures
4. Using visual representations

Print / save as PDF

Follow one thread: constrain the connections, understand the resulting spatial behavior, make deep models trainable, then evaluate whether the representation transfers. Pause after the spatial arithmetic before introducing architecture history. The paper-reading slide can support a short discussion or a longer guided reading, depending on the available time.

Part 1 · Images and local visual evidence

1. Images and local visual evidence
2. Spatial control and feature hierarchies
3. Deep convolutional architectures
4. Using visual representations

Why local connectivity and shared parameters fit many visual tasks.

From an MLP to a convolutional network

1

Prediction

$\widehat{\mathbf y}=f(\mathbf x;\theta)$

→

2

Mean mini-batch objective

$\mathcal L_{\mathcal B}=\frac{1}{B}\sum_{i\in\mathcal B}\ell_i$

→

3

Gradient

$\nabla_\theta\mathcal L_{\mathcal B}$

→

4

Parameter update

$\theta\leftarrow\theta-\eta\nabla_\theta\mathcal L_{\mathcal B}$

Week 04 · MLP

Every output unit can connect to every input value.

architecture changes →

Week 05 · CNN

Connections are local; the same parameters are shared across positions.

The learning loop stays the same; the architecture constrains which predictions it can express.

Recall that $\theta$ includes weights and biases, $B=|\mathcal B|$, and $\eta$ is the learning rate. Walk around the loop once, emphasizing that backpropagation and the optimizer are unchanged. Ask which part must change when an image is represented by local shared connections: the forward map and its parameterization.

Images are structured measurements

One batch: $\mathbf X\in\mathbb R^{B\times C_{\mathrm{in}}\times H\times W}$

$C_{\mathrm{in}}$ measured channels $H,W$ spatial axes

Grayscale

$C_{\mathrm{in}}=1$

One intensity at each location.

RGB

$C_{\mathrm{in}}=3$

Three measurements at each location in one image.

Hyperspectral

$C_{\mathrm{in}}\gg3$

Many wavelength bands at each location.

Channels describe what is measured; height and width describe where it is measured.

Read the shape from left to right: examples, channels, rows, columns. The colored sheets depict aligned measurements of the same scene, not separate examples. Ask where four RGB images would appear in the tensor: $B=4$, $C_{\mathrm{in}}=3$. Array storage order varies across libraries; these slides use PyTorch’s channel-first convention.

Dense connectivity spends parameters on every location

Both layers map the same input to the same output shape:

$1\times28\times28\ \longrightarrow\ 8\times28\times28$

Dense layer

784 inputs≋≋≋≋≋6272 outputs

Each output has its own weight for every input.

$$784\times6272+6272=\mathbf{4{,}923{,}520}$$

Conv2d: 8 filters, $3\times3$, padding 1

local $3\times3$same filter
at every position8 maps

Eight local filters are shared over the image.

$$8\times1\times3\times3+8=\mathbf{80}$$

Equal output dimensions do not mean equal models: the convolution imposes local, shared connectivity.

Count one bias per output unit for the dense layer, but one bias per output channel for Conv2d. Before revealing the convolution card, ask whether doubling image width changes the number of learned filter weights. It does not, although compute and activation storage grow. This comparison assumes stride 1, dilation 1, and ungrouped convolution; the dense output is reshaped to the displayed grid.

A historical experiment on visual selectivity

What was measured

Invasive single-cell recordings from visual cortex in lightly anaesthetized cats while patterned light stimulated the retina.

What this drawing shows

The spikes and tuning curve are schematic illustrations, not data reproduced from the paper.

Historical study: (Hubel and Wiesel 1959). Supplied schematic shown unaltered.

The 1959 study used 24 cats. The published methods describe anaesthesia, immobilization of the eyes using a muscle relaxant, artificial respiration, and extracellular single-unit recording. Keep the discussion factual and brief. The schematic is a teaching illustration and does not reproduce a particular recorded cell.

A receptive field links a response to a region of input

A receptive field is the region of the visual input that can influence the measured response.

Illustrative response patterns—not measured traces. The biological finding is that response can depend on both stimulus position and orientation.

Compare position first, then orientation at a fixed position. A receptive field says where an input can matter; selectivity says which pattern evokes a response there. The sparse marks are schematic and do not mean a neuron must be completely silent outside the drawn condition. The historical evidence is the single-cell study (Hubel and Wiesel 1959).

Biological evidence and the CNN design

Measured observation

Some recorded cortical cells responded selectively to visual patterns in restricted regions.

motivates architectural questions →

⌗

Local connectivity

Each unit initially uses a small spatial neighborhood.

↔︎

Shared weights

One learned local rule is reused at many positions.

▱

Composition through layers

Deeper layers combine earlier local responses.

These are engineered CNN assumptions. The cat experiment did not prove them or prescribe a CNN.

Separate observation from design: restricted visual sensitivity motivates locality, but it does not establish exact weight sharing. Sharing is an engineering constraint that lets one detector operate across the grid. Ask which assumption would fail if an identical patch meant different things at different image locations.

Notation and vocabulary

Image batch

$\mathbf X\in\mathbb R^{B\times C_{\mathrm{in}}\times H\times W}$

Conv2d weights

$\mathbf W\in\mathbb R^{C_{\mathrm{out}}\times C_{\mathrm{in}}\times k_H\times k_W}$

Preactivations

$\mathbf z\in\mathbb R^{B\times C_{\mathrm{out}}\times H_{\mathrm{out}}\times W_{\mathrm{out}}}$

Kernel slice

Spatial weights for one input channel.

Filter

All channel slices that produce one output channel.

Response map

Output of a fixed kernel before an activation.

Activation map

One channel of a learned representation after activation.

Single-layer notation suppresses $(\ell)$: $\mathbf h=\phi(\mathbf z)$, with $\mathbf h^{(0)}=\mathbf X$.

This is the notation contract for the remaining slides. $\mathbf W,\mathbf b,\mathbf z,\mathbf h$ abbreviate the Week 04 layer-indexed quantities; the first input is $\mathbf h^{(0)}=\mathbf X$. A filter contains all input-channel slices for one output channel. “Kernel” is used less consistently in libraries, so follow the tensor shape rather than the word alone.

Fixed kernels ask local questions

Local mean

$\frac19$$\frac19$$\frac19$ $\frac19$$\frac19$$\frac19$ $\frac19$$\frac19$$\frac19$

Asks: what is the average intensity nearby?

Smooths small local changes.

Vertical contrast

$-1$$0$$1$ $-1$$0$$1$ $-1$$0$$1$

Asks: does the right side exceed the left?

Produces a signed edge response.

A kernel defines one local question; scanning asks that same question at every position.

Apply both kernels mentally to a constant patch. The mean kernel returns the constant; the contrast kernel returns zero because its weights sum to zero. “Vertical contrast” here means sensitivity to a vertical edge: it compares right against left. A strong negative value signals the reversed edge direction.

One kernel produces one grayscale response map

One input channel

$C_{\mathrm{in}}=1$

One fixed kernel

One signed scalar at each valid position.

Scan the image

Those scalars form one response map.

The map records where the kernel’s local pattern is present—and with which sign.

Trace the highlighted patch through elementwise products to the signed sum, then locate that one value in the response map. A negative response is evidence for the opposite contrast, not an error. Ask what would happen if every kernel sign were reversed.

The optional scan animation repeats this operation along one row; the static figure contains the full calculation.

Learning chooses the local question

Forward pass

The same $w_0,w_1$ participate at every position.

Backpropagation

Gradient contributions from all uses accumulate into $\nabla_{w_0}\mathcal L$ and $\nabla_{w_1}\mathcal L$.

Update

One update changes the local question everywhere.

Weight sharing is a parameter identity built into the architecture; learning still uses the Week 04 loop.

The example uses $(w_0,w_1)=(2,-1)$: the first window gives $2\cdot1-2=0$. These are repeated uses of the same two parameters, not separately trained copies. Backpropagation adds contributions at shared nodes, just as in Week 04. Nonlinear activations between learned layers are still needed to build nonlinear representations.

Each RGB filter combines all input channels

One filter has shape $C_{\mathrm{in}}\times k_H\times k_W$. For RGB, its three channel slices jointly produce one output value at each position.

RGB is one image with three measured channels; a filter combines them rather than classifying them independently.

Follow each color branch to its scalar contribution, then add the three contributions and one bias. Ask how many output maps ten such filters produce: ten, not thirty. This is ordinary ungrouped convolution; grouped and depthwise convolutions deliberately restrict which input channels each filter can use.

Part 2 · Spatial control and feature hierarchies

1. Images and local visual evidence
2. Spatial control and feature hierarchies
3. Deep convolutional architectures
4. Using visual representations

How spatial sampling changes representations and how depth combines local evidence.

What does Conv2d compute?

input
$B\times C_{\mathrm{in}}\times H\times W$ + weights
$C_{\mathrm{out}}\times C_{\mathrm{in}}\times k_H\times k_W$ → preactivations
$B\times C_{\mathrm{out}}\times H_{\mathrm{out}}\times W_{\mathrm{out}}$

groups=1, dilation=1; $n$ indexes examples, $s_H,s_W$ are strides.

$$z_{n,o,r,c} =b_o+ \sum_{i=0}^{C_{\mathrm{in}}-1} \sum_{u=0}^{k_H-1} \sum_{v=0}^{k_W-1} W_{o,i,u,v}\, \widetilde h_{n,i,\,r s_H+u,\,c s_W+v}.$$

$i$ sums over all input channels.

$u,v$ sum over a local spatial window.

$o$ selects one learned output filter.

PyTorch uses unflipped cross-correlation; $\widetilde{\mathbf h}$ is the padded input, $\mathbf z$ the preactivation.

Operation convention: PyTorch Conv2d.

Fix $n,o,r,c$ and read the right-hand side as one scalar calculation. The three sums combine channel slices and spatial offsets; stride changes the window start. Padding is already included in $\widetilde{\mathbf h}$, so no padding offset is subtracted again. For symmetric padding and dilation 1, $H_{\mathrm{out}}=\lfloor(H+2p_H-k_H)/s_H\rfloor+1$, and similarly for width. Activation follows separately.

A structured linear map under the hood

Zeros encode locality

Each output ignores distant inputs.

Repeated labels encode sharing

The same parameter appears at shifted positions.

The operation is still linear

Before bias and activation, a convolution is a structured matrix multiplication.

Convolution restricts a dense linear map to a sparse, repeated pattern of coefficients.

Each row computes one output position. Zeros remove distant connections, and repeated labels tie parameters across rows. Bias makes the operation affine; an activation makes it nonlinear. The matrix is an explanatory representation: efficient implementations need not construct this large sparse matrix explicitly.

Translation equivariance

Pattern at the left

filter →

translate input
four pixels →

The same pattern at the right

same filter →

$$f(T_{\Delta}\mathbf x)=T_{\Delta}f(\mathbf x)$$

For stride 1, shared weights make responses shift with the input, away from boundary effects.

$T_\Delta$ translates by an integer displacement and $f$ is the stride-1 filtering operation here. Predict where the response moves before tracing the second panel. Exact equality needs a compatible domain and boundary convention; cropping and padding can break it near the edges. Strided sampling will add another restriction on the next slides.

Equivariance and classification invariance

Equivariant representation

shift input → feature map shifts

location is preserved

Invariant prediction

shift input → same class score

location is discarded

Convolution supplies a moving representation. Pooling, aggregation, data augmentation, and training determine whether the final decision becomes stable to a shift.

Equivariance is a property of the representation; invariance is a property of the final output.

Ask whether locating an object and naming its class require the same treatment of position. Dense prediction often needs a moving output, while classification may benefit from a stable score. Equivariance and invariance are general transformation properties; the two rows illustrate their roles in this particular pipeline. Pooling and augmentation encourage stability without guaranteeing it.

Padding and boundary assumptions

A $3$-value window centred at the left boundary needs one value outside the measured input.

No padding

outside47

Do not evaluate this boundary-centred window.

Zero padding

047

Assume the unmeasured value is zero.

Reflect padding

747

Assume the signal mirrors at the boundary.

Padding supplies values beyond the observed domain; its width controls output size and its rule changes border evidence.

Keep the measured values 4 and 7 fixed and change only the value supplied outside the domain. Reflect padding here excludes the edge value itself, hence 7,4,7. Padding preserves height for a stride-1, width-3 kernel only when one value is added on each side; padding in general does not guarantee equal input and output sizes.

Stride and spatial sampling

Stride $s=1$

Evaluate every valid start: $0,1,2,3,4,5,6$.

Stride $s=2$

Evaluate starts $0,2,4,6$; the others are skipped.

The filter is unchanged. The output samples fewer spatial positions, so a one-pixel input shift can land on a different sampling phase.

Stride reduces resolution by choosing where to evaluate the local rule.

The dots mark possible window starts, not the individual values within a kernel. At stride 2, moving the input by one position changes the alignment with the sampled grid. A two-position input shift corresponds to one output step away from boundaries. Ask why using fewer positions saves computation but can lose fine detail.

Pooling and information loss

Max pooling keeps the largest value.

Average pooling keeps the local mean.

Both reduce a region to one number, discarding the exact arrangement inside that region.

Pooling can make some small shifts less consequential, but it does not guarantee translation invariance.

Pooling trades spatial detail for a compact summary.

Read the actual maximum, including its sign; max pooling does not choose the largest absolute magnitude. Permuting the four values inside this window preserves both its maximum and its mean. Moving a value across a window boundary can change the result, which explains why local pooling is not exact translation invariance.

Receptive fields through depth

Receptive-field size: four input positions can influence one layer-2 output.

Output spacing: adjacent layer-2 outputs begin two input positions apart.

Depth composes local dependencies into larger input regions; stride also changes how far adjacent outputs jump.

Trace one top node through both intermediate nodes to the input. Count distinct input positions, not paths: the overlap makes the union four positions wide. Then compare adjacent top nodes to see their two-position spacing. This is structural dependence; learned zero weights or inactive units can make actual influence smaller.

Receptive-field arithmetic

Track two quantities

One axis; sequential local layers; dilation 1.

$$\begin{aligned} j_\ell &= j_{\ell-1}s_\ell,\\ r_\ell &= r_{\ell-1}+(k_\ell-1)j_{\ell-1}. \end{aligned}$$

$r_\ell$: receptive-field size
$j_\ell$: input spacing between adjacent outputs

Stage	$k$	$s$	$r$	$j$
input	—	—	1	1
layer 1	3	1	3	1
layer 2	2	2	4	2

For this two-layer path, one output sees four input positions while adjacent outputs are spaced two positions apart.

Initialize $r_0=j_0=1$. After layer 1, $r_1=3$ and $j_1=1$; ask students to compute layer 2 before revealing the table. The width-2 layer adds one previous-layer spacing, so $r_2=4$, while stride gives $j_2=2$. With dilation $d_\ell$, replace $k_\ell-1$ by $d_\ell(k_\ell-1)$. Global operations require a different dependency analysis.

Feature hierarchies in a trained network

Early block

Fine spatial detail remains visible.

Deeper blocks

Resolution shrinks while channels combine earlier responses.

Interpret carefully

Strong activation does not give every channel a simple human-readable meaning.

Composition through layers changes both what a channel represents and how precisely it preserves location.

This reuses the Week 01 ImageNet-pretrained MobileNetV3-Small on a resized, three-channel MNIST digit. Six channels at each depth were selected by mean absolute activation. Compare spatial resolution and response structure, not brightness across separately displayed maps. These observations do not establish that a particular channel detects a named object part; the digit also differs from the pretraining domain.

Spatial aggregation and class logits

$C\times H\times W$
activation maps →

$\displaystyle \frac{1}{HW}\sum_{r,c}$

one mean
per channel →

$[\bar h_1,\ldots,\bar h_C]$

→

$K$ class logits

Retained: which channels responded strongly.

Discarded: where within each map the responses occurred.

Global average pooling creates location-insensitive summaries by explicitly removing spatial coordinates.

Average each channel separately, then use a learned linear head to map the $C$ means to $K$ class logits. Ask whether swapping two locations within one map changes its mean: no. This invariance of the aggregation does not guarantee invariance of the entire network to an input translation, because borders and strided layers may change the maps.

Convolution in one, two, and three dimensions

━━━━

Conv1d

Slides along one spatial axis: time, sequence position, or distance.

Input: $(B,C,T)$

▦

Conv2d

Slides along height and width: images and spatial fields.

Input: $(B,C,H,W)$

▧

Conv3d

Slides along three spatial axes: volumes or space–time grids.

Input: $(B,C,D,H,W)$

The dimension counts convolved spatial axes; channels remain measurements or learned features at each location.

The number in Conv1d/2d/3d counts the axes along which a kernel slides, not the tensor rank. A multichannel time series is still Conv1d. For video, $D$ can index time while $H,W$ index image position; channels remain separate. This extends the same locality-and-sharing hypothesis to a new domain.

Evidence from the tutorial experiment

Accuracy · validation-selected epoch 30

Model	Validation	Test
MLP	0.536	0.521
Conv1d	0.675	0.654

Same split, seed, optimizer, and 30-epoch budget.

Both validation losses were still decreasing. This is one non-converged run on a synthetic benchmark—not a general ranking of CNNs and MLPs.

Here the local, shared architecture reached higher accuracy within 30 epochs; this does not establish faster wall-clock training.

The table reports accuracy, not loss. Each checkpoint was selected by validation loss, then evaluated once on the held-out test set. Both selected epoch 30, at the budget boundary. Read the learning curves before ranking the models: further training could change the comparison. One matched seed does not measure seed variability, and equal epochs do not imply equal compute. See the tutorial experiment.

When the convolutional bias fits

A useful match

• nearby values interact meaningfully;
• similar patterns recur across positions;
• shifting a pattern should reuse the same rule;
• larger structures can compose from local evidence.

A weak match

• feature order is arbitrary;
• absolute position changes the rule;
• distant relationships dominate immediately;
• the same pattern means different things by location.

Treat locality and sharing as hypotheses. Compare against credible alternatives with validation data and inspect failures under the shifts that matter.

CNNs work when their architectural assumptions resemble the structure of the task—not merely because the input can be arranged as a grid.

Here “bias” means an inductive assumption, not the additive parameter $b$. Invite one task that fits each column. Arbitrarily arranging tabular columns as pixels does not create meaningful neighborhoods. Position-aware inputs and broader interactions can relax a CNN’s assumptions, but those choices should be justified by validation evidence.

Part 3 · Deep convolutional architectures

1. Images and local visual evidence
2. Spatial control and feature hierarchies
3. Deep convolutional architectures
4. Using visual representations

Architectural choices that made deep visual models trainable.

AlexNet: a coordinated design

RGB crop→5 convolutional layers→3 fully connected layers→1000 classes

Use the available scale

1.2 million training images and two GPUs made a large model feasible.

Make optimization practical

ReLUs trained faster than saturating nonlinearities; pooling reduced spatial resolution.

Control overfitting

Random crops and reflections, colour augmentation, weight decay, and dropout.

ILSVRC 2012 top-5 test error: AlexNet ensemble 15.3% · second-best entry 26.2%

AlexNet’s result came from a coordinated model–data–compute–optimization system, not from convolution alone.

Architecture, training choices, and reported competition result: (Krizhevsky et al. 2012).

Read the result as a system-level comparison. Top-5 error means the correct class is absent from the five highest-scoring labels. The 15.3% result is an ensemble competition entry, not the error of one network. The paper reports training on two GTX 580 GPUs; its ReLU speed comparison used a smaller CIFAR-10 model, so do not transfer that speedup numerically to ImageNet (Krizhevsky et al. 2012).

Reading the AlexNet paper

Follow the authors’ argument

Problem→design constraint→intervention→evidence

Open the original paper

Questions to carry into the paper

1. Which choices increase representational capacity?
2. Which choices make that capacity trainable on the available hardware?
3. Which comparisons support each claimed benefit?

Architecture: five convolutional + three fully connected learned layers Optimization: ReLU comparison on CIFAR-10 Generalization: augmentation, dropout, held-out error

Read an architecture paper as a chain of design claims and evidence—not as a list of layers to memorize.

Guided reading: (Krizhevsky et al. 2012).

Open the paper and use Sections 3–6. Ask students to name the constraint addressed by each choice before discussing the reported result.

Original AlexNet and the Torchvision variant

2012 paper reportTraced Torchvision implementation

Input convention$224\times224\times3$ stated; first response size needs an additional boundary convention$1\times3\times224\times224$; padding 2 gives a reproducible $55\times55$ first response

Channels96 → 256 → 384 → 384 → 25664 → 192 → 384 → 256 → 256

Historical detailstwo-GPU grouped connectivity and local response normalizationsingle module graph; no local response normalization

Shared ideaconvolutional feature extractor followed by a large classifierthe same broad pattern, with implementation-specific dimensions

The name identifies a model family. It does not make a library implementation a literal reconstruction of the paper.

Paper specification: (Krizhevsky et al. 2012). Torchvision values come from the repository’s reviewed FX trace of alexnet(weights=None).

Compare the channel rows first: the library model is a variant, not an exact reproduction. For the first layer, floor((224 + 2×2 − 11)/4) + 1 = 55. Without padding, 224 gives 54. The paper’s stated input and response sizes therefore need an additional convention.

Open the complete traced AlexNet graph for a zoomable shape ledger; the trace uses untrained weights. Torchvision source specifies this variant.

The optimization problem with increasing depth

If added layers can represent identity

trained shallow mapping+identity layers=same function

So its best achievable training error should be no worse.

But plain networks were harder to optimize

20-layer plain56-layer plain

On CIFAR-10, the deeper plain network showed higher training error.

Higher training error diagnoses an optimization problem; overfitting would instead describe a widening train–test gap.

When the shallow function can be preserved, extra depth need not worsen the best fit—but optimization may still struggle.

Constructed identity solution and 20/56-layer degradation experiment: (He et al. 2016). Bars are schematic, not digitized paper data.

First ask what extra identity layers should do to a trained shallow function. The argument assumes compatible shapes and that the added layers can represent identity on the relevant inputs. Then compare training errors: worse training fit points to optimization difficulty rather than explaining the result by overfitting. The bars show only the direction of the paper’s CIFAR-10 comparison, not measured magnitudes (He et al. 2016).

Residual learning

$\mathbf x$

↗
↘

$F(\mathbf x;\theta_F)$learned correction$\mathbf x$identity reference

+

$\mathbf y$

$$\mathbf y=F(\mathbf x;\theta_F)+\mathbf x$$

If the desired mapping is near identity, learn a small correction. If identity is adequate, drive $F(\mathbf x;\theta_F)$ toward zero.

The block learns what should change relative to its input, while the shortcut preserves a reference path.

Residual sum before a following activation; $\theta_F$ collects branch parameters (He et al. 2016).

Follow the split, the learned branch, and elementwise addition. The paper’s $\mathbf x,\mathbf y$ mean block input and summed output here; they are not the sample and label symbols from supervised learning. If the target mapping is identity, a zero residual suffices. In the illustrated ResNet-v1 implementation, ReLU follows the sum; the next slide differentiates the sum itself.

Information and gradients through a shortcut

Forward signal

$\mathbf y=\mathbf x+F(\mathbf x)$

The block can preserve $\mathbf x$ directly while adding a learned transformation.

Backward sensitivity

$\dfrac{\partial\mathbf y}{\partial\mathbf x}=\mathbf I+\dfrac{\partial F}{\partial\mathbf x}$

At the sum, the Jacobian has an identity term. A following ReLU adds its own gate.

This creates a useful route for signals and gradients. It does not guarantee convergence, prevent every vanishing gradient, or make arbitrary depth beneficial.

Addition lets the direct path and learned path coexist; optimization can use both.

Residual block definition: (He et al. 2016). The derivative follows from the block equation.

Treat flattened tensor coordinates as vectors when reading the Jacobian. The identity term arises from differentiating the direct path; it is not an extra loss or a separate optimizer. A ReLU after addition multiplies this Jacobian by its activation gate. Even at the sum, branch derivatives can cancel or amplify the direct term, so the shortcut is not a guarantee against vanishing or exploding gradients.

Identity and projection shortcuts

Identity shortcut: shapes already match

$\operatorname{shape}(\mathbf x)=\operatorname{shape}(F(\mathbf x))$

Projection shortcut: shape must change

$\operatorname{shape}(W_s\mathbf x)=\operatorname{shape}(F(\mathbf x))$

Elementwise addition requires equal batch, channel, and spatial dimensions; a projection is used when they differ.

Automatically traced, untrained Torchvision ResNet-18 graphs; shortcut equations from (He et al. 2016).

Use the diagrams to follow the two branches rather than read every operation label aloud. The projection block changes resolution and channel count, so its shortcut uses a stride-2, 1×1 convolution plus BatchNorm in this implementation. $W_s\mathbf x$ is the paper’s simplified projection notation. Open the identity graph or projection graph to inspect the small labels.

The ResNet family

Model	Block	Blocks per stage
ResNet-18	basic, 2 conv	2 · 2 · 2 · 2
ResNet-34	basic, 2 conv	3 · 4 · 6 · 3
ResNet-50	bottleneck, 3 conv	3 · 4 · 6 · 3
ResNet-101	bottleneck, 3 conv	3 · 4 · 23 · 3
ResNet-152	bottleneck, 3 conv	3 · 8 · 36 · 3

Across stages: spatial resolution falls while channel capacity rises.

The family scales depth by repeating residual blocks within a stable four-stage spatial design.

Family configurations and stage rules: (He et al. 2016). ResNet-18 overview is an automatically traced Torchvision implementation.

The table counts blocks within four stages, not individual convolution layers. ResNet-18 has one stem convolution, sixteen block convolutions, and one classifier in its conventional depth count. Bottleneck blocks use three convolutions and different internal channel widths; increasing the name’s number is not simply widening the same block. BatchNorm in the graph motivates the next section (He et al. 2016).

Why normalize activations?

1 · Center$x-\mu$→2 · Scale$\div\sqrt{\sigma^2+\epsilon}$→3 · Restore flexibility$\gamma\widehat{x}+\beta$

$\mu,\sigma^2$ describe a chosen set of activations. $\epsilon>0$ prevents division by zero and limits extreme scaling. $\gamma,\beta$ are learned with the rest of $\theta$.

Normalization changes the parameterization and optimization path. It does not guarantee unit variance after the learned affine step, nor does it make every deeper network train well.

The equation is shared; the method is defined by which activations contribute to $\mu$ and $\sigma^2$.

Batch-normalizing transform and learned affine parameters: (Ioffe and Szegedy 2015).

Centering and scaling use a specified set of values; learning $\gamma,\beta$ restores adjustable scale and offset afterward. With positive $\epsilon$, even the normalized values need not have exactly unit variance. The original BatchNorm paper proposed an internal-covariate-shift explanation; this slide makes the narrower claim that normalization changes optimization, without presenting that historical explanation as a settled mechanism.

BatchNorm axes for image tensors

$B$examplesincluded

$C$channelskept separate

$H$rowsincluded

$W$columnsincluded

For each channel $c$:

$$\mu_c,\sigma_c^2\text{ use all }x_{b,c,h,w}\text{ over }(b,h,w).$$

One learned pair $(\gamma_c,\beta_c)$ is shared across spatial positions.

channel 0: its own statisticschannel 1: different statisticschannel 0 at another location: same statistics

BatchNorm2d(C) couples examples and locations within each channel; it does not mix channels into one mean.

Convolutional BatchNorm axes: (Ioffe and Szegedy 2015).

Choose one channel and sweep across examples, rows, and columns: that is its statistics set during training. There are $C$ means, $C$ variances, and normally $2C$ affine parameters. Batch size one can still supply many spatial values, though neighboring pixels are correlated. Ask whether another example in the training batch can change this example’s normalized output: yes.

BatchNorm during training and evaluation

Training mode

current mini-batch→$\mu_{\mathcal B},\sigma^2_{\mathcal B}$→normalized output

The current batch affects every example’s output. Running estimates are updated for later evaluation.

Evaluation mode

one input→stored running statistics→deterministic output

The learned $\gamma, \beta$ remain active, but the current evaluation batch does not set the statistics.

Small or correlated batches: noisy, unrepresentative training statistics. Domain shift: stored statistics may describe the pretraining domain poorly. Wrong mode: training and evaluation can silently disagree.

BatchNorm carries state: correct train/eval mode and representative batches are part of the model’s behavior.

Default track_running_stats=True: PyTorch BatchNorm2d; (Ioffe and Szegedy 2015).

The slide describes the PyTorch default with running statistics enabled. With track_running_stats=False, evaluation also uses current-batch statistics. eval() changes module behavior; it does not disable gradient recording. Conversely, no_grad() does not stop training-mode running-statistic updates. A frozen backbone therefore needs an explicit mode/state policy, which becomes important in transfer learning.

LayerNorm for one example

Values$\mathbf x=(1,3)$

→

Statistics$\mu=2$$\sigma^2=1$

→

Normalize$\epsilon=0.25$$\widehat{\mathbf x}=(-0.894,0.894)$

→

Learned affine$\gamma=(2,0.5)$$\beta=(1,-1)$$\mathbf y=(-0.789,-0.553)$

$$\widehat{x}_j=\frac{x_j-2}{\sqrt{1+0.25}}, \qquad y_j=\gamma_j\widehat{x}_j+\beta_j.$$

$\epsilon=0.25$ is deliberately large so its numerical role is visible; production defaults are usually much smaller.

LayerNorm computes statistics from the chosen coordinates of this example, then learns how each normalized coordinate should be scaled and shifted.

LayerNorm construction: (Ba et al. 2016).

Compute the population variance: the squared deviations are both one, so their mean is one. Ask for the normalized pair before the first reveal, then apply the two different affine scales and offsets before the second. Round only the final results: $\sqrt{1.25}\approx1.118$. The final values need neither zero mean nor unit variance. No other example contributes to this calculation.

LayerNorm axes for visual tensors

normalized_shape names the trailing axes that share statistics. For tokens, $N$ is token count and $D$ is embedding width.

Image tensor $(B,C,H,W)$LayerNorm((C,H,W))one $\mu,\sigma^2$ over every channel and pixel of each image

Per-channel spatial mapLayerNorm((H,W))separate $\mu,\sigma^2$ for each $(b,c)$ map

Vision-transformer tokens $(B,N,D)$LayerNorm(D)separate $\mu,\sigma^2$ across features for each token

The affine parameter shape follows normalized_shape; changing the axes changes both the statistics and the learned degrees of freedom.

LayerNorm is batch-independent, but its meaning is incomplete until the normalized axes are stated.

Single-example LayerNorm principle: (Ba et al. 2016). Axis notation: PyTorch LayerNorm.

Read each call together with its tensor layout. On (B,C,H,W), LayerNorm(C) does not normalize channels unless channels are moved to the trailing axis. LayerNorm((H,W)) has spatial affine parameters shared over leading batch/channel axes; it is not identical to InstanceNorm’s usual channelwise affine parameterization. Whole-image LayerNorm also ties its parameter shape to the chosen resolution.

Choosing a normalization method

Choice	Statistics over	Batch coupling	Train / eval	Useful starting point
BatchNorm2d(C)	$B,H,W$ for each $c$	yes	different statistics	CNN with representative batches
LayerNorm((C,H,W))	$C,H,W$ for each $b$	no	same computation	batch-independent whole-image features
LayerNorm(D) on $(B,N,D)$	$D$ for each $(b,n)$	no	same computation	token features (e.g. ViT)
no normalization	none	no	same computation	baseline to test on validation data

Choose axes that preserve distinctions the task needs, then test the choice under the actual batch size and deployment distribution.

Normalization is an architectural hypothesis about which activation scales may be treated as nuisance variation.

BatchNorm: (Ioffe and Szegedy 2015). LayerNorm: (Ba et al. 2016).

Use this as a decision recap, not a claim that the listed choices are exhaustive. Whole-image LayerNorm is one batch-independent option, not a universal replacement for BatchNorm in a CNN. Ask which statistics could change when inference uses one example at a time. Compare actual validation behavior and respect the normalization configuration expected by pretrained weights.

Part 4 · Using visual representations

1. Images and local visual evidence
2. Spatial control and feature hierarchies
3. Deep convolutional architectures
4. Using visual representations

Match invariances, pretraining, and model choice to the evidence available.

Augmentation expresses an invariance assumption

A plausible label-preserving change

CAT↔︎CAT

Horizontal reflection can preserve an object-class label.

A label-changing change

←↔︎→

Reflection reverses the meaning of a direction sign.

Training on $T(\mathbf x)$ with the original label asserts that $y(T(\mathbf x))=y(\mathbf x)$ for the task.

Use augmentation only when the transformation preserves the target and resembles variation expected after deployment.

AlexNet used random crops, reflections, and colour augmentation as task-specific regularization (Krizhevsky et al. 2012).

The words CAT stand for an object-class example; they are not an OCR task, where mirrored lettering changes the input semantics. The arrows make the direction-label counterexample literal: reflection exchanges left and right. For segmentation or localization, a valid augmentation may require transforming the target along with the image instead of keeping it unchanged.

Frozen-feature transfer

Pretrained backbonesource-task features→$\mathbf h$fixed representation→New task headrandomly initialized→Target prediction

Frozen

Backbone parameters and normalization state stay fixed; use evaluation mode.

Trainable

Only the new head follows the gradient update.

Strong starting point when target labels or compute are limited and the source representation matches the target domain.

Frozen transfer tests whether an existing representation already separates the new task.

Feature transfer becomes less reliable as source and target tasks diverge (Yosinski et al. 2014).

Connect this pipeline to BatchNorm state: disabling parameter gradients alone does not make the representation fixed. Put the frozen backbone in evaluation mode to stop running-statistic updates and disable dropout, while training the new head. Frozen features can be cached if preprocessing is also fixed. A linear head tests linear separability; a nonlinear head tests a broader form of task usability.

Fine-tuning a pretrained representation

Strategy	Updated parameters	Adaptation	Main risk
Head only	new task head	low	representation mismatch
Partial	head + later blocks	medium	unstable frozen/trainable boundary
Full	entire network	high	overfitting or destructive updates

start with head only→unfreeze later blocks→fine-tune all layers if evidence supports it

Compare strategies on the same validation split. Use smaller learning rates for pretrained parameters and inspect domain-specific errors.

Fine-tuning trades preservation of reusable features for adaptation to the target task; validation chooses the useful amount.

Transferred initialization can improve generalization even after target-task fine-tuning (Yosinski et al. 2014).

The route is a useful experimental starting point, not a mandatory sequence. Use validation to decide whether additional adaptation helps; keep the test split out of that decision. Specify the BatchNorm policy again when unfreezing blocks. A smaller learning rate for pretrained weights is a starting heuristic, and preserving source features may be harmful when the source and target domains differ substantially.

CNNs, MLPs, and Vision Transformers

Starting assumption	CNN	Image MLP	Vision Transformer
Input units	pixels or features on a grid	flattened pixels or patches	embedded image patches
Early interaction	local, shared filters	dense or token-mixing layers	self-attention can connect distant patches
Spatial prior	strong locality and translation sharing	depends on design	position must be represented explicitly
Evidence to seek	task matches local repeated patterns	simple baseline is competitive	pretraining, data, and compute justify weaker image-specific bias

These are typical starting assumptions, not permanent boundaries: hybrids and modern variants combine them.

Choose the architecture whose assumptions and resource needs match the task, then compare it against credible baselines.

ViT represents images as patch sequences and uses position embeddings; its original study found stronger dependence on pretraining scale than comparable CNNs (Dosovitskiy et al. 2021).

Explain only the contrast needed here: convolution builds local sharing into connectivity; patch attention can connect distant tokens immediately. ViT still needs spatial position information, and variants can add local or convolutional structure. The original paper’s scale findings concern its tested models and pretraining regimes; they are not a timeless ranking of model families (Dosovitskiy et al. 2021).

Evidence that a visual model works

1 · Held-out performancetask metric, calibration, and uncertainty

2 · Error structureper-class and subgroup failures

3 · Shortcut probesbackground, crop, colour, and occlusion tests

4 · Shifted conditionsnew devices, sites, seasons, or populations

Decision question: does the model keep using the right evidence when conditions change in ways that matter?

A high average score is the beginning of evaluation; deployment claims require failure analysis under realistic shifts.

Give one concrete probe, such as changing background while preserving the object. A prediction change is a clue to inspect, not conclusive proof that every background cue is invalid. Select development probes on validation data and reserve held-out evaluation for final claims. Estimate uncertainty across examples and, when feasible, across training seeds.

Lecture synthesis

Local evidencesmall receptive fields→Shared filtersreuse across positions→Compositionlarger structures through depth→Aggregationtask-level prediction→Validationtest assumptions on data

Why CNN ideas work

They reduce the search space when locality, repetition, and hierarchy describe the task.

When they fail

The imposed invariances or connectivity can discard distinctions the target needs.

Architecture encodes a hypothesis about visual structure; training fits it, and held-out evidence decides whether it was useful.

Ask students to connect one design choice to one failure mode: locality can miss distant interactions, sharing can ignore position-specific rules, and aggregation can discard needed location information. Close the loop with Week 04: the optimizer still fits parameters, but architecture determines the hypotheses it can search.

Key dates and publications

1959 · Measured visual selectivity

Hubel and Wiesel characterize receptive fields in cat visual cortex.

(Hubel and Wiesel 1959)

1980 · Neocognitron

Fukushima proposes a hierarchical network designed for shift-tolerant pattern recognition.

(Fukushima 1980)

1989 · Learned convolutional recognition

LeCun and colleagues train a constrained network for handwritten ZIP codes.

(LeCun et al. 1989)

1998 · LeNet-5 and documents

Gradient-based convolutional systems are evaluated in practical document recognition.

(LeCun et al. 1998)

2012–2016 · Scale and depth

AlexNet demonstrates GPU-scale ImageNet training; residual learning supports much deeper models.

(Krizhevsky et al. 2012; He et al. 2016)

2021 · Vision Transformer

Patch tokens and self-attention offer a scalable alternative visual architecture.

(Dosovitskiy et al. 2021)

Progress came from changing architectural assumptions together with data, optimization, and compute.

Read the timeline as selected milestones, not a claim that any one paper invented the whole field. Fukushima’s 1980 neocognitron used a different learning scheme from later backpropagation-trained CNNs. ResNet appeared as a 2015 preprint before its 2016 conference publication; ViT appeared as a 2020 preprint before ICLR 2021 (ResNet record, ViT record).

Person of the day: Yann LeCun

Yann LeCun helped make trainable convolutional networks a practical approach to visual recognition.

1989 · Learn the filters

Task-specific architectural constraints and backpropagation support handwritten ZIP-code recognition (LeCun et al. 1989).

1998 · Build a recognition system

LeNet-5 connects learned convolutional features to document-reading applications (LeCun et al. 1998).

2018 · ACM A.M. Turing Award

LeCun shares the award with Geoffrey Hinton and Yoshua Bengio for conceptual and engineering breakthroughs in deep neural networks (Association for Computing Machinery, n.d.).

Local connectivity and shared parameters become useful when they are trained end to end for the task.

Connect LeCun’s contribution to the trainable, shared filters already studied, while crediting the collaborators named in the papers. The award is the 2018 Turing Award; ACM announced the recipients on 27 March 2019, so the two dates refer to different events (ACM announcement).

The existing instructor-supplied portrait is retained; its prior provenance/reuse confirmation is unchanged.

References

Association for Computing Machinery. n.d. “Yann LeCun: 2018 ACM A.M. Turing Award Laureate.” Accessed September 30, 2026. https://amturing.acm.org/award_winners/lecun_6017366.cfm.

Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. “Layer Normalization.” arXiv Preprint arXiv:1607.06450. https://arxiv.org/abs/1607.06450.

Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, et al. 2021. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” International Conference on Learning Representations. https://arxiv.org/abs/2010.11929.

Fukushima, Kunihiko. 1980. “Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position.” Biological Cybernetics 36: 193–202. https://doi.org/10.1007/BF00344251.

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. “Deep Residual Learning for Image Recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–78. https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html.

Hubel, David H., and Torsten N. Wiesel. 1959. “Receptive Fields of Single Neurones in the Cat’s Striate Cortex.” The Journal of Physiology 148 (3): 574–91. https://doi.org/10.1113/jphysiol.1959.sp006308.

Ioffe, Sergey, and Christian Szegedy. 2015. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” Proceedings of the 32nd International Conference on Machine Learning 37: 448–56. https://proceedings.mlr.press/v37/ioffe15.html.

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems 25. https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html.

LeCun, Yann, Bernhard Boser, John S. Denker, et al. 1989. “Backpropagation Applied to Handwritten Zip Code Recognition.” Neural Computation 1 (4): 541–51. https://doi.org/10.1162/neco.1989.1.4.541.

LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86 (11): 2278–324. https://doi.org/10.1109/5.726791.

Yosinski, Jason, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. “How Transferable Are Features in Deep Neural Networks?” Advances in Neural Information Processing Systems 27. https://papers.nips.cc/paper_files/paper/2014/hash/532a2f85b6977104bc93f8580abbb330-Abstract.html.

References — continued