Discovering Structure
with Unsupervised Learning

Week-10 · ML-2026

Compress, map, group — and learn a latent model

Alexei Kornaev,
Dr. Sc., Prof.

Innopolis University

Researcher at the RC for AI,
National RC for Oncology n.a. N.N. Blokhin

Educational draft · Instructor review pending

Lecture map

1. Compress: PCA
2. Map: SNE and t-SNE
3. Group: k-means
4. Learn a reconstruction
5. Train a latent model

Print / save as PDF

Four questions, four tests

Purpose	Evidence
Compress	Reconstruction on new observations
Map	Neighbor identities preserved
Group	Metric, stability, usefulness
Generate	Sampling law and model diagnostics

Choose the question before judging the result.

Chapter 10

Use this table as the recurring evidence contract. MNIST: pixels /255; seed-42 shuffle of official training IDs; 6,000 training, 1,000 validation. Official test untouched. No fitting labels. t-SNE later fits those 1,000 map coordinates jointly.

Compress: PCA

1. Compress: PCA
2. Map: SNE and t-SNE
3. Group: k-means
4. Learn a reconstruction
5. Train a latent model

One number locates a point on a line

Filled blue circles: observations; open green circles: projections; dashed orange segments: lost residuals. Point numbers identify the same observations in both views. Move the angle with the slider or arrow keys; the data stay fixed.

Move the slider to rotate the projection line.

$$h_i=v^\top(x_i-\bar x)$$ $$\widehat x_i=\bar x+h_iv$$

Filled circles mark observations. Open circles mark projections. Dashed lines show residuals.

At 45°: retained 32, lost 8.

Chapter 10

Draw the right triangle for one observation. Unit direction is shared; score belongs to the observation.

Rotate coordinates; then discard one

The same observations are centered, rotated into principal coordinates, and projected onto one coordinate.

Follow the same numbered observations through the panels.

Changing basis is reversible; dropping coordinates loses information.

Chapter 10

Identify original, centered, rotated and reduced states. Colors track a score, not classes. Ask which operation merges distinct observations.

Which line loses least?

For $\widetilde x_i=x_i-\bar x$ and $e_i=x_i-\widehat x_i$: $$\|\widetilde x_i\|^2=h_i^2+\|e_i\|^2.$$

$$\underbrace{\sum_i\|\widetilde x_i\|^2}_{\text{fixed}}= \underbrace{\sum_i h_i^2}_{\text{kept}}+\underbrace{\sum_i\|e_i\|^2}_{\text{lost}}.$$

Maximum retained variance gives minimum squared projection error.

Chapter 10

Use board space to draw the right triangle. The result requires orthogonal projection and squared Euclidean error; centered scores have mean zero.

Covariance records joint variation

$$C=\frac1nX_c^\top X_c.$$

Each row of $X_c$ is one centered observation. Diagonal entries are feature variances. Off-diagonal entries measure how two features vary together.

Matching signs contribute positively; opposite signs negatively.

Changing metres to centimetres multiplies that feature’s variance by 10,000.

Feature scales define what counts as large variation.

Chapter 10

Use paired measurements from the same observation. Zero covariance does not prove independence. Centering and standardizing are different choices; save training statistics.

Why an eigenvector?

$$\operatorname{Var}(h)=v^\top Cv,\qquad v^\top v=1.$$

$\nabla_v$ is the vector of partial derivatives with respect to the components of $v$.

$$\nabla_v\{v^\top Cv-\lambda(v^\top v-1)\}=0\ \Rightarrow\ Cv=\lambda v.$$

$$v^\top Cv=\lambda.$$

The largest eigenvalue gives the most retained variance.

Chapter 10

Derive the multiplier step on the board. In an orthonormal eigenbasis, a unit direction gives a weighted average of eigenvalues. Hence choose the largest, not just any stationary direction.

Algorithm 1 · Fit PCA and encode

Input: training rows $X:n\times D$ and retained dimension $d$.

1. Learn $\bar x$; center $X_c=X-\mathbf1\bar x^\top$.
2. Form $C=X_c^\top X_c/n$; find its orthonormal eigenvectors.
3. Sort eigenvalues; retain $V=[v_1,\ldots,v_d]$.
4. Encode $H=X_cV$; save mean, basis and preprocessing.

New observation: $h=V^\top(x-\bar x)$ with saved statistics.

Chapter 10

SVD is an alternative numerical solver. Feature scaling is chosen before fitting. Do not fit a new mean for a new observation.

Algorithm 2 · Reconstruct

Input: scores $H$ and the same saved mean and basis.

1. Expand $\widehat X_c=HV^\top$.
2. Restore the center: $\widehat X=\widehat X_c+\mathbf1\bar x^\top$.
3. Reverse any earlier scaling using saved statistics.
4. Compare in the same units: $\mathrm{MSE}=\|X-\widehat X\|_F^2/(nD)$.

Restoring the feature count does not restore discarded information.

Chapter 10

Reconstruction does not fit another model. Full basis is reversible; a reduced code generally loses information.

Worked PCA · Observe and center

1 · Observations $$X=\begin{pmatrix}5&3\\3&5\\1&-1\\-1&1\end{pmatrix}.$$ $n=4,\ D=2,\ d=1$.

2 · Center $$\bar x=\binom22,\quad X_c=\begin{pmatrix}3&1\\1&3\\-1&-3\\-3&-1\end{pmatrix}.$$ Track point 1: $(5,3)\to(3,1)$.

Chapter 10

Reuse all six panels of textbook Figure 4 across this and the next two frames. Ask students to compute the mean before revealing the centered matrix.

Worked PCA · Covariance and directions

3 · Covariance $$C=\tfrac14X_c^\top X_c=\begin{pmatrix}5&3\\3&5\end{pmatrix}.$$

4 · Principal directions $$v_1=\tfrac1{\sqrt2}\binom11,\quad\lambda_1=8.$$ $$v_2=\tfrac1{\sqrt2}\binom1{-1},\quad\lambda_2=2.$$

Chapter 10

Compute C11 from the four centered first coordinates. Check C v1 = 8 v1. Retain the direction with largest eigenvalue.

Worked PCA · Encode and reconstruct

5 · Retain and encode $$V=v_1,\quad H=X_cV=\begin{pmatrix}2\sqrt2\\2\sqrt2\\-2\sqrt2\\-2\sqrt2\end{pmatrix}.$$ Point 1: $h_1=(3+1)/\sqrt2$.

6 · Expand and restore $$\widehat X=HV^\top+\mathbf1\bar x^\top=\begin{pmatrix}4&4\\4&4\\0&0\\0&0\end{pmatrix}.$$ Point 1: $(5,3)\to(4,4)$.

$40=32+8$: retain 80% of variation; per-feature MSE $=1$.

Chapter 10

Ask why the first two originals reconstruct identically. Error is 8/(4 times 2); squared variation is not semantic information.

A direction is a signed pixel pattern

MNIST mean image and signed principal-direction pixel weights.

$D=784$ measurements; a score weights one shared pattern.

A component need not be a named visual concept.

Chapter 10

The mean is an image; loadings can be negative. Follow a positive and negative score. Do not interpret a loading as an ordinary grayscale digit.

More coordinates recover more detail

The same five MNIST images reconstructed with increasing numbers of principal coordinates.

Same held-out images; one nested basis.

More retained directions remove squared residual components.

Chapter 10

Read selected rows at d=2,16,64,128,784. Full basis gives exact recovery but no coordinate compression. Validation excluded from fitting. At d=64 training variance 86.4%, validation MSE .00959; these are not recognition scores.

Large variation can hide the useful clue

Spirals and two separated strips illustrate distinctions lost by one-dimensional PCA.

Predict the retained direction for the two horizontal strips.

High variance and useful class information can point in different directions.

Chapter 10

Focus on lower panels: the strips are linearly separable vertically yet share PC1 scores. Top spirals show another limitation. Neither example says PCA always fails on nonlinear data.

Map: SNE and t-SNE

1. Compress: PCA
2. Map: SNE and t-SNE
3. Group: k-means
4. Learn a reconstruction
5. Train a latent model

SNE · Give each point a neighborhood

Stochastic neighbor embedding (SNE) places each observation at a point on a map.

For input point $x_i$, turn distances to the other points into weights. Normalize across candidates $j\ne i$: $$p_{j\mid i}= \frac{\exp(-\|x_i-x_j\|^2/(2\tau_i^2))} {\sum_{\ell\ne i}\exp(-\|x_i-x_\ell\|^2/(2\tau_i^2))}.$$

$p_{j\mid i}$ is the probability of choosing $j$ as a neighbor of $i$.

SNE · Choose the neighborhood scale

Gaussian neighbor weights and normalized probabilities for four candidates.

Small $\tau_i$ concentrates probability on close neighbors.

A larger $\tau_i$ spreads probability over more points.

Perplexity is the effective number of neighbors. Choose a target and adjust $\tau_i$ separately for each point.

For uniform probabilities on k neighbors, perplexity equals k. More generally it is exp of the entropy of the conditional distribution, using natural logs. A candidate’s Gaussian weight must be divided by the sum of all candidate weights.

SNE · Move the map to match neighborhoods

Give each observation a map position $y_i$. Use Gaussian weights with a fixed scale on the map: $$q_{j\mid i}=\frac{\exp(-\|y_i-y_j\|^2)} {\sum_{\ell\ne i}\exp(-\|y_i-y_\ell\|^2)}.$$

Move all $y_i$ to minimize the sum of neighborhood mismatches: $$L=\sum_i\sum_{j\ne i}p_{j\mid i}\log\frac{p_{j\mid i}}{q_{j\mid i}}.$$

If $p_{j\mid i}$ is large, a tiny $q_{j\mid i}$ is costly.

t-SNE modifies SNE

The same MNIST images mapped by t-SNE at several neighborhood scales and initializations.

Symmetric pairs: combine both input neighbor directions.

Student kernel: use $(1+\|y_i-y_j\|^2)^{-1}$ on the map. Its slower decay helps ease crowding.

Read the map through local neighbors. Island gaps depend on the fit.

Hinton and Roweis, SNE · van der Maaten and Hinton, t-SNE · Chapter 10

The example fits coordinates for all 1,000 displayed MNIST images jointly. Digit colors are added afterward. A different perplexity or initialization can rearrange islands. Compare neighbor identities across fits.

Group: k-means

1. Compress: PCA
2. Map: SNE and t-SNE
3. Group: k-means
4. Learn a reconstruction
5. Train a latent model

K-means · Assign, update, repeat

1. Initialize $K$ centers $\mu_1,\ldots,\mu_K$ at random.
2. Assign each point $x_i$ to its nearest center: $c_i=\arg\min_{1\le k\le K}\|x_i-\mu_k\|^2$.
3. Update each center to the arithmetic mean of its assigned points.
4. Repeat steps 2–3 until assignments stabilize.

$$L=\frac1m\sum_{i=1}^m\|x_i-\mu_{c_i}\|^2.$$

There are $m$ points. Each squared distance sums over the $D$ feature coordinates.

Each step can only decrease or preserve L. If a cluster is empty, retain its old center for this demonstration. The final partition can depend on the starting centers.

Watch assignment and update separately

Assign: points change groups while centers stay fixed.

Update: centers move to group means while memberships stay fixed.

Chapter 10

Start Assign 0, ask where centers move, then press Next: Update 1. Ask which points change membership before the following Next. Use native buttons or slider; no autoplay. Print shows the six static endpoints.

What shape can a center represent?

Identical curved observations colored by generating group and by nearest-center clustering.

Nearest-center regions favor compact groups.

Compactness, stability and usefulness need separate checks.

Chapter 10

Voronoi cells have linear pair boundaries. Curved generating groups need not match this objective. Global minimum inertia falls as K increases, so minimum alone cannot choose K; finite restarts can behave differently.

When k-means works well

Two compact synthetic groups shown before and after k-means. Circles and triangles mark fitted assignments and crosses mark fitted centers.

With $K=2$, the fit recovers both generating groups in this sample.

The groups are compact, well separated, and have equal spread in comparable feature units. Fit on coordinates alone. Compare assignments to the generating groups afterward, ignoring cluster ID order. Here the adjusted Rand index is 1.

When brightness is the wrong feature

Original chest X-ray and three color-coded intensity maps for K=2, 3 and 4. Dark blue denotes the lowest fitted intensity group and pale yellow the highest.

The dataset labels this image Pneumonia.

K = 2, 3, 4: each fit divides brightness into more bands. Panel titles give fitted centers from darkest to brightest.

Dark blue marks the lowest band. Pale yellow marks the highest. Bright hardware and tissue can share a band.

Data provider: NIH Clinical Center · NIH ChestX-ray14 download · Wang et al., CVPR 2017

Ask whether the bright group could be read as a disease mask. This partition comes entirely from pixel brightness and ignores location. The image-level label comes from dataset metadata and supplies no pixel-level reference. The demonstration illustrates a mismatch between an intensity objective and disease localization.

Learn a reconstruction

1. Compress: PCA
2. Map: SNE and t-SNE
3. Group: k-means
4. Learn a reconstruction
5. Train a latent model

Training an autoencoder

Schematic encoder, numerical code, decoder, reconstruction loss and weight updates.

The encoder maps an image to code $h$. The decoder reconstructs the input. Reconstruction error updates both networks.

Chapter 10

Follow the input, code, reconstruction, original-image target, and update arrows. Code values and digit outputs are illustrative. Shared encoder/decoder parameters learn; codes depend on images.

Algorithm · Train, check, use

Stage	Operation
Specify	Encoder, decoder, bottleneck and loss
Forward	$H=f_\psi(X)$; $\widehat X=g_\theta(H)$
Score	$L=\frac1{BD}\sum_{i,j}(x_{ij}-\widehat x_{ij})^2$
Update	Clear gradients; backpropagate; update both networks
Check	Measure held-out reconstruction with fixed weights
Use	Encode/reconstruct a new image with fixed weights

Chapter 10

Separate training updates from use. Pixel-mean MSE averages B images and D pixels. A linear AE at a suitable global optimum recovers the PCA subspace, but not necessarily ordered orthonormal scores.

Why require a bottleneck?

$$X:(B,D)\ \longrightarrow\ H:(B,d)\ \longrightarrow\ \widehat X:(B,D),\qquad d<D.$$

A constraint makes copying harder and forces information selection.

A small code still does not guarantee meaningful features.

Chapter 10

Flexible networks may memorize finite examples or preserve nuisance variation. New images use fixed network weights; they do not require refitting their code.

What did this AE preserve?

Original MNIST images and deterministic autoencoder reconstructions at two bottleneck dimensions.

Same held-out images; bottlenecks of 2 and 16.

Measure recovery separately from recognition.

Chapter 10

MNIST 784→128→d→128→784, ReLU hidden, sigmoid output. 6,000 training/1,000 validation; seed0,20 epochs, Adam .001,batch128. A small CPU illustration, not a tuned comparison. PCA has a different output family.

Does reconstruction help recognition?

Freeze encoder → encode → train labeled probe → validate choices → test once.

Ask where labels enter this experiment.

Downstream usefulness needs a declared task.

Chapter 10

Train probe preprocessing on training codes. Use labels only at the disclosed supervised stage. No new MNIST recognition result is claimed. Earlier tutorial uses 8×8 optical digits. Next ask: where should a new code come from?

Train a latent model

1. Compress: PCA
2. Map: SNE and t-SNE
3. Group: k-means
4. Learn a reconstruction
5. Train a latent model

VAE · Learn inference and reconstruction together

The encoder gives each image a distribution of possible codes.

A sampled code enters the decoder; the same input supplies the reconstruction target.

No digit label or correct code is provided.

Textbook · VAE

Follow the chapter order: three distributions; training overview; two jobs; network outputs; derive loss; calculate terms; sample differentiably; train; inspect evidence.

Three distributions, three roles

Distribution	Question
$p(\mathbf z)$	Codes before observing this image?
$p_\theta(\mathbf z\mid\mathbf x)$	Which codes explain it under the model?
$q_\psi(\mathbf z\mid\mathbf x)$	What can the encoder approximate?

These describe image codes, rather than network-weight uncertainty.

Chapter 10

Fix one image throughout. Before/after refers to observing it, not training epochs. Exact posterior depends on the current decoder and is generally intractable.

From a code to a code distribution

Schematic VAE training with per-image code distributions, prior, sampling and ELBO.

1. Encode: the image gives a mean and spread for its latent code.
2. Sample and decode: draw a code and predict the image’s pixel means.
3. Train: explain the original image and penalize departure from the fixed prior.

The diagram uses $\phi$ for the encoder weights we call $\psi$.

Chapter 10

Follow one input to its cloud, one sampled code and reconstruction. Yellow is the fixed prior. Tilted ellipses are conceptual; our encoder covariance is diagonal. Match-prior pressure does not force all posteriors to coincide.

The loss has two jobs

Reconstruction cost: retain information that helps explain this image.

Prior KL cost: compare this image’s approximate posterior with the shared prior.

We will derive why these terms belong together.

Textbook · VAE

Preview only; do not introduce a second derivation. The prior is fixed; image-specific distributions depend on shared encoder weights.

Encoder · Two vectors for every image

$$\boldsymbol\mu(\mathbf x),\ \log\boldsymbol\sigma(\mathbf x)\in\mathbb R^d.$$

$$\boldsymbol\sigma=\exp(\log\boldsymbol\sigma),\qquad\boldsymbol\sigma^2=\exp(2\log\boldsymbol\sigma).$$

$$q_\psi(\mathbf z\mid\mathbf x)=\mathcal N(\mathbf z;\boldsymbol\mu,\operatorname{diag}(\boldsymbol\sigma^2)).$$

Each batch output has shape $(B,d)$: $2d$ scalars per image.

Textbook · VAE

psi is shared across images; predicted means and spreads depend on the image. These describe code uncertainty, not network weights.

Decoder · Means, then an observation model

$$g_\theta(\mathbf z)\in\mathbb R^D\quad\text{is a vector of pixel means.}$$

$$p_\theta(\mathbf x\mid\mathbf z)=\mathcal N(\mathbf x;g_\theta(\mathbf z),s_x^2\mathbf I).$$

Evaluating the distribution at an image gives one scalar density.

Latent $\boldsymbol\sigma$ and pixel spread $s_x$ have different roles.

Textbook · VAE

Decoder input is a code. Its neural output is not a single scalar likelihood. Fixed sx shared across pixels; a chosen continuous observation model.

What should training improve?

$$p_\theta(\mathbf x)=\int p_\theta(\mathbf x\mid\mathbf z)p(\mathbf z)\,d\mathbf z.$$

We would like to increase $\log p_\theta(\mathbf x)$.

The nonlinear decoder makes this integral difficult.

The encoder proposes image-specific codes for a computable bound.

Chapter 10

No correct latent-code labels or posterior targets are supplied. Preview the two jobs: explain input and control departures from prior. Derive the bound once in the next slides.

Expose the bound with Bayes’ rule

Fix $\mathbf x$; expectations are over $q_\psi(\mathbf z\mid\mathbf x)$. $$\begin{aligned} D_{\mathrm{KL}}(q_\psi(\mathbf z\mid\mathbf x)\|p_\theta(\mathbf z\mid\mathbf x))\\ =\mathbb E_{q_\psi(\mathbf z\mid\mathbf x)}[\log q_\psi(\mathbf z\mid\mathbf x)-\log p_\theta(\mathbf z\mid\mathbf x)]. \end{aligned}$$

$$\log p_\theta(\mathbf z\mid\mathbf x)=\log p_\theta(\mathbf x\mid\mathbf z)+\log p(\mathbf z)-\log p_\theta(\mathbf x).$$

The posterior comparison is a derivation tool.

Chapter 10

Leave board room. Substitute the second equality into the first and group log q minus log prior. Keep the full encoder distribution notation in the KL, expectation and log-density term.

Substitute Bayes and group the terms

For fixed $\mathbf x$, abbreviate $q=q_\psi(\mathbf z\mid\mathbf x)$.

$$\begin{aligned} D_{\mathrm{KL}}(q\|p_\theta(\mathbf z\mid\mathbf x)) &=\mathbb E_q[\log q-\log p(\mathbf z)]\\ &\quad-\mathbb E_q[\log p_\theta(\mathbf x\mid\mathbf z)]\\ &\quad+\log p_\theta(\mathbf x). \end{aligned}$$

The first expectation is the prior KL.

Textbook · VAE

This missing algebraic step now remains visible. The KL definition is exact; no single-code density comparison replaces its expectation. Log evidence is constant with respect to sampled codes.

Evidence equals bound plus inference gap

$$\mathcal E(\mathbf x)=\mathbb E_q[\log p_\theta(\mathbf x\mid\mathbf z)]-D_{\mathrm{KL}}(q\|p(\mathbf z)).$$

$$\log p_\theta(\mathbf x)=\mathcal E(\mathbf x)+\underbrace{D_{\mathrm{KL}}(q\|p_\theta(\mathbf z\mid\mathbf x))}_{\text{nonnegative gap}}.$$

Thus $\mathcal E(\mathbf x)\le\log p_\theta(\mathbf x)$.

The prior KL belongs in the bound; posterior KL measures its gap.

Chapter 10

Name ELBO: evidence lower bound. Exact encoder posterior makes the bound tight for fixed model; it does not require zero prior KL.

Change the sign to get a loss

$$\mathcal L(\mathbf x)=-\mathcal E(\mathbf x) =\underbrace{\mathbb E_q[-\log p_\theta(\mathbf x\mid\mathbf z)]}_{\text{reconstruction cost}}+\underbrace{D_{\mathrm{KL}}(q\|p(\mathbf z))}_{\text{prior cost}}.$$

Both networks learn through one objective.

Chapter 10

Only these two terms are evaluated or estimated; the posterior gap is not a third penalty. Natural logs give nats per image.

Reconstruction · Multiply pixel densities

For one fixed image and sampled code:

$$p_\theta(\mathbf x\mid\mathbf z)=(2\pi s_x^2)^{-D/2} \exp\!\left[-\frac{\|\mathbf x-g_\theta(\mathbf z)\|^2}{2s_x^2}\right].$$

Conditional pixel independence gives a product over pixels.

Negative logarithms turn that product into a sum.

Textbook · VAE

Evaluate each pixel density at its observed value. This is conditional independence given z, not unconditional independence of pixels.

Why squared reconstruction error?

For the fixed Gaussian observation model: $$\operatorname{NLL}(\mathbf x,\mathbf z)=\frac{\|\mathbf x-g_\theta(\mathbf z)\|^2}{2s_x^2}+\frac D2\log(2\pi s_x^2).$$

Pixel-density product → log-density sum.

For each image, estimate the average over encoder codes with one draw during training.

The observation model determines the reconstruction term.

Chapter 10

Week07 Bernoulli pixels give BCE; continuous pixels here use Gaussian NLL. Pixel sum differs from latent expectation. Fixed constant can be omitted for gradients but retained when reporting NLL.

Four pixels · Encode, sample, decode

Schematic image: $\mathbf x=(.7,.4,.1,.9)$; $D=4$, $d=2$.

$\boldsymbol\mu=(2,-1)$, $\boldsymbol\sigma=(.5,.2)$, $\boldsymbol\epsilon=(1,-.5)$.

$$\mathbf z=\boldsymbol\mu+\boldsymbol\sigma\odot\boldsymbol\epsilon=(2.5,-1.1).$$

Suppose $g_\theta(\mathbf z)=(.8,.2,.2,.8)$; $s_x=.25$.

Textbook · VAE

These are illustrative outputs, not measurements. Introduce shift-and-scale sampling here as the operation used; explain its gradient mechanism later in the chapter sequence.

Four pixels · Score the original image

Pixel	Observed	Mean	Scaled squared residual
1	.7	.8	.08
2	.4	.2	.32
3	.1	.2	.08
4	.9	.8	.08

$\operatorname{NLL}=.56+C$, where $C=2\log(2\pi s_x^2)$.

Textbook · VAE

Each row divides the squared residual by 2 times .25². Product of four densities gives sum of their negative logs. One whole-image cost for one code; changing code changes the cost.

Prior KL · Use the Gaussian formula

The encoder gives a diagonal Gaussian with means $\mu_k$ and standard deviations $\sigma_k$. The prior is standard normal.

$$D_{\mathrm{KL}}\!\left(q_\psi(\mathbf z\mid\mathbf x)\,\|\,p(\mathbf z)\right) =\frac12\sum_{k=1}^d \left[\mu_k^2+\sigma_k^2-1-2\log\sigma_k\right].$$

Compute this directly from the encoder outputs. It penalizes means away from zero and variances away from one.

Why neither term works alone

Reconstruction alone: no explicit prior cost for distant, extremely narrow image-specific code distributions.

KL alone: every image could receive the same standard normal and pay zero.

Together, the terms balance image information and prior cost.

Textbook · VAE

An image of a 3 must retain its strokes. Low KL alone is not evidence of good representation; reconstruction alone is not a sampling model. Do not claim every posterior should equal the prior.

Shift and stretch fixed noise

Fixed standard-normal points shifted by a mean and stretched by coordinate standard deviations.

$\mathbf z=\boldsymbol\mu+\boldsymbol\sigma\odot\boldsymbol\epsilon$, $\boldsymbol\sigma=\exp(\log\boldsymbol\sigma)$.

Chapter 10

Hold the noise fixed while differentiating one pass; fresh noise later. Both encoder heads get reconstruction gradients. Halve log variance before exponentiation if translating that parameterization.

Algorithm · One VAE training step

1. Encode: predict $\boldsymbol\mu$ and $\log\boldsymbol\sigma$.
2. Sample: $\mathbf z=\boldsymbol\mu+\exp(\log\boldsymbol\sigma)\odot\boldsymbol\epsilon$.
3. Reconstruct: decode $\mathbf z$; score the original image.
4. Score: add NLL and prior KL; average over the batch.
5. Update: backpropagate; update both networks; repeat.

Sum pixel and latent contributions within each image.

Chapter 10

Return mentally to Figure 22: encoder heads, noise path, decoder, two objective terms. Prior fixed. One fresh noise draw per image. Weights shared; no separately stored training code optimized.

Batch loss · Average the per-image costs

For image $i$, draw $\mathbf z_i$ from $q_\psi(\cdot\mid\mathbf x_i)$.

$$R_i=-\log p_\theta(\mathbf x_i\mid\mathbf z_i),\qquad K_i=D_{\mathrm{KL}}\!\left(q_\psi(\cdot\mid\mathbf x_i)\,\|\,p\right).$$ $$\widehat{\mathcal L}_{\mathrm{batch}}=\frac1B\sum_{i=1}^B(R_i+K_i).$$

Sum over pixels in $R_i$ and over latent coordinates in $K_i$. Then average over images.

Generation and interpolation

How would you choose a method tomorrow?

PCA: saved linear coordinates. t-SNE: inspect neighbors.

K-means: compact groups.

AE: nonlinear reconstruction. VAE: latent probabilistic learning.

Next: construct training relationships for useful representations.

Chapter 10

Week11 self-supervised learning revisits where targets come from and downstream evaluation. These methods are alternatives for purposes, not a compulsory pipeline.

Questions that changed the picture

2006: learned low-dimensional reconstruction — Hinton & Salakhutdinov.

2008: local-affinity maps — van der Maaten & Hinton.

2013/2014: joint latent generation and inference — Kingma & Welling.

History connects methods to the problems they addressed.

AE · t-SNE · AEVB

2006 paper is not the invention of all autoencoders. AEVB: 2013 preprint, ICLR2014. Credit collaborators; no citation-count ranking.

People of the day: Kingma and Hinton

Diederik P. (Durk) Kingma

Variational autoencoders with Max Welling.

Geoffrey Hinton

Autoencoders with Ruslan Salakhutdinov; t-SNE with van der Maaten.

Portraits reused from Weeks 07/04. AEVB · Autoencoders (2006) · t-SNE (2008).

References: geometry, grouping and maps

• Bishop, Pattern Recognition and Machine Learning, Chapters 9, 12.
• Hinton & Roweis, Stochastic Neighbor Embedding.
• van der Maaten & Hinton (2008), Visualizing Data using t-SNE.
• Chapter 10: derivations, experimental protocols and sources.

References: learned latent representations

• Hinton & Salakhutdinov (2006), Reducing the Dimensionality of Data with Neural Networks.
• Kingma & Welling, Auto-Encoding Variational Bayes.
• Goodfellow, Bengio & Courville, Deep Learning, Chapters 13–14.
• MNIST source · Chapter 07 foundations.
• Earlier tutorial: 8×8 optical digits; distinct from this chapter’s MNIST experiment.

VAE supplement · Generation and interpolation

Return to the VAE overview

After training · Use the fixed decoder

Schematic prior sampling and a chosen interpolation path.

Draw a code from the prior. The trained decoder gives observation parameters. A chosen path between codes illustrates interpolation.

Textbook · VAE

No input image, encoder or weight update for prior generation. Schematic clean digits are not measured outputs. Interpolation is a selected path, not random prior sampling.

Three operations with fixed weights

Operation	Algorithm	Label
Reconstruct	Encode an image; use mean or draw; decode	Mean-code or posterior-draw reconstruction
Generate	Draw prior code; decode; optionally draw observation	Decoder mean or observation draw
Interpolate	Choose two codes and $(1-\alpha)\mathbf z^{(a)}+\alpha\mathbf z^{(b)}$	Chosen path, usually decoded means

Textbook · VAE

Generation needs no input or encoder; reconstruction does. Label whether observations are actually drawn.

Prior codes: decoded means and observation draws

Fresh prior code decoder means above Gaussian observation draws using those same codes.

Fixed trained MNIST model · fresh prior codes · $s_x=.25$.

Chapter 10

No input images or encoder. Means are not actual observation draws. Gaussian draws can leave [0,1]; grayscale saturation is display-only. Eight prespecified codes, not curated best samples.

A grid explores the decoder

Seven-by-seven grid of decoded mean images at chosen latent coordinates.

A uniform coordinate grid is not probability-weighted prior sampling.

Textbook · VAE

Coordinates range -2 to 2. This probes response at stated locations, including poor regions; it does not prove distributional fidelity.

A chosen path between two inferred codes

Eight decoded means along a line between the first two held-out mean codes.

Endpoints are reconstructions; intermediate outputs are not random samples.

Textbook · VAE

Smooth neural outputs do not guarantee plausibility or named semantic factors. Ordinary AE may also interpolate well. VAE provides a specified sampling model and variational objective.