Four questions about model behavior

Start from the definitions supplied in each task.

1. Attention: who bites whom?
2. Ensembles: why can more models stop helping?
3. Evidence: how can we construct a lower bound?
4. Latent codes: what does good reconstruction reveal?

Give time to identify what is fixed and what varies before each solution. Each variant has three subparts. The review derives every answer from the supplied assumptions.

Variant 1, Task 30.

Can attention tell who bites whom? · Exact examination statement · 1/2

Consider one self-attention layer on three token embeddings $x_1,x_2,x_3$. There are no positional features, masks or dropout. Queries and keys have dimension $d_k$. The same matrices are used at every position: $$q_i=W_Qx_i,\quad k_i=W_Kx_i,\quad v_i=W_Vx_i,\qquad a_{ij}=\frac{\exp(q_i^\top k_j/\sqrt{d_k})} {\sum_{r=1}^3\exp(q_i^\top k_r/\sqrt{d_k})},\quad h_i=\sum_{j=1}^3a_{ij}v_j.$$ Compare A: “dog bites man” with B: “man bites dog”. Each word has a fixed embedding, so B exchanges $x_1,x_3$ and keeps $x_2$. A classifier receives only $h_2$, the contextual representation of “bites”.

Variant 1, Task 30. Questions

Can attention tell who bites whom? · Exact examination statement · 2/2

(a) Express B’s three weights for query 2 in terms of A’s weights. Use this to prove that the classifier receives the same vector in A and B.

(b) Is the entire ordered output $(h_1,h_2,h_3)$ also necessarily unchanged? Give its exact relation to A’s output and justify it.

(c) Now replace each input by $x_i+p_i$, where $p_i$ is a fixed vector attached to position $i$. Explain precisely why the previous permutation argument need no longer hold when $p_1\ne p_3$. Does this guarantee that every choice of $W_Q,W_K,W_V$ distinguishes the sentences? Justify.

Follow the words through the swap

Variant 1 · A word keeps its embedding when its slot changes

dog · $e_1$bites · $e_2$man · $e_3$

man · $e_3$bites · $e_2$dog · $e_1$

The middle query stays fixed; keys and values move together.

The gear is only a placeholder for the word bites, not a computation. These arrows show the permutation of inputs, not the attention matrix. The fixed word vectors can be one-hot in this example; no positional vectors are supplied.

Can attention tell who bites whom?

Variant 1 · Fixed word embeddings · Shared weights · No positions, masks or dropout

Sentence	Position 1	Position 2	Position 3
A	dog	bites	man
B	man	bites	dog

$$q_i=W_Qx_i,\quad k_i=W_Kx_i,\quad v_i=W_Vx_i,\qquad a_{ij}=\frac{\exp(q_i^\top k_j/\sqrt{d_k})}{\sum_{r=1}^3\exp(q_i^\top k_r/\sqrt{d_k})},\quad h_i=\sum_j a_{ij}v_j.$$

Vocabulary [dog, bites, man]: $x_{\mathrm{dog}}=e_1$, $x_{\mathrm{bites}}=e_2$, $x_{\mathrm{man}}=e_3$. Word identity stays fixed when its slot changes. The classifier reads only $h_2$.

The middle word stays in place: $q'_2=q_2$.

Keys and values exchange slots 1 and 3 together.

Primes label sentence B, not derivatives. Token embeddings are columns in R^d. W_Q and W_K have shape d_k by d, W_V has shape d_v by d. The attention dot product sums over query/key coordinates; softmax sums over token positions. Pause before each reveal.

The middle representation

Variant 1(a) · A has weights $(a_{21},a_{22},a_{23})$

$$h_2=a_{21}v_1+a_{22}v_2+a_{23}v_3.$$

Illustration with one-hot values: $(\tfrac12,e_1),(\tfrac13,e_2),(\tfrac16,e_3)$ move as pairs. Their sum stays $(\tfrac12,\tfrac13,\tfrac16)$.

Swapping the keys reorders the denominator’s three terms. Its sum stays fixed.

$$(a'_{21},a'_{22},a'_{23})=(a_{23},a_{22},a_{21}).$$

$$h'_2=\underbrace{a_{23}v_3}_{\text{new slot 1}}+ \underbrace{a_{22}v_2}_{\text{slot 2}}+ \underbrace{a_{21}v_1}_{\text{new slot 3}}=h_2.$$

A deterministic classifier of $h_2$ receives the same vector.

Ask students to pair each moved weight with its moved value. Swapping weights alone would not prove the result.

Outputs follow the words

Variant 1(b) · Rows are query slots; columns are key/value slots

Stack token transposes as rows: $X\in\mathbb R^{3\times d}$, $Q=XW_Q^\top$, $K=XW_K^\top$, $V=XW_V^\top$.

$$P=\begin{pmatrix}0&0&1\\0&1&0\\1&0&0\end{pmatrix},\quad X'=PX,\quad A_{\mathrm{att}}=\operatorname{softmax}_{\mathrm{rows}}(QK^\top/\sqrt{d_k}).$$

Both query rows and key columns move. Row denominators keep the same terms: $$A'_{\mathrm{att}}=P A_{\mathrm{att}}P^\top,\qquad V'=PV.$$

$$H'=P A_{\mathrm{att}}\underbrace{P^\top P}_{I}V=PH, \qquad (h'_1,h'_2,h'_3)=(h_3,h_2,h_1).$$

The sequence is equivariant; the selected middle row is unchanged.

P is a permutation of slots. The attention matrix mixes values, with rows summing to one; it is not a next-word transition model. S’=PSP^T because queries and keys move together. Softmax commutes with this relabeling: a column permutation preserves each denominator, while row permutation moves whole normalized rows. H has shape 3 by d_v.

Position supplies additional information

Variant 1(c) · Replace each word input by $x_i+p_i$, with $p_1\ne p_3$

	A’s endpoints	B’s endpoints
Augmented inputs	$(x_1+p_1,\ x_3+p_3)$	$(x_3+p_1,\ x_1+p_3)$

Position vectors stay attached to slots when words move. $$X'_{\mathrm{aug}}=PX+E_{\mathrm{pos}},\qquad PX_{\mathrm{aug}}=PX+PE_{\mathrm{pos}}.$$

Their difference is $E_{\mathrm{pos}}-PE_{\mathrm{pos}}\ne0$ when $p_1\ne p_3$. The swap proof no longer guarantees the output relation.

A counterexample to a guarantee: $W_V=0$ makes every output zero.

Position can enable a distinction; learned weights must use it.

Variant 2, Task 30.

Why more models can stop helping · Exact examination statement · 1/2

At a fixed input $x$, repeated random training runs produce predictions $F_1,\ldots,F_M$, where $M\ge2$. For a fixed target value $y$, assume $$\mathbb E[F_m]=y+b,\qquad \operatorname{Var}(F_m)=\sigma^2>0,\qquad \operatorname{Cov}(F_m,F_r)=\rho\sigma^2\quad(m\ne r),\quad 0\le\rho\le1.$$ The ensemble prediction is $\overline F=M^{-1}\sum_m F_m$. All expectations concern randomness in training, not different inputs. You may use $\operatorname{Var}(Z)=\mathbb E[(Z-\mathbb E Z)^2]$ and $\operatorname{Cov}(Z,T)=\mathbb E[(Z-\mathbb E Z)(T-\mathbb E T)]$.

Variant 2, Task 30. Questions

Why more models can stop helping · Exact examination statement · 2/2

(a) Expand the square in the variance definition to derive $\operatorname{Var}(\overline F)$. Show how many diagonal and off-diagonal terms occur; do not quote a memorized ensemble formula.

(b) By adding and subtracting $\mathbb E[\overline F]$, derive $\mathbb E[(\overline F-y)^2]$. Find its limit as $M\to\infty$ and explain which two obstacles cannot be removed merely by increasing $M$.

(c) Suppose $\rho=1/4$. Can any finite $M$ make the ensemble variance at most $\sigma^2/4$? Explain. What does the formula say when $\rho=1$?

Averaging random predictions

Variant 2 · Fixed input $x$ and target $y$ · Randomness comes from training

$$\mathbb E[F_m]=y+b,\qquad \operatorname{Var}(F_m)=\sigma^2>0,$$ $$\operatorname{Cov}(F_m,F_r)=\rho\sigma^2\quad(m\ne r),\qquad 0\le\rho\le1.$$ $$\overline F=\frac1M\sum_{m=1}^M F_m,\qquad M\ge2.$$

Fixed target $y$ → common center $y+b$ → predictions $F_1,\ldots,F_M$. Averaging changes their spread, while the common center stays fixed.

Center each prediction: $G_m=F_m-(y+b)$.

$$\mathbb E[G_m]=0,\qquad \mathbb E[\overline F]=y+b.$$

Here $b$ is common prediction bias and $\sigma^2$ is prediction variance across training runs.

Counting covariance terms

Variant 2(a) · Expand the square before simplifying

$$\operatorname{Var}(\overline F)=\frac1{M^2}\mathbb E\left[\left(\sum_m G_m\right)^2\right].$$

$$=\frac{\overbrace{M\sigma^2}^{\text{diagonal}}+ \overbrace{M(M-1)\rho\sigma^2}^{\text{off-diagonal}}}{M^2}.$$

$$=\sigma^2\left(\rho+\frac{1-\rho}{M}\right).$$

Example: $M=4$

σ²	ρσ²	ρσ²	ρσ²
ρσ²	σ²	ρσ²	ρσ²
ρσ²	ρσ²	σ²	ρσ²
ρσ²	ρσ²	ρσ²	σ²

4 diagonal + 12 ordered off-diagonal terms

Correlated fluctuations survive averaging.

Off-diagonal terms are ordered: G_m G_r and G_r G_m both appear. Alternatively count unordered pairs twice. The grid is only a four-model illustration.

Bias and variance in prediction error

Variant 2(b) · What remains after averaging?

Error budget: common bias² + shared fluctuation + averaged fluctuation.

$$\overline F-y=(\overline F-\mathbb E[\overline F])+b.$$

$$\mathbb E[(\overline F-y)^2] =\operatorname{Var}(\overline F)+2b\underbrace{\mathbb E[\overline F-\mathbb E\overline F]}_{0}+b^2.$$

$$=b^2+\sigma^2\left(\rho+\frac{1-\rho}{M}\right).$$

$$\lim_{M\to\infty}\mathbb E[(\overline F-y)^2] =\underbrace{b^2}_{\text{common bias}}+\underbrace{\rho\sigma^2}_{\text{correlation floor}}.$$

The target is fixed. No extra observation-noise variance belongs in this task. Increasing M removes neither the shared bias nor the correlated part of prediction variance.

Can a finite ensemble reach the floor?

Variant 2(c) · Set $\rho=1/4$

$$\frac{\operatorname{Var}(\overline F)}{\sigma^2} =\frac14+\frac{3}{4M}>\frac14\quad\text{for every finite }M.$$

At $\rho=1$, the variance stays $\sigma^2$ for every $M$.

The M=1 endpoint in the figure illustrates the formula outside the question’s M>=2 setup. Ask whether looking very close to the floor establishes equality. Emphasize the strict inequality.

Variant 3, Task 30.

Build a lower bound from a weighted average · Exact examination statement · 1/2

Fix an observation $x$. A latent variable $u$ takes values $1,\ldots,K$. The model specifies strictly positive numbers $p_\theta(x,u)$, and $p_\theta(x)=\sum_u p_\theta(x,u)$. Choose any probabilities $q_u>0$ with $\sum_u q_u=1$. Define $$\mathcal B(q)=\sum_u q_u\ln\frac{p_\theta(x,u)}{q_u}.$$ You may use the strict concavity of the natural logarithm: $\ln(\sum_u q_u t_u)\ge\sum_u q_u\ln t_u$ for $t_u>0$, with equality exactly when all $t_u$ are equal.

Variant 3, Task 30. Questions

Build a lower bound from a weighted average · Exact examination statement · 2/2

(a) Rewrite $p_\theta(x)$ as a $q$-weighted average and prove $\mathcal B(q)\le\ln p_\theta(x)$. Find the unique $q$ that gives equality and identify it probabilistically.

(b) Let $r_u=p_\theta(x,u)/p_\theta(x)$ and define $D(q\|r)=\sum_u q_u\ln(q_u/r_u)$. Show by expansion that $\ln p_\theta(x)-\mathcal B(q)=D(q\|r)$. What does part (a) tell you about the sign of $D$?

(c) If $p_\theta(x,u)=p(u)p_\theta(x\mid u)$, rewrite $\mathcal B(q)$ as an expected log-likelihood minus $D(q\|p)$, using the same definition of $D$. Explain the two terms in one sentence. This bound is called the ELBO.

From a hidden cause to its posterior

Variant 3 · Fixed observation $x$ · Circle: $u=1$ · Diamond: $u=2$

Prior $p(u)$
$(0.5,0.5)$

Likelihood $p_\theta(x\mid u)$
$(0.2,0.8)$

Observed $x$

Joint $p_\theta(x,u)$

0.10.4

$u=1\qquad u=2$

Posterior $r(u)$

0.20.8

$u=1\qquad u=2$

Sum: $p_\theta(x)=0.5$
Normalize: $r(u)=p_\theta(x,u)/0.5$

Trial $q$: $(0.5,0.5)$
Chosen separately from the model

Evidence is a scalar. The posterior is a distribution over hidden states.

The generated bars are covered with exact editable bars. Heights use the displayed probabilities. The likelihood is a function of u for this fixed x, not generally a normalized distribution over u. This example happens to sum to one. The other observation has joint slice (0.4,0.1), completing a normalized model. Model stays fixed while trial q varies. The original corrected illustration connects the observation only to the likelihood.

Evidence is a sum over hidden states

Variant 3 · Fix an observation $x$ · Latent state $u=1,\ldots,K$

$$p_\theta(x)=\sum_u p_\theta(x,u),\qquad p_\theta(x,u)>0.$$

Joint masses $(0.1,0.4)$ → sum $0.5$ → normalize to posterior $(0.2,0.8)$.

Choose $q_u>0$, $\sum_u q_u=1$:

$$\mathcal B(q)=\sum_u q_u\ln\frac{p_\theta(x,u)}{q_u}.$$

Supplied tool: concavity of the natural logarithm

$$\ln\left(\sum_u q_u t_u\right)\ge\sum_u q_u\ln t_u,\qquad t_u>0.$$

Equality holds exactly when all $t_u$ are equal.

A weighted average inside the logarithm

Variant 3(a) · How can the supplied inequality apply to the evidence?

$$p_\theta(x)=\sum_u q_u\frac{p_\theta(x,u)}{q_u}.$$

Set $t_u=p_\theta(x,u)/q_u$. The weights sum to one.

$$\ln p_\theta(x)=\ln\left(\sum_u q_u t_u\right) \ge\sum_u q_u\ln t_u=\mathcal B(q).$$

Every admissible $q$ gives a lower bound on log evidence.

Pause on the inserted q_u/q_u. Strict positivity makes every ratio well-defined. The inequality comes directly from the supplied concavity statement.

When does the bound touch the evidence?

Variant 3(a) · Use the equality condition and normalization

$$\frac{p_\theta(x,u)}{q_u}=c\quad\text{for every }u.$$

$$p_\theta(x)=\sum_u c q_u=c.$$

$$q_u=\frac{p_\theta(x,u)}{p_\theta(x)}=p_\theta(u\mid x).$$

The unique equality distribution is the exact posterior.

The model is fixed. Varying $q$ changes the bound; log evidence stays fixed.

The gap is a posterior KL

Variant 3(b) · Define $r_u=p_\theta(u\mid x)$ and $D(q\|r)=\sum_u q_u\ln(q_u/r_u)$

Fixed log evidence = chosen bound + nonnegative inference gap. The gap compares $q$ with the posterior, not the prior.

$$D(q\|r)=\sum_u q_u\left[\ln q_u-\ln p_\theta(x,u)+\ln p_\theta(x)\right].$$

Since $\sum_u q_u=1$,

$$D(q\|r)=\ln p_\theta(x)-\mathcal B(q).$$

$$D(q\|r)\ge0,\qquad D(q\|r)=0\iff q=r.$$

The posterior KL measures how far the bound falls below log evidence.

Nonnegativity follows from the bound already proved. No external KL theorem is needed. Preserve the order q || posterior. The next slide uses a different reference distribution inside KL.

Two KLs, two different questions

Variant 3(c) · Prior $p(u)$ · Trial $q(u)$ · Posterior $r(u)=p_\theta(u\mid x)$

Posterior KL: how much does inference miss? Prior KL: what does the code cost?

$$p_\theta(x,u)=p(u)p_\theta(x\mid u),\qquad \ln\frac{p_\theta(x,u)}{q_u}=\ln p_\theta(x\mid u)-\ln\frac{q_u}{p(u)}.$$

$$\mathcal B(q)=\underbrace{\mathbb E_q[\ln p_\theta(x\mid u)]}_{R:\ \text{explain the observation}} -\underbrace{D(q\|p)}_{C:\ \text{deviation from prior}}.$$

$$\ln p_\theta(x)=\mathcal B(q)+\underbrace{D(q\|r)}_{\text{posterior gap}}.$$ Course notation: $\mathcal E=\mathcal B(q)$ is the ELBO, maximized during fitting.

An exact posterior closes the gap; it need not equal the prior.

At a fixed model, optimizing q targets the posterior. This can increase prior KL while improving the ELBO. These are not two distances that must decrease together. The likelihood at fixed x is not a normalized distribution over latent states. In a VAE the encoder supplies q_psi(u|x); phi remains the activation notation. KL is asymmetric, not a Euclidean distance or the area between curves.

An exact two-state check

Additional example · Joint masses $(0.1,0.4)$ · Evidence $0.5$ · Posterior $(0.2,0.8)$

$q=(0.5,0.5)$: $\mathcal B=\tfrac12\ln0.2+\tfrac12\ln0.8=\ln0.4\approx-0.9163$.

$\ln p_\theta(x)=\ln0.5\approx-0.6931$; gap $=\ln1.25\approx0.2231$.

$q=(0.2,0.8)$: both joint-to-$q$ ratios equal $0.5$, so the gap is zero.

The curve visualizes all trial distributions of this two-state model. Pause before calculating the marked q=(0.5,0.5) case. These are joint masses for one observation, not a complete joint distribution by themselves. The other observation can have masses (0.4,0.1), giving a normalized full model.

Variant 4, Task 30.

Good reconstruction or a useful latent variable? · Exact examination statement · 1/2

Let both the observation $x$ and latent variable $u$ be binary. The prior is $p(u=0)=p(u=1)=1/2$. Compare two complete models:

• Model A: $p_A(x=u\mid u)=1$ and $p_A(x\ne u\mid u)=0$; its encoder assigns $q_A(u=x\mid x)=1$.
• Model B: $p_B(x=0\mid u)=p_B(x=1\mid u)=1/2$ for either $u$; its encoder assigns $q_B(u=0\mid x)=q_B(u=1\mid x)=1/2$.

For the observation $x=1$, define the reconstruction score and latent cost $$R=\sum_{u:q(u\mid1)>0}q(u\mid1)\ln p(1\mid u),\qquad C=\sum_{u:q(u\mid1)>0}q(u\mid1)\ln\frac{q(u\mid1)}{p(u)}.$$ The training score is $R-C$, which is maximized. Zero-probability encoder states are omitted from these sums. Natural logarithms are used. You may use $p(x)=\sum_u p(x\mid u)p(u)$ and $p(u\mid x)=p(x\mid u)p(u)/p(x)$.

Variant 4, Task 30. Questions

Good reconstruction or a useful latent variable? · Exact examination statement · 2/2

(a) Compute $p_A(x=1)$ and $p_B(x=1)$ by summing over $u$. Can likelihood of this observation distinguish the two models?

(b) Compute $R$, $C$ and $R-C$ for each model. Explain why the model with better reconstruction does not obtain a better total score here.

(c) Use Bayes’ rule to find $p_A(u\mid x=1)$ and $p_B(u\mid x=1)$. Which latent variable carries information about $x$, and which model ignores it? Explain why equal likelihood need not imply equally informative latent representations.

Same observations, different hidden relationships

Variant 4 · Hidden states: circle $u=0$, diamond $u=1$ · Observations: square $x=0$, triangle $x=1$

Model A: copies the hidden state

Model B: ignores the hidden state

Allowed links: $p_A(x=u\mid u)=1$.
Other links: probability $0$.

Every link: $p_B(x\mid u)=1/2$.

Both start from $p(u)=(1/2,1/2)$. Does the observed distribution reveal the difference?

Two binary generative models

Variant 4 · $x,u\in\{0,1\}$ · Prior $p(u=0)=p(u=1)=1/2$

Joint masses $p(u,x)$: hidden-state rows, observation columns.

	A: $x=0$	A: $x=1$	B: $x=0$	B: $x=1$
$u=0$	$1/2$	$0$	$1/4$	$1/4$
$u=1$	$0$	$1/2$	$1/4$	$1/4$

A copies $u$; $q_A(u=x\mid x)=1$. B ignores $u$; $q_B(u\mid x)=1/2$.

For $x=1$, sum only over states with $q>0$:

$$R=\sum_{u:q(u\mid1)>0}q(u\mid1)\ln p(1\mid u),\qquad C=\sum_{u:q(u\mid1)>0}q(u\mid1)\ln\frac{q(u\mid1)}{p(u)}.$$

Maximize $R-C$: reconstruction score minus latent cost.

Marginalizing the latent state

Variant 4(a) · Can the likelihood of $x=1$ distinguish the models?

Use $p(x)=\sum_u p(x\mid u)p(u)$.

$$p_A(1)=\underbrace{0\cdot\tfrac12}_{u=0}+\underbrace{1\cdot\tfrac12}_{u=1}=\tfrac12.$$

$$p_B(1)=\underbrace{\tfrac12\cdot\tfrac12}_{u=0}+\underbrace{\tfrac12\cdot\tfrac12}_{u=1}=\tfrac12.$$

Both log evidences equal $-\ln2$.

The observation has equal likelihood under both models.

Reconstruction and latent cost

Variant 4(b) · Which model achieves the larger score?

Model	Reconstruction R	Cost C	Score R − C
A	$\ln1=0$	$\ln(1/(1/2))=\ln2$	$-\ln2$
B	$\ln(1/2)=-\ln2$	$\ln1=0$	$-\ln2$

A: only $u=1$ contributes, with encoder probability one.

B: both states have the same log likelihood and the encoder equals the prior.

A’s reconstruction advantage is exactly offset by its latent cost.

R is expected log likelihood, generally nonpositive here. C is nonnegative and is subtracted. Do not reverse these signs when later switching to a minimized loss.

Does the latent code carry information?

Variant 4(c) · Apply $p(u\mid x)=p(x\mid u)p(u)/p(x)$

Conditioning = select the $x=1$ column, then divide by its sum $1/2$.

$$p_A(u\mid1)=(0,1).\qquad \text{The latent state identifies }x.$$

$$p_B(u\mid1)=(\tfrac12,\tfrac12)=p(u).\qquad \text{The decoder ignores }u.$$

Equal likelihood can accompany different amounts of latent information.

Both encoders are exact posteriors: both ELBOs equal log evidence. There is no approximation gap in either model.

In A, the same deterministic relation also holds for x=0. In B, the posterior stays equal to the prior for either observation. This unused-latent model helps motivate posterior collapse, but the example does not establish how a particular optimizer would reach it.

Four distinctions to carry forward

1. Equivariant outputs and an invariant selected readout
2. Reduced variance and remaining bias/correlation
3. Log evidence and a lower bound that depends on inference
4. Accurate reconstruction and information in a latent code

Week 10 connects the ELBO and the usefulness of representations.

The ELBO connects inference to evidence. A useful representation requires evaluation suited to the downstream task. Invite students to name the assumption that made each review result possible.