Sequential
representation
learning

Week-06 · ML-2026

RNNs and Transformer encoders

Alexei Kornaev,
Dr. Sc., Prof.

Innopolis University

Researcher at the RC for AI,
National RC for Oncology n.a. N.N. Blokhin

Lecture map

1. Sequence problems and recurrent state
2. Learning through time and gated memory
3. Self-attention and contextual representations
4. Inside a Transformer encoder

Print / save as PDF

Session 1: Parts 1–2. Session 2: Parts 3–4.
Code inspection remains a separate, flexible part of each session.

What should you be able to explain?

• Trace a recurrent state forward and shared-parameter gradients backward
• Calculate query–key scores and a weighted value mixture
• Reconstruct an encoder block and choose visibility for its task

Track positions, feature widths, parameter sharing, and permitted inputs.

The main lecture was reviewed by the instructor before finalization. Earlier authoring review marker retained: TODO(instructor): Review the gated-memory bridge and attention arithmetic. Review the complete encoder calculations and task boundaries before the final synthesis. The complete teaching content and fifth question are now implemented.

Notation: positions and representations

Symbol	Meaning
$t=0,\ldots,T-1$	Position within one sequence
$x_t\in\mathbb R^D$	Input row vector at position $t$
$z_t,h_t\in\mathbb R^H$	Preactivation and recurrent state
$h^{(\ell)}\in\mathbb R^{T\times D}$	All token representations at encoder layer $\ell$
$\theta,\ \mathcal L$	Trainable parameters and the stated loss

Subscripts locate a position. Superscripts locate a layer.

Row vectors throughout, as in the course’s row-major batch convention. Batch axis B is omitted in these design samples. A scalar coordinate has an additional index j. In the neuron sample D=H=2. Input vectors are given numerical features; token lookup will be introduced in Batch 2 or 3. No tutorial vocabulary is assumed.

Part 1 · Sequence problems and recurrent state

1. Sequence problems and recurrent state
2. Learning through time and gated memory
3. Self-attention and contextual representations
4. Inside a Transformer encoder

How can the current prediction use earlier observations?

Sequences contain ordered observations

The position axis adds structure to the feature axis.

Measurements may arrive in time, words have an order in a sentence, and patches have positions in an image. A supplied patch order is not physical time. Here each x_t will be a row of numerical features. Token lookup and positional encoding come in later batches. A sequence may have variable length, while each position has the same feature width D in these examples.

The task determines where outputs belong

Task	Observed input	Output	Example
Sequence classification	$x_0,\ldots,x_{T-1}$	One label	Classify an utterance
Position-wise labeling	$x_0,\ldots,x_{T-1}$	One label per position	Label each word’s role
Next-observation prediction	$x_0,\ldots,x_t$	Predict position $t+1$	Forecast a measurement

Specify the information available at the moment of prediction.

The first two tasks may legitimately observe a complete sequence. A future predictor may use only the observed prefix. An architecture alone does not define this information boundary. This taxonomy does not require every output to be a class: regression heads may output continuous values. Position-wise labeling can be causal or bidirectional depending on the deployment task.

Student question 1 · Does order change the answer?

Task: classify the direction of the trend. Can the mean distinguish these sequences?

Both means are $0$. Their ordered changes have opposite signs.

Allow 20–30 seconds before revealing the explanation. These are conceptual classroom checks, not assessment answers. The trend can also be recognized by a fixed-window linear model using the endpoints: recurrence is not required. Order would not matter for a different target such as the sequence mean. Later sequence representations must be judged against the actual task.

A fixed window is already a valid input

$$x_{\mathrm{flat}}=[x_0\mid x_1\mid\cdots\mid x_{T-1}]\in\mathbb R^{TD}$$

$$\widehat y=f(x_{\mathrm{flat}};\theta)$$

An MLP can use order

Each window position occupies distinct input coordinates.

Its input width is fixed

A new window length needs an explicit handling rule.

An RNN offers a shared update for successive positions.

Flattening preserves the observed values and their order. Padding, truncation, or resampling can also give an MLP a fixed input width; no universal claim of recurrent superiority is made. Compare models with access to the same evidence. The vertical bars here mean concatenation, not conditioning.

A recurrent state carries the earlier computation

$$h_t=f_{\mathrm{cell}}(x_t,h_{t-1};\theta)$$

The state changes at each position. The cell parameters are shared.

Goodfellow, Bengio & Courville (2016), Chapter 10.

The function named cell is a vector transformation, not one scalar neuron. State h_t is a learned representation, not a stored copy of the whole prefix. Reset the state at independent sequence boundaries in this lecture. State can be carried across chunks when a task calls for it, but that changes the information available. theta is fixed during this forward pass.

One recurrent unit combines input and memory

Two input coordinates, two state coordinates. Here $j\in\{1,2\}$.

1 · Current input $$I_{t,j}=\sum_{k=1}^{2}W_{x,jk}x_{t,k}$$

2 · Previous state $$R_{t,j}=\sum_{r=1}^{2}W_{h,jr}h_{t-1,r}$$

3 · Affine sum $$z_{t,j}=I_{t,j}+R_{t,j}+b_{h,j}$$

4 · Activation $$h_{t,j}=\tanh(z_{t,j})$$

Every state coordinate has its own weights and bias.

RNN formulation: Goodfellow, Bengio & Courville, Chapter 10. Original course diagram.

Four clicks synchronize edges and equations. The unit is one scalar tanh neuron, not the complete vector state. Both state neurons receive both current input coordinates and both previous state coordinates. W_x and W_h each have shape (2,2), with rows indexing output units. Bias has shape (2,). The full row-vector formula is z_t=x_t W_x^T+h_(t-1) W_h^T+b_h. I_(t,j) and R_(t,j) name the current-input and recurrent contributions only on this slide.

Two ordinary units form one recurrent state

$$z_t=x_tW_x^\top+h_{t-1}W_h^\top+b_h$$

$$h_t=\tanh(z_t)$$

Quantity	Shape	Role
$x_t$, $h_{t-1}$	$D$, $H$	Current features, previous state
$W_x$, $W_h$	$H\times D$, $H\times H$	One row per state unit
$b_h$, $z_t$, $h_t$	$H$	Bias, affine sum, activation

The same row-vector convention extends Week 04’s affine layer.

Coordinate labels j,k,r start at one on the scalar neuron slide, while positions t start at zero. The shape H×H does not mean that a state coordinate connects only to itself: the recurrent matrix mixes all previous-state coordinates. tanh acts on each coordinate separately. h_-1=0 is our declared initial state, not a universal requirement.

Small numbers make the forward pass visible

Three observations from two numerical features. Illustrative, untrained parameters.

$$W_x=\begin{bmatrix}0.3&-0.2\\0.1&0.4\end{bmatrix}$$ $$W_h=\begin{bmatrix}0.5&0.1\\-0.2&0.3\end{bmatrix}$$ $$b_h=(0.05,-0.10),\quad h_{-1}=(0,0)$$

Position	Input $x_t$
$0$	$(1.0,-0.5)$
$1$	$(0.2,0.8)$
$2$	$(-0.4,0.3)$

Both input coordinates may be signed, as with standardized sensor features.

All three steps reuse these same ten cell parameters.

TODO(instructor): Review the standalone sequence example and forward-pass storyboard in Batch 2.

This microscope has D=H=2. The recurrent cell has 4+4+2=10 parameters. The synthetic input values do not represent a claimed real sensor experiment. Keep all displayed parameters fixed while tracing forward and backward. Calculations use full precision; slide values round to four decimal places.

The first observation creates a state

$x_0=(1.0,-0.5)$ and $h_{-1}=(0,0)$.

1 · Current-input contribution $$x_0W_x^\top=(0.4000,-0.1000)$$

2 · Previous-state contribution $$h_{-1}W_h^\top=(0,0)$$

3 · Add the bias $$z_0=(0.4000,-0.1000)+(0.05,-0.10)=(0.4500,-0.2000)$$

4 · Apply tanh to each coordinate $$h_0=(0.4219,-0.1974)$$

A zero initial state leaves the input and bias paths active.

The next observation combines with the earlier state

$x_1=(0.2,0.8)$ and the computed $h_0=(0.4219,-0.1974)$.

1 · Current-input contribution $$x_1W_x^\top=(-0.1000,0.3400)$$

2 · Previous-state contribution $$h_0W_h^\top=(0.1912,-0.1436)$$

3 · Add both contributions and the bias $$z_1=(-0.1000,0.3400)+(0.1912,-0.1436)+(0.05,-0.10)$$ $$z_1=(0.1412,0.0964)$$

4 · Apply tanh $$h_1=(0.1403,0.0961)$$

Earlier x_0 affects this step through h_0. A state coordinate can change sign as input and recurrent contributions combine. Stored rounded values are only for display; the generator carries full precision through each step. Students can calculate one output coordinate while the other serves as a check.

Unrolling makes the dependency graph visible

$t$	$x_t$	$z_t$	$h_t$
0	$(1.0000, -0.5000)$	$(0.4500, -0.2000)$	$(0.4219, -0.1974)$
1	$(0.2000, 0.8000)$	$(0.1412, 0.0964)$	$(0.1403, 0.0961)$
2	$(-0.4000, 0.3000)$	$(-0.0502, -0.0192)$	$(-0.0502, -0.0192)$

Each state depends on a prefix through repeated uses of one cell.

The boxes show state outputs of the repeated cell. They are not three separate parameter sets. The affine preactivation is distinct from the activated state. z_2 and h_2 happen to round to the same four-decimal values because tanh is almost linear near zero; they differ at full precision. The recurrent state is a compressed learned representation of the prefix, not guaranteed lossless memory.

Student question 2 · Do longer sequences add parameters?

Our cell has $D=H=2$.

$$W_x:\ 2\times2,\qquad W_h:\ 2\times2,\qquad b_h:\ 2$$

If we process six positions instead of three, how many trainable cell parameters do we use?

Still $4+4+2=10$. There are more computations and saved states.

Pause before revealing. The count excludes an output head and any learned initial state. With full BPTT, saved activations normally grow with length; checkpointing/recomputation can trade memory for work. The same weights have many uses, each contributing to one gradient.

An output head gives the state a task

$$a_t=h_tW_o^\top+b_o\in\mathbb R^C$$

$$p_t=\operatorname{softmax}(a_t)$$

Classification

$C$ logits support a distribution over $C$ classes.

Regression

An affine head can predict a continuous target directly.

Hidden coordinates, logits, and probabilities have different roles.

C is the number of output classes here, distinct from input width D and state width H. W_o has shape C×H and b_o shape C. Choose the output head and loss from the task. tanh hidden-state coordinates are not probabilities. In a sequence-to-label model a head can read the final state; position-wise prediction can apply the same head to every state. The next backward example first uses an identity head on the final state so that the chain rule is transparent, then adds a three-class head and mean per-position cross-entropy.

Part 2 · Learning through time and gated memory

1. Sequence problems and recurrent state
2. Learning through time and gated memory
3. Self-attention and contextual representations
4. Inside a Transformer encoder

A shared parameter receives a contribution from every use.

A final-state loss starts a simple backward pass

First microscope: identity readout, target $y=(0.20,-0.15)$.

$$\mathcal L=\frac12\|h_2-y\|_2^2$$

$$h_2=(-0.0502,-0.0192),\qquad \mathcal L\approx0.039853$$

$$g^h_2=\frac{\partial\mathcal L}{\partial h_2}=h_2-y=(-0.2502,0.1308)$$

A loss derivative describes sensitivity, not a parameter update.

This identity readout makes the state itself a two-dimensional regression output. It is an arithmetic teaching example, not a claim that these untrained states solve a task. There are no intermediate-position losses in this first objective. The factor 1/2 cancels the derivative’s factor two, and no averaging over time is applied. Gradient row shapes are the same as the differentiated rows.

The activation supplies a local derivative

$$h_{t,j}=\tanh(z_{t,j})$$

$$\frac{\partial h_{t,j}}{\partial z_{t,j}}=1-h_{t,j}^2$$

$$g^z_t=g^h_t\odot(1-h_t^2)$$

At $t=2$: $(1-h_2^2)=(0.9975,0.9996)$,

$$g^z_2=(-0.2496,0.1307).$$

The same local chain rule was used in Week 04. The elementwise square and product avoid an explicit diagonal matrix. When a tanh coordinate is near ±1 its derivative is small; near zero its derivative is near one. That does not by itself guarantee stable recurrent gradients because each step also multiplies by W_h.

One final loss travels backward through time

Three steps, $h_{-1}=0$. Here $\mathcal L=\tfrac12\|h_2-y\|^2$ for a target vector $y$.

1 · Loss and final activation $$g^h_2=h_2-y,\quad g^z_2=g^h_2\odot(1-h_2^2)$$

2 · Previous state receives a signal $$g^h_t=g^z_{t+1}W_h,\quad t=1,0$$

3 · Repeat the local chain rule $$g^z_t=g^h_t\odot(1-h_t^2),\quad t=1,0$$

4 · Sum contributions to one shared matrix $$\frac{\partial\mathcal L}{\partial W_h}=\sum_{t=0}^{2}(g^z_t)^\top h_{t-1}$$

$g^h_t=\partial\mathcal L/\partial h_t$ and $g^z_t=\partial\mathcal L/\partial z_t$ are row vectors.

Chain rule and parameter sharing: Goodfellow, Bengio & Courville, Chapter 10. $\odot$ and the square act coordinate-wise.

This is a final-state regression loss with an identity readout, deliberately small enough to inspect independently of any tutorial. The second click is the recurrent vector-Jacobian product. On the third click repeat g^h_0=gz_1 W_h before applying tanh’s derivative at t=0. At t=0 the recurrent parameter contribution is zero because h_-1=0. All other contributions already contain the effect of the final loss. The generator verifies this two-unit chain rule and the summed W_h gradient with central finite differences. The later per-position classification objective uses a separate output head and requires additional direct-loss terms. The optimizer update happens after backward propagation, not between unrolled steps.

The same chain rule reaches earlier states

Backward order	$g^h_t$	$1-h_t^2$	$g^z_t$
$t=2$	$(-0.2502, 0.1308)$	$(0.9975, 0.9996)$	$(-0.2496, 0.1307)$
$t=1$	$(-0.1509, 0.0143)$	$(0.9803, 0.9908)$	$(-0.1480, 0.0141)$
$t=0$	$(-0.0768, -0.0106)$	$(0.8220, 0.9610)$	$(-0.0631, -0.0101)$

$$g^h_{t-1}=g^z_tW_h,\qquad g^z_{t-1}=g^h_{t-1}\odot(1-h_{t-1}^2)$$

Read the ledger from the final position back to the first.

This table belongs only to the final-state squared-error objective. There are no direct intermediate loss signals to add. The rows do not show an optimizer running at each time: parameters remain fixed. Numbers are full-precision calculations rounded for display. The gradient can have either sign and may cancel along different paths.

One weight use contributes an outer product

$$z_{t,j}=\sum_k W_{x,jk}x_{t,k}+\sum_r W_{h,jr}h_{t-1,r}+b_{h,j}$$

$$\left.\frac{\partial\mathcal L}{\partial W_{h,jr}}\right|_{\text{use at }t}=g^z_{t,j}h_{t-1,r}$$

$$G_t=(g^z_t)^\top h_{t-1}\in\mathbb R^{H\times H}$$

Rows index receiving units. Columns index previous-state coordinates.

G_t is the contribution from this use of the shared matrix, with the previous state treated as a local input. The backward signal g^z_t already includes downstream paths. Outer product means H×1 times 1×H. Using an inner product would produce one scalar and lose the individual parameter derivatives.

Shared weights receive the sum of their uses

Use at t = 0

$$\begin{bmatrix}0.0000 & 0.0000\\0.0000 & 0.0000\end{bmatrix}$$

Use at t = 1

$$\begin{bmatrix}-0.0624 & 0.0292\\0.0060 & -0.0028\end{bmatrix}$$

Use at t = 2

$$\begin{bmatrix}-0.0350 & -0.0240\\0.0183 & 0.0126\end{bmatrix}$$

Sum

$$\begin{bmatrix}-0.0974 & 0.0052\\0.0243 & 0.0098\end{bmatrix}$$

$$\frac{\partial\mathcal L}{\partial W_h}=G_0+G_1+G_2$$

The contributions add to one matrix. They may reinforce or cancel.

TODO(instructor): Review the gradient derivation in Batch 2 and the short gated-memory comparison in Batch 3.

All matrices use the final-state squared-error objective. G_0 is zero because h_-1=0. Earlier input still affects the loss via W_x at t=0. Rounded matrices may sum differently in the last decimal; calculations and checks use full precision. This is the same branch-addition rule from Week 04 applied to parameter sharing through time.

Student question 3 · Which shared-weight uses matter?

Only the final state has a loss. The matrix $W_h$ appears at $t=0,1,2$.

Should its gradient include just the last use, or all three uses?

Sum all uses. In this example, the $t=0$ term is zero because $h_{-1}=0$.

Wait before revealing. The zero contribution comes from the local recurrent input, not from a rule that the first state is irrelevant. A nonzero initial state could give a nonzero G_0. W_x at t=0 does have a contribution. The final loss reaches earlier cell uses through the unrolled paths.

A loss at every position adds direct signals

Second microscope: $C=3$ classes, target IDs $(y_0,y_1,y_2)=(0,2,1)$.

$$a_t=h_tW_o^\top+b_o,\quad p_t=\operatorname{softmax}(a_t)$$

$$\ell_t=-\log p_t[y_t],\qquad \mathcal L=\frac1T\sum_{t=0}^{T-1}\ell_t$$

$$g^a_t=\frac{p_t-\operatorname{onehot}(y_t)}{T}$$

For this mean objective, divide by $T$ once at the loss derivative.

Here y_t is a class ID at position t, not the two-coordinate target y of the identity-head example. Targets are arbitrary illustrative labels, not a fitted sensor-label dataset. Natural log gives cross-entropy in nats. This formula assumes T real scored positions; padded batches require the actual valid-position count. In an autoregressive task y_t could be the next token ID, but this example is position-wise classification. No mean over a batch is included here.

The same states produce three-class predictions

$$W_o=\begin{bmatrix}0.4&-0.2\\-0.3&0.5\\0.1&0.2\end{bmatrix}$$ $$b_o=(0.02,-0.04,0.01)$$

At the final position:

$$p_2=(0.3379,0.3252,0.3370)$$

Target ID: $y_2=1$.

Mean loss across all three positions:

$$\mathcal L\approx1.027996\ \text{nats}$$

The output head changes the objective and its direct gradient paths.

State computation and W_x,W_h,b_h are unchanged. This head adds 3×2+3=9 parameters, giving 19 total. The forward and backward ledgers in the JSON include logits and probabilities at every position, and mean cross-entropy. The head weights are untrained and probabilities are close to uniform. Do not interpret these values as model accuracy or learned sensor semantics.

A hidden state receives direct and future gradients

1 · Direct signal through this position’s output head $$g^h_{t,\text{direct}}=g^a_tW_o$$

2 · Add the signal from the next recurrent use $$g^h_t=g^a_tW_o+g^z_{t+1}W_h$$

3 · Pass through the local activation $$g^z_t=g^h_t\odot(1-h_t^2)$$

At the final position, the future contribution is zero.

Goodfellow, Bengio & Courville (2016), Chapter 10. Derivatives independently checked.

Set g^z_T=0 as a backward boundary. The future term already aggregates losses after t via the recurrence; do not add each future loss again. With only a final-state loss, the direct term is zero at earlier positions, which recovers the previous example. Matrix shapes: g^a_t is 1×C, W_o C×H; g^z_(t+1) 1×H, W_h H×H. Only after both paths are added does the activation derivative gate the signal. The current-position cross-entropy derivative already contains the factor 1/T.

The recurrent parameters collect local contributions

$$\frac{\partial\mathcal L}{\partial W_x}=\sum_t(g^z_t)^\top x_t$$

$$\frac{\partial\mathcal L}{\partial W_h}=\sum_t(g^z_t)^\top h_{t-1}$$

$$\frac{\partial\mathcal L}{\partial b_h}=\sum_t g^z_t$$

The local input changes by position. The parameter is shared.

Optional depth for the timed route: Extend the mean-cross-entropy derivation only if time allows; the primary shared-weight sum is already established.

The sums run t=0,…,T-1. This formula works for both objectives when g^z_t is calculated from the corresponding objective. Dimensions are H×D, H×H, and H. For a batch add contributions from its examples with the loss’s declared mean scaling. No second division by T belongs in these sums.

The output head also shares its parameters

$$\frac{\partial\mathcal L}{\partial W_o}=\sum_t(g^a_t)^\top h_t$$

$$\frac{\partial\mathcal L}{\partial b_o}=\sum_t g^a_t$$

$$\theta\leftarrow\theta-\eta\nabla_\theta\mathcal L$$

Compute the complete gradient before the optimizer changes parameters.

Optional depth for the timed route: Optional extension of the per-position objective; keep the head/target distinction even if this calculation is skipped.

The final line illustrates ordinary gradient descent with learning rate eta; other optimizers use their own state and update rule. Backpropagation computes derivatives; it is not itself an optimizer. The backward recurrence used the same forward snapshot of every parameter. Updating between time steps would differentiate a different computation than the displayed unrolled model.

Two independent checks support the derivation

Objective	Checked parameters	Max. finite-difference error	Max. autograd error
Final-state squared error	10 scalars	$3.19\times10^{-12}$	$5.55\times10^{-17}$
Mean three-position CE	19 scalars	$1.71\times10^{-10}$	$6.94\times10^{-17}$

$$\frac{\partial\mathcal L}{\partial\theta_j}\approx\frac{\mathcal L(\theta_j+\epsilon)-\mathcal L(\theta_j-\epsilon)}{2\epsilon},\quad \epsilon=10^{-6}$$

Finite differences and PyTorch autograd agree with manual BPTT.

Original float64 calculations. Full numerical ledger.

Optional depth for the timed route: Brief numerical verification reference; code inspection can use this later without interrupting the lecture narrative.

The finite difference perturbs one parameter while keeping every other parameter fixed, and reexecutes the full sequence and objective. PyTorch differentiates an independently written forward loop. All 10 active cell parameters in the identity-head objective and all 19 parameters in the CE objective are checked. Small errors support this example’s implementation, not a universal proof or an empirical learning claim. Rounding displayed values must not be used inside the checks.

Long paths multiply local Jacobians

$$D_t=\operatorname{diag}(1-h_t^2)$$

$$g^h_{t-1}=g^h_tD_tW_h$$

$$g^h_s=g^h_t(D_tW_h)(D_{t-1}W_h)\cdots(D_{s+1}W_h)$$

Along one future-loss path, every transition changes the sensitivity.

Pascanu, Mikolov & Bengio (2013), §2.

This expression isolates a single downstream path with no intervening direct-loss additions. In the full objective, sum such paths as in the direct-plus-future recurrence. D_t is H×H and each g^h is a row, so the factors are ordered backward from t toward s. The source paper uses a different state/activation parameterization in its theoretical section; its footnote explains equivalence. This slide derives the factors from our declared tanh recurrence rather than copying their matrix order.

Repeated factors can shrink or amplify a signal

Toy scalar case: a constant local derivative $r$ gives $|r|^n$.

Original illustrative calculation; Pascanu et al. (2013) analyzes the matrix case.

This chart assumes the same scalar local derivative on every transition. It is not measured gradient data from the two-unit microscope. Matrix products depend on directions, changing activations, and singular values. A bounded tanh output does not bound W_h or the entire Jacobian product. Long-range signal can vanish despite nonzero forward states. Avoid asserting that hidden values or scalar eigenvalues alone determine learnability in a nonlinear model.

Gradient clipping limits a large update direction

$$g=\nabla_\theta\mathcal L,\qquad c>0$$

$$\widetilde g=\begin{cases}g,&\|g\|_2\le c,\\\displaystyle\frac{c}{\|g\|_2}g,&\|g\|_2>c.\end{cases}$$

$$\|\widetilde g\|_2\le c$$

Clipping caps large gradients. It cannot recover a vanished signal.

Pascanu, Mikolov & Bengio (2013), §3.2.

Clip the full computed gradient vector before the optimizer step. Norm clipping preserves a nonzero vector’s direction but limits its magnitude. The cap applies to the gradient, not necessarily the final parameter-step norm of an adaptive optimizer. Zero gradient falls in the first case, avoiding division by zero. Tanh saturation, shared mixing, and long path products motivate changes in the state update, including gates, which the next slides introduce.

A GRU learns how much old state to retain

$$h_t=u_t\odot h_{t-1}+(1-u_t)\odot\widetilde h_t$$

A gate near one retains that coordinate. A gate near zero favors the candidate.

PyTorch GRU equations. Here $u_t$ names its update gate $z_t$.

TODO(instructor): Review the short GRU/LSTM comparison in Batch 3.

Three clicks reveal retention, writing, and addition. The illustrative gate, old state, and candidate are supplied scalar values, not trained model outputs. Their checked arithmetic is 0.9×0.5+0.1×(-0.4)=0.41. All operations in the vector equation act coordinate-wise. Keep course z_t for preactivations, so PyTorch’s update gate z_t is translated to u_t. Other references sometimes define the update gate with the opposite mixture convention; inspect which path receives it before comparing formulas.

GRU gates depend on the input and previous state

$u_t,r_t,\widetilde h_t,h_t\in\mathbb R^H$. $\sigma$ is the logistic sigmoid.

Retention/update gate $$u_t=\sigma(x_tW_{xu}^\top+b_{xu}+h_{t-1}W_{hu}^\top+b_{hu})$$

Reset gate $$r_t=\sigma(x_tW_{xr}^\top+b_{xr}+h_{t-1}W_{hr}^\top+b_{hr})$$

Candidate state $$\widetilde h_t=\tanh\!\left(x_tW_{xn}^\top+b_{xn}+r_t\odot(h_{t-1}W_{hn}^\top+b_{hn})\right)$$

Retention controls the mixture. Reset controls recurrent input to the candidate.

PyTorch GRU: reset after the recurrent affine operation.

Optional depth for the timed route: The retention mixture and short LSTM diagram are essential; full GRU gate algebra can be read afterward.

Input weights have shape H×D, recurrent weights H×H, and biases H. The separate input and recurrent biases follow PyTorch’s implementation, including the recurrent candidate bias inside the reset product. Reset-after-affine generally differs from first gating h_(t-1) and then multiplying by W_hn. The generator checks this exact convention against GRUCell with fixed two-unit weights and no training. Gates also depend on h_(t-1), so the full Jacobian includes their derivatives; a direct carry branch does not guarantee stable learning.

An LSTM separates cell storage from its visible state

$$c_t=f_t\odot c_{t-1}+i_t\odot\widetilde c_t$$ $$h_t=o_t\odot\tanh(c_t)$$

Forget, input, and output gates control retention, writing, and reading.

PyTorch LSTM equations. Plain LSTM, without projection or peephole connections.

Three clicks trace the retained cell path, gated writing, and the visible readout. c_t and h_t each have width H. Gates f_t,i_t,o_t are sigmoid functions of separate learned affine transformations of x_t and h_(t-1); the candidate c_tilde is a tanh affine transformation. In course row notation, for g in {f,i,o}, g_t=sigmoid(x_t W_xg^T+h_(t-1) W_hg^T+b_g); candidate uses the analogous c weights with tanh. PyTorch calls the candidate g_t; we call it c_tilde. With gates and candidate held fixed, the direct cell carry derivative is diag(f_t). The full derivative has additional paths; LSTM does not guarantee perfect memory or eliminate every vanishing-gradient problem. The generator verifies the plain cell update against LSTMCell.

Gates change the state update; recurrence still links positions

Cell	Carried state	Update structure
Simple RNN	$h_t$	Affine mixing followed by activation
GRU	$h_t$	Gated mixture of old state and candidate
LSTM	$(c_t,h_t)$	Gated storage, writing, and visible readout

Attention offers another route: combine source representations directly.

This closes the short gated-memory bridge. Compare mechanisms rather than claiming a universal accuracy ranking or training a new model. In the usual unidirectional recurrence, each state still depends on the previous state, so positions remain sequential. A bidirectional recurrent network has two separate directional passes; it still is not parallel self-attention. Session 1 can end here, with flexible code inspection and a verbal synthesis before Session 2.

Part 3 · Self-attention and contextual representations

1. Sequence problems and recurrent state
2. Learning through time and gated memory
3. Self-attention and contextual representations
4. Inside a Transformer encoder

Each position forms a weighted combination of permitted source representations.

The same token can need different context

A contextual representation depends on the surrounding sequence.

The language example motivates the computation, without claiming any specific trained attention map or embedding geometry. The arrows suggest useful contextual associations, not measured model weights. The numeric example will use the independent tokens the, cat, sleeps with hand-chosen vectors. Context also matters for measurements and image patches, so the mechanism is not exclusive to language.

Token lookup supplies an initial representation

$$s_t\ \text{is a token ID},\qquad e_t=E[s_t]\in\mathbb R^D$$

$$h_t^{(0)}=e_t+p_t^{\mathrm{pos}}$$

Token information

A learned table $E$ maps categories to vectors.

Position information

$p_t^{\mathrm{pos}}$ identifies where a token occurs.

Token IDs are labels; their numerical magnitude has no semantic meaning.

Vaswani et al. (2017), §3.4–3.5.

p_t was used for class probabilities earlier; p_t^pos explicitly labels a position vector. We distinguish learned position embeddings from the paper’s fixed sinusoidal position encoding in Batch 4. An embedding has D coordinates regardless of the token ID. The next numeric microscope takes X as a supplied sublayer input and omits position features, normalization, and dropout to expose attention arithmetic. Its arbitrary vectors are not claimed to be trained embeddings. Tokenization may produce words, subwords, or other discrete symbols; no tokenizer is implemented here.

A query matches keys and combines values

The query row chooses weights over source value rows.

Vaswani et al. (2017), §3.2.

Three clicks reveal scaling, softmax, and the weighted value path. Query q_t corresponds to the position being updated. Source key k_j participates in matching; source value v_j supplies the combined content. Queries, keys, and values are different learned projections. The source at j=t is also allowed in self-attention. Attention weights depend on input representations and model parameters; they are not a fixed learned table. Dot-product scores are not generally cosine similarities because norms also matter.

One input has three learned projection maps

$$X\in\mathbb R^{T\times D}$$

$$Q_{\mathrm{att}}=XW_Q^\top,\quad K_{\mathrm{att}}=XW_K^\top$$

$$V_{\mathrm{att}}=XW_V^\top$$

Weights	Shape	Projected rows
$W_Q,W_K$	$d_k\times D$	Queries and keys: $T\times d_k$
$W_V$	$d_v\times D$	Values: $T\times d_v$

Matching needs equal query/key widths. Value width may differ.

The matrices named Q_att,K_att,V_att avoid confusion with vocabulary size K in the wider course contract. The paper right-multiplies by D×d projection matrices; our parameter matrices are their transposes, preserving one row per output coordinate. Biases are omitted in this example. In self-attention all three maps receive the same X. Cross-attention would use different source/query representations, and is outside this batch. X may already contain normalization/position information in a full block.

A three-token microscope fixes every input number

Positions $0,1,2$: “the”, “cat”, “sleeps”. $T=3$, $D=4$, $d_k=d_v=2$.

$$X=\begin{bmatrix}1&0&1&0\\0&1&0&1\\1&1&0&0\end{bmatrix}$$

Query weights

$$W_Q=\begin{bmatrix}1&0&0&0\\0&1&0&0\end{bmatrix}$$

Key weights

$$W_K=\begin{bmatrix}0&0&1&0\\0&0&0&1\end{bmatrix}$$

Value weights

$$W_V=\begin{bmatrix}1&0&1&0\\0&1&0&2\end{bmatrix}$$

These illustrative parameters expose arithmetic, not learned language semantics.

TODO(instructor): Review the independent token example and attention arithmetic in Batch 3.

X contains given representations for one head without added positional features. All three positions are visible, including each query’s own source row. These are artificial fixed weights, not a trained model. Projectors are deliberately sparse to make each output coordinate calculable. Every full-precision number and parameter is stored in the generated ledger. No attention dropout or projection bias is used.

Queries, keys, and values keep different information

$Q_{\mathrm{att}}$

$$\begin{bmatrix}1.0000 & 0.0000\\0.0000 & 1.0000\\1.0000 & 1.0000\end{bmatrix}$$

$K_{\mathrm{att}}$

$$\begin{bmatrix}1.0000 & 0.0000\\0.0000 & 1.0000\\0.0000 & 0.0000\end{bmatrix}$$

$V_{\mathrm{att}}$

$$\begin{bmatrix}2.0000 & 0.0000\\0.0000 & 3.0000\\1.0000 & 1.0000\end{bmatrix}$$

$$q_1=(0,1),\qquad k_1=(0,1),\qquad v_1=(0,3)$$

The same input row produces three vectors with different roles.

The displayed matrices correspond to Q_att, K_att, V_att. q_1, k_1, and v_1 are their row at position 1, the token cat. Each row projection applies the same corresponding weights at every position. This is parameter sharing over positions without recurrent state dependence. The value does not need to equal the input embedding, query, or key.

One query scores all permitted source keys

Update position $t=1$: $q_1=(0,1)$. Source index $j=0,1,2$.

Raw dot-product scores $$q_1K_{\mathrm{att}}^\top=(0,1,0)$$

Scale by the square root of query/key width $$S_{1,:}=\frac{(0,1,0)}{\sqrt2}\approx(0,0.7071,0)$$

Rows locate queries; columns locate sources $$S_{tj}=\frac{q_tk_j^\top}{\sqrt{d_k}}$$

A score measures query–key compatibility before normalization.

Scaling controls a dot product’s width dependence

$$qk^\top=\sum_{r=1}^{d_k}q_rk_r$$

$$\operatorname{Var}(qk^\top)=d_k$$

$$\operatorname{Var}\!\left(\frac{qk^\top}{\sqrt{d_k}}\right)=1$$

Toy assumptions: all query/key coordinates are independent, with mean $0$ and variance $1$.

Scaling counters larger score variance as the head width grows.

Vaswani et al. (2017), §3.2.1, scaling footnote.

Under the stated independence assumptions, each q_r k_r has variance one and cross terms vanish. Learned projections need not satisfy those assumptions exactly; this is a motivation, not a guarantee that trained scores have variance one. Large score differences can make softmax concentrated and sensitivities small. Our sparse deterministic microscope is not a random sample from these assumptions. Scaling does not turn dot products into cosine similarity.

Row softmax turns the scores into mixing weights

$$A_{1j}=\frac{\exp(S_{1j})}{\sum_{m=0}^{2}\exp(S_{1m})}$$

$$\exp(S_{1,:})\approx(1,2.0281,1)$$

$$A_{1,:}\approx(0.2483,0.5035,0.2483)$$

Weights are nonnegative and sum to one within a query row.

Softmax uses the source-position axis, not the feature axis or all entries of the entire score matrix together. m is the source-index variable in the denominator. Implementations subtract each row’s maximum before exponentiating for stability; this leaves weights unchanged. The weight at source 1 is greatest here, but our untrained example does not show a learned grammatical relationship. No claim that a row of attention weights explains an entire model prediction.

The weights combine values, not keys

Source values $$v_0=(2,0),\quad v_1=(0,3),\quad v_2=(1,1)$$

Weighted combination $$O_1\approx0.2483(2,0)+0.5035(0,3)+0.2483(1,1)$$

One contextual output row $$O_1\approx(0.7448,1.7587)$$

The output is a feature vector; its coordinates need not sum to one.

Full-precision weights produce the displayed output, with rounding only after computation. Multiplying displayed four-decimal weights can change the last digit. O_1 has value width d_v=2, not sequence length T=3. Weighted values can contain signed coordinates and are not class probabilities. Attention reads every allowed source via its own weight, including a source with a low or zero weight.

Matrix attention performs the same calculation for every query

$$S=\frac{Q_{\mathrm{att}}K_{\mathrm{att}}^\top}{\sqrt{d_k}},\quad A=\operatorname{softmax}_{\mathrm{rows}}(S),\quad O=AV_{\mathrm{att}}$$

Weights A

$$\begin{bmatrix}0.5035 & 0.2483 & 0.2483\\0.2483 & 0.5035 & 0.2483\\0.4011 & 0.4011 & 0.1978\end{bmatrix}$$

Outputs O

$$\begin{bmatrix}1.2552 & 0.9930\\0.7448 & 1.7587\\1.0000 & 1.4011\end{bmatrix}$$

$A:T\times T$ mixes positions. $O:T\times d_v$ contains output features.

Original checked arithmetic. Full numeric ledger.

The middle row is the row just calculated. All query rows can be computed in parallel in this sublayer. Cross-checks compare a scalar-loop calculation and PyTorch scaled_dot_product_attention with the matrix implementation. This does not imply autoregressive generation of multiple future tokens is parallel. The attention map may be nonsymmetric even in self-attention because query and key projections differ. In this particular input pattern some weights coincide for arithmetic reasons; that is not an architecture constraint.

Student question 4 · Which axis sums to one?

$A$ has shape $T\times T$: rows are queries, columns are source positions.

Which axis sums to one, and which vectors form $O_t$?

Each query row sums to one across sources. $O_t$ combines the source value vectors.

Pause before revealing. If students answer that columns sum to one, return to the one-query denominator. Distinguish the feature coordinates of O_t from the scalar weights A_tj. Attention values can be any learned vectors; they need not be probability distributions.

A causal mask changes which sources are available

$$M_{tj}=\begin{cases}0,&j\le t,\\-\infty,&j>t,\end{cases}\qquad A=\operatorname{softmax}_{\mathrm{rows}}(S+M)$$

$$A^{\mathrm{causal}}_{1,:}\approx(0.3302, 0.6698, 0.0000)$$

$$O^{\mathrm{causal}}_1\approx(0.6605, 2.0093)$$

Position 1 can use sources 0 and 1, while source 2 receives zero weight.

The main encoder example has bidirectional visibility. This comparison recomputes a causal row to establish a task boundary, not to redefine all encoders as causal. A next-token predictor must exclude future input tokens containing its target. Exponentiating negative infinity contributes zero; softmax renormalizes the remaining scores, so dropping source 2 also changes earlier weights. At least one permitted source is needed in every scored row. Padding restrictions and the complete model’s input/output alignment come in Batch 4. The implementation is checked against is_causal=True in PyTorch attention. Source 1 is the current observation; its shifted target, if any, is at the next position.

Multiple heads use separate projection parameters

$$O^{[i]}=\operatorname{Attention}(XW_{Q,i}^\top,XW_{K,i}^\top,XW_{V,i}^\top)$$

Each head can form a different combination of source features.

Vaswani et al. (2017), §3.2.2.

The bracketed superscript [i] indexes a head; t indexes a query position and j a source position. N_heads=2 in this example and d_k=d_v=2 per head; neither the original RNN state width H nor vocabulary K is the head count. Separate parameters allow different representations, but do not force heads to learn different grammatical roles or guarantee that every head is useful. Heads project the full input width D=4; they are not restricted to disjoint slices of the input. The concatenation combines features within each query row.

Concatenation collects features within each query

Query position	Head 1 output	Head 2 output
0	$(1.2552, 0.9930)$	$(1.0000, 0.6044)$
1	$(0.7448, 1.7587)$	$(1.0000, 0.1978)$
2	$(1.0000, 1.4011)$	$(1.0000, 0.3333)$

$$[O^{[1]}\mid O^{[2]}]\in\mathbb R^{T\times(N_{\mathrm{heads}}d_v)}$$

$$[O^{[1]}\mid O^{[2]}]_{t=1}\approx(0.7448, 1.7587, 1.0000, 0.1978)$$

Two width-2 outputs concatenate to width 4 at every query position.

$O^{[1]}$ and $O^{[2]}$ are complete head-output matrices; their row t=1 is shown numerically below. The earlier subscript O_t names a query row. Bracketed head indices keep those roles distinct. W_O has shape D×(N_heads d_v)=4×4. Its entries in the ledger are [[1,0,0.5,0],[0,1,0,0.5],[0.5,0,1,0],[0,0.5,0,1]]. Head 2 has separate fixed projections, also saved in the ledger. The numeric row shows query t=1. Head 2 uses W_Q2=W_K1, W_K2=W_Q1, and W_V2=[[0,1,1,0],[1,0,0,-1]]. This is still the attention sublayer’s output, before residual addition or the position-wise MLP. The next batch traces those encoder operations. No output probability head is shown here.

The output projection mixes head features

$$W_O=\begin{bmatrix}1&0&0.5&0\\0&1&0&0.5\\0.5&0&1&0\\0&0.5&0&1\end{bmatrix}$$

$$\operatorname{MHA}(X)=[O^{[1]}\mid O^{[2]}]W_O^\top$$

$$\operatorname{MHA}(X)_{t=1}\approx(1.2448, 1.8576, 1.3724, 1.0771)$$

The projection mixes features within a row and returns width $D=4$.

Use the previous concatenated row, calculated at full precision. The first output coordinate is 0.7447652348 + 0.5 times 1 = 1.2447652348. The output projection applies independently to each query row; source positions were mixed inside the heads. W_O stores one output unit per row and has shape D by (N_heads d_v). This example happens to concatenate to D already, but the general output projection also restores D when N_heads d_v differs. These fixed weights illustrate feature mixing, without a trained semantic interpretation. Residual addition follows in Batch 4. Two clicks reveal the map and its numeric result.

Part 4 · Inside a Transformer encoder

1. Sequence problems and recurrent state
2. Learning through time and gated memory
3. Self-attention and contextual representations
4. Inside a Transformer encoder

Track the representation through one block, then through the stack.

One encoder block has two residual branches

Pre-norm teaching variant

Attention branch $$U=h^{(\ell)}+\operatorname{MHA}(\operatorname{LN}_1(h^{(\ell)}))$$

MLP branch $$h^{(\ell+1)}=U+\operatorname{MLP}(\operatorname{LN}_2(U))$$

Input, both additions, and output: $T\times D$.

Each branch computes a change and adds it to its own input.

Pre/post-norm distinction: Xiong et al. (2020).

TODO(instructor): Review the pre-norm teaching convention and diagram scale.

TODO(instructor): Review the complete encoder walkthrough and task boundaries in Batch 4.

This expands the earlier design sample. Trace the two orange bypasses with the stylus. The first adds the unnormalized h^(ell); the second adds U. LN_1 and LN_2 are separate modules. General attention has multiple heads; the numerical microscope below uses one head to make the entire block calculable. The original paper uses post-norm; its comparison is later in this section. Projection/MLP dropout is omitted in this deterministic example. These two clicks establish the architecture before enlarging its operations.

Attention, LayerNorm, and MLP act on different axes

Operation	What interacts?	Parameters across positions
Attention	Permitted source rows	Shared projection maps
LayerNorm	Features within one row	Shared scale and shift
MLP	Features within one row	Shared affine maps

Only attention introduces new interactions between positions inside this block.

The MLP input is already contextual because attention precedes it. Saying the MLP is position-wise does not make its input independent of the rest of the sequence. LayerNorm never averages across tokens or examples in this construction. Residual additions are coordinate-wise. This diagram is structural, not a measured attention pattern.

A complete block microscope

Three positions, $D=4$, one head with $d_k=d_v=2$. Fixed, untrained values.

$$h^{(\ell)}=\begin{bmatrix}1.0000 & 0.0000 & -1.0000 & 2.0000\\0.0000 & 2.0000 & 1.0000 & -1.0000\\2.0000 & 1.0000 & 0.0000 & 1.0000\end{bmatrix}$$

$$W_Q=\begin{bmatrix}1&0&0.5&0\\0&1&0&0.5\end{bmatrix}$$

$$W_K=\begin{bmatrix}0.5&0&1&0\\0&0.5&0&1\end{bmatrix}$$

$$W_V=\begin{bmatrix}1&0&0&0\\0&0&1&0\end{bmatrix}$$

Track row 0 through both branches; every row is computed.

Both LN modules: $\gamma=\mathbf 1$, $\beta=\mathbf 0$, $\epsilon=10^{-5}$. No attention bias or dropout.

Reuse the initial Batch 1 encoder input and parameters, now completing its MLP branch. This is a separate example from Batch 3’s unnormalized the/cat/sleeps projection example. h^(ell) is a given representation at one encoder depth, with no trained semantic claim. Epsilon=1e-5; each LN has gamma=(1,1,1,1), beta=(0,0,0,0). Attention is bidirectional. No projection biases, dropout, or added position vectors appear in this arithmetic microscope. MLP biases are explicit later. Every full-precision intermediate and parameter is in the Batch 4 ledger.

LayerNorm computes statistics within one token row

Mean and population variance across D features $$\mu_t=\frac1D\sum_{j=1}^{D}h_{t,j},\quad \sigma_t^2=\frac1D\sum_{j=1}^{D}(h_{t,j}-\mu_t)^2$$

Normalize, then apply learned feature scale and shift $$X_{t,j}=\gamma_j\frac{h_{t,j}-\mu_t}{\sqrt{\sigma_t^2+\epsilon}}+\beta_j$$

$\gamma,\beta\in\mathbb R^D$; one statistic pair per row.

LayerNorm uses the current row in both training and inference.

Ba, Kiros & Hinton (2016), Layer Normalization.

Here h abbreviates h^(ell) only in the expanded coordinate equations. The feature index j starts at one, matching the scalar-neuron convention; token positions t start at zero. This is not BatchNorm. Gamma/beta are shared across tokens, not recomputed statistics. Epsilon protects the denominator; with epsilon the normalized variance is slightly below one before affine scale/shift. For constant rows gamma/beta still apply and the normalized centered part is zero. LN_2 has the same definition but its own learned parameters and statistics from U.

LayerNorm numbers for all three rows

$$h^{(\ell)}_0=(1,0,-1,2)$$

$$\mu_0=0.5,\quad \sigma_0^2=1.25,\quad X_0\approx(0.4472, -0.4472, -1.3416, 1.3416)$$

Position	Mean	Population variance	Normalized row
0	0.5000	1.2500	$(0.4472, -0.4472, -1.3416, 1.3416)$
1	0.5000	1.2500	$(-0.4472, 1.3416, 0.4472, -1.3416)$
2	1.0000	0.5000	$(1.4142, 0.0000, -1.4142, 0.0000)$

Changing another token does not change this row’s LayerNorm statistics.

Do the centering live: (0.5,-0.5,-1.5,1.5). Divide by sqrt(1.25+1e-5). The third row has different statistics from the first two. This table is rounded only after full-precision computation. The input is not rescaled in the residual bypass. Distinguish the scalar variance sigma_0² from the value vector v_0 used in attention.

The normalized rows enter attention

$$Q_{\mathrm{att}}=XW_Q^\top,\quad K_{\mathrm{att}}=XW_K^\top,\quad V_{\mathrm{att}}=XW_V^\top$$

$$A=\operatorname{softmax}_{\mathrm{rows}}\!\left(\frac{Q_{\mathrm{att}}K_{\mathrm{att}}^\top}{\sqrt2}\right),\quad O=AV_{\mathrm{att}}$$

$$q_0\approx(-0.2236, 0.2236),\quad k_0\approx(-1.1180, 1.1180),\quad v_0\approx(0.4472, -1.3416)$$

$$A_{0,:}\approx(0.4176, 0.2545, 0.3279),\quad O_0\approx(0.5366, -0.9101)$$

The attention output has width 2; the residual input has width 4.

Return to the same query/source axes from Batch 3, now with X=LN_1(h). q_0 comes from row 0, but its weights and O_0 use every allowed source row. The numeric ledger contains every key/value row and the complete attention matrix, not only this highlighted row. This is one head, so the following output projection is 4 by 2. General multiple-head concatenation uses the same restore-width principle. Three clicks show projections, source mixing, and the computed row.

Project the attention output and add the bypass

$$W_O=\begin{bmatrix}1&0\\0&1\\0.5&0\\0&0.5\end{bmatrix}$$

$$B_{\mathrm{att}}=OW_O^\top\in\mathbb R^{T\times D}$$

$$B_{\mathrm{att},0}\approx(0.5366, -0.9101, 0.2683, -0.4550)$$

$$U_0=h^{(\ell)}_0+B_{\mathrm{att},0}\approx(1.5366, -0.9101, -0.7317, 1.5450)$$

Add the attention branch to $h^{(\ell)}$, with matching position and feature axes.

The initial h row, not normalized X, travels around the first branch. Calculate its first coordinate live: 1+0.5366122=1.5366122. The two extra projection rows expose how a width-two head becomes a width-four change. B_att names a branch result and avoids confusing it with batch size B. All four U coordinates remain representations, not output probabilities.

The MLP contains ordinary feedforward units

Every token uses the same network; no connection crosses token rows here.

Vaswani et al. (2017), §3.3.

Connect to Week 04 units and affine/nonlinear/affine composition. D=4 and F=3 are intentionally tiny; F need not be greater than D. Larger models often choose a wider inner representation. The nodes represent scalar features of one row, not four tokens. MLP biases enter each corresponding affine unit. This example uses ReLU; ViT’s GELU choice is stated at the patch bridge. The mathematical structure remains position-wise in either case.

Two affine maps surround the MLP activation

$$R=\operatorname{LN}_2(U)\in\mathbb R^{T\times D}$$

$$Z_{\mathrm{mlp}}=RW_1^\top+b_1,\qquad W_1\in\mathbb R^{F\times D},\quad b_1\in\mathbb R^F$$

$$J_{\mathrm{mlp}}=\phi(Z_{\mathrm{mlp}})\in\mathbb R^{T\times F}$$

$$B_{\mathrm{mlp}}=J_{\mathrm{mlp}}W_2^\top+b_2,\qquad W_2\in\mathbb R^{D\times F},\quad b_2\in\mathbb R^D$$

Bias vectors broadcast across rows; the output returns to width $D$.

Z_mlp is the preactivation matrix and J_mlp is its activated hidden matrix. They are not attention weights. Parameter rows correspond to output units, consistently with Week 04. The two affine maps cannot generally collapse to one linear map when a nonlinear activation separates them. Phi=max(0,z) in this example. Each encoder layer normally has its own W_1/W_2 even though positions within that layer share them.

The microscope fixes both MLP maps

$$W_1=\begin{bmatrix}1.0000 & 0.0000 & -0.5000 & 0.0000\\0.0000 & 1.0000 & 0.0000 & 0.5000\\0.5000 & -0.5000 & 0.5000 & 0.0000\end{bmatrix}$$

$$b_1=(0.0000, 0.1000, -0.1000)$$

$$W_2=\begin{bmatrix}0.5000 & 0.0000 & 0.2000\\0.0000 & 0.5000 & -0.2000\\0.3000 & -0.1000 & 0.1000\\-0.2000 & 0.2000 & 0.4000\end{bmatrix}$$

$$b_2=(0.0000, -0.0500, 0.0500, 0.0000)$$

Here $F=3$ and $\phi=\operatorname{ReLU}$; all parameters are illustrative.

Matrices are generated from the exact parameters in the ledger. W_1 is 3 by 4 and W_2 is 4 by 3. The hidden and output biases are different. These are not training results or recommended model settings. Leave room to calculate one unit manually on the next slide. Projection biases were omitted in attention, while these MLP biases are retained to connect to the Week 04 affine unit.

Normalize U and activate the MLP units

$$R_0\approx(0.9950, -1.0740, -0.9231, 1.0021)$$

$$Z_{\mathrm{mlp},0}\approx(1.4566, -0.4730, 0.4730)$$

$$J_{\mathrm{mlp},0}\approx(1.4566, 0.0000, 0.4730)$$

$$J_{\mathrm{mlp},0}=\max(0,Z_{\mathrm{mlp},0})$$

ReLU removes the negative preactivation in this row.

The generator recomputes LN_2 statistics from U, not from h. Gamma/beta equal one/zero here but are a separate module’s parameters. Calculate hidden unit one: R_0,1 -0.5 R_0,3=1.4566123. Unit two has negative preactivation and becomes zero. Three clicks reveal the normalized row, preactivations, and activated units. The other tokens use exactly the same maps but produce their own activations. None of the chosen preactivations is exactly zero, so the finite-difference derivative check avoids the ReLU kink.

Project the MLP units and add the second bypass

$$B_{\mathrm{mlp}}=J_{\mathrm{mlp}}W_2^\top+b_2$$

$$B_{\mathrm{mlp},0}\approx(0.8229, -0.1446, 0.5343, -0.1021)$$

$$h^{(\ell+1)}_0=U_0+B_{\mathrm{mlp},0}\approx(2.3595, -1.0547, -0.1974, 1.4428)$$

The second addition carries $U$ around the normalized MLP branch.

Calculate first branch coordinate: 0.5 times 1.4566123 +0.2 times 0.4729518=0.8228965. Add to U_0,1=1.5366122 to obtain 2.3595087. The second bypass does not carry the original h directly and does not add the normalized R. The block output is still T by D. A task head has not yet appeared.

All rows survive both residual additions

Position	Input $h^{(\ell)}_t$	After attention $U_t$	Block output $h^{(\ell+1)}_t$
0	$(1.0000, 0.0000, -1.0000, 2.0000)$	$(1.5366, -0.9101, -0.7317, 1.5450)$	$(2.3595, -1.0547, -0.1974, 1.4428)$
1	$(0.0000, 2.0000, 1.0000, -1.0000)$	$(0.5650, 0.9637, 1.2825, -1.5182)$	$(0.5963, 0.8824, 1.3481, -1.4556)$
2	$(2.0000, 1.0000, 0.0000, 1.0000)$	$(2.3129, 0.4733, 0.1565, 0.7366)$	$(3.4916, 0.3136, 0.9027, 0.5284)$

$$h^{(\ell)}\longrightarrow U\longrightarrow h^{(\ell+1)},\qquad T\times D\longrightarrow T\times D\longrightarrow T\times D$$

Encoder depth changes the representations while preserving their axes.

Original checked example. Complete parameters and intermediate values.

Optional depth for the timed route: Reference ledger after completing the highlighted row; no need to recalculate all rows live.

PyTorch LayerNorm, fused attention, affine maps, and ReLU independently reproduce the complete block. A separate scalar upstream objective checks derivatives with respect to every input entry against central finite differences. This is validation of this fixed example, not a full Transformer backward lecture. Rounding is applied only when displaying values. Revisit both bypass origins in the full map if a student adds X or R instead of h or U.

Residual addition supplies a direct derivative route

$\delta_{ab}=1$ when $a=b$, otherwise $0$; $t,r$ are positions, $j,i$ are features.

$$U=h+\mathcal F(h)$$

$$\frac{\partial U_{t,j}}{\partial h_{r,i}}=\underbrace{\delta_{tr}\delta_{ji}}_{\text{direct identity route}}+\frac{\partial\mathcal F(h)_{t,j}}{\partial h_{r,i}}$$

Bypass

The input can influence the output without passing through the branch’s transformations.

Branch

Gradients also follow the learned transformations and their cross-position dependencies.

An identity contribution helps preserve a route; it does not guarantee stable gradients.

Optional depth for the timed route: Coordinate derivative extension; the two bypass origins and additions remain essential.

Kronecker delta equals one when the paired indices match and zero otherwise. Here r is another position index, while i/j are feature coordinates. Mathcal F denotes the entire attention-after-LN branch, distinct from MLP hidden width F. This coordinate derivative avoids silently switching the course row-vector Jacobian convention. Branch contributions may amplify or cancel identity contributions. Residual connections do not remove the need to train the branch or solve all optimization failures.

Pre-norm and post-norm place LayerNorm differently

$$U=h+\mathcal F(\operatorname{LN}(h))$$

LayerNorm is inside the branch.

$$U=\operatorname{LN}(h+\mathcal F(h))$$

LayerNorm follows the addition.

The diagrams describe different computations, even with matching parameter shapes.

Original post-norm: Vaswani et al. (2017), §3.1. Comparison: Xiong et al. (2020).

The simplified branch mathcal F can mean attention or the position-wise MLP. Repeat the corresponding placement for both sublayers. The original paper also includes dropout around sublayer outputs; this schematic isolates normalization placement. Our main example omits dropout. Pre-norm has an uninterrupted bypass before final normalization; post-norm sends the sum through LN. Do not claim one placement universally wins or that pre-norm makes warm-up unnecessary in every training configuration.

Stack blocks, then choose a task head

$$h^{(\ell+1)}=\operatorname{Block}_{\ell}(h^{(\ell)}),\quad \ell=0,\ldots,N_{\mathrm{layers}}-1$$ $$h^{\mathrm{final}}=\operatorname{LN}_{\mathrm{final}}(h^{(N_{\mathrm{layers}})})$$

Sharing over positions does not imply sharing parameters over depth.

The two-block diagram is schematic, not an assertion that this course uses exactly two blocks. Each block normally has its own projection, MLP, and LN parameters. N_layers counts depth; T counts positions. Final LN is an explicit choice for this pre-norm stack, also used in the ViT construction discussed later. It is outside the individual block. Positional information enters h^(0), and the task head determines whether outputs are per-token or pooled. No head is implied merely by the term encoder.

Without position information, row permutations follow the input

$\Pi$ permutes matrix rows; $\pi(t)$ identifies the input row moved to position $t$.

$$\operatorname{Block}(\Pi h)=\Pi\operatorname{Block}(h)$$

$$h'_t=h_{\pi(t)}\quad\Longrightarrow\quad h'^{\mathrm{out}}_t=h^{\mathrm{out}}_{\pi(t)}$$

Equivariance

Reordering input rows reorders output rows when no position-dependent features or masks are present.

Pooling

A mean of those rows gives the same sequence representation after a permutation.

A bidirectional block needs a position signal to distinguish supplied order.

Pi is a permutation matrix and pi its row-index mapping. This statement assumes shared row maps, no position features, no position-dependent masks, and deterministic evaluation without a new dropout realization. Attention scores transform as Pi S Pi^T, row softmax as Pi A Pi^T, and outputs as Pi O. Row-wise LN/MLP and residuals preserve this property. A causal or positional mask breaks the stated unrestricted permutation symmetry. The complete numeric block passes the permutation check. Mean pooling is invariant; per-token output remains equivariant rather than unchanged.

Position vectors give rows a location signal

Even $D$; position-vector feature indices start at $0$. $r=0,\ldots,D/2-1$.

$$h_t^{(0)}=e_t+p_t^{\mathrm{pos}}$$

$$p_{t,2r}^{\mathrm{pos}}=\sin\!\left(\frac{t}{10000^{2r/D}}\right),\quad p_{t,2r+1}^{\mathrm{pos}}=\cos\!\left(\frac{t}{10000^{2r/D}}\right)$$

$$p^{\mathrm{pos}}_0=(0.0000, 1.0000, 0.0000, 1.0000),\quad p^{\mathrm{pos}}_1\approx(0.8415, 0.5403, 0.0100, 1.0000)$$

Move a token to another position and its initial representation changes.

Sinusoidal example: Vaswani et al. (2017), §3.5.

Optional depth for the timed route: Exact sinusoidal formula is optional; retain the need for order information from the preceding slide.

D is even here, r=0,…,D/2-1, and these position-vector coordinate indices start at zero to preserve the published even/odd pairing. Other feature-coordinate examples use one-based indices; state this translation. D=4 gives frequency divisors 1 and 100. The formula is one choice, not a requirement of every Transformer; learned position vectors are also possible. Permuting content while keeping row-position vectors fixed need not permute outputs. Moving content and its position vectors together is a different operation. This example does not guarantee extrapolation to arbitrary lengths.

Padding is a visibility and output-selection problem

$m_t=1$ for a real token and $0$ for padding.

$$M^{\mathrm{pad}}_{tj}=\begin{cases}0,&m_j=1,\\-\infty,&m_j=0,\end{cases}\qquad A=\operatorname{softmax}_{\mathrm{rows}}(S+M^{\mathrm{pad}})$$

Mask padded sources at every layer; omit padded outputs from loss and pooling.

m_j is one for a real input token and zero for padding. The shown input has three real tokens and one pad. A padded query may still produce nonzero output by reading real sources, so a source mask alone does not remove its loss/pooling contribution. Mask padded sources at every layer to prevent those outputs from contaminating later real queries. At least one source must remain allowed per scored row; completely empty sequences need separate handling. Adding a pad row with arbitrary values leaves the three real block outputs unchanged in the independent check.

Task heads select rows or combine them

Token classification

$$a_t=h_t^{\mathrm{final}}W_o^\top+b_o$$ $$p_t=\operatorname{softmax}(a_t)$$

One class distribution per real token.

Sequence classification

$$\bar h=\frac{\sum_t m_t h_t^{\mathrm{final}}}{\sum_t m_t}$$ $$a=\bar hW_o^\top+b_o$$

One distribution after pooling.

Both examples may use the complete input sequence when the task permits it.

These are illustrative head designs, not the only ones. Softmax of pooled logits produces a sequence class distribution. W_o has C output rows and D input columns; lowercase subscript o distinguishes the task head from attention projection W_O. A class token is another pooling design and is introduced in ViT. Mean pooling includes only real tokens and needs a positive denominator. Shared token-classification weights act on each row. Training losses must also select valid labels and padded queries. Bidirectional visibility is legitimate when full input is available for the chosen target.

Next-token prediction changes visibility and target alignment

$K$: vocabulary size. $\mathcal I$: positions with a valid next-token target.

$$y_t=s_{t+1},\qquad M^{\mathrm{causal}}_{tj}=\begin{cases}0,&j\le t,\\-\infty,&j>t.\end{cases}$$

$$a_t=h_t^{\mathrm{final}}W_o^\top+b_o,\qquad W_o\in\mathbb R^{K\times D}$$

$$\mathcal L=-\frac1{|\mathcal I|}\sum_{t\in\mathcal I}\log\operatorname{softmax}(a_t)_{s_{t+1}}$$

Position $t$ may read $s_0,\ldots,s_t$ while predicting the unseen $s_{t+1}$.

Mathcal I contains positions with a valid next-token target; for a length-T sequence without appended EOS it is 0,…,T-2. K is vocabulary size, not attention keys. Padded targets are excluded. Causal and padding restrictions combine before softmax; apply restrictions at every attention layer. These equations describe a causal self-attention stack for autoregressive prediction, not the bidirectional encoder used for the main microscope. A full encoder-decoder translation model would additionally use cross-attention, outside this lecture batch. Final LN and MLP do not restore hidden future information when causal visibility is respected.

Student question 5 · Which operation mixes positions?

Which operation mixes source positions inside one block?

If row $t$ predicts $s_{t+1}$, which sources must be hidden?

Attention mixes positions. Hide sources $j>t$ at every layer; keep the target shifted to $s_{t+1}$.

Pause before revealing. Ask a student to identify the LN/MLP feature axis on the architecture map. If they answer that MLP mixes positions, revisit the diagram of four-feature units within one row. If they hide the current token, distinguish the observed s_t from target s_(t+1). For a padded batch also exclude padded sources and targets. This is a conceptual classroom check, not assessed quiz material.

Vision Transformer: image patches become token rows

$$N_{\mathrm{patch}}=\frac{H_{\mathrm{img}}W_{\mathrm{img}}}{P^2},\quad x_t^{\mathrm{patch}}\in\mathbb R^{P^2C_{\mathrm{img}}},\quad e_t=x_t^{\mathrm{patch}}W_{\mathrm{patch}}^\top$$

Patch embeddings change the input; the encoder still uses width $D$.

Dosovitskiy et al. (2021), §3.1.

The schematic grayscale image is 6 by 6 by 1 with four 3 by 3 patches. Flatten patches in raster order; nine pixel features project to D=4 in the checked example. Prepend a learned class-token embedding, then add a position vector to every input row, giving T=N_patch+1=5. Class-token output after final LN feeds the classifier. Original ViT uses pre-norm and GELU in its MLP; our arithmetic example used ReLU. These are explicitly different activation choices, not an assertion that every Transformer equals the microscope. Image dimensions must be divisible by P for this nonoverlapping example. The drawing is original, not copied from the paper. No comparison of trained performance is asserted.

Full attention connects positions at a quadratic arithmetic cost

$$S,A\in\mathbb R^{T\times T},\qquad QK^\top:\ O(T^2d_k),\quad AV:\ O(T^2d_v)$$

$$T\longrightarrow2T\quad\Rightarrow\quad T^2\longrightarrow4T^2$$

Connection structure	Consequence within one layer
Recurrent state chain	State updates depend on earlier updates
Full self-attention	Query rows can be computed in parallel

Short interaction paths trade against the cost of considering all source pairs.

Structural comparison: Vaswani et al. (2017), §4.

Optional depth for the timed route: Optional one-minute comparison; keep task visibility and autoregressive alignment in the main route.

The displayed costs isolate score formation and value aggregation per head, excluding Q/K/V projections and the MLP. Total attention also includes O(T D²) projection terms for usual widths; MLP costs depend on D and F. Materializing each head’s score/weight matrices uses quadratic space, but fused/tiled attention implementations can avoid storing the full matrices while full-pair arithmetic remains quadratic. Do not claim every implementation must allocate T² storage or that attention is faster for every sequence length. Parallel processing of a known input sequence does not make autoregressive generation of unseen tokens parallel. No new computational library is added.

Two ways to build a contextual representation

Question	Recurrent state	Transformer block
How do positions interact?	Shared state update along the sequence	Input-dependent mixtures of permitted source values
What is shared?	Cell parameters across positions	Projection and MLP parameters across positions
How does learning reach parameters?	Backpropagation through all uses of the cell	Backpropagation through projections, mixing, row maps, and residual branches
What determines visibility?	Direction of the state computation	Input availability and the attention mask

Representation, task head, target alignment, and visibility must agree.

Close the conceptual loop with the lecture’s opening sequence tasks. Recurrence is not intrinsically causal in every architecture: a bidirectional recurrent model has both directions. The current RNN microscope is left-to-right. Transformer blocks do not automatically define language generation; a task head and appropriate visibility must be specified. Ask students to mentally trace one input coordinate into a contextual representation, without adding a sixth formal classroom question.

Selected milestones in sequence learning

1997 · Gated recurrent storage

Hochreiter and Schmidhuber introduce LSTM.

Original paper

2003 · Learned word representations

Bengio, Ducharme, Vincent, and Jauvin jointly learn word vectors and a neural language model.

JMLR paper

2014/2015 · Attention with recurrence

Bahdanau, Cho, and Bengio learn soft alignment for neural translation.

2014 preprint; ICLR 2015

2017 · Attention without a recurrent state chain

The eight-author Transformer team builds sequence transduction around attention.

Attention Is All You Need

The Transformer combines a sequence of advances in representation and information access.

Selected milestones, not an exhaustive priority history. The 1997 LSTM paper is not identical to every modern LSTM variant: later modifications include the forget gate shown in our brief modern cell diagram. The neural language model in 2003 is a fixed-context feedforward model, not a Transformer or an invention of every distributed representation. Bahdanau et al. appeared as a 2014 preprint and at ICLR 2015; both dates are intentional. Its attention reads a recurrent encoder for translation, distinct from Transformer self-attention. All names receive collaborator credit.

The Transformer was a team contribution

Ashish Vaswani · Noam Shazeer · Niki Parmar · Jakob Uszkoreit
Llion Jones · Aidan N. Gomez · Łukasz Kaiser · Illia Polosukhin

Contribution recorded in the paper	Researchers
Proposed replacing RNNs with self-attention	Jakob Uszkoreit
Designed and implemented the first Transformer models	Ashish Vaswani with Illia Polosukhin
Proposed scaled dot products, multiple heads, and the fixed positional representation	Noam Shazeer

The paper declares equal contribution and randomized author order.

Original author contribution note.

These selected roles connect directly to the architecture studied; they are not a ranking of the eight contributors. The same note credits Niki Parmar’s model design/evaluation, Llion Jones’s variants/code/inference/visualizations, and Łukasz Kaiser and Aidan Gomez’s tensor2tensor work. Credit the complete team. Bengio is the person spotlight for the earlier learned-representation and attention lineage, not an author of this Transformer paper.

Person of the day: Yoshua Bengio

A pioneer of learned representations for language

2003 · Learn representations with the task

Word vectors and next-word probabilities are learned together with Ducharme, Vincent, and Jauvin. Paper

2014/2015 · Learn which source information to read

With Bahdanau and Cho, soft alignment becomes part of neural translation. Paper

2018 · ACM A.M. Turing Award

Shared with Geoffrey Hinton and Yann LeCun for breakthroughs in deep neural networks. ACM

Learn useful vectors, then learn how context should shape them.

The instructor selected Bengio and supplied this illustration after reviewing the main lecture. An informal “godfather” nickname is not an exclusive inventor claim; the slide emphasizes concrete work with named collaborators. The award is the 2018 award, announced in 2019. His 2003 model jointly learned a distributed word representation and a probability function in a fixed context. The attention paper has Bahdanau, Cho, and Bengio as coauthors. Bengio is not one of the eight authors of Attention Is All You Need. No current affiliation or company role is asserted. The instructor supplied the portrait; do not describe it as a documentary photograph.

Historical review marker retained for the authoring record: TODO(instructor): Select the historical spotlight and review the final synthesis in Batch 5. The instructor’s finalization request selects Bengio and authorizes this ending.

References · recurrent learning and language representations

• Goodfellow, Bengio & Courville (2016). Deep Learning, Chapter 10: Sequence Modeling.
• Hochreiter & Schmidhuber (1997). Long Short-Term Memory.
• Bengio, Ducharme, Vincent & Jauvin (2003). A Neural Probabilistic Language Model.
• Pascanu, Mikolov & Bengio (2013). On the difficulty of training recurrent neural networks.
• Bahdanau, Cho & Bengio (2015; preprint 2014). Neural Machine Translation by Jointly Learning to Align and Translate.
• PyTorch documentation. GRU and LSTM cell equations.

References · attention and Transformer encoders

• Vaswani et al. (2017). Attention Is All You Need.
• Ba, Kiros & Hinton (2016). Layer Normalization.
• Xiong et al. (2020). On Layer Normalization in the Transformer Architecture.
• Dosovitskiy et al. (2021). An Image is Worth 16×16 Words.
• ACM SIGAI (2019). Announcement of the 2018 Turing Award recipients.