Week 01 · From a Learning Problem to a Fitted Model
Central question. What does it mean for a model to learn, and how can a fitted line help us distinguish prediction, optimization and generalization?
Scope. Linear regression, mean squared error (MSE), gradient descent and honest evaluation are the core. The “3 and 7” tutorial supplies a short case study of representations. The Gaussian-likelihood derivation of MSE belongs to Week 07.
A sensor returns a voltage, but the experiment needs a displacement in millimetres. Can a few paired readings teach us how to convert one into the other? Fitting a line will give us a concrete calculation. Deciding whether that line is useful will require a more careful question: useful for which future readings, and supported by what evidence?
Learning outcomes and prerequisites
After reading this chapter, readers should be able to:
- specify experience, task and performance measure, and locate supervised, unsupervised and reinforcement learning;
- distinguish inputs, representations, targets, predictions, parameters and hyperparameters;
- compute linear predictions, residuals, MSE and one simultaneous gradient update;
- separate training, validation-based model selection and final test evaluation;
- explain why a representation changes what a simple classifier can learn.
Prerequisites: elementary algebra, vectors and scalar derivatives. We will introduce matrix notation after a scalar example; probability theory is not required.
Companions: lecture resources, tutorial, Lab 1 instructions, and the syllabus.
Notation contract
| Symbol | Meaning and convention |
|---|---|
| \(n,d\); \(i,j\) | Number of observations and features; observation and feature indices |
| \(x_i\in\mathbb R^{d\times1}\), \(y_i\), \(\widehat y_i\) | One input column vector, observed scalar target, predicted scalar target; for one feature we write the scalar \(x_i\) |
| \(X\in\mathbb R^{n\times d}\) | Row \(i\) is \(x_i^\top\); the stacked targets and predictions \(y,\widehat y\) have shape \((n,1)\) |
| \(w\in\mathbb R^{d\times1}\), \(b\), \(\theta=(w,b)\) | Weights, scalar intercept and collected fitted parameters; \(w\) is scalar in the one-feature example |
| \(r_i=\widehat y_i-y_i\) | Prediction-minus-target residual; retain this sign in derivatives |
| \(L_i=r_i^2\), \(\bar L=n^{-1}\sum_i L_i\) | Squared error and mean squared error; \(L=\sum_i L_i\) denotes a sum when needed |
| \(\eta\), \(g(x)\) | Positive learning rate and an explicitly named feature map |
| \(k\), \(\theta^{(k)}\) | Update index, starting at zero, and parameters after \(k\) simultaneous updates; \((k)\) is an index, not a power |
Here \(\mathbb R\) denotes real numbers; a superscript \(\top\) transposes a column into a row. In the calibration example, \(x\) is measured in volts (V), \(y\) and \(\widehat y\) in millimetres (mm), \(w\) in mm/V and \(b\) in mm. Residuals have units mm and MSE has units mm². The learning rate below operates on numerical values in these declared coordinates; changing units changes the appropriate update scale.
The original PDF’s parameter symbol \(\phi\) is translated to \(\theta\); its prediction \(h\) is written \(\widehat y\). We keep the intercept explicit; an equivalent augmented-input convention will be explained with the linear model. In the digit tutorial, an embedding called \(z\) is a representation, not the classification logit introduced in Week 02. Reserve \(\phi\) for activations and \(h^{(\ell)}\) for later network representations. Later chapters sometimes write \(L\) for a generic objective; here the bar in \(\bar L\) explicitly identifies a mean rather than a sum.
What would count as learning?
Experience, task, performance and intended use
Imagine calibrating a displacement sensor. For each calibration observation we record the sensor’s voltage and a reference measurement of displacement. Later, the reference instrument will be absent: the voltage must supply a prediction of the displacement. This is regression, a task whose output is a numerical quantity rather than a class label.
There are several ways to construct the conversion. A mechanistic model might start from the sensor’s physical design. A fitted model instead chooses a relationship using observed input–target pairs. Physical knowledge can guide that choice, and the two approaches can be combined. Both may run as fully deterministic computer programs; determinism does not distinguish them.
To describe what learning would accomplish, we use Mitchell’s experience–task– performance formulation (Mitchell 1997, chap. 1). It asks us to identify the experience available to a program, the task it performs, and the measure by which improvement will be assessed. For our proposed calibration system:
| Part of the learning problem | Calibration interpretation |
|---|---|
| Experience \(E\) | Paired voltage and reference-displacement measurements available for fitting |
| Task \(T\) | Predict a reference displacement from a new voltage |
| Performance measure \(P\) | Mean squared prediction error on fresh reference pairs, with lower values better |
| Intended scope \(S\) | The same sensor, voltages within 0–2 V, and comparable operating conditions |
Scope is the course’s addition to the three-part formulation. It names where we intend to use the result; it requires evidence of its own. A small error on one sensor does not establish performance on another sensor, after a temperature change, or outside the measured voltage range. Even within the stated scope, the fitting data alone cannot establish a reliable error rate.
Notice that “use linear regression” is absent from the task definition. It is one proposed method of solving the problem. We can compare methods only after deciding what prediction they must produce and how to judge it.
Where do the target values come from?
The calibration pairs make this supervised learning: each training input comes with a target. A target is the recorded quantity we ask the prediction to match. It need not be a perfect physical truth; a reference instrument can also have measurement error. We will not infer an unobserved true displacement or a noise distribution from our miniature table.
In unsupervised learning, there is no supplied target for each input; we might group sensor traces by their patterns. In reinforcement learning, an agent chooses actions and learns from their consequences and rewards; for example, a controller might choose motions while trying to maintain a desired displacement. These settings describe the available feedback. Regression describes the required output. The two distinctions answer different questions about the learning problem.
Three pairs we can follow by hand
Table 1 lists a deliberately tiny synthetic training example. These numbers were chosen for explanation; they are not readings from a real instrument. Each row is one observation. The voltage is the single feature available when making a prediction; the reference displacement is the target available during fitting and later evaluation.
| ID | Input \(x_i\) (V) | Target \(y_i\) (mm) | Role |
|---|---|---|---|
| S1 | 0 | 1 | Training |
| S2 | 1 | 2 | Training |
| S3 | 2 | 2 | Training |
For now the representation is simply the measured voltage itself. A representation is the numerical description passed to the model; adding \(x^2\) or a recent voltage history would change that description. We will first fit the line \(\widehat y=wx+b\). Its slope \(w\) and intercept \(b\) are parameters: the fitting procedure changes them using the training pairs. The learning rate and number of updates are hyperparameters: choices that control that procedure. Choosing a representation or a model family is also part of model selection, guided by validation evidence.
We can already ask a useful question: S2 and S3 have the same recorded target but different voltages. Must a useful line pass through every point? The answer will depend on how we measure error across the observations, not on whether one particular point looks convincing.
Data roles before model choice
A project starts by defining the intended prediction and collecting suitable measurements. Before fitting, we decide which observations may influence the model and which will supply evidence about future use. The roles are:
| Data role | What it may influence | What its error tells us |
|---|---|---|
| Training | Model parameters and any feature means or scales learned from training inputs | How well the system matches cases used to fit it |
| Validation | Choice of features, model family and training settings | How well those choices work on new development cases excluded from fitting |
| Test | Final assessment after development choices are fixed | Evidence on cases kept out of development |
The three rows in Table 1 all have the training role. We keep them together so every prediction and update is easy to inspect. They do not constitute three evaluation sets. A real calibration study would need additional measurements and a split appropriate to the intended use. If deployment is on later recording sessions, holding out whole sessions may be more informative than mixing neighboring readings into a random split.
Suppose we compare the raw voltage with a polynomial representation. We fit each candidate on training data, then use validation data to decide between them. Once the representation and training settings are fixed, the test set provides the final assessment. Trying many variants against the test set would let that set influence the method; its score would lose the intended independence from development. Repeated validation choices can overfit the validation set too, which is one reason to preserve a separate final test.
Preprocessing follows the same boundary. If a feature is standardized, its mean and scale are calculated from training observations. Validation and test inputs receive that already-fitted transformation. Calculating the mean from all inputs first lets held-out observations influence the fitted system. Another, more direct form of leakage would be to include the reference instrument’s reading among the inputs even though it will be absent during use. Both examples break the correspondence between the development procedure and the prediction system we intend to evaluate.
A calibration table contains voltage, reference displacement and recording session ID. The future system will receive voltage only. Which columns may be used as prediction inputs, which supplies the target, and which could help define an evaluation split? Explain your choices before selecting a model.
The workflow now has a purpose: define the claim, collect and separate the data, choose a representation and model, fit parameters, select settings with validation, and assess the chosen system on reserved test observations. Deployment can reveal new conditions that require monitoring and another development cycle. A falling training error is one useful event inside this workflow; the final claim concerns performance beyond the fitting table.
A line makes predictions
One feature: slope and intercept
Return to the calibration pairs in Table 1. We want a rule that accepts a voltage and returns a displacement prediction. Our first model is
\[ \widehat y_i=wx_i+b. \]
The slope \(w\) says how much the prediction changes when the input increases by one volt. The intercept \(b\) is the prediction at zero volts. Multiplying a voltage in V by a slope in mm/V gives mm, so both \(wx_i\) and \(b\) have the same units as the target. This check is part of understanding the model, not merely a bookkeeping detail.
Strictly, a nonzero intercept makes this an affine function of the input. We retain the usual name linear regression. Fixing \(b=0\) would constrain the line to pass through the origin; the measurement problem has not given us a reason to impose that constraint.
Once \(w\) and \(b\) are fixed, the line can predict at any supplied voltage. The ability to calculate a number does not supply evidence for that number: using the rule outside the intended calibration range is extrapolation.
Keep one tiny dataset throughout
Before learning parameters, compare three candidate settings. These are chosen examples, not three steps of an optimizer.
| Line | \(w\) | \(b\) | Prediction |
|---|---|---|---|
| A | 0 | 0 | \(\widehat y=0\) |
| B | 1 | 0 | \(\widehat y=x\) |
| C | 0 | 1 | \(\widehat y=1\) |
Candidate A always predicts zero. B increases with voltage. C always predicts one millimetre. For B at observation S2, substituting the input gives
\[ \begin{aligned} \widehat y_2&=1\cdot1+0=1\ \mathrm{mm},\\ y_2&=2\ \mathrm{mm}. \end{aligned} \]
Its residual is the prediction-minus-target difference,
\[ r_2=\widehat y_2-y_2=1-2=-1\ \mathrm{mm}. \]
A negative residual means the prediction is below its target; a positive residual means it is above. The magnitude gives the vertical gap at that same input. It is not the shortest perpendicular distance from the point to the line: this task predicts a target from a supplied input.
The introduction lecture writes its residual as target minus prediction. This chapter uses prediction minus target. The values differ by a minus sign, while their squares are identical. We will retain the chapter’s sign when we differentiate the objective.
For the three training observations, B’s calculation is now fully visible:
| ID | \(y_i\) | \(\widehat y_i\) | \(r_i\) | \(r_i^2\) |
|---|---|---|---|---|
| S1 | 1 | 0 | \(-1\) | 1 |
| S2 | 2 | 1 | \(-1\) | 1 |
| S3 | 2 | 2 | 0 | 0 |
Targets, predictions and residuals are in mm; the last column is in mm². B matches S3 exactly but misses S1 and S2. A rule for comparing models must account for all three rows, rather than select whichever row looks best.
MSE turns residuals into a fitting objective
Combine errors without cancelling their signs
A signed average can hide large errors: residuals of \(+1\) and \(-1\) mm average to zero although neither prediction is correct. One way to measure the size of each error is to square it. Define the per-observation squared-error loss, its sum, and its mean:
\[ \begin{aligned} L_i&=r_i^2,\\ L&=\sum_{i=1}^{n}L_i, \qquad \bar L=\frac{L}{n}. \end{aligned} \]
The mean \(\bar L\) is the mean squared error, or MSE. For our line, substitute \(r_i=wx_i+b-y_i\) to obtain the fitting objective
\[ \bar L(w,b)=\frac1n\sum_{i=1}^{n}(wx_i+b-y_i)^2. \]
Now a parameter choice determines predictions, the predictions determine residuals, and the residuals determine one loss value. We choose MSE here as an explicit fitting objective. The probability-model derivation comes later in Week 07.
For B, the last column of the residual table sums to \(2\ \mathrm{mm}^2\). Dividing by the three observations gives
\[ \bar L_B=\frac{1+1+0}{3}=\frac23\ \mathrm{mm}^2. \]
The same computation compares all three candidates:
| Line | Residuals (mm) | \(L\) (mm²) | \(\bar L\) (mm²) |
|---|---|---|---|
| A | \((-1,-2,-2)\) | 9 | 3 |
| B | \((-1,-1,0)\) | 2 | \(2/3\) |
| C | \((0,-1,-1)\) | 2 | \(2/3\) |
B and C tie under this objective. They make different predictions and match different observations exactly. A single aggregate score does not preserve that detail; inspecting individual residuals remains useful.
A loss contour collects parameter settings with equal error
Figure 1 shows the same three settings in two spaces. In data space the axes are input and output, and a model is a line. In parameter space the axes are slope and intercept, and that entire line becomes one point. The observations stay fixed while candidate parameters vary.
A contour joins parameter settings with the same value of the objective. For this exact training table, the loss surface is
\[ \begin{aligned} \bar L(w,b)&=\frac13\Big[(b-1)^2\\ &\quad+(w+b-2)^2\\ &\quad+(2w+b-2)^2\Big]. \end{aligned} \]
Each squared term comes from one row of the calibration table. The contour labels are MSE values in mm², not displacement measurements or probabilities. B and C occupy different points on the \(2/3\) contour. A lies on the contour with value 3. The inner contours show that other choices can improve on all three candidates. Finding such choices is the next task.
What changes between candidates is the parameter setting and therefore the predictions. The training pairs, residual convention and objective stay fixed. The resulting comparison establishes a difference in training fit; it supplies no held-out evaluation of future sensor readings.
Two parameter settings lie on the same MSE contour. Must they predict the same value for every input? Must their largest residuals occur at the same training observation? Use B and C to explain what the common loss retains and what it loses.
Squaring makes some errors matter more
A residual of \(2\) mm contributes \(4\) mm², while a residual of \(1\) mm contributes \(1\) mm². Doubling a residual’s magnitude quadruples its contribution to the sum. The contribution to the mean is divided by the same fixed \(n\), so it too quadruples. This does not mean the whole dataset’s MSE must quadruple when only one residual changes.
Large residuals can therefore dominate MSE. An unusual measurement may be a recording mistake, a rare valid case or evidence that the model family is inadequate. A large loss alone cannot decide among those explanations. Inspect the data and the intended task before changing the objective or discarding an observation. Squared error is a modeling choice, not a guarantee that every important kind of mistake receives an appropriate cost.
Mean and sum have the same parameter minimizers on a fixed nonempty dataset: \(\bar L=L/n\) multiplies the objective by a positive constant independent of \(w,b\). Their gradients and numerical values differ by that factor. Some texts use half-MSE, \(\tfrac12\bar L\), to simplify derivatives. This chapter uses full MSE; the factor will remain explicit in the update calculation.
MSE can also serve as an evaluation measure. Its role depends on which data we measure it on and which decisions it may influence. Training MSE is the objective fitted here. Validation MSE can guide development, and test MSE can assess the chosen system on reserved cases. The shared formula does not make those data roles interchangeable. If a result is reported as root mean squared error, \(\operatorname{RMSE}=\sqrt{\bar L}\), it has units mm again; it is not the same numerical quantity as MSE.
Several features: one row per observation
The scalar line extends by adding weighted contributions from several input features. For one observation, write the features and weights as columns, \(x_i,w\in\mathbb R^{d\times1}\). Their dot product is a scalar:
\[ \widehat y_i=w^\top x_i+b =\sum_{j=1}^{d}w_jx_{ij}+b. \]
Here \(x_{ij}\) is feature \(j\) of observation \(i\). Each \(w_j\) has the units needed to convert its own feature into the target’s units. We collect observations in rows of \(X\in\mathbb R^{n\times d}\) and stack predictions in a column:
\[ \widehat y=Xw+b\mathbf1, \qquad \widehat y\in\mathbb R^{n\times1}. \]
The vector \(\mathbf1\) contains \(n\) ones. Multiplying the scalar intercept by it adds the same \(b\) to every prediction. The row \(x_i^\top\) in \(X\) produces \(x_i^\top w\), equal to the scalar \(w^\top x_i\) above. Thus rows index observations and columns index features; no observation is summed into another.
The residual column is \(r=\widehat y-y\), also of shape \((n,1)\). The same squared-error objective can be written compactly as
\[ \bar L=\frac1n r^\top r =\frac1n\lVert Xw+b\mathbf1-y\rVert_2^2. \]
The notation \(\lVert r\rVert_2^2=r^\top r=\sum_i r_i^2\) means the sum of squared entries. It introduces no new loss. In an array implementation, keep targets and predictions in the same shape: mixing a prediction column \((n,1)\) with a target array \((n,)\) can broadcast into an unintended \((n,n)\) array of pairwise differences.
The older lecture notation can also absorb the intercept into the matrix. Define the augmented row \(\widetilde x_i^\top=(1,x_i^\top)\) and parameter column \(\widetilde w=(b,w^\top)^\top\). Then \(\widetilde X\in\mathbb R^{n\times(d+1)}\) and \(\widehat y=\widetilde X\widetilde w\). The leading column of ones accounts for the intercept; it is not another measured feature. We keep \(b\) separate so its role stays visible.
Linear in the parameters can mean curved in the input
We can change the representation before applying the weighted sum. For the single voltage \(x\), choose a fixed feature map
\[ \begin{aligned} g(x)&=\begin{bmatrix}x\\x^2\end{bmatrix},\\ \widehat y&=w_1x+w_2x^2+b. \end{aligned} \]
The prediction can curve as \(x\) changes. It is still linear in its coefficients (with an explicit intercept): the features \(x\) and \(x^2\) are fixed while the coefficients vary. A coefficient does not multiply another fitted coefficient or appear inside a nonlinear operation. The choice to include \(x^2\) changes the model family; merely adjusting \(w\) in the original straight-line model cannot introduce curvature (Bishop 2006, chap. 3).
For the same three inputs, the transformed matrix and an illustrative coefficient setting give
\[ X_g=\begin{bmatrix}0&0\\1&1\\2&4\end{bmatrix}, \]
\[ w=\begin{bmatrix}3/2\\-1/2\end{bmatrix},\qquad b=1, \]
\[ X_gw+b\mathbf1 =\begin{bmatrix}1\\2\\2\end{bmatrix}=y. \]
In this display we substitute the numerical voltages in V. The first weight has units mm/V, the second mm/V², and the intercept mm. These coefficients illustrate a member of the richer family; we have not run a fitting algorithm to obtain them. Zero training MSE on three pairs establishes that the model can interpolate those pairs. It does not establish that its curved predictions will be better on new measurements. Choosing whether that flexibility is useful requires the validation evidence introduced earlier.
Derivatives tell us how to update the line
Differentiate the operations that produced the loss
The contour picture lets us compare parameter settings, but it does not yet provide a method for finding better ones. A gradient collects the partial derivatives of the objective: how its value responds to a small change in each parameter while the other parameters stay fixed.
For one residual \(r_i=wx_i+b-y_i\), the inputs and targets are fixed during fitting. The parameter derivatives of the residual are
\[ \frac{\partial r_i}{\partial w}=x_i, \qquad \frac{\partial r_i}{\partial b}=1. \]
Apply the chain rule to its squared loss: differentiating the square gives \(2r_i\), and differentiating the residual supplies the remaining factor.
\[ \frac{\partial L_i}{\partial w}=2r_ix_i, \qquad \frac{\partial L_i}{\partial b}=2r_i. \]
The dataset objective is the mean of these per-observation losses. Summation and division by the fixed count \(n\) give
\[ \begin{aligned} \frac{\partial\bar L}{\partial w} &=\frac2n\sum_{i=1}^{n}r_ix_i,\\ \frac{\partial\bar L}{\partial b} &=\frac2n\sum_{i=1}^{n}r_i. \end{aligned} \]
The intercept derivative adds the signed residuals. The slope derivative weights each residual by its input. S1, whose input is zero, therefore contributes to the intercept derivative but not to the slope derivative. Its loss is still included in the objective. The factor 2 belongs to our choice of full MSE; half-MSE would cancel it.
For the two scalar parameters, the gradient is the column \(\nabla_\theta\bar L=(\partial\bar L/\partial w, \partial\bar L/\partial b)^\top\). To interpret its direction, take a small parameter change \(\Delta\theta\). The first-order loss change is approximately \(\nabla_\theta\bar L^\top\Delta\theta\). Choosing \(\Delta\theta=-\eta\nabla_\theta\bar L\) makes this approximation \(-\eta\lVert\nabla_\theta\bar L\rVert_2^2\), which is negative when the gradient is nonzero and \(\eta>0\). This is a local argument: neglected higher-order terms matter when the step is large.
Take one simultaneous update
Gradient descent uses that direction to update the parameters. The parenthesized superscript \((k)\) labels the number of updates already taken. At each iteration, first compute predictions, residuals and both derivatives at \(w^{(k)},b^{(k)}\); then apply
\[ \begin{aligned} w^{(k+1)}&=w^{(k)} -\eta\left.\frac{\partial\bar L}{\partial w}\right|_{\theta^{(k)}},\\ b^{(k+1)}&=b^{(k)} -\eta\left.\frac{\partial\bar L}{\partial b}\right|_{\theta^{(k)}}. \end{aligned} \]
The vertical bar means “evaluate at the indicated parameters.” Both derivatives use the same pre-update state. Recomputing the intercept derivative after changing \(w\) would implement a different procedure.
Start from \(w^{(0)}=b^{(0)}=0\), the same line as candidate A in Figure 1. Its residuals are \((-1,-2,-2)\) mm. The slope’s weighted residual sum is \((-1)0+(-2)1+(-2)2=-6\); the intercept’s residual sum is \(-1-2-2=-5\). Substituting those sums gives
\[ \begin{aligned} \left.\frac{\partial\bar L}{\partial w}\right|_{\theta^{(0)}} &=\frac23(-6)=-4,\\ \left.\frac{\partial\bar L}{\partial b}\right|_{\theta^{(0)}} &=\frac23(-5)=-\frac{10}{3}. \end{aligned} \]
These are derivatives with respect to numerical coordinates in the declared V/mm units. With learning rate \(\eta=0.1\), the two updates are
\[ \begin{aligned} w^{(1)}&=0-0.1(-4)=\frac25,\\ b^{(1)}&=0-0.1\left(-\frac{10}{3}\right)=\frac13. \end{aligned} \]
The new line predicts \((1/3,11/15,17/15)\) mm. Its residuals are \((-2/3,-19/15,-13/15)\) mm, giving
\[ \begin{aligned} \bar L^{(1)}&=\frac13\left[ \frac49+\frac{361}{225}+\frac{169}{225}\right]\\ &=\frac{14}{15}\ \mathrm{mm}^2. \end{aligned} \]
Thus the first step lowers training MSE from \(3\) to \(14/15\) mm². The parameters moved because of derivatives of the full training objective, not because we selected one of candidates B or C by inspection.
Watch the fit and the parameter point move together
Figure 2 replays the same calculation for 120 updates. The first three seconds hold the initial state; the following four seconds hold the first updated state. Later frames show the remaining recorded updates, ending with a longer pause on the final state. Use the native player controls to play, pause or replay; the video is silent and starts only when requested.
The line starts at zero, moves to slope 0.4 and intercept one third after the first update, and approaches slope 0.5 and intercept 1.167 during the finite run. The matching point moves across fixed MSE contours toward lower loss. Its path bends: the slope need not change monotonically while slope and intercept adjust together. Both panels and the readout use the same recorded parameters.
The following static sequence preserves the initial, first and final states for reading without motion and for print.
The numerical readout is also available as a compact ledger. All values below use the same full-MSE objective; decimals are rounded to six places.
| \(k\) | \(w\) (mm/V) | \(b\) (mm) | MSE (mm²) |
|---|---|---|---|
| 0 | 0.000000 | 0.000000 | 3.000000 |
| 1 | 0.400000 | 0.333333 | 0.933333 |
| 120 | 0.500387 | 1.166130 | 0.055556 |
Keep the learning rate separate from the direction
The gradient specifies a local direction; the learning rate scales the step. Using \(\eta=0.6\) from the same initial state gives \(w^{(1)}=2.4\) and \(b^{(1)}=2\). The new training MSE is \(149/15\approx9.933333\) mm², larger than the starting value of 3. Following the negative gradient with an overly large step can therefore increase the actual objective despite the first-order argument.
The animation’s \(\eta=0.1\) decreases MSE for this particular dataset and coordinate system. Changing voltage units or feature scales changes derivative magnitudes and the appropriate step size. A very small rate can make progress slow within a fixed update budget. The rate and the number of updates are choices to make and check, not universal constants. Even a well-behaved finite trace does not prove exact convergence or useful predictions on new cases.
The slope initially rises above its final recorded value while the training loss keeps falling. Why is it insufficient to judge progress by the slope alone? Relate your answer to the simultaneous change in intercept and the position on the loss contours.
The matrix gradient collects the same contributions
For \(d\) features, the partial derivative of residual \(r_i\) with respect to weight \(w_j\) is feature value \(x_{ij}\). Therefore
\[ \frac{\partial\bar L}{\partial w_j} =\frac2n\sum_{i=1}^{n}r_ix_{ij}. \]
The \(j\)th entry of \(X^\top r\) is exactly that sum before multiplication by \(2/n\): row \(j\) of \(X^\top\) contains feature \(j\) across the observations. Stacking the weight derivatives produces
\[ \begin{aligned} \nabla_w\bar L&=\frac2n X^\top r,\\ \frac{\partial\bar L}{\partial b}&=\frac2n\mathbf1^\top r,\\ r&=Xw+b\mathbf1-y. \end{aligned} \]
Shapes make the reductions explicit: \(X^\top\) is \((d,n)\), \(r\) is \((n,1)\), so the weight gradient is \((d,1)\) like \(w\). The intercept derivative sums across observations to one scalar. The updates subtract these derivatives from the corresponding parameter arrays using the same pre-update residual column. This is the matrix form of the scalar calculation, not another learning rule.
We used all \(n\) observations for every update: full-batch gradient descent. A minibatch update instead computes a mean gradient on a subset of \(B\) observations, where \(B\) is the minibatch size. With uniform random sampling, that mean is an unbiased estimate of the full-data gradient, but individual steps can fluctuate. Do not expect every minibatch update to lower the full-data loss. The animation uses full batches to isolate the geometry.
Verify derivatives by perturbing one parameter
An independent numerical check estimates the derivative from two objective values near the same parameter setting. For the slope,
\[ \frac{\partial\bar L}{\partial w} \approx\frac{\bar L(w+\epsilon,b)-\bar L(w-\epsilon,b)}{2\epsilon}. \]
Here \(\epsilon>0\) is a small numerical perturbation, not a training step. Perturb \(b\) in the same way while holding \(w\) fixed to check the intercept. Too large a perturbation can give a poor local approximation for general objectives; too small a perturbation can amplify floating-point cancellation. The following self-contained check uses the synthetic training table:
from math import isclose
x, y = (0., 1., 2.), (1., 2., 2.)
def mse(w, b):
return sum((w * xi + b - yi)**2 for xi, yi in zip(x, y)) / len(x)
h = 1e-5
slope_derivative = (mse(h, 0.) - mse(-h, 0.)) / (2 * h)
bias_derivative = (mse(0., h) - mse(0., -h)) / (2 * h)
assert isclose(slope_derivative, -4., abs_tol=1e-8)
assert isclose(bias_derivative, -10/3, abs_tol=1e-8)This checks a derivative; it performs no parameter update. The chapter’s recorded trace also checks both scalar derivatives at all 121 states and the multiple-feature rule on the quadratic representation. Automatic differentiation and its comparison with manual derivatives are developed in Week 04.
Fitting is not the final claim
Change the question, then collect the evidence
The animation answers an optimization question: can the chosen update reduce MSE on the three calibration pairs? It cannot answer an evaluation question: how accurately will the selected model predict new readings? Even the quadratic representation’s exact interpolation of those three targets supplied no new observations with which to answer that second question.
To separate these questions, Figure 4 introduces a different, dimensionless synthetic experiment. It is not a larger sample from our sensor. The generator fixes seed 2026 and creates targets using
\[ \begin{aligned} y&=0.5+0.8x-0.6x^2+\xi,\\ \xi&\sim\operatorname{Uniform}[-0.15,0.15]. \end{aligned} \]
Here \(\xi\) denotes an independently generated random perturbation between −0.15 and 0.15 for each target. We use 12 equally spaced training inputs in \([-1,1]\) and 40 fresh validation inputs sampled uniformly from the same interval. The validation perturbations are also generated independently. These declared sampling choices define this illustration’s intended use.
Before comparing scores, fix three candidate families: polynomials of degree 1, 3 and 9. Within each family, a numerical least-squares solver fits the coefficients using only the training pairs. This solver lets us compare flexibility without confusing a poor fit with an unfinished gradient-descent run. Deriving the solver is not required here; its training residuals satisfy an independent stationarity check. The degree is a hyperparameter; the fitted coefficients are parameters.
| Degree | Training MSE | Validation MSE |
|---|---|---|
| 1 | 0.046127 | 0.043419 |
| 3 | 0.004617 | 0.007746 |
| 9 | 0.001984 | 0.009316 |
The degree-nine family contains every degree-three polynomial: its additional coefficients could be zero. Its best training fit therefore cannot be worse than the best degree-three fit, apart from numerical error. Yet in this sample its lower training MSE accompanies higher validation MSE. Extra flexibility has fitted details that do not transfer as well to these fresh observations. This is an example of overfitting. It is evidence from one generated sample, not a theorem that degree three always wins or an estimate of how reliably it will win across repeated samples.
Choose degree three among these candidates by validation MSE, then freeze the choice before a final test assessment. Validation has influenced the choice, so its winning score is not an untouched final test score. This illustration stops at selection. It supplies neither a final test result nor evidence for extrapolation beyond \([-1,1]\). Notice also that degree one’s validation error is slightly below its training error: finite samples need not obey a universal “validation is always worse” ordering.
Suppose we keep trying degrees, representations and seeds until this validation score looks excellent. What has the validation set now helped us fit? Which new evidence would be needed for a credible final performance claim?
Read a curve only after identifying its horizontal axis
A curve against training updates asks a different question from the degree comparison above. For a fixed model, falling training and validation errors suggest useful progress on both samples. Falling training error with rising validation error suggests that further fitting may hurt transfer; a checkpoint can be selected using validation, before the test is examined. These are interpretations of evidence, not guarantees about every run.
If both errors remain high, distinguish an optimization failure from an unsuitable model family. First check gradients, update scale and whether the training objective has actually stabilized. Only then ask whether the chosen features can express the relationship. A line cannot acquire curvature merely by taking more small gradient steps. Conversely, adding polynomial terms will not repair an incorrect derivative.
Feature scaling also affects the optimization geometry. Replacing a feature by its centered, rescaled version changes the numerical gradient and useful step size. For an affine linear model with an intercept and nonzero scales, it does not enlarge the family of expressible functions: coefficients can be translated between coordinates. In distance-based classification, scaling additionally changes which differences count as large. Fit means and scales on training inputs, then apply those same values to validation and test inputs.
A concrete leakage diagnosis: standardizing all observations before splitting lets the eventual test inputs influence preprocessing. Selecting the degree with test MSE lets test targets influence model selection even if no gradient used them. Repeatedly choosing features after inspecting test mistakes has the same problem. Split according to intended use first—including session or time groups where appropriate—and keep every fitted transformation and selection inside its proper data role.
Short case study: the same 3 and 7 in different spaces
A handwritten 3 and 7 are easy to name, but how should a computer compare them? The Week 01 notebook follows the same classification task through four representations. Unlike the regression example, its targets are digit identities. The central question is which properties of an image become accessible to a simple decision rule.
| Space | Decision and experience |
|---|---|
| Raw pixels | Nearest class mean by mean absolute pixel distance. Training labels determine two mean images. |
| Two ink features | Five neighbours in scaled features. Human-chosen coordinates; training-fitted scaling and labelled neighbours. |
| Frozen CNN features | Five neighbours by cosine distance. External pretraining plus local labelled neighbours. |
| Image–text features | Similarity to two fixed digit descriptions. External pretraining and human-chosen class descriptions. |
The first stage averages training images within each class. A new image is assigned to the closer prototype, using the mean absolute difference across pixels. The mean is learned from the training examples; the distance rule is a modeling choice. A translated stroke may move many pixels without changing the digit’s identity, exposing a limitation of this coordinate system.
The second stage makes human assumptions explicit. Vertical ink balance and horizontal centre of mass compress 784 pixel values into two numbers. The notebook fits a scaler on training features, then uses five-neighbour voting with Euclidean distance. The plot is easy to inspect, but compression may also discard information needed to separate ambiguous digits.
The third stage uses a pretrained MobileNetV3-Small CNN as a frozen feature extractor: its parameters are not fitted on the local MNIST examples. Its 576-dimensional vectors are normalized, and five-neighbour classification uses cosine distance. External pretraining is prior experience, even though local training does not update this encoder. Both neighbour stages use five neighbours, but the representation and the distance change. Their comparison alone cannot attribute a performance difference solely to the representation.
The notebook’s PCA projection is a two-dimensional view of these embeddings. PCA is fitted on training embeddings; the neighbour classifier still operates in the original 576-dimensional embedding space. Apparent overlap or separation in a projection is not a proof about distances or performance in that full space. The projection’s mathematics belongs to the later dimensionality-reduction topic.
Finally, MobileCLIP2 embeds an image and the descriptions “a handwritten digit three” and “a handwritten digit seven.” Normalized dot products compare image and text embeddings; the larger similarity selects the class. This stage fits no classifier using the local training labels. It still relies on external image–text training and class information supplied by the descriptions. “No local label fitting” does not mean “no learning.” CNN and image–text model internals belong to later chapters.
These stages are an investigation, not a controlled architecture benchmark. They differ in features, distance rules and access to external experience. The chapter reports no accuracy ranking. Preserve train-fitted preprocessing; if test images or errors guide prompt or feature choices, those observations have become development evidence and no longer provide an untouched assessment.
If two digits overlap in the PCA picture, must the full embedding classifier confuse them? What comparison would isolate a change of representation from a change of distance rule?
Synthesis, connections and optional exercises
The working loop is now concrete: choose observations and their roles, construct a representation, make a prediction, measure training loss, update parameters, then evaluate the frozen choice using the appropriate evidence. In the sensor example, a single line makes every part of that loop visible. The contour plot shows why changing a parameter changes the objective; the validation example shows why improving that objective is only part of the learning problem.
| Tempting conclusion | More careful interpretation |
|---|---|
| A smaller training loss proves a better predictor | It proves a better fit to that training objective; new-data evidence is separate |
| More training will fix any model | Optimization cannot add a missing feature or relationship to the model family |
| A polynomial is nonlinear in its fitted weights | Fixed polynomial features still give a model linear in its coefficients |
| A test set is safe if no gradient uses it | Test-guided preprocessing or selection also compromises the final assessment |
| A frozen pretrained model has no training experience | Its experience occurred before the local experiment |
| A two-dimensional plot reveals the whole embedding | It is a projection; discarded directions can matter |
Week 02 keeps the fitted-score idea but builds a classification probability and decision boundary around it. Week 04 makes representations learnable and automates derivatives through a richer computation. Week 07 returns to MSE with explicit probabilistic assumptions. Neither probabilities nor neural networks replace the need to distinguish fitting from evaluation.
Optional exercises
- State a learning claim. For a sensor intended for future measurement sessions, specify experience, task, performance measure and intended scope. Which split would reveal a session-specific shortcut that a random row split might hide?
- Check a new candidate. On S1–S3, compute predictions, signed residuals and MSE for \(w=0.5\), \(b=0.5\). Compare with candidates A–C. Which part of that comparison says anything about future measurements?
- Take one simultaneous step. Start from \(w=1\), \(b=0\), use full MSE on S1–S3 and learning rate \(0.05\). Calculate both derivatives before updating either parameter. Explain why updating \(b\) using the already-updated \(w\) would be a different procedure.
- Audit shapes and scale. With four observations and two features, give the shapes of \(X,w,y,\widehat y,r\) and \(\nabla_w\bar L\). Explain how an accidental \((4,)\) target array could change a residual calculation intended to use a \((4,1)\) column. Does changing MSE to a sum preserve the numerical update at the same learning rate?
- Read the evidence. Use Figure 4 to choose a degree. Explain why the smallest training MSE gives a different choice, why the selected validation score is not a final test result, and what repeated searches on the same validation set would change.
- Diagnose leakage. A workflow standardizes all readings, splits rows at random, and chooses a representation after inspecting test mistakes. Identify three distinct concerns for prediction in future sessions and propose an order of operations that addresses them.
- Design a representation comparison. Propose a controlled comparison of handcrafted and frozen features for 3 versus 7. State what to hold fixed, how to select settings, how to preserve the final assessment, and what a PCA plot can and cannot establish. Describe an image perturbation that tests a meaningful invariance rather than merely increasing the dataset size.
References and further reading
Mitchell, Machine Learning, Chapter 1: the experience–task–performance formulation (Mitchell 1997). Scope is an explicit course addition.
Bishop, Pattern Recognition and Machine Learning: linear regression and least squares (Bishop 2006). Read the fitting objective here; return to its probabilistic interpretation in Week 07.
The lecture and tutorial linked above are the teaching companions; the syllabus defines which parts of their broader material belong to this week.
Implementation companions for the representation case: the official Torchvision MobileNetV3-Small documentation describes the pretrained model; Apple’s MobileCLIP2 research page describes the image–text pretraining context. The local notebook is the source for its exact preprocessing, distance rules and classifier choices.