Week 02 · From Scores to Probabilities and Decisions
Central question. How does a learned score become a probability and then a decision—and what changes when the classifier learns a margin instead?
Scope. Introduce binary cross-entropy (BCE) and multiclass cross-entropy as usable objectives here. Their likelihood and information-theory derivations belong to Week 07. SVMs and kernels are Week 02 self-study, used again in the Week 03 lab; full dual/KKT derivations remain optional.
Learning outcomes and prerequisites
- perform training-data EDA and distinguish scaling from added model expressivity;
- compute logits, sigmoid probabilities, log-odds and thresholded decisions;
- interpret BCE/CE, stable logits-based implementations and their input shapes;
- compare confusion-matrix metrics, ROC/PR trade-offs and validation-selected thresholds;
- distinguish one-vs-rest from a coupled softmax classifier;
- explain margins, support vectors, hinge loss, kernels and the roles of \(C\) and \(\gamma\);
- distinguish a decision score, a class-probability estimate and calibration.
Prerequisites: Week 01 data roles, vectors, fitting objectives and gradient updates. Introduce basic probability vocabulary locally; do not assume likelihood or information theory.
Companions: lecture resources, tutorial, Lab 2 instructions, and the syllabus.
Notation contract
| Symbol | Meaning and convention |
|---|---|
| \(x_i\in\mathbb R^{d\times1}\) | One raw input column; observations occupy rows of \(X\) |
| \(g(x_i)\in\mathbb R^{d_g\times1}\) | Fixed feature map with \(d_g\) coordinates; feature rows form \(G\) |
| \(w\in\mathbb R^{d_g\times1}\), \(b\), \(\theta=(w,b)\) | Matching weights, scalar intercept and their collection |
| \(y_i\in\{0,1\}\), \(z_i\) | Observed binary label and scalar affine score (logit) |
| \(q_i=\sigma(z_i)\), \(\widehat c_i\) | Model probability of class one and separate class decision |
| \(\tau\in(0,1)\) | Probability threshold; select a deployed value using validation |
| \(\delta(x)\) | Signed distance to the zero contour in the declared input coordinates |
| \(K\), \(q_{ic}\) | Class count and probability for class \(c\); reserve \(C\) for SVMs |
| \(L_i\), \(\bar L=n^{-1}\sum_i L_i\) | Per-example loss and explicitly mean-reduced objective |
| \(t_i=2y_i-1\), \(f(x)\) | Later SVM signed label and decision score; \(f\) is not a probability |
| \(k(x,x')\), \(C\), \(\gamma\) | Kernel, soft-margin penalty parameter and RBF scale parameter |
Single inputs and weights remain columns, as in Week 01. The batch scores, probabilities and binary targets here use matching \((n,1)\) columns. The notebook’s squeezed \((n,)\) convention is translated below. Later multiclass logits use \((B,K)\) and class-index targets \((B,)\), where \(B\) is batch size; there \(y_i\) denotes a class index rather than a binary label. Shapes and meanings are stated again with each loss.
Translate the lecture’s parameter \(\phi\) to \(\theta\) and probability \(h\) to \(q\). Translate the notebook’s feature map \(\phi(x)\) to \(g(x)\), retaining \(\phi\) for neural-network activations. The SVM label \(t_i\) takes values \(-1,+1\) and is distinct from threshold \(\tau\). All logarithms are natural. Scores and probabilities are dimensionless; the toy inputs below are also dimensionless. A real feature’s units must be matched by its weight.
Inspect the data before drawing a boundary
What should be predicted, and from which observations?
In Week 01, a voltage supplied a continuous prediction. Now suppose an input must be assigned to one of two classes. A binary label \(y=1\) denotes the positive class, and \(y=0\) the other class. “Positive” names the event we choose to track; it does not mean desirable, healthy or more common. A class label is an observed outcome, while the model’s output may be a probability rather than an immediate yes/no decision.
The companion notebook uses two noisy moons: points from two curved bands in a plane. Its coordinates have no physical units. A scatter plot can reveal curved structure that separate one-coordinate histograms conceal. A line is a useful baseline precisely because its systematic errors can expose a limitation of the representation. Later, Iris supplies a separate three-class example with petal measurements in centimetres.
The notebook generates 900 moons observations and uses a stratified 540/180/180 training/validation/test split, with seed 2026 and noise setting 0.23. Stratification uses labels to allocate rows by class; it is not permission to use the held-out outcomes for model selection. Training data supply fitted parameters and preprocessing statistics; validation guides representation, settings and decision thresholds; the test set assesses the frozen pipeline. These are reproducible teaching settings, not a recommended sample size or split for every application. Patient, device, session or time dependence can require a different split from random rows.
EDA and preprocessing are part of the pipeline
Exploratory data analysis (EDA) asks what is in the training data before we fit a complicated explanation. Check feature meanings and units, missing entries, impossible values, label counts, duplicates and grouping identifiers. Inspect distributions and relationships with labels. A correlation or a visual pattern suggests a hypothesis; it does not establish a causal mechanism or performance on new observations. An observed minimum–maximum rectangle is a record of this sample, not a proof of the distribution’s support.
For numeric features, a standardization uses
\[ \widetilde x_j=\frac{x_j-\mu_j}{s_j}, \]
where \(\mu_j\) and \(s_j>0\) are the training mean and scale for feature \(j\). Apply these same fitted values to validation and test observations. If a feature is constant in training, handle its zero scale explicitly rather than divide by zero; it supplies no within-training variation. Scaling changes coordinates and optimization behavior. It does not add observations or create a curved boundary for an affine model in those coordinates. The later capacity section will separate these effects in detail.
A missing numeric entry is not automatically a zero. A simple baseline can replace it by a training median, possibly accompanied by a missingness indicator. The replacement and the decision to use an indicator belong to the pipeline; assess their usefulness with validation. For a nominal feature such as a region name, one-hot indicators express membership without inventing numerical distances. With an intercept, retaining every category gives redundant columns; a declared reference category removes that particular redundancy. An unseen category still needs an explicit handling rule. An ordinal feature has a meaningful order, but encoding levels as 0, 1, 2 additionally assumes equal score increments for a linear term. Order alone does not justify that spacing.
A leakage counterexample: inspect all labels, choose a promising feature, fill missing values using all rows and only then split. Both the selected feature and the fitted replacement have already seen evidence meant to be held out. Split first according to the intended use, fit transformations on training rows, select alternatives with validation and preserve the final assessment. The Lab 2 comparison applies these data roles to medical measurements. Its quadratic PCA-derived features are a different construction from the tutorial’s cubic moons features; neither is the small example below.
Freeze a calculation small enough to inspect
For hand calculations and the later training animation, use a separate, dimensionless synthetic training set. The table groups 16 labelled observations at four input coordinates. Each count represents distinct rows; A–D name coordinates, not individual observations. All 16 rows are training data. There is no validation/test sample or reported held-out performance.
| Site | \((x_1,x_2)\) | Count \(y=0\) | Count \(y=1\) |
|---|---|---|---|
| A | \((1,0)\) | 1 | 3 |
| B | \((0,1)\) | 1 | 3 |
| C | \((-1,0)\) | 3 | 1 |
| D | \((0,-1)\) | 3 | 1 |
Both classes occur at every coordinate. This is deliberate ambiguity, not a plotting error: the supplied features do not determine every label. Any model receiving exactly the same input must return the same probability at that input. The extra point O at \((0,0)\) is an unlabelled probe, not a seventeenth training observation.
Begin with hand-chosen weights \(w=(2,-1)^\top\) and \(b=0\). They have not been fitted. For the later two-parameter animation we will keep \(b=0\) and update only \(w_1,w_2\), so every animated state can lie on one exact two-dimensional loss surface. The general logistic model below still has a trainable intercept. The repeated conflicting labels and two independent coordinate directions avoid relying on a perfectly separable example with diverging fitted weights. The precise training objective and updates follow in the next section.
A score has geometry
Keep the weighted sum; change its interpretation
For a fixed feature map \(g(x)\), start with the affine score
\[ z(x)=w^\top g(x)+b. \]
The weights and intercept are fitted parameters. For our two original coordinates, \(g(x)=x\) and
\[ z(x)=w_1x_1+w_2x_2+b. \]
A score can be any real number; it is neither a class label nor a probability. With the hand-chosen parameters, \(z(x)=2x_1-x_2\). At A the score is 2, at B it is −1, and at O it is 0. An observation’s label does not enter this calculation; it will enter the training loss when we learn the parameters.
For \(n\) inputs, stack the raw input rows in \(X\) and their feature rows in \(G\). The batch calculation is
\[ z=Gw+b\mathbf1. \]
Here \(X\) has shape \((n,d)\), \(G\) has shape \((n,d_g)\), \(w\) has shape \((d_g,1)\), and \(\mathbf1\) is an \((n,1)\) column of ones. Thus the scores \(z\), probabilities \(q\) and targets \(y\) can all use \((n,1)\) columns. With the identity feature map, \(G=X\). The notebook instead squeezes binary logits to \((n,)\) to match its one-dimensional target array. Either convention works when both arrays match; mixing them can silently broadcast an intended vector into a matrix.
The zero contour and its normal direction
With two original features, all points satisfying
\[ w_1x_1+w_2x_2+b=0 \]
form a line when \(w\ne0\). If two points lie on it, their difference \(v\) obeys \(w^\top v=0\), so \(w\) is perpendicular to the line. Moving in the direction of \(w\) increases the score. The intercept moves the zero contour parallel to itself when the weights remain fixed. In our example the zero contour is \(x_2=2x_1\), as shown in Figure 1.
Keep the implicit equation when drawing boundaries. Solving for \(x_2\) by dividing by \(w_2\) fails when \(w_2=0\), even though a perfectly valid vertical line may remain. In more than two features the corresponding set is a hyperplane. Fixed nonlinear features can make its preimage curve in the original input plane; we will examine that distinction after learning the score (Bishop 2006, chap. 4).
A score is not generally a distance
For \(g(x)=x\) and \(w\ne0\), project an input onto the zero contour by defining
\[ x_{\rm boundary}=x-\frac{z(x)}{\lVert w\rVert_2^2}w. \]
Substituting this point into the affine score gives zero. The signed perpendicular distance, positive on the \(z>0\) side, is therefore
\[ \delta(x)=\frac{z(x)}{\lVert w\rVert_2}. \]
The norm \(\lVert w\rVert_2\) is the square root of the sum of squared weight coordinates. For our weights it is \(\sqrt5\). A’s score is 2, but its signed distance in the declared input coordinates is \(2/\sqrt5\approx0.8944\). Distance changes if the input coordinate units change; it is not itself a probability or a measure of how familiar the input is.
If \(w=0\), do not divide by its norm. When \(b\ne0\) the score is the same nonzero number everywhere and the zero contour is empty. When \(b=0\), every input has score zero: the whole plane is the zero set, not a unique separating line.
A probability is not a decision
Turn a real score into a binary probability model
The sigmoid maps a score into \((0,1)\):
\[ \begin{aligned} q(x)&=\sigma(z(x)),\\ \sigma(z)&=\frac{1}{1+e^{-z}}. \end{aligned} \]
The exponential \(e^{-z}\) is positive, so a finite real score gives a value strictly between zero and one. In this binary model, interpret \(q(x)\) as \(P_\theta(Y=1\mid x)\): the model’s conditional probability of label one at input \(x\), with \(\theta=(w,b)\). Its probability for label zero is \(1-q(x)\). The notation states a model, not knowledge of the true data-generating probability. Whether such estimates are accurate requires evaluation.
At \(z=0\), \(q=0.5\). A larger score gives a larger probability of class one; a very negative score gives a small one. Despite the name logistic regression, the response here is a class probability based on a linear score, rather than an unconstrained numerical target as in Week 01. The sigmoid curve does not turn a straight zero-score contour into a curved one.
The odds of class one are \(q/(1-q)\). Rearranging the sigmoid gives
\[ \frac{q}{1-q}=e^z, \]
and taking the natural logarithm gives the log-odds, or logit:
\[ z=\log\frac{q}{1-q}. \]
Thus logistic regression models the log-odds as an affine function of the chosen features. A one-unit score increase multiplies the odds by \(e\); it does not add a fixed amount to the probability. At A, \(z=2\) gives \(q=1/(1+e^{-2})\approx0.8808\). That same estimate applies to all four A rows, including their conflicting labels. Assigning a probability is not reproducing every observed outcome.
Declare a policy before naming a predicted class
Choose a probability threshold \(\tau\in(0,1)\) and define
\[ \widehat c(x)= \begin{cases} 1,&q(x)\ge\tau,\\ 0,&q(x)<\tau. \end{cases} \]
Equality is assigned to class one throughout this chapter. The threshold is a policy setting, separate from the fitted weights. A deployed choice depends on the task’s error consequences and validation evidence; 0.5 is a useful reference, not a universal requirement.
Because sigmoid is strictly increasing, we can invert the threshold:
\[ \begin{aligned} q(x)\ge\tau &\iff \frac{q(x)}{1-q(x)}\ge\frac{\tau}{1-\tau}\\ &\iff z(x)\ge\log\frac{\tau}{1-\tau}. \end{aligned} \]
Only \(\tau=0.5\) yields the zero-score boundary. For \(\tau=0.8\), the score threshold is \(\log4\approx1.3863\) and our boundary becomes \(2x_1-x_2=\log4\). Its position changes while the fitted probability function stays fixed. Increasing the threshold shrinks the region assigned to class one; it performs no parameter update.
| Site | \(z\) | \(q\) | \(\widehat c\) at .5 | \(\widehat c\) at .8 |
|---|---|---|---|---|
| A | 2 | 0.8808 | 1 | 1 |
| B | −1 | 0.2689 | 0 | 0 |
| C | −2 | 0.1192 | 0 | 0 |
| D | 1 | 0.7311 | 1 | 0 |
| O | 0 | 0.5000 | 1 | 0 |
At D, the same score and probability support different decisions under the two policies. At O, the 0.5 decision follows our tie convention; it is not a claim of strong evidence for class one. No true label was supplied for O. At no site should a probability be confused with a count of correct decisions or a held-out performance estimate.
Move from threshold 0.5 to 0.8 in the table. Which objects changed: the input, score, probability, fitted parameters or decision? Would finding the threshold with the best test result preserve an untouched final assessment?
The same boundary can carry different probabilities
Multiply both \(w\) and \(b\) by a positive factor \(a\). The new score is \(az\). Its zero contour and the decisions at threshold 0.5 are unchanged. Its probabilities generally change: doubling our score at A gives \(\sigma(4)\approx0.9820\) rather than \(\sigma(2)\approx0.8808\). A larger positive multiplier pushes nonzero scores’ probabilities closer to zero or one. It preserves rankings, but need not improve the probabilities’ accuracy. At a threshold other than 0.5, this scaling can change decisions.
A model’s binary confidence is often written \(\max(q,1-q)\). It describes concentration within the two known alternatives, not proof of correctness or familiarity. The same geometry can therefore produce different confidence without any new evidence. Later we will return to confident predictions away from the observed training inputs.
Two classifiers have parameters \((w,b)\) and \((10w,10b)\). They agree at threshold 0.5. Must they assign the same probabilities? Must the more confident one be more accurate? What would you need to compare their probability estimates?
Finally, the degenerate \(w=0\) model produces the constant probability \(\sigma(b)\). If \(b=0\), it assigns 0.5 everywhere and our inclusive 0.5 rule assigns class one everywhere. The absence of a unique separating line is therefore compatible with a well-defined probability and decision rule.
BCE trains a score; numerical stability preserves the objective
Give the observed label a role in fitting
The hand-chosen score explains a prediction, but it does not explain how to learn weights. We need a loss that compares a probability \(q_i\) with the observed binary label \(y_i\). Introduce binary cross-entropy (BCE) directly as a fitting objective:
\[ L_i=-y_i\log q_i-(1-y_i)\log(1-q_i). \]
For \(y_i=1\), the loss is \(-\log q_i\); for \(y_i=0\), it is \(-\log(1-q_i)\). The label selects the probability assigned to the observed event. Increasing that probability decreases its loss. A confident mistake is costly: a positive observation assigned \(q=0.9\) has loss about 0.1054, whereas assigning it \(q=0.1\) gives about 2.3026. For a negative observation those values exchange places. The decision threshold appears nowhere in this objective.
Natural logarithms fix the numerical scale. Average over the declared training observations to obtain
\[ \bar L=\frac1n\sum_{i=1}^{n}L_i. \]
There is no factor of one half and no parameter penalty here. Duplicated coordinates still contribute once per labelled observation: the four sites contain 16 rows, not four equally labelled examples. The probabilistic reason for this particular logarithmic loss is developed in Week 07. Here we study its calculation, derivatives and use.
Derive the gradient through the sigmoid
First differentiate the loss with respect to its probability input:
\[ \frac{\partial L_i}{\partial q_i} =-\frac{y_i}{q_i}+\frac{1-y_i}{1-q_i}. \]
Differentiating \(q_i=(1+e^{-z_i})^{-1}\) gives
\[ \frac{\partial q_i}{\partial z_i} =\frac{e^{-z_i}}{(1+e^{-z_i})^2} =q_i(1-q_i). \]
The chain rule multiplies these two local derivatives. Cancelling the probability factors gives
\[ \begin{aligned} \frac{\partial L_i}{\partial z_i} &=-y_i(1-q_i)+(1-y_i)q_i\\ &=q_i-y_i. \end{aligned} \]
For a positive label, \(q_i-y_i<0\), so decreasing loss locally calls for a larger logit. For a negative label it calls for a smaller one. A confident wrong prediction has a logit derivative near −1 or +1, respectively. This simple residual-like expression is specific to BCE composed with sigmoid. Using squared error on sigmoid probabilities would add the sigmoid derivative and a different numerical factor; it is a different objective.
Since \(\partial z_i/\partial w_j=g_j(x_i)\) and \(\partial z_i/\partial b=1\), the mean-loss gradients are
\[ \begin{aligned} \nabla_w\bar L&=\frac1nG^\top(q-y),\\ \frac{\partial\bar L}{\partial b} &=\frac1n\mathbf1^\top(q-y). \end{aligned} \]
\(G\) is \((n,d_g)\), the residual column \(q-y\) is \((n,1)\), and the weight gradient is \((d_g,1)\). The bias derivative is scalar. Evaluate all derivatives at the same old parameters, then take a simultaneous gradient step:
\[ \begin{aligned} w&\leftarrow w-\eta\nabla_w\bar L,\\ b&\leftarrow b-\eta\frac{\partial\bar L}{\partial b}. \end{aligned} \]
Here \(\eta>0\) is the learning rate. Replacing the mean by a sum multiplies these gradients by \(n\), changing the step at a fixed learning rate.
One checked step on the frozen training table
For our identity-feature toy, \(G=X\). The superscript \((k)\) counts completed updates, not powers. Fix \(b=0\) and start from \(w^{(0)}=(2,-1)^\top\). The conflicting class counts give the initial gradients
\[ \begin{aligned} \left.\frac{\partial\bar L}{\partial w_1}\right|_{w^{(0)}} &=\tfrac12\bigl(\sigma(2)-\tfrac34\bigr)\\ &\approx0.065399,\\ \left.\frac{\partial\bar L}{\partial w_2}\right|_{w^{(0)}} &=\tfrac12\bigl(\sigma(-1)-\tfrac34\bigr)\\ &\approx-0.240529. \end{aligned} \]
For example, A contributes \(4\sigma(w_1)-3\) to the sum of slope-one derivatives. C contributes \(-\{4\sigma(-w_1)-1\}\); using \(\sigma(-w_1)=1-\sigma(w_1)\) and dividing their total by 16 gives the first expression above. B and D give the second in exactly the same way.
With learning rate \(\eta=0.5\), use both initial derivatives:
\[ \begin{aligned} w_1^{(1)}&=2-0.5(0.065398539)\\ &\approx1.967301,\\ w_2^{(1)}&=-1-0.5(-0.240529289)\\ &\approx-0.879735. \end{aligned} \]
Mean training BCE decreases from 0.845095 to 0.814781. At B, whose input is \((0,1)\), the probability increases from 0.268941 to 0.293233. Both weights have changed; the boundary rotates. The intercept stays zero by the declared restriction, not because intercepts are generally unlearnable. In this symmetric dataset its bias derivative also happens to cancel.
Watch one update in two spaces
Figure 2 shows 120 simultaneous full-batch updates, with the same learning rate. The initial state holds for three seconds and the first updated state for four seconds. Later frames show actual recorded updates, ending with a longer final hold. Play or pause with the native controls; there is no audio or autoplay.
The initial boundary is 2x₁−x₂=0. The first update decreases w₁ and increases w₂, lowering mean BCE. Later w₂ changes sign; a zero value would give a vertical boundary. The final boundary slopes downward. By update 120 the weights are approximately (1.102331, 1.094002), with mean BCE 0.562337 and probability 0.749135 at B. The loss-contour point and the data-space boundary describe the same parameters at every recorded state. This is fitting on training observations, not a held-out performance result.
The static sequence in Figure 3 supplies the same initial, first and final states for no-motion reading and print.
| \(k\) | \(w_1\) | \(w_2\) | Mean BCE |
|---|---|---|---|
| 0 | 2.000000 | −1.000000 | 0.845095 |
| 1 | 1.967301 | −0.879735 | 0.814781 |
| 120 | 1.102331 | 1.094002 | 0.562337 |
The final recorded point is close to a finite optimum, rather than exactly converged. Both labels occur at every site, so making a probability arbitrarily close to zero or one would severely penalize its conflicting observation. This differs from an unregularized, perfectly separable dataset, where weights can keep growing while BCE approaches its infimum. A finite animation endpoint would not prove a finite optimum in that different problem.
Here \(b=0\) throughout. Can training translate the boundary away from the origin? What happens as \(w_2\) crosses zero? Does the falling loss establish better performance on new observations, or only a better fit to these training rows?
Stable arithmetic preserves the objective
The probability formula is useful for interpretation, but a computer may round a sigmoid output to exactly zero or one. Taking logarithms afterward can produce infinities, and terms such as zero times an infinite logarithm can produce NaNs. For binary labels an equivalent logit-based form is
\[ \begin{aligned} L_i&=\max(z_i,0)-z_iy_i\\ &\quad+\log\bigl(1+e^{-|z_i|}\bigr). \end{aligned} \]
This follows by writing BCE as \(\log(1+e^{z_i})-y_iz_i\) and factoring the large exponential when \(z_i>0\). The remaining exponential has a nonpositive argument. A log1p operation computes \(\log(1+u)\) accurately when \(u\) is small. Although the stable expression uses a maximum, the combined loss remains smooth at \(z_i=0\); use the framework’s fused implementation for differentiation.
PyTorch’s BCEWithLogitsLoss combines the sigmoid and BCE computation with stable arithmetic. Pass logits and matching floating-point binary targets, with reduction="mean" explicitly selected here. Passing a sigmoid probability as its input would apply the wrong composition. The following standalone check reproduces the first update on CPU using 64-bit arithmetic:
import torch
X = torch.tensor([[1., 0.], [0., 1.], [-1., 0.], [0., -1.]],
dtype=torch.float64).repeat_interleave(4, dim=0)
y = torch.tensor([0., 1., 1., 1., 0., 1., 1., 1.,
0., 0., 0., 1., 0., 0., 0., 1.],
dtype=torch.float64).reshape(-1, 1)
w = torch.tensor([[2.], [-1.]], dtype=torch.float64, requires_grad=True)
criterion = torch.nn.BCEWithLogitsLoss(reduction="mean")
z = X @ w # b is fixed at zero
assert z.shape == y.shape == (16, 1)
initial_loss = criterion(z, y)
initial_loss.backward()
expected_gradient = torch.tensor([[0.065398539], [-0.240529289]],
dtype=torch.float64)
assert torch.allclose(w.grad, expected_gradient, atol=1e-9, rtol=0.)
with torch.no_grad():
w -= 0.5 * w.grad
updated_loss = criterion(X @ w, y)
assert abs(initial_loss.item() - 0.845094849) < 1e-9
assert abs(updated_loss.item() - 0.814781427) < 1e-9This performs one update, so no gradient reset is needed between iterations. A repeated training loop must clear accumulated gradients before each new backward pass, as the notebook does. Sigmoid probabilities can be computed separately for interpretation. The generator also checks all 121 states by central finite differences, comparing nearby losses with the analytic weight and bias derivatives; no numerical derivative drives the animation.
Scaling and added features change different things
A change of coordinates is not a larger model family
If optimization struggles, should we scale the inputs, add features or train longer? These choices address different problems. Scaling can improve the numerical journey; a new feature map can change which relationships are expressible. Neither guarantees better performance on new observations.
To make the first distinction exact, write a training-fitted coordinate change as \(x_j=\mu_j+s_j\widetilde x_j\), with \(s_j>0\). Substitution into an affine score gives
\[ \begin{aligned} z(x)&=\sum_jw_js_j\widetilde x_j+\widetilde b,\\ \widetilde b&=b+\sum_jw_j\mu_j. \end{aligned} \]
Thus the new weights \(\widetilde w_j=s_jw_j\) and intercept reproduce exactly the old scores. With a free intercept and invertible affine scaling, the unpenalized family has not gained a new boundary. Recentring would require special care in our earlier fixed-intercept animation: its restriction is not invariant to an arbitrary translation of input coordinates.
The gradient journey still changes. Multiplying a feature by 20 multiplies its contribution to a weight derivative by 20 at matching predictions. A fixed learning rate and finite update budget can therefore behave differently. The notebook’s stretched-coordinate experiment illustrates this issue; a poor fixed-budget fit is not proof that the line family cannot express a solution. A coefficient penalty also depends on the coordinate convention: keeping the same numerical penalty after rescaling generally changes the objective.
A fixed nonlinear map can bend the boundary
The moons example needs more geometric freedom than a single line. The notebook uses the nine nonconstant monomials through degree three:
\[ \begin{aligned} g(x)=(&x_1,x_2,x_1^2,x_1x_2,x_2^2,\\ &x_1^3,x_1^2x_2,x_1x_2^2,x_2^3)^\top. \end{aligned} \]
We keep a separate intercept rather than include a constant feature. The score \(w^\top g(x)+b\) is linear in the nine fitted coefficients, while its zero contour can curve in the original two-coordinate plane. For example, an \(x_1x_2\) term responds to a joint relationship that neither input contributes on its own. This map introduces no new measurements, but enlarges the set of functions the weighted sum can express.
Figure 4 reruns the notebook’s declared moons setup: 540 training and 180 validation rows, training-fitted standardization, zero initial logits, full-batch SGD with learning rate 0.1 and 1500 updates. The degree-three map is fixed before comparison, and its nine columns are standardized using training statistics only. Both models have a trainable intercept. No test predictions or scores are computed for this figure.
The parameter count includes the intercept; both score columns report mean BCE.
| Map | Params | Train | Validation |
|---|---|---|---|
| Linear | 3 | 0.2537 | 0.2894 |
| Cubic | 10 | 0.1983 | 0.2254 |
Here the curved model has lower validation BCE and makes 16 validation errors at threshold 0.5, compared with 22 for the line, out of 180 observations. This supports the curved representation in this particular comparison. It does not prove exact optimization, universal superiority of polynomials or performance outside the declared distribution. More terms can also fit noise. Select a representation and its settings with validation, then freeze the pipeline before its final test assessment. The earlier four-site training animation is a different example; extra features cannot resolve its contradictory labels at identical inputs.
Evaluate decisions without losing the scores
Count what each decision did
BCE evaluates probability predictions. A confusion matrix evaluates decisions at a declared threshold. To see the arithmetic without hiding it behind a training run, use the following constructed validation table. These eight labels and probabilities are illustrative values, separate from both the moons experiment and the animation. They are not predictions exported from either model, and they provide no claim about calibration or final test performance.
| Row | Label \(y\) | Probability \(q\) |
|---|---|---|
| V1 | 1 | 0.9 |
| V2 | 0 | 0.8 |
| V3 | 1 | 0.7 |
| V4 | 0 | 0.6 |
| V5 | 1 | 0.4 |
| V6 | 0 | 0.3 |
| V7 | 1 | 0.2 |
| V8 | 0 | 0.1 |
Keep class one positive and retain the rule \(\widehat c=1\) when \(q\ge\tau\). A true positive (TP) is a positive observation predicted positive; a false positive (FP) is a negative observation predicted positive. A false negative (FN) is a positive observation predicted negative; a true negative (TN) is a negative observation predicted negative.
At \(\tau=0.5\), V1–V4 are predicted positive, giving TP=2 and FP=2. V5–V8 are predicted negative, giving FN=2 and TN=2. The counts sum to eight. The orientation below is explicit: rows are observed labels and columns are predicted labels.
| Observed / predicted | Class 0 | Class 1 |
|---|---|---|
| Class 0 | TN = 2 | FP = 2 |
| Class 1 | FN = 2 | TP = 2 |
Accuracy, precision, recall and F1 ask different questions
The four common summaries are
\[ \begin{aligned} \mathrm{accuracy}&=\frac{\mathrm{TP}+\mathrm{TN}}{n_{\rm eval}},\\ \mathrm{precision}&=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\\ \mathrm{recall}&=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}. \end{aligned} \]
Here \(n_{\rm eval}\) counts evaluation rows, distinct from the training sample size. Accuracy asks how often the decision is correct. Precision asks which fraction of predicted positives are positive; recall asks which fraction of observed positives are retrieved. A model predicting only the common class can have high accuracy while missing every rare positive.
When precision and recall are positive, their harmonic mean is
\[ \begin{aligned} \mathrm{F1}&=\frac{2}{1/\mathrm{precision}+1/\mathrm{recall}}\\ &=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}. \end{aligned} \]
The second expression follows by substituting the count definitions; it also makes the zero cases explicit. F1 balances these two kinds of positive-class success, but ignores TN and is not a universal measure of application utility. At threshold 0.5 all four summaries happen to equal 0.5 in our table.
Do not silently divide by zero. With no predicted positives, precision is undefined. With no observed positives, recall is undefined. The count formula for F1 gives zero when TP=0 and FP+FN>0; if all three counts vanish, it too is undefined. We write an em dash for an undefined ratio. Software may substitute a declared value for reporting; compare conventions as well as numbers.
Change the threshold while keeping every probability fixed
Recompute decisions under several thresholds. Equality is included, so V3 is positive at threshold 0.7 and V5 is positive at threshold 0.4.
| \(\tau\) | TP | FP | FN | TN |
|---|---|---|---|---|
| 0.95 | 0 | 0 | 4 | 4 |
| 0.9 | 1 | 0 | 3 | 4 |
| 0.7 | 2 | 1 | 2 | 3 |
| 0.5 | 2 | 2 | 2 | 2 |
| 0.4 | 3 | 2 | 1 | 2 |
| 0.2 | 4 | 3 | 0 | 1 |
Acc. and Prec. abbreviate accuracy and precision.
| \(\tau\) | Acc. | Prec. | Recall | F1 |
|---|---|---|---|---|
| 0.95 | 0.500 | — | 0.000 | 0.000 |
| 0.9 | 0.625 | 1.000 | 0.250 | 0.400 |
| 0.7 | 0.625 | 0.667 | 0.500 | 0.571 |
| 0.5 | 0.500 | 0.500 | 0.500 | 0.500 |
| 0.4 | 0.625 | 0.600 | 0.750 | 0.667 |
| 0.2 | 0.625 | 0.571 | 1.000 | 0.727 |
For this illustration, declare the policy before selection: among candidate thresholds, choose the highest one with validation recall at least 0.75. This is a teaching scenario, not a course or application requirement. Sweeping all score-change thresholds selects \(\tau=0.4\). It retrieves three of the four positives while predicting five positives in total. Lowering the threshold further retrieves the fourth positive but also admits more negatives. The probability function and its BCE are unchanged.
Selecting a threshold uses validation labels even though no gradient is taken. Repeatedly revising the rule, features or candidates after seeing validation results can overfit that evidence. Freeze the chosen representation, fitted preprocessing, model and decision policy before a final test evaluation. A validation recall of 0.75 does not guarantee the same recall on future samples.
ROC and precision–recall preserve the threshold sweep
A receiver operating characteristic (ROC) plot places false-positive rate on the horizontal axis and true-positive rate on the vertical axis:
\[ \begin{aligned} \mathrm{FPR}&=\frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}},\\ \mathrm{TPR}&=\mathrm{recall}. \end{aligned} \]
Lowering a threshold can only add predicted positives, so both rates are nondecreasing along the sweep. Precision need not change monotonically: its numerator and denominator grow together. A precision–recall (PR) plot therefore shows recall horizontally and precision vertically for the same sequence of decision policies. Neither plot shows a new training run.
The area under the ROC curve (ROC-AUC) summarizes ranking across thresholds, not correctness at one chosen threshold. In this table, each positive outranks respectively 4, 3, 2 and 1 negatives: 10 successes among 16 positive–negative pairs, giving AUC=0.625. Equal scores receive half credit; a threshold cannot split a tied group under our rule. AUC does not establish calibrated probabilities, select a policy or say how costly its errors are.
Precision also depends on prevalence, the positive fraction in the intended population. If class-conditional score distributions stay fixed, changing that fraction leaves TPR/FPR unchanged but changes the fraction of predicted positives that are truly positive. For example, TPR=0.75 and FPR=0.5 with 40 positives and 60 negatives would give TP=30 and FP=30, hence precision=0.5 rather than 0.6 at a 50% positive fraction. These are hypothetical counts under the fixed-rate assumption, not a new experimental result. A PR reference must therefore use the relevant prevalence, and even ROC behavior can change when the class-conditional distributions themselves shift. See the scikit-learn metrics guide for implementation conventions and further reading.
Several classes require a declared model
Several binary questions or one categorical distribution?
The notebook’s Iris task has three mutually exclusive species. It uses petal length and width, with a separate 90/30/30 split and a training-fitted scaler. We use that task to motivate multiclass models; the numerical logits below are hand-chosen and are not fitted Iris results.
Let \(c=0,\ldots,K-1\) index classes, reserving \(C\) for the later SVM penalty. Each class has a score \(z_c=w_c^\top g(x)+b_c\). A one-vs-rest (OvR) model asks \(K\) binary questions: “class \(c\) or another class?” Apply sigmoid to each score and train using BCE against one-hot binary indicators. The indicator for the observed class is one and the others are zero. The questions are fitted independently; their raw sigmoid outputs need not sum to one. To produce one class, the notebook chooses the largest score, equivalently the largest sigmoid value. Normalizing the outputs afterward is an extra convention, not the softmax training objective.
A softmax model instead couples the scores into one categorical distribution:
\[ q_c=\frac{e^{z_c}}{\sum_{j=0}^{K-1}e^{z_j}}. \]
Every value is positive and the values sum to one. Increasing one score while holding the others fixed changes the whole distribution. Adding the same constant \(a\) to every score leaves it unchanged: the factor \(e^a\) cancels from numerator and denominator. Multiplying the scores generally changes their separation and the resulting probabilities.
Consider the same illustrative scores under these two output maps:
| Class | Logit | OvR sigmoid | Softmax |
|---|---|---|---|
| 0 | \(\log2\) | \(2/3\) | \(4/7\) |
| 1 | \(0\) | \(1/2\) | \(2/7\) |
| 2 | \(-\log2\) | \(1/3\) | \(1/7\) |
| Sum | — | \(3/2\) | \(1\) |
Softmax normalizes \(2,1,1/2\) by their total \(7/2\). Both maps rank class zero highest here, but their training losses and probability interpretations differ. This table compares transformations of supplied scores, not two trained models’ accuracy. A softmax decision chooses the largest \(q_c\), equivalently \(z_c\); for ties we choose the lowest class index. The tie surface \(z_c=z_j\) is an actual decision boundary only where the tied scores dominate the alternatives.
Categorical cross-entropy and stable loss inputs
For observed class index \(y_i\), introduce the per-example categorical cross-entropy (CE) loss directly:
\[ L_i=-\log q_{i,y_i}. \]
Equivalently, with indicator \(u_{ic}=1\) when \(y_i=c\) and zero otherwise, \(L_i=-\sum_cu_{ic}\log q_{ic}\). Use the mean of these per-row losses for \(\bar L\). If the observed class in our table is one, the loss is \(-\log(2/7)=\log(7/2)\approx1.252763\). This is a penalty for the probability assigned to the observed class, not the winning class. Its likelihood and information-theory derivations remain in Week 07.
For stable computation, subtract the largest score \(m=\max_cz_c\) before exponentiation:
\[ \begin{aligned} q_c&=\frac{e^{z_c-m}}{\sum_je^{z_j-m}},\\ L_i&=-(z_{i,y_i}-m_i)\\ &\quad+\log\sum_je^{z_{ij}-m_i}. \end{aligned} \]
The common shift leaves the distribution unchanged while preventing large positive exponential arguments. PyTorch’s CrossEntropyLoss accepts logits directly. In the class-index convention used here, logits are \((B,K)\) floating-point values and targets are \((B,)\) integer class indices. Do not apply softmax before this fused loss. For OvR, BCEWithLogitsLoss instead receives \((B,K)\) logits and matching floating-point one-hot indicators; its mean reduction averages all \(BK\) binary entries, whereas mean categorical CE averages the \(B\) categorical losses. Their numerical scales are not directly comparable.
If weights are stored as \(W\in\mathbb R^{K\times d_g}\) with rows \(w_c^\top\), and the multiclass bias is a \((K,1)\) column \(b\), batch logits are \(Z=GW^\top+\mathbf1b^\top\). Here \(G\) is \((B,d_g)\), \(\mathbf1\) is \((B,1)\), and \(Z\) is \((B,K)\). This vector bias replaces the earlier scalar binary bias; softmax normalizes each observation’s row across classes.
from math import log
import torch
scores = torch.tensor([[log(2), 0., -log(2)]], dtype=torch.float64)
target = torch.tensor([1], dtype=torch.long)
criterion = torch.nn.CrossEntropyLoss(reduction="mean")
q = torch.softmax(scores, dim=1)
expected = torch.tensor([[4/7, 2/7, 1/7]], dtype=torch.float64)
assert torch.allclose(q, expected)
assert abs(criterion(scores, target).item() - log(3.5)) < 1e-12
assert torch.allclose(criterion(scores + 1000, target),
criterion(scores, target), atol=1e-12, rtol=0.)
assert abs(torch.sigmoid(scores).sum().item() - 1.5) < 1e-12This checks a transformation and loss, not model training. The notebook supplies the full OvR and softmax experiments. Selecting the model and decision policy still requires validation; a normalized distribution alone establishes neither calibration nor correctness.
Self-study: SVMs and kernels
Logistic regression learns a score that enters a probability model. A support vector machine (SVM) asks a different fitting question: can we separate the classes with a large margin, while allowing violations when the data overlap? The input representation still matters. A linear SVM cannot turn the two moons into a curved boundary merely by changing its loss.
This self-study supplies the margin and kernel vocabulary needed for the Week 03 comparison. Read the geometry and loss first, then use the small comparison to connect the formulas to a fitted model. Bishop’s kernel and sparse-kernel chapters develop the theory further (Bishop 2006). The full dual optimization and its optimality conditions (Karush–Kuhn–Tucker, or KKT, conditions) are optional extensions here.
Learn a margin rather than a probability
Translate the labels first. Return to binary observations \(y_i\in\{0,1\}\) and define signed labels \(t_i=2y_i-1\in\{-1,+1\}\). Start with a linear score in raw coordinates,
\[ f(x)=w^\top x+b,\qquad m_i=t_i f(x_i). \]
Here \(m_i\) is the functional margin: positive means that the score points towards the observed class, negative means the opposite. We declare class one when \(f(x)\geq0\); a score exactly zero is a tie resolved by this convention. Unlike \(q\), \(f\) can be any real number. It does not state a class probability.
For \(w\ne0\), the signed geometric distance of \(x\) to the hyperplane is \(f(x)/\lVert w\rVert_2\). Multiplying by \(t_i\) gives distance measured towards the correct side. Why divide by the norm? Moving a distance \(a\) along the unit normal \(w/\lVert w\rVert_2\) changes the score by \(a\lVert w\rVert_2\). Consequently the displacement needed to reach score zero has magnitude \(|f(x)|/\lVert w\rVert_2\).
For example, take \(w=(3,4)^\top\), \(b=-2\), \(x=(2,0)^\top\) and \(y=1\). Then \(f(x)=4\), \(\lVert w\rVert_2=5\), and the correctly signed distance is \(4/5=0.8\). Multiplying both \(w\) and \(b\) by seven changes the functional margin to \(28\), but the distance remains \(28/35=0.8\). Maximizing the unnormalized score would therefore mistake a change of scale for an improvement in separation.
Fix a scale before maximizing a margin. For linearly separable training data, the hard-margin SVM uses
\[ \begin{aligned} \underset{w,b}{\operatorname{minimize}}\quad &\frac12\lVert w\rVert_2^2,\\ \text{subject to}\quad &t_i f(x_i)\geq1\quad\text{for all }i. \end{aligned} \]
The constraints set the score scale. The two supporting planes \(f(x)=+1\) and \(f(x)=-1\) are each \(1/\lVert w\rVert_2\) from the decision plane, so the full margin width is \(2/\lVert w\rVert_2\). Minimizing the squared norm maximizes this width subject to separating every training observation. This statement assumes separability and uses the coordinates supplied to the model; it is not a promise of good performance on new data.
Allow overlap with hinge loss. No separating line can satisfy those constraints for contradictory labels at the same input. A soft-margin SVM introduces a nonnegative slack \(\xi_i\) and relaxes the constraint to \(t_i f(x_i)\geq1-\xi_i\). For fixed scores, the smallest feasible slack is
\[ \xi_i=\max(0,1-m_i)=L_i^{\mathrm{hinge}}. \]
| Signed margin \(m_i\) | Hinge loss | Interpretation |
|---|---|---|
| \(-0.5\) | \(1.5\) | Wrong side |
| \(0\) | \(1\) | On the decision plane |
| \(0.4\) | \(0.6\) | Correct side, inside the margin |
| \(1\) | \(0\) | On its supporting plane |
| \(1.6\) | \(0\) | Beyond its supporting plane |
Hinge loss penalizes a correctly classified point when its margin is too small. It is therefore different from counting classification errors. The zero-score row incurs loss one regardless of which class the tie rule selects.
We use the summed-loss convention for \(C\):
\[ \begin{aligned} J(w,b)&=\frac12\lVert w\rVert_2^2 +C\sum_{i=1}^{n}L_i^{\mathrm{hinge}},\\ &C>0. \end{aligned} \]
The intercept is unpenalized. Larger \(C\) gives margin violations more weight relative to the norm penalty; smaller \(C\) allows more violations in exchange for a smaller norm. Neither setting guarantees a particular validation accuracy. For comparison with mean losses elsewhere in this chapter, divide the entire objective by \(nC\), a positive constant:
\[ \begin{aligned} \frac{J}{nC} &=\frac1n\sum_i L_i^{\mathrm{hinge}} +\frac{\lambda}{2}\lVert w\rVert_2^2,\\ \lambda&=\frac{1}{nC}. \end{aligned} \]
Thus \(C\) is not a portable numerical measure of regularization across arbitrary loss reductions, sample counts or estimators. Read the objective before copying a value. The SVC mathematical formulation uses this sum convention; our comparison below uses SVC(kernel="linear"), not another estimator with a potentially different loss or intercept penalty.
Which observations support the fitted boundary? At an optimum, the score can be expressed using training observations with nonzero coefficients; these are the support vectors. In a separable hard-margin fit they sit on the supporting planes. In a soft-margin fit they can also lie inside the margin or be misclassified. A point far beyond its correct supporting plane has zero hinge loss and contributes no support-vector term. Degenerate cases can put a zero-coefficient observation exactly on a supporting plane, so location alone need not identify the stored support-vector set. Noise and overlap can leave many support vectors; “support” does not mean “correct” or “typical.”
Similarity can define a feature space implicitly
Earlier we explicitly constructed a cubic feature map \(g(x)\). An SVM can also use \(f(x)=w^\top g(x)+b\). The same margin argument then lives in feature space. Its distances are not generally Euclidean distances in the original \(x_1,x_2\) plot. Even ordinary standardization changes the metric in which a margin is measured, although a linear boundary remains a line in raw coordinates.
A kernel computes feature-space inner products without requiring us to construct every feature coordinate:
\[ k(x,x')=\langle g(x),g(x')\rangle. \]
For example, the homogeneous degree-two polynomial kernel has the explicit map
\[ \begin{aligned} g(x)&=(x_1^2,\sqrt{2}x_1x_2,x_2^2)^\top,\\ g(x)^\top g(x')&=(x^\top x')^2. \end{aligned} \]
The factor \(\sqrt{2}\) makes the cross term match the squared inner product. With \(x=(1,2)^\top\) and \(x'=(3,-1)^\top\), their raw inner product is \(1\); both the kernel and the explicit feature inner product equal \(1\). This map contains only degree-two monomials, unlike the earlier cumulative cubic map.
Three useful choices are
\[ \begin{aligned} k_{\mathrm{linear}}(x,x')&=x^\top x',\\ k_{\mathrm{poly}}(x,x')&=(\gamma x^\top x'+r)^p,\\ k_{\mathrm{RBF}}(x,x')&=\exp(-\gamma\lVert x-x'\rVert_2^2). \end{aligned} \]
For the polynomial expression here, use integer degree \(p\geq1\), \(\gamma>0\) and \(r\geq0\); these choices define a valid inner-product kernel. The radial basis function (RBF) kernel uses \(\gamma>0\) and corresponds to an infinite-dimensional feature space. A kernel must yield a positive semidefinite Gram matrix on any finite set of inputs; an arbitrary similarity score need not have that property. The explicit polynomial calculation illustrates why inner products, rather than visual similarity alone, are the key requirement.
The fitted binary score has the form
\[ \begin{aligned} f(x)&=\sum_{i\in\mathrm{SV}}\alpha_i t_i k(x_i,x)+b,\\ &\alpha_i>0. \end{aligned} \]
Here \(\mathrm{SV}\) indexes support vectors and \(\alpha_i\) are fitted coefficients. This prediction formula is enough to use kernels without deriving the full dual problem. Each query compares with the support vectors in the same preprocessed coordinates. In scikit-learn’s binary SVC, dual_coef_ stores \(\alpha_i t_i\), and decision_function returns the resulting score; the official guide explains these attributes. Applying a sigmoid to that score by itself does not supply a justified probability model or establish calibration.
Scaling and \(\gamma\) work together. For the two inputs above, \(\lVert x-x'\rVert_2^2=13\). At \(\gamma=0.5\), their RBF similarity is \(e^{-6.5}\approx0.00150\), whereas a point’s similarity to itself is \(1\). Larger \(\gamma\) makes similarity fall more quickly with distance and permits more localized changes in the boundary. Very large values can fit small noisy neighborhoods; very small values can make many distinct inputs almost alike. Changing a feature’s units changes these distances. Fit scaling on training rows and select both \(C\) and \(\gamma\) using validation, rather than interpreting \(\gamma\) independently of the representation.
Prepare an honest baseline for Week 03
Return to the capacity experiment: the same 540 training and 180 validation rows, with the test partition unused. Keep its already fitted linear logistic model as a reference. Fit one training-only standardization of the two raw features and reuse it for every SVM candidate; its statistics match the earlier linear logistic transform. Predictions are displayed in the original coordinates.
Declare the search before inspecting results: linear SVM uses \(C\in\{0.1,1,10\}\); RBF SVM uses the same values and \(\gamma\in\{0.1,1,10\}\). Select each family’s lowest validation error count, breaking ties by lower \(C\), then lower \(\gamma\). Do not refit after selection. Use zero SVM scores and logistic threshold \(\tau=0.5\) for decisions. Validation scores in this experiment contain no exact zero ties. All fits use unweighted classes and solver tolerance \(10^{-7}\); probability estimation is disabled.
| Model | Settings | Val. errors | Supports |
|---|---|---|---|
| Linear logistic reference | Frozen earlier fit | 22 | — |
| Linear SVM | \(C=0.1\) | 21 | 172 |
| RBF SVM | \(C=1\), \(\gamma=1\) | 9 | 94 |
“Val. errors” counts mistakes on 180 validation rows; “Supports” counts support vectors. The logistic reference is the earlier unpenalized fit with 1,500 full-batch updates, not a newly tuned logistic competitor. SVMs use a different loss, norm penalty and solver, and the RBF family has nine candidates versus three for the linear SVM. This is an exploratory boundary comparison, not a controlled claim that the kernel alone caused the error difference or that the search budgets were equal. RBF changes the available geometry; changing the fitting criterion within the linear family leaves a straight boundary.
The full candidate ledger records every validation count, selected support indices and fitted coefficients. Its generator reconstructs each selected score from the kernel expansion and checks it against the library. No test input is transformed or scored. The reported minima are selection results on one reused validation set, not independent estimates of future performance. A small score difference may be unstable; repeatedly widening the search after seeing validation results also adapts the procedure to that set.
For a new task, specify preprocessing, candidate families, search budget, selection metric and decision rule before comparison. If cross-validation is used, fit preprocessing separately within each training fold. Keep the test set outside that loop. Compare classification metrics on the same rows; do not feed raw SVM scores into BCE as if they were probabilities. The SVC API documents score access and explicit parameters; calibration is a separate later topic. Kernel methods also require storing and evaluating similarities with support vectors, a practical consideration as datasets grow.
Check your understanding. Explain why scaling both \(w\) and \(b\) does not improve the geometric margin, why a correct prediction may have positive hinge loss, and why an RBF boundary can curve without explicitly listing infinitely many features. Then state which comparison choices would have to be frozen before inspecting a final test result. These questions prepare the Week 03 baseline discussion without prescribing a solution to its assessed task.
Confidence, synthesis and optional exercises
A confident extrapolation is still an extrapolation
For a binary prediction, call \(\max(q,1-q)\) the model’s confidence in its more probable class. This is a property of the output distribution. It does not say how much relevant training evidence exists near the input.
Return to the four training locations in the small example. The training calculation gives the exact stationary model \(w=(\log3,\log3)^\top\), \(b=0\). Use this analytic model, rather than rounding the last animation frame. At A and B it assigns class one probability \(3/4\), matching the observed three-to-one label counts; at C and D it assigns \(1/4\). Now construct an unlabelled probe \(x=(r,r)^\top\), where \(r>1\):
\[ \begin{aligned} z(r,r)&=2r\log3,\\ q(r,r)&=\frac{3^{2r}}{1+3^{2r}}. \end{aligned} \]
All observed training coordinates lie in \([-1,1]\). At \((3,3)^\top\), the model nevertheless gives \(q=729/730\approx0.99863\). Moving farther along this ray makes the probability approach one, even though no new label is observed. The affine score extrapolates and the sigmoid saturates; neither operation checks whether the input resembles the training observations.
from math import exp, isclose, log
w = (log(3), log(3)) # exact stationary weights of the four-site toy
probe = (3., 3.) # constructed input; its label is unknown
z = sum(a * x for a, x in zip(w, probe))
q = 1 / (1 + exp(-z))
assert isclose(q, 729/730, abs_tol=1e-12)
assert q > .99The tutorial’s confidence experiment makes the same conceptual point with a ring of constructed out-of-distribution (OOD) probes around the moons data. Its ring is deliberately different from the training cloud, not a held-out sample from the moons process. Our exact probe is likewise a counterexample construction, not a benchmark or a validated OOD detector. Being outside an observed coordinate range alone does not establish which inputs a real population can generate.
Neither construction establishes whether a probe’s class prediction is right: there is no observed probe label. Nor does it show that distant points must receive high confidence; directions that keep the score near zero do not. The conclusion is narrower and useful: high class confidence alone cannot certify familiarity or correctness. Calibration asks whether predicted probabilities agree with observed class frequencies across suitable groups of labelled cases. Evidence of calibration on one population is not automatically evidence under a changed input distribution. Week 12 develops calibration and distribution shift; no uncertainty-method survey is needed to recognize the failure here.
One learning loop, several distinct objects
The chapter’s central question has three answers. A score describes the current model; a probability map interprets that score within a declared class model; a decision rule turns the output into an action. Fitting and evaluating these objects require different information.
- Choose inputs and a representation. Inspect training data, fit preprocessing there, and declare \(g(x)\). Scaling changes coordinates; polynomial features or kernels change the available geometry.
- Fit parameters with a declared objective. Logistic models use BCE or CE; the SVM uses a norm penalty and hinge loss. In the animation, reducing mean BCE moves the boundary through parameter updates. A different threshold does not perform another update.
- Select the model and decision policy on validation data. Check the metric that answers the task’s question. A probability loss, ranking measure and thresholded error count assess different properties. Include the representation, \(C\), \(\gamma\) and threshold among development choices where applicable.
- Freeze the procedure before final evaluation. Apply the learned preprocessing and frozen model/policy to held-out data. Interpret that result as evidence for the stated sampling conditions, rather than a guarantee for arbitrary future inputs.
The straight and curved moons boundaries show why fitting and representation must be considered together. The threshold sweep shows why a single fitted score can produce several policies. The unlabelled probe shows why even an extreme probability is a model output that still needs evidence.
Common confusions to keep separate
| Distinction | What to check |
|---|---|
| Score and distance | Divide a nonconstant linear score by the weight norm to obtain signed distance in the declared coordinates. |
| Probability and decision | State the positive class and threshold; changing the threshold leaves fitted probabilities unchanged. |
| Scaling and added capacity | Rescaling with a free intercept preserves the affine family; nonlinear features can change boundary shape. |
| Loss and accuracy | Loss uses scores or probabilities; accuracy counts decisions after a rule is applied. |
| OvR and softmax | Independent sigmoid outputs need not sum to one; softmax couples mutually exclusive alternatives. |
| SVM score and probability | A signed margin score is not itself a calibrated class probability. |
| Confidence and familiarity | An extreme predicted probability does not certify that the input is represented by training evidence. |
Optional exercises
Use natural logarithms, the chapter’s column-vector notation and inclusive binary thresholds. These are self-study questions, not additional assessed lab requirements. Show calculations or explain the evidence behind a conclusion.
Exercise 1 — Change the decision without fitting
For \(w=(2,-1)^\top\), \(b=0\) and \(x=(1,1)^\top\), calculate the score, signed distance and sigmoid probability. Give the decisions at \(\tau=0.5\) and \(0.8\). Now multiply both weights and intercept by two. Which of those quantities and decisions change? Explain why preserving the zero contour is insufficient to preserve a decision made at a nonzero score threshold.
Exercise 2 — One visible BCE update
Two training rows are \(x_1=(1,0)^\top\), \(y_1=1\) and \(x_2=(0,1)^\top\), \(y_2=0\). Start from \(w=(0,0)^\top\), \(b=0\). Using mean BCE, calculate both probabilities, the weight gradient and the intercept gradient. Perform one gradient-descent step with learning rate \(0.2\), updating \(b\) as well as \(w\). State the shapes of \(X\), \(z\), \(y\) and the weight gradient under the \((n,1)\) output convention. How would using a summed loss change this step at the same learning rate?
Exercise 3 — Change the units, then check the claim
An input coordinate is replaced by \(x'_1=100x_1\), with \(x'_2=x_2\). Find weights and intercept in the new coordinates that preserve every score of \(z=2x_1-x_2+3\). Explain why the new coordinates do not add an affine boundary shape, yet can change the behavior of gradient descent at a fixed learning rate. Would imposing the same numerical L2 penalty on the new weights preserve the original fitting objective? Justify your answer.
Exercise 4 — Counts before metrics
Use the eight-row validation table. At thresholds \(0.5\) and \(0.4\), calculate TP, FP, FN, TN, precision, recall and F1. Under the illustrative requirement of recall at least \(0.75\), which of these two policies is feasible? Explain why selecting it on this table does not turn its reported metrics into an independent test result. What must be reported when a threshold predicts no positive examples?
Exercise 5 — Three mutually exclusive classes
For logits \((0,\log2,\log3)\), calculate the three independent sigmoid outputs and their sum, then the softmax probabilities. With observed zero-based class index \(2\), calculate categorical CE and state the logits/target shapes for a single-row call to CrossEntropyLoss. Explain the effect of adding \(100\) to all logits and why the fused loss should receive the original logits.
Exercise 6 — Margin, penalty and similarity
For \(w=(3,4)^\top\), \(b=-2\), \(x=(0,0.75)^\top\) and \(y=0\), calculate the SVM score, signed label, functional margin, correctly signed geometric distance and hinge loss. For \(n=100\), rewrite \(\lVert w\rVert^2/2+C\sum_i L_i^{\mathrm{hinge}}\) as mean hinge plus a norm penalty, up to a positive constant. Separately, compare the RBF similarities of \((0,0)^\top\) and \((1,1)^\top\) at \(\gamma=0.5\) and \(2\). Which value makes the similarity more localized?
Exercise 7 — Far away, but which confidence?
Keep the exact stationary toy model used above. Move along \(x=(r,-r)^\top\) instead of \((r,r)^\top\). Determine the probability and model confidence as \(r\) grows. Reconcile this result with the high-confidence probe: what does each establish, and what does neither establish about unseen labels? State what kind of labelled evidence would be needed to assess calibration.
Exercise 8 — Repair an evaluation procedure
A researcher fits a scaler on all rows, searches several polynomial degrees and SVM settings using validation error, then checks test accuracy after every new search round and keeps whichever model has the best test score. Identify the two ways held-out information enters development. Describe a procedure with training-fitted preprocessing, validation-based selection and a frozen final evaluation. If using cross-validation, where must the scaler be fitted? If the test set has already guided selection, explain why renaming its best score a “final test” does not restore an independent evaluation.
Reading routes and connections
For geometry and probability decisions, revisit the score and threshold sections before Bishop’s linear classification chapter (Bishop 2006, chap. 4). For fitting, revisit the checked BCE update and Prince’s supervised-learning/loss chapters (Prince 2023). Use the tutorial for executable experiments; its likelihood and information-theory sections are later readings rather than prerequisites for this chapter.
For SVM self-study, work through the margin and kernel examples, then Bishop’s kernel and sparse-kernel chapters (Bishop 2006, chaps. 6–7) and the official SVM guide below. Full dual/KKT derivations remain optional.
Week 03 will replace a single score boundary with recursive partitions. Week 04 learns representations instead of fixing a polynomial map. Week 07 derives classification losses from observation models and develops the probability perspective. Its multiclass class-count symbol \(C\) corresponds to this chapter’s \(K\); here \(C\) was reserved for the SVM penalty.
References and further reading
- Bishop, Pattern Recognition and Machine Learning: linear classification, kernel methods and sparse kernel methods (Bishop 2006).
- Prince, Understanding Deep Learning: supervised learning and loss functions (Prince 2023); use the losses now, their likelihood derivations later.
- scikit-learn SVM guide: practical kernel choices and decision-score conventions.