Nonlinear representation learning
with feedforward neural networks

Week-04 · ML-2026

Alexei Kornaev,
Dr. Sc., Prof.

Innopolis University

Researcher at the RC for AI,
National RC for Oncology n.a. N.N. Blokhin

Lecture map

1. Learned representations and MLPs
2. Forward propagation, outputs, and tasks
3. Backpropagation and autograd
4. Training and generalization

Print / save as PDF

Part 1 · Learned representations and MLPs

1. Learned representations and MLPs
2. Forward propagation, outputs, and tasks
3. Backpropagation and autograd
4. Training and generalization

From artificial neurons to nonlinear multilayer architectures.

Where are we in machine learning?

Model	Prediction	What is learned?	Source of flexibility
Linear regression	$\mathbf w^\top\mathbf x+b$	Coefficients	Linear response
Logistic regression	$\sigma(\mathbf w^\top\mathbf x+b)$	Decision boundary	Linear boundary in feature space
Decision tree	Leaf value	Splits and leaves	Recursive partitioning
Forest / boosting	Ensemble prediction	Multiple trees	Combined partitions
MLP	$f^{(L)}\!\circ\cdots\circ f^{(1)}(\mathbf x)$	Transformations and predictor	Learned nonlinear representation

The final row adds learned features between the supplied input and the prediction.

The learning algorithm remains the same

Logistic regression

$$\widehat y=\sigma(\mathbf w^\top\mathbf x+b)$$

Two-layer network

$$\mathbf h=\phi(\mathbf W^{(1)}\mathbf x+\mathbf b^{(1)}), \qquad \widehat y=g(\mathbf W^{(2)}\mathbf h+\mathbf b^{(2)})$$

$$\theta \longrightarrow \widehat y \longrightarrow \mathcal L \longrightarrow \nabla_\theta\mathcal L \longrightarrow \theta-\eta\nabla_\theta\mathcal L$$

$\theta=\{\mathbf W^{(\ell)},\mathbf b^{(\ell)}\}_{\ell=1}^{L}$ collects all trainable parameters. The output transformation $g$ depends on the prediction task.

The model becomes richer; the learning loop remains familiar.

The network learns the representation

Earlier layers learn the features used by later layers.

Natural and artificial neurons

Natural neuron. Receives electrochemical signals through dendrites and synapses, integrates them across space and time, and may generate an action potential transmitted along the axon.

Artificial neuron. Receives numerical inputs, scales them by learned weights, adds a bias, and applies an activation function to produce an output.

Biological terminology	Artificial abstraction
Incoming signals	Inputs $x_j$
Synaptic influence	Weights $w_j$
Signal integration	Weighted sum and bias
Firing response	Activation function
Axon signal	Output $h$

This is a functional analogy, not a mechanistic model: biological neurons are dynamic, spiking, adaptive cells; an artificial neuron is a simplified differentiable computation.

An artificial neuron accumulates evidence

$$z=\mathbf w^\top\mathbf x+b = \sum_{j=1}^{d}w_jx_j+b, \qquad h=\phi(z).$$

The weighted sum collects signed evidence. The activation shapes the signal passed onward.

Activation functions shape the signal

$\phi(z)=z$
Range: $\mathbb R$
Preserves the affine signal.

$\phi(z)=\frac{1}{1+e^{-z}}$
Range: $(0,1)$
Maps scores to bounded positive outputs.

$\phi(z)=\tanh z$
Range: $(-1,1)$
Produces a zero-centred bounded signal.

$\phi(z)=\max(0,z)$
Range: $[0,\infty)$
Keeps positive evidence and clips negative values.

Variants such as Leaky ReLU modify the inactive side while preserving a piecewise-linear form.

Without nonlinearity, depth disappears

$$\mathbf h=\mathbf W^{(1)}\mathbf x+\mathbf b^{(1)}, \qquad \mathbf y=\mathbf W^{(2)}\mathbf h+\mathbf b^{(2)}.$$

$$\begin{aligned} \mathbf y &=\mathbf W^{(2)}(\mathbf W^{(1)}\mathbf x+\mathbf b^{(1)})+\mathbf b^{(2)}\\ &=(\mathbf W^{(2)}\mathbf W^{(1)})\mathbf x +(\mathbf W^{(2)}\mathbf b^{(1)}+\mathbf b^{(2)}) \end{aligned}$$

$$\mathbf y=\widetilde{\mathbf W}\mathbf x+\widetilde{\mathbf b}.$$

Two affine layers reduce to one equivalent affine layer.

Nonlinearity prevents a deep network from collapsing into a single affine transformation.

From one neuron to a layer

Each neuron has its own weights and bias:

$$h_i=\phi(\mathbf w_i^\top\mathbf x+b_i).$$

Stacking the neuron parameters gives one vectorized layer:

$$\mathbf h=\phi(\mathbf W\mathbf x+\mathbf b).$$

The rows of $\mathbf W$ contain the neurons’ weight vectors.

Anatomy of a multilayer perceptron

$$4\longrightarrow5\longrightarrow3\longrightarrow2, \qquad \mathbf h^{(0)}=\mathbf x, \qquad \mathbf h^{(\ell)}= \phi^{(\ell)}\!\left(\mathbf W^{(\ell)}\mathbf h^{(\ell-1)}+\mathbf b^{(\ell)}\right).$$

Part 2 · Forward propagation, outputs, and tasks

1. Learned representations and MLPs
2. Forward propagation, outputs, and tasks
3. Backpropagation and autograd
4. Training and generalization

Follow values through the network and interpret its outputs.

Shapes are part of the mathematics

One sample

$$\mathbf z^{(\ell)}=\mathbf W^{(\ell)}\mathbf h^{(\ell-1)}+\mathbf b^{(\ell)}.$$

$$\begin{aligned} \mathbf h^{(\ell-1)}&\in\mathbb R^{d_{\ell-1}},\\ \mathbf W^{(\ell)}&\in\mathbb R^{d_\ell\times d_{\ell-1}},\\ \mathbf b^{(\ell)},\mathbf z^{(\ell)}&\in\mathbb R^{d_\ell}. \end{aligned}$$

Batch of $B$ samples

$$\mathbf Z^{(\ell)}=\mathbf H^{(\ell-1)}\mathbf W^{(\ell)\top}+\mathbf b^{(\ell)} \in\mathbb R^{B\times d_\ell}.$$

PyTorch stores each neuron’s weights as one row of weight, with shape $d_{\text{out}}\times d_{\text{in}}$.

One neuron: a numerical forward pass

$$\mathbf x=\begin{bmatrix}2\\-1\end{bmatrix},\qquad \mathbf w_1=\begin{bmatrix}0.5\\1\end{bmatrix},\qquad b_1=0.5.$$

$$\mathbf w_1^\top\mathbf x+b_1=0.5(2)+1(-1)+0.5.$$

$$z_1=0.5.$$

$$h_1=\operatorname{ReLU}(0.5)=0.5.$$

Calculate the weighted contributions before revealing the result.

One layer: neurons work in parallel

$$\mathbf W^{(1)}= \begin{bmatrix} 0.5&1\\-1&0.5\\1&1 \end{bmatrix},\qquad \mathbf b^{(1)}=\begin{bmatrix}0.5\\0\\-0.5\end{bmatrix}.$$

$$\mathbf z^{(1)}=\mathbf W^{(1)}\mathbf x+\mathbf b^{(1)} =\begin{bmatrix}0.5\\-2.5\\0.5\end{bmatrix}.$$

$$\mathbf h^{(1)}=\operatorname{ReLU}(\mathbf z^{(1)}) =\begin{bmatrix}0.5\\0\\0.5\end{bmatrix}.$$

A layer performs several neuron calculations in one matrix operation.

A complete MLP forward pass

$$\mathbf W^{(2)}=\begin{bmatrix}2&-1&1\end{bmatrix}, \qquad b^{(2)}=-0.5.$$

$$\begin{aligned} \hat y &=\mathbf W^{(2)}\mathbf h^{(1)}+b^{(2)}\\ &=2(0.5)-1(0)+1(0.5)-0.5\\ &=1. \end{aligned}$$

Blue labels mark stored parameters; circles show intermediate values and the final prediction.

Forward propagation composes affine transformations with hidden-layer activations; the output transformation depends on the task.

From network outputs to predictions

Task	Network output	Interpretation	Typical loss
Regression	$K$ real values	Predicted quantities	MSE
Binary classification	One logit $z$	$p(y=1\mid x)=\sigma(z)$	BCE with logits
Multiclass classification	$K$ logits	Softmax probabilities; class $\arg\max_k z_k$	Cross-entropy

BCEWithLogitsLoss and CrossEntropyLoss expect logits and apply the required stable transformation internally.

$\arg\max_k\operatorname{softmax}(\mathbf z)_k=\arg\max_k z_k$.

What does nn.Linear compute?

$$\mathbf Z=\mathbf X\mathbf W^\top+\mathbf b.$$

X                  (B, 2)
layer.weight       (3, 2)
layer.bias         (3,)
Z                  (B, 3)

layer = nn.Linear(
    in_features=2, out_features=3
)

Z = layer(X)
Z_explicit = (
    X @ layer.weight.T + layer.bias
)

torch.testing.assert_close(Z, Z_explicit)

nn.Linear performs the affine transformation. The activation is a separate operation.

Source: PyTorch nn.Linear documentation (PyTorch contributors, n.d.-c).

Softmax turns logits into probabilities

$$\mathbf z_i=[z_{i1},\ldots,z_{iK}]\in\mathbb R^K, \qquad q_{ik}=q_\theta(y=k\mid\mathbf x_i) =\frac{\exp(z_{ik})}{\sum_{j=1}^{K}\exp(z_{ij})}.$$

$$q_{ik}>0,\qquad\sum_{k=1}^{K}q_{ik}=1,\qquad \operatorname{softmax}(\mathbf z+c\mathbf 1)=\operatorname{softmax}(\mathbf z).$$

Softmax converts relative evidence into a categorical probability distribution.

Numerically stable implementations subtract the largest logit before exponentiation.

Softmax and cross-entropy complete the classifier

Model output

$$q_{ik}=\operatorname{softmax}(\mathbf z_i)_k.$$

$$\mathbf z=[2,1,0]\quad\longrightarrow\quad\mathbf q\approx[0.665,0.245,0.090].$$

Observed class

$$\mathbf y_i=[y_{i1},\ldots,y_{iK}],\quad y_{ik}\in\{0,1\},\quad\sum_{k=1}^{K}y_{ik}=1.$$

$$\mathbf y=[0,1,0].$$

$$\ell_i=-\sum_{k=1}^{K}y_{ik}\log q_{ik}=-\log q_{i,c_i}, \qquad\ell=-\log(0.245)\approx1.41.$$

The one-hot target selects the log-probability assigned to the observed class. Higher probability gives a smaller loss; lower probability gives a larger loss.

loss = F.cross_entropy(logits, targets)

logits $(B,K)$ · targets $(B,)$ integer class indices

cross_entropy expects logits. It applies a stable log-softmax and negative log-likelihood internally (PyTorch contributors, n.d.-b).

How to read an architecture

1. What does one input represent?
2. What does the output dimension represent?
3. Where are the wide and narrow layers?
4. Is there a bottleneck or symmetry?
5. Which output transformation and loss complete the model?

An architecture describes a mapping; the data, output interpretation, and loss define the task.

Guess the task: function approximator

Scalar input $x$ and target $y=f(x)$ suggest regression or function approximation.

Linear output; typical loss: MSE.

Guess the task: image classifier

$784=28\times28$ grayscale pixels enter two learned hidden representations. Ten output logits suggest multiclass classification; typical loss: cross-entropy.

The dimensions suggest an MNIST-like task, but architecture alone does not identify the dataset.

Guess the task: autoencoder

$$\mathbf x\xrightarrow{\text{encoder}}\mathbf z \xrightarrow{\text{decoder}}\hat{\mathbf x}.$$

Matching input and output dimensions, approximate symmetry, and a $32$-dimensional bottleneck suggest reconstruction. The learned representation can also support dimensionality reduction or anomaly detection.

The bottleneck asks the network to preserve useful information in a smaller representation.

Part 3 · Backpropagation and autograd

1. Learned representations and MLPs
2. Forward propagation, outputs, and tasks
3. Backpropagation and autograd
4. Training and generalization

Use the chain rule to assign sensitivity to every parameter.

Learning asks who caused the error

$$\hat{\mathbf y}=f_\theta(\mathbf x),\qquad L=\mathcal L(\hat{\mathbf y},\mathbf y),$$

$$\nabla_\theta L=\left[ \frac{\partial L}{\partial\theta_1},\ldots, \frac{\partial L}{\partial\theta_P} \right].$$

How much would the loss change if this parameter changed slightly?

A computation graph remembers the forward pass

$$z=wx+b=5\;\longrightarrow\; h=\operatorname{ReLU}(z)=5\;\longrightarrow\; \hat y=ah+c=1\;\longrightarrow\; L=\tfrac12(\hat y-y)^2=0.5.$$

Supplied: $x=3,y=2$. Trainable: $w=2,b=-1,a=0.4,c=-1$.

The backward pass uses the graph structure and selected values produced during the forward pass.

Every operation contributes a local derivative

$$\frac{\partial L}{\partial\hat y}=\hat y-y=-1, \qquad \frac{\partial\hat y}{\partial h}=a=0.4.$$ $$\frac{\partial\hat y}{\partial a}=h=5, \qquad \frac{\partial\hat y}{\partial c}=1.$$

$$\frac{\partial h}{\partial z}=1\quad(z=5>0),$$ $$\frac{\partial z}{\partial w}=x=3, \qquad \frac{\partial z}{\partial b}=1.$$

Each operation combines its local derivative with the sensitivity arriving from later operations.

The chain rule follows a path

$$\frac{\partial L}{\partial w} =\frac{\partial L}{\partial\hat y} \frac{\partial\hat y}{\partial h} \frac{\partial h}{\partial z} \frac{\partial z}{\partial w} =(-1)(0.4)(1)(3)=-1.2.$$

Along a dependency path, local derivatives multiply.

Branches accumulate responsibility

$$h\longrightarrow\hat y_1\longrightarrow L_1, \qquad h\longrightarrow\hat y_2\longrightarrow L_2, \qquad L=L_1+L_2.$$ $$\frac{\partial L}{\partial h} =\frac{\partial L_1}{\partial h} +\frac{\partial L_2}{\partial h}.$$

Along paths, derivatives multiply; where paths meet, gradient contributions add.

One complete forward and backward pass

Forward values

Quantity	Value
$z$	$5$
$h$	$5$
$\hat y$	$1$
$L$	$0.5$

Backward sensitivities

Gradient	Value	Gradient	Value
$\partial L/\partial\hat y$	$-1$	$\partial L/\partial a$	$-5$
$\partial L/\partial h$	$-0.4$	$\partial L/\partial c$	$-1$
$\partial L/\partial z$	$-0.4$	$\partial L/\partial w$	$-1.2$
$\partial L/\partial x$	$-0.8$	$\partial L/\partial b$	$-0.4$

Forward propagation computes values. Backpropagation computes sensitivities.

Backpropagation through one layer

$$\mathbf Z=\mathbf H_{\mathrm{in}}\mathbf W^\top+\mathbf b, \qquad \mathbf H=\phi(\mathbf Z), \qquad \mathbf G_H=\frac{\partial L}{\partial\mathbf H}.$$

$$\begin{aligned} \mathbf G_Z&=\mathbf G_H\odot\phi'(\mathbf Z),\\ \frac{\partial L}{\partial\mathbf W}&=\mathbf G_Z^\top\mathbf H_{\mathrm{in}},\\ \frac{\partial L}{\partial\mathbf b}&=\sum_{i=1}^{B}\mathbf G_{Z,i},\\ \frac{\partial L}{\partial\mathbf H_{\mathrm{in}}}&=\mathbf G_Z\mathbf W. \end{aligned}$$

Quantity	Shape
$\mathbf H_{\mathrm{in}}$	$B\times d_{\mathrm{in}}$
$\mathbf W$	$d_{\mathrm{out}}\times d_{\mathrm{in}}$
$\mathbf G_Z$	$B\times d_{\mathrm{out}}$
$\partial L/\partial\mathbf W$	$d_{\mathrm{out}}\times d_{\mathrm{in}}$
$\partial L/\partial\mathbf H_{\mathrm{in}}$	$B\times d_{\mathrm{in}}$

Incoming sensitivity $\mathbf G_H$ → parameter gradients → sensitivity to the previous layer.

Backpropagation through an MLP

$$\begin{aligned} \mathbf G_Z^{(\ell)} &=\mathbf G_H^{(\ell)}\odot\phi'(\mathbf Z^{(\ell)}),\\ \nabla_{\mathbf W^{(\ell)}}L &=\mathbf G_Z^{(\ell)\top}\mathbf H^{(\ell-1)},\\ \mathbf G_H^{(\ell-1)} &=\mathbf G_Z^{(\ell)}\mathbf W^{(\ell)}. \end{aligned}$$

Backpropagation repeats the same layer-level calculation in reverse order.

Reverse-mode differentiation computes vector–Jacobian products without constructing every full Jacobian explicitly.

Autograd executes the graph backward

x = torch.tensor(3.0)
y = torch.tensor(2.0)

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(-1.0, requires_grad=True)
a = torch.tensor(0.4, requires_grad=True)
c = torch.tensor(-1.0, requires_grad=True)

z = w * x + b
h = torch.relu(z)
y_hat = a * h + c
loss = 0.5 * (y_hat - y) ** 2

loss.backward()

$$\frac{\partial L}{\partial L}=1$$

The scalar loss seeds the backward pass.

w.grad = -1.2
b.grad = -0.4
a.grad = -5.0
c.grad = -1.0

PyTorch accumulates .grad on leaf tensors with requires_grad=True. Intermediate gradients can be retained explicitly (PyTorch contributors, n.d.-a).

What does x.grad compute?

$$x.\mathrm{grad}=\frac{\partial L}{\partial x} =\frac{\partial L}{\partial z} \frac{\partial z}{\partial x} =(-0.4)(2)=-0.8.$$

x = torch.tensor(3.0, requires_grad=True)
# same forward pass as before
loss.backward()
x.grad  # tensor(-0.8000)

Tensor	Interpretation
w.grad	Loss sensitivity to a trainable weight; used by the optimizer
x.grad	Loss sensitivity to the input
z.grad	Not stored by default: $z$ is a non-leaf intermediate
loss	Seeds backward with $\partial L/\partial L=1$

Call z.retain_grad() before backward() to store an intermediate gradient. Input gradients support sensitivity analysis, saliency, adversarial methods, and input optimization (PyTorch contributors, n.d.-a).

Checking and interpreting gradients

Central finite difference

$$\frac{\partial L}{\partial\theta_i} \approx \frac{L(\theta_i+\varepsilon)-L(\theta_i-\varepsilon)}{2\varepsilon}.$$

For the running example:

$$\frac{\partial L}{\partial w}\approx-1.2.$$

Observation	Possible interpretation
Gradient near zero	Saturation, inactive path, convergence, or weak sensitivity
Very large gradient	Exploding gradient or poor scaling
No gradient stored	Backward has not run, gradients are disabled, the tensor is disconnected, or a non-leaf gradient was not retained
NaN or Inf	Numerical instability

A gradient measures local sensitivity and provides a diagnostic signal; its negative gives the steepest-descent direction.

Part 4 · Training and generalization

1. Learned representations and MLPs
2. Forward propagation, outputs, and tasks
3. Backpropagation and autograd
4. Training and generalization

Turn gradients into useful learning and evaluate what generalizes.

From gradients to a training loop

$$\hat{\mathbf Y}=f_\theta(\mathbf X_B),\qquad L_B=\frac{1}{B}\sum_{i\in B}\mathcal L(\hat{\mathbf y}_i,\mathbf y_i),\qquad \theta_{t+1}=\theta_t-\eta\,\widehat{\nabla_\theta L}_{B_t}.$$

1. Sample a mini-batch.
2. Compute predictions.
3. Compute the loss.
4. Clear old gradients.
5. Backpropagate.
6. Update parameters.
7. Repeat.

for X, y in loader:
    optimizer.zero_grad()

    y_hat = model(X)
    loss = criterion(y_hat, y)

    loss.backward()
    optimizer.step()

Autograd computes gradients; the optimizer uses them to change the parameters.

PyTorch accumulates gradients, so clear them before the next backward pass (PyTorch contributors, n.d.-d, n.d.-e).

The loss surface is not the training trajectory

$$\nabla_\theta L(\theta_t)\quad\text{local information}$$ $$\theta_0,\theta_1,\ldots,\theta_T\quad\text{resulting trajectory}$$

The objective defines the landscape; the training procedure determines how we move through it.

The surface depends on the model, data, and objective. The trajectory also depends on initialization, mini-batches, learning rate, and optimizer state. A two-dimensional contour is only a slice or projection.

The learning rate controls every step

$$g_t=\widehat{\nabla_\theta L}_{B_t},\qquad \Delta\theta_t=-\eta g_t.$$

The gradient measures local sensitivity; its negative supplies the descent direction, and the learning rate controls the update scale.

Too small can be slow; too large can overshoot or diverge. There is no universally optimal learning rate.

What does an optimizer change?

Optimizer	Main mechanism	Practical consequence
SGD	Current mini-batch gradient	Simple; sensitive to learning rate and scale
Momentum	Accumulates a moving direction	Reduces oscillation and preserves motion
Adam	Adaptive first- and second-moment estimates	Often convenient and fast early in training
AdamW	Adam with decoupled weight decay	Shrinkage is separate from adaptive gradient scaling

$$g_t=\widehat{\nabla_\theta L}_{B_t},\qquad v_t=\beta v_{t-1}+g_t,\qquad \theta_{t+1}=\theta_t-\eta v_t.$$

An optimizer transforms gradients—and often their history—into parameter updates.

AdamW applies decoupled parameter shrinkage; it is not merely Adam plus an L2 term (Kingma and Ba 2015; Loshchilov and Hutter 2019).

Mini-batches estimate the full gradient

$$\nabla_\theta L_{\mathrm{full}}=\frac1N\sum_{i=1}^{N}\nabla_\theta\mathcal L_i, \qquad \widehat{\nabla_\theta L}_{B}=\frac1B\sum_{i\in B}\nabla_\theta\mathcal L_i.$$

Smaller batch	Larger batch
Noisier estimate	More stable estimate
More updates per data pass	Fewer updates per data pass
Lower memory requirement	Higher memory requirement
Less parallel efficiency	Better utilization, up to a limit

Mini-batches trade gradient certainty for computational efficiency and more frequent updates.

The estimate is not an “incorrect gradient.” Its exact variance and unbiasedness depend on the sampling procedure.

Initialization determines what can propagate

$$\mathbf h^{(\ell-1)}\longrightarrow\mathbf z^{(\ell)}\longrightarrow\mathbf h^{(\ell)}$$

$$\operatorname{Var}(z_j^{(\ell)})\approx d_{\mathrm{in}}\operatorname{Var}(W_{jk}^{(\ell)}) \operatorname{Var}(h_k^{(\ell-1)}).$$

$$\text{Xavier: }\operatorname{Var}(W)\approx\frac{2}{d_{\mathrm{in}}+d_{\mathrm{out}}}$$

$$\text{He: }\operatorname{Var}(W)\approx\frac{2}{d_{\mathrm{in}}}\quad\text{for ReLU-like activations}$$

Initialization should keep signals and sensitivities informative across layers.

These variance rules use simplifying assumptions; they do not guarantee stable training in every architecture (Glorot and Bengio 2010; He et al. 2015).

Validation protects the final claim

Split	Role
Training	Fit model parameters
Validation	Select epoch, architecture, and hyperparameters
Test	Estimate final performance after decisions are fixed

A decreasing training loss proves successful optimization, not successful generalization.

Early stopping selects an epoch from validation performance. The test set is not monitored as a training curve.

Regularization constrains how the model learns

Mechanism	Where it acts	Main effect
L1/L2 penalty	Objective	Discourages parameter configurations
Weight decay	Update rule	Shrinks selected parameters
Dropout	Network during training	Randomly masks activations or units during training
Data augmentation	Training data	Encodes plausible invariances
Early stopping	Training duration	Limits training-specific adaptation

Regularization constrains parameters, representations, data, or training time.

L2 penalty and weight decay match for ordinary SGD under the corresponding update, but not generally for adaptive optimizers. Dropout is off at evaluation; the framework handles scaling. Biases and normalization parameters are often excluded from weight decay (Srivastava et al. 2014; Loshchilov and Hutter 2019).

A brief history of artificial neural networks

1943 · A mathematical neuron

McCulloch and Pitts propose a logical model of neural computation.

(McCulloch and Pitts 1943)

1958 · A neuron that learns

Rosenblatt introduces the perceptron and its learning rule.

(Rosenblatt 1958)

1969 · Limits of a single layer

Minsky and Papert analyze what single-layer perceptrons cannot represent.

(Minsky and Papert 1969)

1970s–1986 · Learning through many layers

Reverse differentiation becomes a practical way to assign credit through composed computations.

(Werbos 1974; Rumelhart et al. 1986)

2006–2012 · The deep-learning revival

Better training, larger datasets, GPUs, and new architectures make deep networks practical.

(Hinton et al. 2006; Krizhevsky et al. 2012)

Today · Differentiable programming

PyTorch constructs computation graphs and calculates gradients automatically.

(PyTorch contributors, n.d.-a)

Neural networks became transformative when expressive hidden representations met a practical method for assigning credit through many layers.

Person of the day: Geoffrey Hinton

Geoffrey Hinton helped turn backward credit assignment into a practical foundation of modern deep learning.

1986 · Learning internal representations

Rumelhart, Hinton, and Williams showed how hidden representations can be learned by propagating error derivatives backward (Rumelhart et al. 1986).

2006 · Reviving deep networks

Hinton, Osindero, and Teh introduced an influential layer-wise training method for deep belief networks (Hinton et al. 2006).

2012 · Deep learning at scale

Krizhevsky, Sutskever, and Hinton demonstrated deep convolutional networks trained with GPUs on ImageNet (Krizhevsky et al. 2012).

$$\mathcal{L}(f_\theta(\mathbf x),y) \quad\xrightarrow{\text{reverse-mode differentiation}}\quad \nabla_\theta \mathcal{L}.$$

Backpropagation organizes the chain rule through a computation graph. Autograd is the software machinery that performs this differentiation automatically.

2018 ACM Turing Award (Association for Computing Machinery 2019) · 2024 Nobel Prize in Physics (Nobel Prize Outreach 2024)

References

Association for Computing Machinery. 2019. “Fathers of the Deep Learning Revolution Receive ACM A.M. Turing Award.” https://awards.acm.org/about/2018-turing.

Glorot, Xavier, and Yoshua Bengio. 2010. “Understanding the Difficulty of Training Deep Feedforward Neural Networks.” Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of machine learning research, vol. 9: 249–56. https://proceedings.mlr.press/v9/glorot10a.html.

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.” Proceedings of the IEEE International Conference on Computer Vision, 1026–34. https://openaccess.thecvf.com/content_iccv_2015/html/He_Delving_Deep_into_ICCV_2015_paper.html.

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. 2006. “A Fast Learning Algorithm for Deep Belief Nets.” Neural Computation 18 (7): 1527–54. https://doi.org/10.1162/neco.2006.18.7.1527.

Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” International Conference on Learning Representations. https://arxiv.org/abs/1412.6980.

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems 25, 1097–105. https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html.

Loshchilov, Ilya, and Frank Hutter. 2019. “Decoupled Weight Decay Regularization.” International Conference on Learning Representations. https://arxiv.org/abs/1711.05101.

McCulloch, Warren S., and Walter Pitts. 1943. “A Logical Calculus of the Ideas Immanent in Nervous Activity.” The Bulletin of Mathematical Biophysics 5: 115–33. https://doi.org/10.1007/BF02478259.

Minsky, Marvin, and Seymour A. Papert. 1969. Perceptrons: An Introduction to Computational Geometry. The MIT Press. https://mitpress.mit.edu/9780262130431/perceptrons/.

Nobel Prize Outreach. 2024. “The Nobel Prize in Physics 2024.” https://www.nobelprize.org/prizes/physics/2024/summary/.

PyTorch contributors. n.d.-a. “Autograd Mechanics.” Accessed September 22, 2026. https://docs.pytorch.org/docs/stable/notes/autograd.html.

PyTorch contributors. n.d.-b. “CrossEntropyLoss — PyTorch Documentation.” Accessed September 22, 2026. https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html.

PyTorch contributors. n.d.-c. “Linear — PyTorch Documentation.” Accessed September 22, 2026. https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html.

PyTorch contributors. n.d.-d. “torch.optim — PyTorch Documentation.” Accessed September 22, 2026. https://docs.pytorch.org/docs/stable/optim.html.

PyTorch contributors. n.d.-e. “Zeroing Out Gradients in PyTorch.” Accessed September 22, 2026. https://docs.pytorch.org/tutorials/recipes/recipes/zeroing_out_gradients.html.

Rosenblatt, Frank. 1958. “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.” Psychological Review 65 (6): 386–408. https://doi.org/10.1037/h0042519.

Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. 1986. “Learning Representations by Back-Propagating Errors.” Nature 323 (6088): 533–36. https://doi.org/10.1038/323533a0.

Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” Journal of Machine Learning Research 15 (56): 1929–58. https://www.jmlr.org/papers/v15/srivastava14a.html.

Werbos, Paul John. 1974. “Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences.” PhD thesis, Harvard University. https://www.proquest.com/docview/302768593.

References — continued