Lecture 7: Neural Networks

Summary of Key Ideas

In the earlier lectures, we made linear models more flexible by transforming the inputs using basis functions. We chose those functions ourselves and learnt how to combine them.

In this lecture, we take the next step: learn the features as well.

A neural network learns representations through a composition of parameterised computations. Each computation transforms the representation produced by the previous one. We will study the multilayer perceptron, or MLP, where each layer computes weighted sums, adds biases, and applies an activation function.

Learning still uses the same ingredients: data, a model, and a loss. The output and loss depend on the task, just as they did for regression and classification. To train every layer, we need derivatives through the composed computations. Backpropagation computes them efficiently using the chain rule. For an ordinary weighted connection, the gradient has a simple form:

\[\boxed{ \text{weight gradient} = \text{error signal at the receiving unit} \;\times\; \text{input to the connection}. }\]

We will then discuss why this approach can work. The Universal Approximation Theorem (UAT) tells us that a sufficiently wide network with one hidden layer can approximate any continuous function on a compact domain arbitrarily well. Depth can make some functions much cheaper to represent by reusing earlier computations.

Representing a function and learning it from data are separate questions. We will show why gradient descent with a suitable learning rate is guaranteed to decrease a smooth training loss, and why stochastic gradient descent (SGD) can use a few sampled examples to estimate the full gradient. With a fixed step size, sampling noise can keep SGD fluctuating near a minimum.

Finally, reducing training loss is only part of machine learning. We want good predictions on new data. Both the architecture and the training process influence which solution we learn.

Learning Objectives

By the end of this lesson, you should be able to:

  1. Explain a neural network as a composition of functions to compute features.
  2. Compute a Multilayer Perceptron (MLP) prediction and choose a suitable output and loss.
  3. Write a weight gradient as error signal × input, and use it in an update.
  4. Explain the Universal Approximation Theorem (UAT) and the benefit of depth over shallow networks.
  5. Interpret the convergence properties of gradient descent and fixed-step stochastic gradient descent (SGD).

We will first connect neural networks to basis functions, then build an MLP and learn its parameters. We will then study what networks can represent and what gradient-based learning can guarantee.

We focus on the notation, the form of each weight gradient, and how the gradients become parameter updates. The notes include the gradient-descent and SGD arguments in full. Tutorial 6 provides practice with these arguments and the detailed backpropagation calculations. Architecture and inductive bias are a short preview of Lecture 8.

What Is a Neural Network?

From Fixed to Adaptive Basis Functions

Recall the model

\[y(\mathbf{x}) = \mathbf{w}^{\top}\boldsymbol{\phi}(\mathbf{x}) = \sum_{j=1}^{M}w_j\phi_j(\mathbf{x}).\]

Previously, we chose each \phi_j before fitting the model. Learning only changed the output weights \mathbf{w}.

Suppose instead that the features have parameters of their own:

\[\boxed{ y(\mathbf{x}) = \mathbf{w}^{\top}\boldsymbol{\phi}_{\theta}(\mathbf{x}). }\]

Now learning can change both the feature map and the way its outputs are combined.

For a concrete example, consider a squared exponential basis function:

\[\phi_j(x) = \exp\left[ -\frac{(x-c_j)^2}{2s_j^2} \right], \qquad s_j>0.\]

The centre c_j determines where the bump is located. The width s_j determines how broadly it responds.

If these quantities are fixed, the model can only adjust the height of each bump through w_j. If we learn them, the bumps can move and change width to fit the data.

Try it: use the adaptive-basis demonstration on Slide 42. Compare fixed and learned centres and widths using the same data and number of features. Watch where the features move and how the fitted curve changes. The demonstration uses gradient descent for the centres and widths, and least squares for the output weights.

The main idea is:

Learning can choose which features to compute, as well as how to combine them.

This becomes especially useful when good features are difficult to design. An image contains many pixel values. We might want features describing edges, textures, parts, or objects, but writing all of them by hand is difficult.

Learning Representations Through Computation

A representation is the collection of values we use to describe an input at some stage of a computation. In the basis-function example, the representation is the vector of feature values.

We can write that example as two functions. The first computes the features:

\[\mathbf z^{(2)} =\boldsymbol\phi_{\theta_1}(x) =f_{1,\theta_1}(x).\]

Its parameters \theta_1 are the centres and widths. The second combines the feature values:

\[y=\mathbf w^\top\mathbf z^{(2)} =f_{2,\theta_2}(\mathbf z^{(2)}).\]

Here \theta_2 contains the output weights. The full prediction is

\[\boxed{ y=f_{2,\theta_2}(f_{1,\theta_1}(x)) =(f_{2,\theta_2}\circ f_{1,\theta_1})(x). }\]

The symbol \circ means function composition. Apply the rightmost function first, then pass its output to the next function.

We can continue with a third function:

\[\begin{aligned} \mathbf z^{(2)}&=f_{1,\theta_1}(\mathbf x),\\ \mathbf z^{(3)}&=f_{2,\theta_2}(\mathbf z^{(2)}),\\ y&=f_{3,\theta_3}(\mathbf z^{(3)}). \end{aligned}\]

Each function uses the output of the previous one. The intermediate values \mathbf z^{(2)} and \mathbf z^{(3)} are the representations. They are the results of the computations.

For L functions, the general form is

\[\boxed{ f_\theta =f_{L,\theta_L}\circ\cdots\circ f_{2,\theta_2}\circ f_{1,\theta_1}, \qquad \theta=(\theta_1,\ldots,\theta_L). }\]

We choose the computations and learn their parameters from data. The functions need not be simple: they can combine information, keep track of a state, or resemble steps in an algorithm.

Here L counts the functions we compose. For the MLP notation below, the input is layer 1, and function f_l computes the outputs of layer l+1.

For the rest of these notes, \theta denotes the collection of all trainable parameters in the network.

The Multilayer Perceptron

We will begin with the multilayer perceptron (MLP). An MLP repeatedly applies two operations:

  1. compute weighted sums and add biases
  2. apply a nonlinear activation function.

Each layer receives the representation produced by the previous layer. The final layer produces the prediction.

An MLP is one example of a neural network. Other architectures change the computations and the connections between them. We will return to this idea at the end of the lecture.

Try it: in the TensorFlow Playground linked on Slide 47, change the number of hidden units, the number of layers, and the activation. Watch how the hidden representations and the final decision boundary change.

How Does an MLP Work?

One Artificial Neuron

We start with one unit receiving an input vector \mathbf{x}. It first computes

\[a = \mathbf{w}^{\top}\mathbf{x}+b.\]

The quantity a is called the pre-activation. The weights determine how the inputs are combined, and the bias shifts the result.

The unit then applies an activation function:

\[\boxed{ z=\sigma(a). }\]

Here z is the unit's output. We can interpret it as one feature computed from \mathbf{x}.

With a single input, the calculation becomes

\[a=wx+b, \qquad z=\sigma(a).\]

Changing w changes how the unit responds to the input. Changing b shifts where the response occurs.

Activation Functions

Three common choices are sigmoid, tanh, and ReLU.

Sigmoid

\[\sigma(a) = \frac{1}{1+e^{-a}}.\]

Its output lies between 0 and 1. Large negative inputs give values near 0, and large positive inputs give values near 1.

We already used sigmoid in logistic regression to turn a real-valued score into a probability.

Tanh

\[\tanh(a) = \frac{e^{2a}-1}{e^{2a}+1}.\]

Its output lies between -1 and 1. Like sigmoid, it flattens out for inputs with large magnitude.

ReLU

The rectified linear unit, or ReLU, is

\[\operatorname{ReLU}(a) = \max(0,a).\]

It returns zero for a negative input and leaves a positive input unchanged. ReLU is piecewise linear: it has one linear piece on each side of zero.

For a layer of units, we apply these activation functions separately to each component.

Layers and Notation

Let's make the notation clear. We use the same layer and connection indices as Tutorial 6.

The input is layer 1:

\[\mathbf{z}^{(1)}=\mathbf{x}.\]

For a connection from layer l to layer l+1, write

\[\boxed{ \begin{aligned} \mathbf{a}^{(l+1)} &= \mathbf{W}^{(l)}\mathbf{z}^{(l)} + \mathbf{b}^{(l+1)},\\ \mathbf{z}^{(l+1)} &= \sigma_{l+1}\left(\mathbf{a}^{(l+1)}\right). \end{aligned} }\]

The notation tells us where each quantity belongs:

If layer l has M_l units and layer l+1 has M_{l+1} units, then

\[\mathbf{W}^{(l)} \in \mathbb{R}^{M_{l+1}\times M_l}.\]

Each row computes the weighted input to one receiving unit.

For one hidden layer and a scalar regression output, we have

\[\mathbf{z}^{(2)} = \sigma_2\left( \mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(2)} \right),\]

followed by

\[y = \mathbf{W}^{(2)}\mathbf{z}^{(2)} + b^{(3)}.\]

Here \mathbf{W}^{(2)} is a row matrix. The hidden activations \mathbf{z}^{(2)} are the learned features used by the output.

Forward Propagation

Forward propagation means computing these quantities in order:

  1. start with the input
  2. compute the first layer's pre-activations and outputs
  3. use those outputs as inputs to the next layer
  4. continue until we obtain the prediction.

For a numerical example, consider an MLP with two inputs, hidden layers of three and two ReLU units, and one linear output. Let

\[\mathbf{x} = \begin{bmatrix}1\\-1\end{bmatrix}.\]

The first hidden layer uses

\[\mathbf{W}^{(1)} = \begin{bmatrix} 1&0.5\\ -0.5&1\\ 1&-1 \end{bmatrix}, \qquad \mathbf{b}^{(2)} = \begin{bmatrix}0\\0.5\\-1\end{bmatrix}.\]

Multiplying by the input and adding the bias gives

\[\mathbf{a}^{(2)} = \begin{bmatrix}0.5\\-1\\1\end{bmatrix}, \qquad \mathbf{z}^{(2)} = \operatorname{ReLU}\left(\mathbf{a}^{(2)}\right) = \begin{bmatrix}0.5\\0\\1\end{bmatrix}.\]

The next hidden layer uses

\[\mathbf{W}^{(2)} = \begin{bmatrix} 1&-1&0.5\\ -0.5&0.5&1 \end{bmatrix}, \qquad \mathbf{b}^{(3)} = \begin{bmatrix}0\\-0.25\end{bmatrix}.\]

Both pre-activations are positive, so ReLU leaves them unchanged:

\[\mathbf{a}^{(3)} = \mathbf{z}^{(3)} = \begin{bmatrix}1\\0.5\end{bmatrix}.\]

The output layer uses

\[\mathbf W^{(3)}=\begin{bmatrix}1.2&-0.8\end{bmatrix}, \qquad b^{(4)}=0.1.\]

Its prediction is

\[y = 1.2(1)-0.8(0.5)+0.1 = 0.9.\]

During training, we keep the intermediate values because the backward pass will use them.

Try it: use Next layer on Slide 53 to follow this calculation alongside the network diagram. Match each matrix row to a receiving unit.

Why Do We Need Nonlinear Activations?

Suppose we remove the activations and compose two affine layers:

\[\mathbf{W}^{(2)} \left( \mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(2)} \right) + \mathbf{b}^{(3)}.\]

Expanding gives

\[\underbrace{ \mathbf{W}^{(2)}\mathbf{W}^{(1)} }_{\mathbf{W}} \mathbf{x} + \underbrace{ \mathbf{W}^{(2)}\mathbf{b}^{(2)} + \mathbf{b}^{(3)} }_{\mathbf{b}}.\]

This is just

\[\mathbf{W}\mathbf{x}+\mathbf{b}.\]

So stacking affine layers still gives an affine map. Nonlinear activations are what allow the composition to represent more complicated functions of the input.

Learning the Parameters

Choose the Output and Loss

The output and loss should match the target. We can use the probability models from Lectures 2 and 3 to derive a suitable loss function in the same way as before.

Regression

For a real-valued target, a common choice is a linear output,

\[y=a,\]

with squared loss,

\[\ell(y,t)=\frac12(y-t)^2.\]

As before, this corresponds to a Gaussian target model with fixed variance, up to a constant and a positive scaling of the loss.

Binary Classification

For t\in\{0,1\}, use a sigmoid output,

\[y=\sigma(a),\]

and binary cross-entropy,

\[\ell(y,t) = -t\log y-(1-t)\log(1-y).\]

The output y represents the probability of class 1.

Multiclass Classification

For mutually exclusive classes, use one score a_k per class and apply softmax:

\[y_k = \frac{e^{a_k}}{\sum_j e^{a_j}}.\]

With a one-hot target vector \mathbf{t}, the loss is

\[\ell(\mathbf{y},\mathbf{t}) = -\sum_k t_k\log y_k.\]

Only the correct class contributes to this sum. The loss is the negative logarithm of the probability assigned to that class.

The hidden layers can use ReLU even when the output uses sigmoid or softmax. The hidden activations and the output transformation have different jobs.

The Training Objective

For a dataset

\[\mathcal{D} = \{(\mathbf{x}_n,t_n)\}_{n=1}^{N},\]

define the loss for example n as

\[\ell_n(\theta) = \ell\left(f_{\theta}(\mathbf{x}_n),t_n\right).\]

We use the average training loss:

\[\boxed{ \mathcal{L}(\theta) = \frac1N\sum_{n=1}^{N}\ell_n(\theta). }\]

Gradient descent updates all the parameters using

\[\theta \leftarrow \theta-\eta\nabla_{\theta}\mathcal{L}(\theta),\]

where \eta>0 is the step size, also called the learning rate.

Every layer can learn from the final loss. The question is how to compute the derivatives through the entire network.

Recall: Gradients for Linear Models

We have already seen the main pattern in simpler models.

For each data point (\mathbf{x},t), let

\[a=\mathbf{w}^{\top}\mathbf{x}+b.\]

In linear regression with squared loss,

\[y=a, \qquad \ell=\frac12(y-t)^2.\]

Differentiating gives

\[\frac{\partial\ell}{\partial a}=y-t.\]

In logistic regression with binary cross-entropy, we also obtained

\[\frac{\partial\ell}{\partial a}=y-t.\]

The output transformation and loss are different, but the two derivatives happen to have the same form.

Since

\[\frac{\partial a}{\partial w_i}=x_i,\]

the chain rule gives

\[\boxed{ \frac{\partial\ell}{\partial w_i} = \underbrace{(y-t)}_{\text{error signal}} \cdot \underbrace{x_i}_{\text{input}}. }\]

We multiply two scalars here. The input tells us how strongly this weight affected the score. The error signal tells us how changing the score would affect the loss.

Each Connection Has a Weight

Now consider a hidden connection in an MLP. We write

\[w^{(l)}_{j,i}\]

for the weight from unit i in layer l to unit j in layer l+1. The superscript gives the source layer. The subscripts give the receiving unit first, then the input unit.

For training example n, the receiving unit computes

\[a^{(l+1)}_{n,j} = \sum_i w^{(l)}_{j,i}z^{(l)}_{n,i} + b^{(l+1)}_j.\]

The subscript n identifies the training example. The weights are shared across examples; the activations depend on the example.

For instance, w^{(2)}_{1,2} connects unit 2 in layer 2 to unit 1 in layer 3.

Define the Error Signal

For each unit, define

\[\boxed{ \delta^{(l+1)}_{n,j} \equiv \frac{\partial\ell_n} {\partial a^{(l+1)}_{n,j}}. }\]

This error signal measures how the final loss changes when we change the unit's pre-activation.

For a hidden unit, it includes the effect of all the later computations. It is generally more complicated than simply subtracting a target from that unit's output.

Apply the chain rule:

\[\frac{\partial\ell_n}{\partial w^{(l)}_{j,i}} = \frac{\partial\ell_n}{\partial a^{(l+1)}_{n,j}} \cdot \frac{\partial a^{(l+1)}_{n,j}} {\partial w^{(l)}_{j,i}}.\]

The second factor is the saved input, z^{(l)}_{n,i}. Therefore,

\[\boxed{ \frac{\partial\ell_n}{\partial w^{(l)}_{j,i}} = \underbrace{\delta^{(l+1)}_{n,j}}_{\text{error signal}} \cdot \underbrace{z^{(l)}_{n,i}}_{\text{input}}. }\]

This is the same pattern as the gradient for a linear model. The main additional work is finding the error signal at each hidden unit.

Collect the Gradients in a Matrix

We can connect the individual weight gradients to their matrix form. Start with the entry for weight w^{(l)}_{j,i}, then identify the row and column vectors that generate every entry.

We can collect all the connection gradients into a matrix:

\[\boxed{ \frac{\partial\ell_n}{\partial\mathbf{W}^{(l)}} = \boldsymbol{\delta}^{(l+1)}_n \cdot \left(\mathbf{z}^{(l)}_n\right)^{\top}. }\]

This is an outer product:

\[(M_{l+1}\times1) \;\cdot\; (1\times M_l) = M_{l+1}\times M_l.\]

The dimensions match the weight matrix. Entry (j,i) is exactly

\[\delta^{(l+1)}_{n,j}\cdot z^{(l)}_{n,i}.\]

For the bias, the local derivative is 1, so

\[\frac{\partial\ell_n}{\partial\mathbf{b}^{(l+1)}} = \boldsymbol{\delta}^{(l+1)}_n.\]

For the average training loss, average the per-example gradients:

\[\boxed{ \frac{\partial\mathcal{L}}{\partial\mathbf{W}^{(l)}} = \frac1N \sum_{n=1}^{N} \boldsymbol{\delta}^{(l+1)}_n \cdot \left(\mathbf{z}^{(l)}_n\right)^{\top}. }\]

Tutorial 6 uses a sum of losses, so its expression omits 1/N. Its worked network omits biases for simplicity. The layer and connection indices follow the same convention.

As practice, write out the matrix for a layer with two inputs and three receiving units. Check that every entry is the receiving unit's error signal multiplied by the corresponding input. Then write the bias gradient and the gradient-descent update.

Backpropagation and the Optimiser

Backpropagation applies the chain rule backwards through the network.

First, the forward pass computes the activations and the loss. The backward pass starts from the loss and computes how it depends on each preceding quantity. When several later units depend on the same hidden output, their contributions are added.

Once the error signals are available, each weight gradient uses the saved input from the forward pass. Reusing these intermediate results makes the calculation efficient.

The calculation has four stages:

  1. save the inputs and activations during the forward pass
  2. compute error signals during the backward pass
  3. combine each error signal with the corresponding input
  4. pass the gradients to the optimiser.

Backpropagation computes the gradients. The optimiser uses them to change the parameters. For ordinary gradient descent, the update is

\[\mathbf{W}^{(l)} \leftarrow \mathbf{W}^{(l)} - \eta \frac{\partial\mathcal{L}}{\partial\mathbf{W}^{(l)}}.\]

We repeat the forward pass, loss calculation, backward pass, and update over the training data. Tutorial 6 works through the detailed backward calculations.

What Can a Neural Network Represent?

We have now defined a network and a way to train it. Before analysing the updates, let us ask what functions the model can represent.

Universal Approximation

One version of the Universal Approximation Theorem says the following.

Let f:[0,1]\rightarrow\mathbb{R} be continuous. For any desired accuracy \varepsilon>0, there is a finite width M and a choice of parameters such that

\[\hat f(x) = b^{(3)}_1+\sum_{j=1}^{M}w^{(2)}_{1,j}\sigma(w^{(1)}_{j,1}x+b^{(2)}_j)\]

satisfies

\[\boxed{ \max_{x\in[0,1]} |f(x)-\hat f(x)| < \varepsilon. }\]

Here \sigma is the sigmoid function. This is a network with one hidden layer and a linear output.

The maximum means that the approximation is accurate throughout the interval [0,1]. The result also extends to continuous functions on compact domains in higher dimensions. Here, you can think of a compact domain as a closed, bounded set: it includes its boundary and does not extend to infinity. A closed interval or a closed box is an example.

The required width can depend on the target function and on how accurate we want the approximation to be.

A Proof Sketch

We can understand the idea in one dimension by building an approximation in four steps. This is an informal construction; a full proof also controls the approximation error.

1. Approximate the Target With Small Steps

Divide the interval into small pieces and use a constant value on each piece. The result is a staircase.

Because the target is continuous on a closed, bounded interval, sufficiently short pieces make the variation of the target within each piece small. This lets the staircase approximate the target closely.

2. Write the Staircase as a Sum of Jumps

Let \tau_j be the location of jump j, and let c_j be the height after that jump. Define

\[\Delta_j=c_j-c_{j-1}.\]

Then

\[g(x) = c_0 + \sum_{j=1}^{M} \Delta_j\,\mathbb{1}[x>\tau_j].\]

Each indicator switches on one change in height. The jumps can be positive or negative.

3. Replace a Hard Step With a Sigmoid

Consider

\[\sigma\bigl(k(x-\tau)\bigr) = \frac{1}{1+\exp[-k(x-\tau)]}.\]

The centre \tau determines where the transition occurs. Increasing k makes the transition steeper. Multiplying by \Delta gives a smooth change from approximately 0 to approximately \Delta.

4. Add the Sigmoid Units

Replace the hard steps by sigmoid units:

\[\hat f(x) = c_0 + \sum_{j=1}^{M} \Delta_j \sigma\bigl(k(x-\tau_j)\bigr).\]

This has the required neural-network form. The first-layer weights and biases compute the sigmoid units. The second-layer weights and output bias combine them:

\[\begin{aligned} w^{(1)}_{j,1}&=k, & b^{(2)}_j&=-k\tau_j,\\ w^{(2)}_{1,j}&=\Delta_j, & b^{(3)}_1&=c_0. \end{aligned}\]

First make the staircase steps small, then make the sigmoid transitions steep. A continuous sigmoid cannot uniformly match a fixed hard jump. Small jumps let us keep the smoothed curve close to the original continuous target. This explains the idea in one dimension; the theorem also covers higher-dimensional inputs.

Try it: on Slide 74, increase the number of sigmoid units and adjust their steepness. Compare the resulting curve with the target. More units provide more transitions; greater steepness makes each transition sharper.

What the Theorem Guarantees

The theorem is an existence result. It says that suitable weights and a sufficiently large width exist. But it does not tell us:

A model can represent an excellent predictor even when learning that predictor is difficult.

Deep Learning

Some of you may now be wondering: if one hidden layer is enough, why do we need deep neural networks? One answer is efficiency of representation. Some functions can be represented with far fewer units when we use several layers.

Hierarchy and Compositionality

Depth allows later computations to reuse earlier results.

For an image, a simple hierarchy might combine local changes in brightness into edges, edges into parts, and parts into larger patterns. We do not need to prescribe every intermediate feature. The network learns features that help with the task.

Two ideas are useful here:

These ideas explain why depth can help when the target itself has reusable structure. They do not imply that every problem needs a deep network.

An early example comes from Lee and colleagues (2009). Their convolutional deep belief network learned edge detectors from natural images. Building on these features, later layers learned face parts and more complete face patterns from unlabelled face images. This illustrates how later representations can build on earlier ones.

Weighted ReLUs Build Piecewise-Linear Functions

In one dimension, a ReLU unit can create a bend:

\[\max(0,x-\tau_j).\]

It is zero to the left of \tau_j and has slope 1 to the right. Multiplying by w_j changes the slope contribution.

A shallow network can therefore form

\[y(x) = b+\sum_{j=1}^{M}w_j\max(0,x-\tau_j).\]

Each unit changes the slope at one location. The slope of the sum changes by w_j when x crosses \tau_j.

Try it: on Slide 79, compare the individual weighted ReLU curves with their sum. Identify which unit creates each change in slope.

Comparing Depth and Width

Consider a target built by repeatedly applying a small piecewise-linear computation. This gives us a concrete way to compare shallow and deep representations.

On [0,1], define the triangular function

\[q(u) = 2\operatorname{ReLU}(u) - 4\operatorname{ReLU}\left(u-\frac12\right).\]

It rises from 0 to 1 and then returns to 0. Its output is again in [0,1], so we can apply it repeatedly.

Let

\[u_0(x)=x, \qquad u_r(x)=q(u_{r-1}(x)).\]

Each composition introduces more bends. Take the fixed target

\[f(x) = \sum_{r=1}^{6} 0.6^{r-1}u_r(x).\]

A deep network can reuse the preceding triangular computation and carry a running sum forward. With d hidden layers, it computes the first d terms of the target. Two units in the first layer begin the triangular computation. Each later layer uses two units for the next triangular computation and a third to carry the accumulated sum. At six hidden layers, this gives 2+5(3)=17 hidden units.

A shallow network can represent the same target directly as a piecewise-linear function. The target is linear between the grid points j/64, for j=0,\ldots,64. One ReLU unit supplies the initial slope and another supplies each of the 63 interior slope changes. This construction uses 64 hidden units. Both counts exclude the input and output units.

The example shows how composition can save units. These weights are constructed directly; we have not shown that a training algorithm will find them. We also have not proved that the shallow construction is the smallest possible one, or that depth helps equally for every target.

Try it: on Slide 80, increase the depth while keeping the target fixed. Then select Match the deep curve. Compare the approximation curves and the numbers of hidden units.

Why Can We Learn It?

Approximation tells us what a network can represent. We now ask what gradient-based learning can guarantee.

We will first analyse full gradient descent, then replace the full gradient with a sampled estimate.

Full gradient descent starts from initial parameters. At each step, it averages the gradients of all training examples, then subtracts this gradient multiplied by the step size.

We use s for the iteration index and T for the total number of updates. The symbol t_n is the target for training example n.

A Gradient-Descent Step

For the parameter vector \theta_s at iteration s, gradient descent uses

\[\theta_{s+1} = \theta_s - \eta\nabla\mathcal{L}(\theta_s).\]

The negative gradient points in the direction of greatest local decrease per unit distance. But the size of the step matters. For example, let

\[\mathcal{L}(\theta)=\frac12\theta^2.\]

At \theta=2, the gradient is 2. With \eta=0.5,

\[\theta_{\mathrm{new}}=2-0.5(2)=1.\]

The loss decreases from 2 to 0.5. A sufficiently large step can overshoot and increase the loss. For example, \eta=3 moves the parameter from 2 to -4, increasing the loss from 2 to 8.

Try it: on Slide 83, change the step size and compare the updates. Identify when the loss decreases and when a step overshoots.

To make a general statement about descent, we need to control how quickly the gradient changes. The next few sections are about controlling this step size.

The Goal of the Descent Proof

Our goal is to find a range of learning rates for which every gradient-descent step reduces the training loss whenever the gradient is nonzero.

First, we need the idea of a Lipschitz Gradient. We say that the loss has a \beta-Lipschitz gradient, or is \beta-smooth, if

\[\boxed{ \|\nabla\mathcal{L}(\mathbf{u}) - \nabla\mathcal{L}(\mathbf{v})\| \leq \beta\|\mathbf{u}-\mathbf{v}\|. }\]

Here, \|\cdot\| is the Euclidean norm. The constant \beta bounds how much the gradient can change when the parameters move. For a given change in the parameters, a smaller \beta gives a tighter bound on the change in the gradient. A larger \beta permits faster changes, so a smaller step size may be needed.

For a \beta-smooth loss, we will show that

\[0<\eta\leq\frac1\beta \quad\Longrightarrow\quad \mathcal{L}(\theta_{s+1})\leq\mathcal{L}(\theta_s).\]

The decrease is strict when \nabla\mathcal{L}(\theta_s)\neq\mathbf{0}. At a zero gradient, the update leaves the parameters unchanged.

The proof has two steps. First, use Taylor's theorem and smoothness to bound the loss after a step. Then substitute the gradient-descent update and choose the learning rate so that the loss decreases.

Start From Taylor's Theorem

For a twice continuously differentiable loss, expand around \theta using a displacement \mathbf{d}:

\[\mathcal{L}(\theta+\mathbf{d}) = \mathcal{L}(\theta) + \nabla\mathcal{L}(\theta)^{\top}\mathbf{d} + \frac12\mathbf{d}^{\top}\mathbf{H}(\xi)\mathbf{d}.\]

Here

\[\mathbf{H}=\nabla^2\mathcal{L}\]

is the Hessian, the matrix of second derivatives. The point \xi lies on the line segment between \theta and \theta+\mathbf{d}. In other words,

\[\xi=\theta+\alpha\mathbf{d}, \qquad 0<\alpha<1.\]

Taylor's theorem guarantees that such an intermediate point exists. We do not need to find it, because the smoothness bound applies throughout the segment.

The first two terms give a linear prediction of how the loss changes. The last term accounts for curvature.

Why Does Smoothness Bound Curvature?

The Hessian tells us how the gradient changes. The Lipschitz condition limits that change. We can connect the two by taking a limit.

First, consider one parameter. Smoothness gives

\[|\mathcal{L}'(x+h)-\mathcal{L}'(x)| \leq \beta|h|.\]

For h\neq 0, divide by |h|:

\[\left| \frac{\mathcal{L}'(x+h)-\mathcal{L}'(x)}{h} \right| \leq \beta.\]

Now let h approach zero. The fraction approaches the second derivative, so

\[|\mathcal{L}''(x)|\leq\beta.\]

This is the curvature bound in one dimension: the slope cannot change faster than \beta.

For several parameters, use the same idea along a direction. Start at any point \mathbf{u} and move to \mathbf{u}+h\mathbf{d}, where h is a scalar. The Lipschitz condition gives

\[\|\nabla\mathcal{L}(\mathbf{u}+h\mathbf{d}) -\nabla\mathcal{L}(\mathbf{u})\| \leq \beta\|h\mathbf{d}\| = \beta|h|\|\mathbf{d}\|.\]

Divide by |h|:

\[\left\| \frac{\nabla\mathcal{L}(\mathbf{u}+h\mathbf{d}) -\nabla\mathcal{L}(\mathbf{u})}{h} \right\| \leq\beta\|\mathbf{d}\|.\]

As h approaches zero, the fraction becomes the rate of change of the gradient in direction \mathbf{d}. By the chain rule, this is the Hessian multiplied by \mathbf{d}:

\[\lim_{h\to 0} \frac{\nabla\mathcal{L}(\mathbf{u}+h\mathbf{d}) -\nabla\mathcal{L}(\mathbf{u})}{h} = \mathbf{H}(\mathbf{u})\mathbf{d}.\]

Therefore,

\[\boxed{ \|\mathbf{H}(\mathbf{u})\mathbf{d}\| \leq\beta\|\mathbf{d}\|. }\]

Finally, bound the term in Taylor's theorem. The result above holds at the intermediate point \xi. We apply the Cauchy–Schwarz inequality:

\[|\mathbf{a}^{\top}\mathbf{b}| \leq\|\mathbf{a}\|\|\mathbf{b}\|.\]

Apply it with \mathbf{a}=\mathbf{d} and \mathbf{b}=\mathbf{H}(\xi)\mathbf{d}:

\[\begin{aligned} |\mathbf{d}^{\top}\mathbf{H}(\xi)\mathbf{d}| &\leq \|\mathbf{d}\|\,\|\mathbf{H}(\xi)\mathbf{d}\|\\ &\leq \|\mathbf{d}\|\,\beta\|\mathbf{d}\|\\ &= \beta\|\mathbf{d}\|^2. \end{aligned}\]

One factor of \|\mathbf{d}\| comes from the bound on the gradient's change. The other comes from taking the inner product with \mathbf{d}.

The curvature term can be positive or negative. For an upper bound on the loss, we use

\[\mathbf{d}^{\top}\mathbf{H}(\xi)\mathbf{d} \leq |\mathbf{d}^{\top}\mathbf{H}(\xi)\mathbf{d}| \leq \beta\|\mathbf{d}\|^2.\]

Therefore,

\[\boxed{ \mathcal{L}(\theta+\mathbf{d}) \leq \mathcal{L}(\theta) + \nabla\mathcal{L}(\theta)^{\top}\mathbf{d} + \frac{\beta}{2}\|\mathbf{d}\|^2. }\]

This is an upper bound on the loss after a step. It also holds for a differentiable loss with a Lipschitz gradient without requiring a continuous Hessian.

Substitute the Gradient-Descent Step

Write

\[\mathbf{g}=\nabla\mathcal{L}(\theta), \qquad \mathbf{d}=-\eta\mathbf{g}.\]

Substitute into the bound:

\[\begin{aligned} \mathcal{L}(\theta-\eta\mathbf{g}) &\leq \mathcal{L}(\theta) + \mathbf{g}^{\top}(-\eta\mathbf{g}) + \frac{\beta}{2}\|-\eta\mathbf{g}\|^2\\ &= \mathcal{L}(\theta) - \eta\|\mathbf{g}\|^2 + \frac{\beta\eta^2}{2}\|\mathbf{g}\|^2\\ &= \mathcal{L}(\theta) - \eta\left(1-\frac{\beta\eta}{2}\right) \|\mathbf{g}\|^2. \end{aligned}\]

If

\[0<\eta\leq\frac1\beta,\]

then 1-\beta\eta/2\geq1/2. We obtain

\[\boxed{ \mathcal{L}(\theta_{s+1}) \leq \mathcal{L}(\theta_s) - \frac{\eta}{2} \|\nabla\mathcal{L}(\theta_s)\|^2. }\]

For a nonzero gradient, the loss decreases. The guaranteed decrease is larger when the gradient is larger.

Note: this argument assumes smoothness. ReLU is nondifferentiable at zero, so a ReLU network can violate this assumption. Even with smooth activations, a single global smoothness constant requires justification. The result tells us what follows when the stated condition holds over the region used by the updates.

What Does Convergence Mean?

The one-step bound tells us that the loss decreases. We now ask what happens after many steps: how small can the gradient become, and do the parameters settle at one point?

Suppose the loss has a finite lower bound \mathcal{L}_{\inf}. Use a fixed step size satisfying 0<\eta\leq1/\beta.

Rearrange the one-step result:

\[\frac{\eta}{2} \|\nabla\mathcal{L}(\theta_s)\|^2 \leq \mathcal{L}(\theta_s)-\mathcal{L}(\theta_{s+1}).\]

Sum from s=0 to T-1:

\[\frac{\eta}{2} \sum_{s=0}^{T-1} \|\nabla\mathcal{L}(\theta_s)\|^2 \leq \mathcal{L}(\theta_0)-\mathcal{L}(\theta_T).\]

The intermediate loss terms cancel. Since \mathcal{L}(\theta_T)\geq\mathcal{L}_{\inf},

\[\frac{\eta}{2} \sum_{s=0}^{T-1} \|\nabla\mathcal{L}(\theta_s)\|^2 \leq \mathcal{L}(\theta_0)-\mathcal{L}_{\inf}.\]

The smallest squared gradient is at most the average, so

\[\boxed{ \min_{0\leq s<T} \|\nabla\mathcal{L}(\theta_s)\|^2 \leq \frac{ 2[\mathcal{L}(\theta_0)-\mathcal{L}_{\inf}] }{ \eta T }. }\]

The displayed bound controls the smallest squared gradient norm among the first T iterates. Its upper bound decreases as O(1/T). With enough steps, at least one iterate has an arbitrarily small gradient. For the gradient norm itself, taking the square root gives an O(1/\sqrt{T}) upper bound on the smallest norm seen so far.

This running-minimum bound alone does not show that every gradient approaches zero. For that stronger conclusion, return to the preceding summed inequality. The partial sums of \sum_s\|\nabla\mathcal{L}(\theta_s)\|^2 are bounded by a constant independent of T. Since every term is nonnegative, the terms must approach zero. Thus, under the stated assumptions, \|\nabla\mathcal{L}(\theta_s)\|\to0 along the sequence. This extra argument does not give an O(1/T) bound for every individual squared gradient norm.

This is a stationarity guarantee. A stationary point has zero gradient. For a nonconvex loss, a stationary point can be a local minimum, a local maximum, or a saddle point. The argument does not guarantee a global minimum or convergence of the parameters to one particular point.

Remark: convergence to a single stationary point (extension). With additional assumptions, we can strengthen the conclusion to

\[\theta_s\to\theta_\star, \qquad \nabla\mathcal L(\theta_\star)=\mathbf 0.\]

The parameter sequence now settles at one point as the number of steps grows. This is beyond the scope of this course.

Stochastic Gradient Descent

Computing the full gradient requires all N training examples:

\[\nabla\mathcal L(\theta) =\frac1N\sum_{n=1}^N\nabla\ell_n(\theta).\]

When N is large, this can make each update expensive. Stochastic gradient descent (SGD) estimates this gradient using sampled examples.

Choose initial parameters \theta_0, a step size \eta>0, and a number of updates T. For s=0,\ldots,T-1:

  1. sample an index n_s uniformly from \{1,\ldots,N\}
  2. compute \mathbf g_s=\nabla\ell_{n_s}(\theta_s) using backpropagation
  3. update \theta_{s+1}=\theta_s-\eta\mathbf g_s.

Return \theta_T. For the analysis, each index is sampled independently of earlier sampling, with replacement. A minibatch averages several sampled gradients before making the update.

SGD Can Keep Fluctuating Near the Minimum

Consider two training examples with targets t_1=+1 and t_2=-1. Our model predicts the same constant \theta for both examples; it does not use the inputs. The loss on each example is squared error:

\[\ell_1(\theta)=\frac12(\theta-1)^2, \qquad \ell_2(\theta)=\frac12(\theta+1)^2.\]

The average training loss is

\[\begin{aligned} \mathcal L(\theta) &=\frac12[\ell_1(\theta)+\ell_2(\theta)]\\ &=\frac14[(\theta-1)^2+(\theta+1)^2]\\ &=\frac12\theta^2+\frac12. \end{aligned}\]

Its minimum is at \theta=0, the mean of the two targets. Its gradient is \theta; the constant 1/2 has no effect on the updates.

Full gradient descent averages both example gradients:

\[\nabla\mathcal L(\theta) =\frac12[(\theta-1)+(\theta+1)]=\theta.\]

For an SGD update with B=1, sample one of the two examples uniformly. The sampled gradient is \theta-1 if the target is +1, and \theta+1 if the target is -1. At \theta=0, these gradients are -1 and +1, even though their average is zero. Each example pulls the prediction towards its own target, so sampling can keep moving the parameter away from the minimum of the average loss.

For a minibatch of size B, independently sample B example indices with replacement. Repeated indices are allowed, so B can exceed the two-example dataset size. Average their gradients:

\[\bar t_s=\frac1B\sum_{b=1}^{B}t_{n_{s,b}}, \qquad g_s=\frac1B\sum_{b=1}^{B}(\theta_s-t_{n_{s,b}}) =\theta_s-\bar t_s.\]

The update is therefore

\[\theta_{s+1}=\theta_s-\eta g_s =(1-\eta)\theta_s+\eta\bar t_s.\]

All the randomness comes from sampling training examples. There is no additional random perturbation.

Smaller steps or larger batches reduce the fluctuations. We will show that the sampled gradient is correct on average, and then account for how far individual samples can differ from that average.

Try it: on Slide 91, compare the parameter values over successive updates. Change the step size and batch size, and observe how the spread of the SGD path changes. The demonstration samples the two examples with replacement and averages their gradients at each update.

The Sampled Gradient is Unbiased

Our training objective is the average loss,

\[\mathcal L(\theta)=\frac1N\sum_{n=1}^{N}\ell_n(\theta).\]

At iteration s, hold the current parameters \theta_s fixed. Each loss is differentiable there, with a finite gradient. There are N possible sampled gradients: one for each training example.

1. Average over the possible sampled examples. Each index n has probability 1/N. If we select it, the sampled gradient is \nabla\ell_n(\theta_s). The conditional expectation is therefore

\[\mathbb E[\mathbf g_s\mid\theta_s] =\sum_{n=1}^{N}\frac1N\nabla\ell_n(\theta_s).\]

2. Move the gradient outside the finite sum. Differentiation is linear, so

\[\sum_{n=1}^{N}\frac1N\nabla\ell_n(\theta_s) =\nabla\left[\frac1N\sum_{n=1}^{N}\ell_n(\theta_s)\right].\]

3. Recognise the training objective. The expression inside the brackets is the average training loss. Thus,

\[\boxed{ \mathbb E[\mathbf g_s\mid\theta_s]=\nabla\mathcal L(\theta_s). }\]

This is what unbiased means: at the current parameters, the sampled gradient equals the full training gradient on average.

The conditioning matters because \theta_s depends on earlier updates. We average over the new sampling at this step.

For a one-parameter example, suppose the four training examples have gradients -2,-1,1,4 at the current parameters. Each has probability 1/4, so the mean is

\[\frac{-2-1+1+4}{4}=0.5.\]

This is also the full training gradient. Individual sampled gradients can be positive or negative, even though their mean is positive.

Why Do Individual Updates Fluctuate?

Unbiasedness describes an average over possible samples. The gradient from one example can point in a different direction from the full gradient.

For example, consider two training losses:

\[\ell_1(\theta)=\frac12(\theta-1)^2, \qquad \ell_2(\theta)=\frac12(\theta+1)^2.\]

Their average is

\[\mathcal L(\theta) =\frac12[\ell_1(\theta)+\ell_2(\theta)] =\frac12\theta^2+\frac12.\]

The constant 1/2 has no effect on the gradient, so the full-gradient update is the same as for the previous quadratic example. The average loss is minimised at \theta=0, but the two individual gradients there are -1 and +1. Sampling either example moves the parameter to +\eta or -\eta. The full gradient is zero, yet an individual update moves away from the minimum.

Each individual loss favours its own minimum. Repeatedly choosing different examples can therefore make the parameter fluctuate around the minimum of their average.

To analyse this, write the sampled gradient as

\[\mathbf g_s=\nabla\mathcal L(\theta_s)+\boldsymbol\epsilon_s.\]

The difference \boldsymbol\epsilon_s is the sampling noise. It is determined by the sampled examples, rather than an additional perturbation. In our two-target example, \epsilon_s=-\bar t_s: the sampled gradient \theta_s-\bar t_s differs from the full gradient \theta_s by exactly the negative sampled-target mean.

From the unbiasedness proof,

\[\mathbb E[\boldsymbol\epsilon_s\mid\theta_s]=\mathbf0.\]

Zero-mean noise follows from uniform sampling. We have proved this property.

The Additional Variance Assumption

For the convergence analysis, we now make an additional assumption: the single-example noise variance is at most v^2 at every iteration. The same constant must work throughout the analysis. For a uniformly sampled index n,

\[\mathbb E_n\!\left[ \|\nabla\ell_n(\theta_s)-\nabla\mathcal L(\theta_s)\|^2 \mid\theta_s \right]\leq v^2 \qquad\text{for every }s.\]

This bounds the average squared distance of a sampled gradient from the full gradient. Individual deviations can be larger than v. The bound does not follow from unbiasedness.

At any fixed parameter setting, a finite collection of finite gradients has a finite maximum size. The unbiasedness proof uses this finite average and needs no global bound on gradient size. Those gradients can change as the parameters move, so a single variance bound throughout training needs its own justification.

For a minibatch of B independently sampled examples, the gradient estimate is their average:

\[\mathbf g_s=\frac1B\sum_{b=1}^B\nabla\ell_{n_{s,b}}(\theta_s).\]

Here n_{s,b} is the index of the bth sampled example at iteration s. Averaging preserves unbiasedness and reduces the noise variance:

\[\boxed{ \mathbb E[\|\boldsymbol\epsilon_s\|^2\mid\theta_s] \leq\frac{v^2}{B}. }\]

To see the factor 1/B, expand the squared norm of the average noise. The B individual squared terms each have expectation at most v^2. The cross terms have expectation zero because the samples are independent and their noise has mean zero. Dividing by B^2 leaves Bv^2/B^2=v^2/B.

Keep the three statements separate:

Here the dataset is fixed. The randomness comes from which examples we sample for each update.

When Does SGD Decrease the Loss on Average?

Assume a \beta-smooth loss, a step size 0<\eta\leq1/\beta, and the variance bound above. Average B independent sampled gradients at each step.

We can derive the bound by averaging the smoothness inequality. The calculation below also supports the optional SGD exercise in Tutorial 6.

Start from the same smoothness inequality, now with the step -\eta\mathbf g_s:

\[\mathcal L(\theta_{s+1}) \leq\mathcal L(\theta_s) -\eta\nabla\mathcal L(\theta_s)^\top\mathbf g_s +\frac{\beta\eta^2}{2}\|\mathbf g_s\|^2.\]

Hold \theta_s fixed and average over the newly sampled gradient. For the middle term, unbiasedness gives

\[\mathbb E[\nabla\mathcal L(\theta_s)^\top\mathbf g_s\mid\theta_s] =\|\nabla\mathcal L(\theta_s)\|^2.\]

For the last term, expand the squared norm:

\[\begin{aligned} \|\mathbf g_s\|^2 &=\|\nabla\mathcal L(\theta_s)+\boldsymbol\epsilon_s\|^2\\ &=\|\nabla\mathcal L(\theta_s)\|^2 +2\nabla\mathcal L(\theta_s)^\top\boldsymbol\epsilon_s +\|\boldsymbol\epsilon_s\|^2. \end{aligned}\]

The cross term has expectation zero. The remaining noise term is bounded by the variance assumption:

\[\mathbb E[\|\mathbf g_s\|^2\mid\theta_s] \leq\|\nabla\mathcal L(\theta_s)\|^2+\frac{v^2}{B}.\]

Substitute both results:

\[\begin{aligned} \mathbb E[\mathcal L(\theta_{s+1})\mid\theta_s] &\leq\mathcal L(\theta_s)-\eta\|\nabla\mathcal L(\theta_s)\|^2 +\frac{\beta\eta^2}{2}\left(\|\nabla\mathcal L(\theta_s)\|^2+\frac{v^2}{B}\right)\\ &=\mathcal L(\theta_s) -\eta\left(1-\frac{\beta\eta}{2}\right)\|\nabla\mathcal L(\theta_s)\|^2 +\frac{\beta\eta^2v^2}{2B}. \end{aligned}\]

Our step-size condition gives 1-\beta\eta/2\geq1/2, so

\[\boxed{ \mathbb E[\mathcal L(\theta_{s+1})\mid\theta_s] \leq\mathcal L(\theta_s) -\underbrace{\frac\eta2\|\nabla\mathcal L(\theta_s)\|^2}_{\text{progress}} +\underbrace{\frac{\beta\eta^2v^2}{2B}}_{\text{noise}}. }\]

The first term we subtract comes from following the true gradient. The positive term accounts for sampling noise in the upper bound.

The expected loss is guaranteed to decrease when the decrease term exceeds the noise term. A sufficient condition is

\[\|\nabla\mathcal L(\theta_s)\|^2>\frac{\beta\eta v^2}{B}.\]

When the gradient becomes small, this guarantee can disappear. Smaller steps or larger batches reduce the noise term relative to the decrease term.

This is a bound on the expected next loss. A particular SGD step may increase the loss. The positive noise term is also an upper-bound term; it does not mean that noise increases the loss by that amount on every step.

SGD Convergence With a Fixed Step Size

For a convergence bound, also assume the loss has a finite lower bound \mathcal L_{\inf}. Keep \eta and B fixed, with the smoothness and variance assumptions holding throughout the updates.

Sum the expected one-step inequalities. Taking expectations over the earlier sampling as well gives

\[\frac\eta2\sum_{s=0}^{T-1}\mathbb E\|\nabla\mathcal L(\theta_s)\|^2 \leq\mathcal L(\theta_0)-\mathcal L_{\inf} +\frac{T\beta\eta^2v^2}{2B}.\]

Divide by \eta T/2. The minimum of the expected squared gradients is at most their average, so

\[\boxed{ \min_{0\leq s<T}\mathbb E\|\nabla\mathcal L(\theta_s)\|^2 \leq \underbrace{\frac{2[\mathcal L(\theta_0)-\mathcal L_{\inf}]}{\eta T}}_{\text{shrinks with time}} +\underbrace{\frac{\beta\eta v^2}{B}}_{\text{noise floor}}. }\]

The minimum is over the first T iterates; the expectation averages over the random sampling. At some iteration among the first T updates, the expected squared gradient is at most the right-hand side.

The terms have the following consequences:

If v^2>0, the bound approaches a positive value as T grows. This is the noise floor in the bound. It does not prove that every problem has persistent fluctuations: the actual noise may be smaller, or vanish near a solution.

The guarantee concerns expected squared gradient size. It gives neither a distance from the optimum nor convergence of the parameters to a single point. A stationary point also need not be a global minimum.

Connection to Tutorial 6

The optional SGD exercise in Tutorial 6 uses a bound on the second moment of the sampled gradient, \mathbb E[g_s^2\mid\theta_s]\leq\sigma^2, in its scalar example. Here we assumed a bound on the noise variance instead. A second-moment bound also bounds the variance, but the two derivations give differently written bounds.

The tutorial writes \alpha for the step size and L for the curvature bound. These correspond to \eta and \beta here. We use s consistently for iterations, keeping t_n for targets.

Returning to the Quadratic Example

We can now calculate the spread of the fluctuations for the two-target dataset. Recall

\[\mathcal L(\theta)=\frac12\theta^2+\frac12, \qquad t_1=+1,\quad t_2=-1.\]

Full gradient descent follows

\[\theta_{s+1}=(1-\eta)\theta_s.\]

For 0<\eta<2, the factor has magnitude less than one, so the parameter approaches zero.

SGD samples B examples independently with replacement at each update. Their target mean \bar t_s gives the sampled gradient g_s=\theta_s-\bar t_s and the update

\[\theta_{s+1}=(1-\eta)\theta_s+\eta\bar t_s.\]

Each sampled target is +1 or -1 with equal probability, so it has mean zero and variance one. Independence of the B samples gives

\[\mathbb E[\bar t_s]=0, \qquad \operatorname{Var}(\bar t_s)=\frac1{B^2}\sum_{b=1}^{B}1=\frac1B.\]

This variance follows from the dataset and sampling procedure; it is not an arbitrary noise assumption. In the notation of the general bound, this example has v^2=1.

The new batch is independent of earlier updates. Thus \mathbb E[\theta_s] approaches zero, while

\[\operatorname{Var}(\theta_{s+1}) =(1-\eta)^2\operatorname{Var}(\theta_s)+\frac{\eta^2}{B}.\]

At the limiting variance V, the value no longer changes between steps:

\[V=(1-\eta)^2V+\frac{\eta^2}{B}.\]

Rearranging gives

\[\boxed{V=\frac{\eta}{B(2-\eta)}.}\]

For any finite B and fixed step size 0<\eta<2, the spread remains positive. Smaller steps or larger batches reduce it. The shaded band on Slide 91 is [-\sqrt V,\sqrt V], one long-run standard deviation on each side of zero. It describes the spread; it is not a boundary that contains every iterate.

This specific example gives the noise ball picture: parameters fluctuate around the minimum. The general SGD convergence bound makes a statement about gradients; it does not establish such a region in parameter space for every loss.

Machine Learning ≠ Optimization

The descent arguments concern the training objective:

\[\mathcal{L}(\theta) = \frac1N \sum_{n=1}^{N} \ell\left(f_\theta(\mathbf{x}_n),t_n\right).\]

Our real goal is good performance on new data:

\[R(\theta) = \mathbb{E}_{(\mathbf{x},t)\sim p} \left[ \ell\left(f_\theta(\mathbf{x}),t\right) \right].\]

In general,

\[\boxed{ \mathcal{L}(\theta)\neq R(\theta). }\]

The training objective is a data-based estimate of the population objective. It is also the quantity we used to select the model. As we saw in Lectures 5 and 6, that selection can favour accidental details of the training sample.

An optimisation method asks how to reduce the objective it is given. Machine learning also asks whether that objective and the available data lead to useful predictions.

Importantly, research has shown that optimisation can act as a form of regularisation. A flexible network may have many parameter settings that fit the training data. The training process influences which one we reach.

For example:

These choices can favour some solutions even when we do not add an explicit penalty to the loss. This is called an implicit bias of the learning process, and it can act as a form of regularisation. However, not every such bias improves generalisation. The useful question is whether the solutions favoured by the training process match the structure of the prediction problem.

Summary

In this lesson, we were introduced to the key concepts behind neural networks. The main ideas are:

  1. Learned features. A neural network composes parameterised functions. Their intermediate outputs are the representations used by later computations. Learning changes both the features and the predictor.

  2. MLP predictions. Compute each layer's weighted sums, biases, and activations in order. Choose the output and loss to match the task: a linear output and squared loss for regression, sigmoid and binary cross-entropy for binary classification, or softmax and cross-entropy for multiclass classification.

  3. Gradients and updates. For an affine connection, the weight gradient is the receiving unit's error signal multiplied by its input. Backpropagation computes these gradients. The optimiser uses them to update the parameters.

  4. UAT and depth. A sufficiently wide network with one sigmoid hidden layer can approximate a continuous target on a compact domain arbitrarily well. Depth can use fewer units for functions with reusable structure. These are representation results; learning and generalisation require further arguments.

  5. Convergence. Under the smoothness, step-size, and lower-bound conditions, full gradient descent drives gradient norms towards zero. With a fixed step size and bounded sampling variance, the SGD bound includes a noise term. SGD can keep fluctuating, as the quadratic example shows. Neither result guarantees a global minimum.

The optimisation results concern the training loss. Machine learning also requires good predictions on new data.