Lecture 5: Overfitting and Bias-Variance

Summary of Key Ideas

In the previous lectures, we focused mainly on how to define and fit a model. This lecture (and the next one) asks a different question:

Once we fit a model, why should it work on new data?

A model can fit its training data extremely well and still make poor predictions on examples it has never seen. The quantity we really care about is therefore the test or generalisation error, not the training error.

For squared loss, the key idea is that the expected generalisation error can be split into three parts:

\[\boxed{ \text{generalisation error} = (\text{bias})^2 + \text{variance} + \text{noise}. }\]

These three terms describe different ways a prediction can be wrong. Bias measures whether the model is systematically wrong on average. Variance measures how much the fitted model changes when we train on a different dataset. Noise is variation in the target that remains even if we knew the best possible prediction.

This decomposition helps us understand underfitting and overfitting. A model that is too simple often has high bias. A model that is very flexible can have low bias but high variance because it reacts strongly to the particular sample it was trained on.

The second half of the lecture asks how we choose a model or a hyperparameter such as polynomial degree. Training error cannot do this because it usually prefers the most flexible model. We therefore hold out data for validation or use cross-validation. We will also see a simple mathematical reason why validation can work: with enough held-out examples, the validation errors of a fixed set of candidate models are likely to be close to their true risks.

Learning Objectives

By the end of this lesson, the main things you should be able to do are:

In addition, you should be able to:

We will first build an intuitive picture using many possible training datasets. We will then derive the bias-variance decomposition. Then we will use it to understand underfitting and overfitting. Finally, we will discuss model selection and cross-validation.

From Model Fitting to Generalisation

We've seen from earlier lectures that a richer model can represent more patterns, but more freedom is not automatically better. With limited data, a flexible model may adapt not only to the underlying pattern but also to accidental details of the training set.

This is the problem we first saw with high-order polynomial regression in Lecture 1. As the polynomial degree increased, the training error kept falling. The error on held-out data eventually started to rise.

At the time, we called this overfitting. We will now develop a more precise way to think about what is happening.

Training Error and Generalisation Error

Suppose our fitted model is written as

\[y(x,\mathcal{D}),\]

where \mathcal{D} is the training dataset. Writing the dependence on \mathcal{D} explicitly is important because a different dataset would generally produce a different fitted model.

The training error is computed on the data we used to fit the model. For squared loss,

\[\frac{1}{N} \sum_{n=1}^{N} \left( y(x_n,\mathcal{D})-t_n \right)^2.\]

This quantity is easy to compute, but it is not what we ultimately care about. We want the model to work on a new example drawn from the same underlying population.

For a fixed fitted model, the generalisation error is

\[\boxed{ R(\mathcal{D}) = \mathbb{E}_{p(x,t)} \left[ \left( y(x,\mathcal{D})-t \right)^2 \right]. }\]

The expectation means that we average the loss over possible future inputs and targets from the data distribution p(x,t).

The key problem is that we do not know p(x,t), so we cannot normally calculate this expectation exactly. We only have a finite dataset sampled from it.

This is why low training error alone tells us very little about whether the model has learnt a useful pattern.

The Training Dataset Is Random

To understand generalisation, imagine repeating the entire learning process many times.

Each time:

  1. draw a new training dataset \mathcal{D}
  2. fit the model using exactly the same learning algorithm
  3. look at the prediction at some fixed input x

Because each dataset is different, the fitted model is different. Therefore

\[y(x,\mathcal{D})\]

is itself a random quantity when we think across possible training datasets.

Use the interactive plot on Slide 18. Keep the model degree fixed and repeatedly sample a new dataset. Each dataset produces a different fitted polynomial. With a simple model, the fitted curves may be fairly similar. With a very flexible model and little data, they can look completely different.

This is the central picture behind variance.

Two Important Averages

There are two different kinds of randomness in this lecture, and therefore two different averages. Keeping them separate is the main difficulty in the derivation.

The Average Target

At a fixed input x, the target may still be random. For example,

\[t = h(x)+\varepsilon,\]

where \varepsilon represents noise.

Define

\[\boxed{ \bar{t}(x) = \mathbb{E}_{p(t\mid x)}[t]. }\]

This is the average target at input x. If the noise has zero mean, \bar{t}(x) is the underlying function h(x).

The Average Model

Now imagine repeatedly drawing a training dataset and fitting the model. Define

\[\boxed{ \bar{y}(x) = \mathbb{E}_{p(\mathcal{D})} \left[ y(x,\mathcal{D}) \right]. }\]

This is the average prediction over all possible training datasets.

The average model is mainly a thinking tool. In a real problem we only have one training dataset, so we cannot actually average over every dataset we might have received.

The interactive plots on Slides 18-19 make these two ideas concrete. At a fixed x, each possible training set gives one fitted prediction. Their average is \bar{y}(x). Separately, repeated targets at the same x vary around \bar{t}(x) because of noise.

Bias, Variance, and Noise

We can now define the three sources of test error.

Bias

The bias at input x is the difference between the average model and the average target:

\[\boxed{ \operatorname{bias}(x) = \bar{y}(x)-\bar{t}(x). }\]

If this quantity is large, then even after averaging over many possible training datasets, the model is systematically wrong.

A common reason is that the model family is too restrictive. For example, if the true relationship is curved but our model is always a straight line, the average fitted line cannot reproduce the true relationship.

Bias therefore reflects a limitation of the model family and learning procedure.

Variance

The variance measures how much the fitted prediction changes from one training dataset to another:

\[\boxed{ \operatorname{var}(x) = \mathbb{E}_{p(\mathcal{D})} \left[ \left( y(x,\mathcal{D})-\bar{y}(x) \right)^2 \right]. }\]

A model has high variance if small changes in the training sample can lead to very different predictions.

Flexible models trained on small datasets often have high variance because they have enough freedom to react strongly to the particular noise in that dataset.

Noise

Even if we knew the average target \bar{t}(x) exactly, the observed target t could still differ from it. The noise at x is

\[\boxed{ \mathbb{E}_{p(t\mid x)} \left[ \left( t-\bar{t}(x) \right)^2 \right]. }\]

This part comes from the data-generating process itself. It does not depend on which model we fit.

In the simple model

\[t=h(x)+\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\sigma^2),\]

the noise term is \sigma^2.

The Bias-Variance Decomposition

We now derive the result formally. The derivation is longer than the idea, but not especially difficult.

We will use the same trick twice: add and subtract an average, then expand the square.

We want the expected squared prediction error, averaged over future examples and over the random training dataset:

\[\mathbb{E}_{p(x,t)} \mathbb{E}_{p(\mathcal{D})} \left[ \left( y(x,\mathcal{D})-t \right)^2 \right].\]

First Split: Variance

Add and subtract the average model \bar{y}(x):

\[y-t = \left( y-\bar{y} \right) + \left( \bar{y}-t \right).\]

Squaring gives

\[(y-t)^2 = (y-\bar{y})^2 + (\bar{y}-t)^2 + 2(y-\bar{y})(\bar{y}-t).\]

Now average over datasets. The cross term disappears because

\[\mathbb{E}_{p(\mathcal{D})} \left[ y(x,\mathcal{D})-\bar{y}(x) \right] = 0.\]

Therefore,

\[\mathbb{E} \left[ (y-t)^2 \right] = \underbrace{ \mathbb{E}_{p(x)} \mathbb{E}_{p(\mathcal{D})} \left[ (y-\bar{y})^2 \right] }_{\text{variance}} + \mathbb{E}_{p(x,t)} \left[ (\bar{y}-t)^2 \right].\]

The first term is the variance. The second term still mixes together systematic model error and target noise.

Second Split: Bias and Noise

Now add and subtract the average target \bar{t}(x):

\[\bar{y}-t = \left( \bar{y}-\bar{t} \right) + \left( \bar{t}-t \right).\]

Expand the square again:

\[(\bar{y}-t)^2 = (\bar{y}-\bar{t})^2 + (\bar{t}-t)^2 + 2(\bar{y}-\bar{t})(\bar{t}-t).\]

The cross term again disappears because

\[\mathbb{E}_{p(t\mid x)} \left[ \bar{t}(x)-t \right] = 0.\]

Putting the two splits together gives

\[\boxed{ \begin{aligned} \mathbb{E} \left[ (y(x,\mathcal{D})-t)^2 \right] &= \underbrace{ \mathbb{E}_{p(x)} \left[ \left( \bar{y}(x)-\bar{t}(x) \right)^2 \right] }_{(\text{bias})^2} \\ &\quad+ \underbrace{ \mathbb{E}_{p(x)} \mathbb{E}_{p(\mathcal{D})} \left[ \left( y(x,\mathcal{D})-\bar{y}(x) \right)^2 \right] }_{\text{variance}} \\ &\quad+ \underbrace{ \mathbb{E}_{p(x)} \mathbb{E}_{p(t\mid x)} \left[ \left( \bar{t}(x)-t \right)^2 \right] }_{\text{noise}}. \end{aligned} }\]

In short,

\[\boxed{ \text{generalisation error} = (\text{bias})^2 + \text{variance} + \text{noise}. }\]

This is an exact identity for squared loss under the expectations above.

How to Read the Decomposition

The decomposition is useful because the three terms respond differently to modelling choices.

Simple Models

A very simple model may not be able to represent the true relationship.

For example, if the true function is sinusoidal and we fit a straight line, the fitted line will miss the curve in roughly the same way no matter which training sample we draw.

This gives:

This is the typical picture of underfitting.

Adding more training data cannot make a straight line become a sine curve. More data can tell us very precisely which straight line is best, but the model family itself is still too limited.

Flexible Models

A very flexible model may be able to represent the true function, so its average prediction can have low bias.

However, with limited data, different samples may produce very different fitted models. The model has enough freedom to chase accidental fluctuations in each sample.

This gives:

This is the typical picture of overfitting.

Unlike high bias, high variance can often be reduced by collecting more data. With more observations, individual sampling accidents have less influence on the fitted model.

The interactive comparison on Slides 26-30 shows these effects with polynomial regression. A low-degree polynomial has a stable but systematically wrong fit. A high-degree polynomial can track the truth well on average but varies strongly from dataset to dataset.

Model Complexity and the Bias-Variance Trade-Off

As we make a model more flexible, bias often decreases because the model family can represent more functions. At the same time, variance often increases because the fitted model has more freedom to react to the training sample. Noise remains unchanged because it belongs to the data-generating process, not the model.

The sum can therefore have a minimum at an intermediate level of model complexity.

However, this U-shaped picture is an explanatory pattern, not a theorem saying that every modern model must behave this way. The slides make this point explicitly. What actually happens may differ depending on the model, training procedure, and data.

Which Term Does a Change Affect?

The following table is a useful guide.

Change Bias^2 Variance Noise
More training data roughly unchanged decreases unchanged
Simpler model usually increases decreases unchanged
Richer model usually decreases increases unchanged
Stronger regularisation usually increases decreases unchanged
Average several fitted models roughly unchanged decreases unchanged

The word "usually" matters. These are common effects, not universal laws.

Regularisation Through the Bias-Variance Lens

In Lecture 1 we introduced ridge regression. In Lectures 2 and 3, we interpreted regularisation as placing a prior on the parameters. We can now give it another interpretation.

Regularisation restricts the fitted model. For example,

\[\frac{\lambda}{2} \|\mathbf{w}\|^2\]

discourages large weights.

This can introduce additional bias because the fitted model is no longer free to choose the parameters that best fit the training data. But the restriction can reduce variance substantially because the model becomes less sensitive to the exact training sample.

This gives another way to think about why regularisation can improve generalisation:

Regularisation deliberately accepts some bias in exchange for lower variance.

The best amount of regularisation depends on the data. We therefore need a principled way to choose hyperparameters such as \lambda, polynomial degree, or model size.

Validation and Cross-Validation

As we have previously seen, fitting a model solely on training error usually results in overfitting. To estimate generalisation performance, we need to evaluate the model on data that was not used to fit it.

A common solution is to split the available data into:

For each candidate model:

  1. fit the parameters on the training data
  2. compute the loss on the validation data
  3. choose the candidate with the smallest validation loss

The validation set acts as a small sample of future data. Because the model was not fitted to those examples, its performance on them gives information about generalisation.

K-Fold Cross-Validation

A single train-validation split can be wasteful when data is scarce. The answer can also depend strongly on which examples happened to be placed in the validation set.

In K-fold cross-validation, we divide the data into K parts, called folds.

For each fold k:

  1. hold out fold k
  2. fit the model using the other K-1 folds
  3. measure the loss on fold k

Then average the K validation losses.

Each example is used for validation once and for training in the other K-1 fits.

Slides 36-38 show this process and how cross-validation can select a polynomial degree that is close to the one with the lowest true generalisation error in the synthetic example.

Cross-validation is extremely useful, but several caveats matter:

For a final estimate of performance, we therefore keep a separate test set that is not used for fitting or model selection. We evaluate on the test set only after the model and hyperparameters have been chosen.

Why Can Validation Work?

We now look at a simple version of model selection that we can analyse mathematically. Full K-fold cross-validation is too difficult for us at the moment. Instead, suppose we have one independent validation set of V examples and a fixed collection of candidate models

\[h_0,h_1,\ldots,h_M.\]

The candidates are fixed before we look at the validation data.

For a candidate h, define its true risk

\[\boxed{ R(h) = \mathbb{E}_{p(x,t)} \left[ \ell(h(x),t) \right], }\]

and its validation score

\[\boxed{ \widehat{R}(h) = \frac{1}{V} \sum_{i=1}^{V} \ell(h(x_i),t_i). }\]

Assume the loss is bounded:

\[0\leq \ell(h(x),t)\leq 1.\]

For example, \ell could be the 0-1 classification loss.

The basic question is:

How close is the validation score \widehat{R}(h) to the true risk R(h)?

Hoeffding's Inequality

For a fixed candidate h, the validation losses

\[\ell(h(x_1),t_1),\ldots,\ell(h(x_V),t_V)\]

are independent draws in [0,1] (the loss is bounded between 0 and 1, inclusive), provided the validation examples are independent and were not used to fit h.

Their average is \widehat{R}(h), and their expectation is R(h).

Hoeffding's inequality tells us that

\[\boxed{ \Pr \left( \left| \widehat{R}(h)-R(h) \right| \geq \varepsilon \right) \leq 2\exp \left( -2V\varepsilon^2 \right). }\]

This is a concentration result: as the number of validation examples V grows, it becomes increasingly unlikely that the validation score is far from the true risk.

There are two useful things to notice:

  1. the bound only requires independent observations and a bounded loss. Notably, it does not assume that the loss is Gaussian.
  2. this is an upper bound. It can be loose, and for small V or very small \varepsilon the numerical bound may even exceed 1, in which case it tells us nothing useful.

The interactive plots on Slides 46-49 show how the bound changes with V and \varepsilon.

Evaluating Multiple Candidate Models

We do not use validation to evaluate only one model. We compare many candidates h_0, h_1, \dots, h_M and choose the one with the lowest validation score.

Even if every candidate individually has a small chance of receiving a misleading score, we care about the event that any candidate receives such a score. This is because we want to prevent the case where one lucky candidate wins based on pure luck.

For candidate j, define the bad event

\[B_j = \left\{ \left| \widehat{R}(h_j)-R(h_j) \right| \geq \varepsilon \right\}.\]

Hoeffding only gives

\[\Pr(B_j) \leq 2e^{-2V\varepsilon^2}\]

for a single candidate h_j. We want to control

\[\Pr \left( B_0\cup B_1\cup\cdots\cup B_M \right).\]

The Union Bound

For any collection of events,

\[\boxed{ \Pr \left( \bigcup_j B_j \right) \leq \sum_j \Pr(B_j). }\]

This is the union bound.

Applying it to our M+1 candidate models gives

\[\Pr \left( B_0\cup\cdots\cup B_M \right) \leq \sum_{j=0}^M \Pr(B_j).\]

and using Hoeffding's inequality gives us, \[\Pr \left( B_0\cup\cdots\cup B_M \right) \leq 2(M+1)e^{-2V\varepsilon^2}.\]

Set the right-hand side equal to a failure probability \delta:

\[2(M+1)e^{-2V\varepsilon^2} = \delta.\]

Solving for \varepsilon gives

\[\boxed{ \varepsilon = \sqrt{ \frac{ \ln\left(2(M+1)/\delta\right) }{ 2V } }. }\]

Therefore, with probability at least 1-\delta, every validation score is within \varepsilon of its true risk at the same time:

\[\left| \widehat{R}(h_j)-R(h_j) \right| \leq \varepsilon \qquad \text{for every }j.\]

More explicitly:

\[\left| \widehat{R}(h_j)-R(h_j) \right| \leq \sqrt{ \frac{ \ln\left(2(M+1)/\delta\right) }{ 2V } } \qquad \text{for every }j.\]

This is a remarkable result! It means that even though we don't know the true risk R(h_j), we can say something about how different it is from the empirical error \widehat{R}(h_j). We see that the upper bound depends on the number of models M, the confidence \delta and the number of samples in the validation set V. Take a moment to analyse how each factor affects the upper bound. For example, what happens when we increase V? Ask yourself: are the effects reasonable?

Note: Despite being remarkable, the bound is loose; it can be significantly larger than the true difference between R(h_j) and \widehat{R}(h_j).

What This Means for the Model We Select

Let

\[\hat{\jmath} = \operatorname*{arg\,min}_j \widehat{R}(h_j)\]

be the candidate selected using the validation data.

Let

\[j^* = \operatorname*{arg\,min}_j R(h_j)\]

be the candidate that actually has the smallest true risk.

Suppose the good event above holds, so every validation score is within \varepsilon of its true risk.

Then

\[R(h_{\hat{\jmath}}) \leq \widehat{R}(h_{\hat{\jmath}}) + \varepsilon.\]

Because h_{\hat{\jmath}} has the smallest validation score,

\[\widehat{R}(h_{\hat{\jmath}}) \leq \widehat{R}(h_{j^*}).\]

And because the score of h_{j^*} is also within \varepsilon of its true risk,

\[\widehat{R}(h_{j^*}) \leq R(h_{j^*}) + \varepsilon.\]

Combining the three inequalities gives

\[\boxed{ R(h_{\hat{\jmath}}) \leq \min_j R(h_j) + 2\varepsilon. }\]

Substituting the expression for \varepsilon,

\[\boxed{ R(h_{\hat{\jmath}}) \leq \min_j R(h_j) + 2 \sqrt{ \frac{ \ln\left(2(M+1)/\delta\right) }{ 2V } } }\]

with probability at least 1-\delta.

This is another remarkable result! It says that if the validation set is large enough, the candidate selected by validation will have true risk close to the best true risk among the candidates we considered.

As before, the bound also shows two useful trends:

However, the result (and the earlier one) depend on an important condition: the candidate models must be fixed before the validation data is used to score them. If we repeatedly inspect validation results, redesign the candidates, and test again on the same validation set, we are adapting to the validation data and this simple guarantee no longer applies.

Cross-Validation and the Simple Bound

The mathematical result above is for a single independent validation set, not full K-fold cross-validation.

In K-fold cross-validation, the training sets overlap heavily and every data point is used for training in some folds and validation in another. The resulting validation errors are therefore dependent in a more complicated way.

The same intuition still motivates cross-validation: evaluate models on data not used to fit that particular model, and average over several held-out splits. But the simple Hoeffding-plus-union-bound above does not transfer directly to K-fold cross-validation.

For this lecture, the important lesson is:

Held-out data works because it gives us an independent sample of losses. With enough held-out examples, averages of those losses are likely to be close to their expectations.

Summary

In this lecture, we shifted attention from fitting the model to understanding how well the fitted model should work on new data.

Training error measures performance on the examples used for fitting. Generalisation error measures expected performance on a fresh example:

\[R(\mathcal{D}) = \mathbb{E}_{p(x,t)} \left[ \left( y(x,\mathcal{D})-t \right)^2 \right].\]

Because the training dataset is itself random, the fitted model is also random. We therefore defined the average model

\[\bar{y}(x) = \mathbb{E}_{p(\mathcal{D})} \left[ y(x,\mathcal{D}) \right]\]

and the average target

\[\bar{t}(x) = \mathbb{E}_{p(t\mid x)}[t].\]

For squared loss, the expected generalisation error decomposes exactly as

\[\boxed{ \text{error} = (\text{bias})^2 + \text{variance} + \text{noise}. }\]

Bias measures the gap between the average model and the average target. Variance measures how much fitted models vary around the average model when the training dataset changes. Noise measures variation in the target itself.

A model that is too simple often has high bias and low variance. A highly flexible model can have low bias but high variance. This gives us a useful way to think about underfitting and overfitting.

The decomposition also helps us understand common interventions:

To choose model complexity, we cannot rely on training error because it rewards flexibility. Instead, we estimate performance using held-out data or cross-validation.

For one independent validation set of V examples and a fixed collection of M+1 candidate models, Hoeffding's inequality and the union bound give

\[\varepsilon = \sqrt{ \frac{ \ln\left(2(M+1)/\delta\right) }{ 2V } }.\]

With probability at least 1-\delta, every validation score is within \varepsilon of its true risk, and the model selected by validation satisfies

\[\boxed{ R(h_{\hat{\jmath}}) \leq \min_j R(h_j) + 2\varepsilon. }\]

The main ideas to take away are:

  1. Low training error is not the goal; low generalisation error is.
  2. Bias describes systematic error, variance describes sensitivity to the training sample, and noise comes from the data itself.
  3. Underfitting and overfitting are different problems and need different fixes
  4. Validation works by testing models on data that was not used to fit them.
  5. Model selection itself can overfit, so final performance should be measured on a separate test set.