Lecture 6: Computational Learning Theory

Summary of Key Ideas

In Lecture 5, we asked why a model that fits the training data should work on new data. We studied this question using the bias-variance decomposition and then used held-out data to estimate generalisation error.

In this lecture, we ask a more basic question:

What can the training data itself tell us about performance on unseen data?

The first answer is: without assumptions, very little. A model can fit every observed example perfectly while behaving in many different ways on inputs that were never observed. To learn from a finite dataset, we need some reason to believe that the observed examples tell us something about future examples.

We will make two main assumptions:

  1. The training examples are drawn independently from one fixed distribution p(x,t).
  2. We restrict ourselves to a hypothesis class \mathcal{H} chosen before seeing the data.

For one hypothesis h fixed before the data is drawn, Hoeffding's inequality tells us that its training error is likely to be close to its generalisation error. Learning is harder because we use the training data to search through \mathcal{H} and choose a hypothesis with small training error. The selected hypothesis therefore depends on the data.

The main idea of the lecture is to control all hypotheses at once. If every hypothesis in \mathcal{H} has training error close to generalisation error, then the hypothesis chosen by the learning algorithm is also covered. For a finite hypothesis class, Hoeffding's inequality together with the union bound gives

\[\Pr \left( \exists h\in\mathcal{H}: \left| L_{\mathcal{D}}(h)-L_p(h) \right| > \varepsilon \right) \leq 2|\mathcal{H}|e^{-2N\varepsilon^2}.\]

This is our first example of a PAC, or probably approximately correct, guarantee. It tells us how the amount of data, the size of the hypothesis class, the allowed error \varepsilon, and the failure probability \delta are related.

Finally, we will briefly see what happens when \mathcal{H} is infinite. Counting hypotheses no longer works, but we can replace the raw number of hypotheses with better measures of what the class can actually do on a finite dataset. We will also see the basic idea behind PAC-Bayes, where a prior gives different hypotheses different complexity costs.

Learning Objectives

By the end of this lesson, the main things you should be able to do are:

We will first ask whether learning is possible at all. We will then analyse a finite hypothesis class carefully. The last section gives a short view of how the same ideas extend beyond finite classes.

Is Learning Possible?

Suppose the input consists of three binary values,

\[x\in\{0,1\}^3,\]

so there are only eight possible inputs. Imagine that our training dataset contains five of these eight inputs and their labels.

Now suppose we consider every possible binary classifier

\[\mathcal{H} = \left\{ h:\{0,1\}^3\rightarrow\{0,1\} \right\}.\]

There may be many hypotheses that agree perfectly with all five training examples. Yet they can assign completely different labels to the three inputs we have not seen.

The demonstration on Slide 9 shows this directly. Every hypothesis displayed has zero training error, but together they produce every possible labelling of the unseen inputs.

The lesson is simple:

A perfect fit to the observed data does not, by itself, tell us what should happen on unobserved data.

We therefore need assumptions that connect the sample we observed to the examples we may see in the future.

Two Assumptions

For the rest of the main analysis, we make two assumptions.

First, the data is sampled independently from one fixed distribution:

\[(x_n,t_n) \sim p(x,t), \qquad n=1,\ldots,N.\]

This means that the training examples and future examples come from the same underlying population.

Second, we choose a hypothesis class \mathcal{H} before seeing the data. The learner searches only inside this class.

For example, \mathcal{H} might be all decision stumps using a fixed set of features, all polynomials up to some fixed degree, or a finite collection of classifiers prepared in advance.

The first assumption lets the training data tell us something about future data. The second controls how many different explanations the learner is allowed to choose from.

We will not obtain a statement that is always exactly correct. Instead, we will obtain a statement that is probably approximately correct.

Training Loss and Generalisation Loss

From this point, the lecture focuses on binary classification with the 0-1 loss:

\[\ell(h(x),t) = \mathbb{1}[h(x)\neq t].\]

The loss is 1 if the classifier is wrong and 0 if it is correct.

For a dataset

\[\mathcal{D} = \{(x_n,t_n)\}_{n=1}^{N},\]

the training loss is

\[\boxed{ L_{\mathcal{D}}(h) = \frac{1}{N} \sum_{n=1}^{N} \ell(h(x_n),t_n). }\]

This is simply the fraction of training examples that h misclassifies.

The generalisation loss is

\[\boxed{ L_p(h) = \mathbb{E}_{p(x,t)} \left[ \ell(h(x),t) \right]. }\]

For 0-1 loss, this is the probability that h misclassifies a fresh example from the underlying distribution.

We can compute L_{\mathcal{D}}(h) from the training data. We cannot directly compute L_p(h) because we do not know p(x,t). Our goal is therefore to understand when

\[L_{\mathcal{D}}(h) \approx L_p(h).\]

From Verification to Learning

A Fixed Hypothesis Is Easy to Verify

Suppose we choose one hypothesis h before drawing the training data.

For each training example, define

\[Z_n = \ell(h(x_n),t_n).\]

Each Z_n is either 0 or 1, and

\[\mathbb{E}[Z_n] = L_p(h).\]

The training loss is their average:

\[L_{\mathcal{D}}(h) = \frac{1}{N} \sum_{n=1}^{N} Z_n.\]

Since the examples are independent and Z_n\in[0,1], Hoeffding's inequality gives

\[\boxed{ \Pr \left( \left| L_{\mathcal{D}}(h)-L_p(h) \right| > \varepsilon \right) \leq 2e^{-2N\varepsilon^2}. }\]

So, for one hypothesis fixed in advance, the training loss is unlikely to be very far from the generalisation loss. This is similar to the Hot-or-Not example from Lecture 5. There we estimated an unknown population fraction using the average of a finite sample. Here, each example tells us whether a fixed classifier was right or wrong, and the sample average estimates its true error rate.

Verification Is Not Learning

There is an important catch. Hoeffding's inequality above applies to a hypothesis that was fixed before the data was drawn. But a learning algorithm does the opposite:

  1. draw the dataset \mathcal{D}
  2. evaluate or fit many hypotheses using \mathcal{D}
  3. choose one that performs well on \mathcal{D}

For empirical risk minimisation,

\[\boxed{ \hat{h} = \operatorname*{arg\,min}_{h\in\mathcal{H}} L_{\mathcal{D}}(h). }\]

The selected hypothesis \hat{h} depends on the dataset. We cannot simply take the fixed-h Hoeffding bound and replace h with \hat{h}.

This is the same selection effect we have seen before. Each fixed hypothesis has a noisy estimate of its true performance. When we search over many hypotheses and keep the one with the smallest training error, the search can favour both genuinely good hypotheses and hypotheses that happened to look unusually good on this particular sample.

The hiring example on Slide 24 gives the same intuition: if many candidates sit a noisy exam and we hire the top scorer, we tend to select someone whose score contains both ability and favourable luck.

The question is therefore:

How can we make a statement that remains valid even after the data is used to choose the hypothesis?

Uniform Convergence and Finite-Class Learning

We do not know in advance which hypothesis the learner will choose. A simple solution is to ask for a statement that covers every hypothesis at once:

\[\boxed{ \left| L_{\mathcal{D}}(h)-L_p(h) \right| \leq \varepsilon \qquad \text{for every }h\in\mathcal{H}. }\]

This property is called uniform convergence.

If it holds, then it automatically holds for \hat{h}, regardless of which hypothesis the learning algorithm selected from \mathcal{H}.

The interactive plot on Slide 26 shows this idea. A dataset is good when all hypotheses have empirical errors close to their true errors. A dataset is bad when even one hypothesis has a large gap.

For a finite hypothesis class, we can control this bad event using two tools we already know:

  1. Hoeffding's inequality for one fixed hypothesis
  2. the union bound to cover the whole class

The Finite-Class Bound

For each hypothesis h\in\mathcal{H}, define the bad event

\[B_h = \left\{ \left| L_{\mathcal{D}}(h)-L_p(h) \right| > \varepsilon \right\}.\]

For one fixed hypothesis, Hoeffding gives

\[\Pr(B_h) \leq 2e^{-2N\varepsilon^2}.\]

The event that uniform convergence fails is the event that at least one hypothesis is bad:

\[\bigcup_{h\in\mathcal{H}} B_h.\]

Using the union bound,

\[\Pr \left( \bigcup_{h\in\mathcal{H}} B_h \right) \leq \sum_{h\in\mathcal{H}} \Pr(B_h).\]

If \mathcal{H} contains |\mathcal{H}| hypotheses, then

\[\boxed{ \Pr \left( \exists h\in\mathcal{H}: \left| L_{\mathcal{D}}(h)-L_p(h) \right| > \varepsilon \right) \leq 2|\mathcal{H}|e^{-2N\varepsilon^2}. }\]

This is the main finite-class result. The important step is not very complicated but can be confusing if you are new to this topic. Go slow and remember that Hoeffding controls one hypothesis. The union bound is used to analyze searching over the whole class. The union bound is "coarse" since we are overcounting.

Reading the PAC Guarantee

Let \delta be the probability that we are willing to assign to the bad event. We ask for

\[2|\mathcal{H}|e^{-2N\varepsilon^2} \leq \delta.\]

Then, with probability at least 1-\delta over the random draw of the training set,

\[\left| L_{\mathcal{D}}(h)-L_p(h) \right| \leq \varepsilon \qquad \text{for every }h\in\mathcal{H}.\]

This is the basic PAC language:

The probability is over which training dataset we happen to draw.

How Much Data Do We Need?

Starting from

\[2|\mathcal{H}|e^{-2N\varepsilon^2} \leq \delta,\]

solve for N:

\[\boxed{ N \geq \frac{1}{2\varepsilon^2} \ln \left( \frac{2|\mathcal{H}|}{\delta} \right). }\]

This is a sample complexity result. It tells us how many training examples are sufficient for the finite-class guarantee.

There are three useful things to notice. Asking for a smaller error tolerance is expensive because N grows as 1/\varepsilon^2. Halving \varepsilon requires four times as much data. The size of the hypothesis class appears only inside a logarithm, so a larger class costs more data, but only logarithmically. Higher confidence also costs only logarithmically through \ln(1/\delta).

The interactive plot on Slide 31 is useful for seeing how these quantities affect the required sample size.

Generalisation Error as Training Error Plus a Penalty

We can instead solve the same inequality for \varepsilon:

\[\boxed{ \varepsilon = \sqrt{ \frac{ \ln(2|\mathcal{H}|/\delta) }{ 2N } }. }\]

Therefore, with probability at least 1-\delta, every hypothesis satisfies

\[\boxed{ L_p(h) \leq L_{\mathcal{D}}(h) + \sqrt{ \frac{ \ln(2|\mathcal{H}|/\delta) }{ 2N } }. }\]

This has a useful interpretation:

\[\boxed{ \text{generalisation loss} \leq \text{training loss} + \text{complexity penalty}. }\]

The penalty grows with the size of the hypothesis class and shrinks with the amount of data. This gives us a formal version of an idea that has appeared throughout the course: a flexible model can obtain a smaller training error, but searching a larger set of possibilities also makes it easier to fit accidents in the training sample.

What Does the Bound Say About the Learned Hypothesis?

Let

\[\hat{h} = \operatorname*{arg\,min}_{h\in\mathcal{H}} L_{\mathcal{D}}(h)\]

be the hypothesis chosen by empirical risk minimisation.

Let

\[h^* = \operatorname*{arg\,min}_{h\in\mathcal{H}} L_p(h)\]

be the best hypothesis in the class according to the true generalisation loss.

Suppose uniform convergence holds, so every hypothesis satisfies

\[\left| L_{\mathcal{D}}(h)-L_p(h) \right| \leq \varepsilon.\]

Then

\[L_p(\hat{h}) \leq L_{\mathcal{D}}(\hat{h}) + \varepsilon.\]

Since \hat{h} has the smallest training loss,

\[L_{\mathcal{D}}(\hat{h}) \leq L_{\mathcal{D}}(h^*).\]

And since uniform convergence also holds for h^*,

\[L_{\mathcal{D}}(h^*) \leq L_p(h^*) + \varepsilon.\]

Combining these gives

\[\boxed{ L_p(\hat{h}) \leq L_p(h^*) + 2\varepsilon. }\]

So the learner's hypothesis has generalisation loss at most 2\varepsilon worse than the best hypothesis in the class.

Note that the guarantee compares \hat{h} with the best model available inside \mathcal{H}. If every hypothesis in \mathcal{H} is poor, learning cannot fix the problem. This is the same basic issue we called high bias in Lecture 5.

The Realizable Case

There is a special case where we can obtain a stronger result.

The problem is realizable if there is some hypothesis in the class that is always correct:

\[L_p(h^*)=0.\]

If such a hypothesis exists, it also has zero training error on every dataset. Suppose our learner returns some hypothesis \hat{h} with zero training error.

Call a hypothesis bad if

\[L_p(h)>\varepsilon.\]

A bad hypothesis has probability at least \varepsilon of making a mistake on a fresh example. Therefore, the probability that it makes no mistakes on N independent training examples is

\[(1-L_p(h))^N < (1-\varepsilon)^N.\]

Using

\[1-\varepsilon \leq e^{-\varepsilon},\]

we get

\[(1-\varepsilon)^N \leq e^{-\varepsilon N}.\]

So one bad hypothesis has probability at most e^{-\varepsilon N} of looking perfect on the training set. Using the union bound over the whole finite class,

\[\boxed{ \Pr \left( L_p(\hat{h})>\varepsilon \right) \leq |\mathcal{H}|e^{-\varepsilon N}. }\]

To make this probability at most \delta, it is sufficient that

\[\boxed{ N \geq \frac{1}{\varepsilon} \ln \left( \frac{|\mathcal{H}|}{\delta} \right). }\]

Compare this with the general finite-class result:

\[N \geq \frac{1}{2\varepsilon^2} \ln \left( \frac{2|\mathcal{H}|}{\delta} \right).\]

The realizable case depends on 1/\varepsilon rather than 1/\varepsilon^2. The stronger assumption lets us obtain a stronger guarantee. The interactive comparison on Slide 38 shows the difference.

Realizable and Non-Realizable Problems

It is useful to keep the two settings separate.

In the realizable setting,

\[\min_{h\in\mathcal{H}}L_p(h)=0.\]

The class contains a perfect classifier.

In the non-realizable, or agnostic, setting,

\[\min_{h\in\mathcal{H}}L_p(h)>0.\]

No classifier in the class is perfect. We then aim to learn something close to the best hypothesis available in the class:

\[L_p(\hat{h}) \approx \min_{h\in\mathcal{H}}L_p(h).\]

The earlier uniform-convergence result covers this more general case.

What the Finite-Class Result Teaches Us

Before moving to more advanced ideas, it is useful to separate the main lessons from the algebra.

A finite dataset cannot determine arbitrary behaviour outside the observed points. We assumed that examples come from one fixed distribution and that the learner searches within a restricted hypothesis class.

For one fixed hypothesis, Hoeffding is enough. Once the learner searches a class, we need a statement that remains true for every hypothesis. For a finite class, the statistical cost of this search appears as

\[\ln|\mathcal{H}|.\]

More data decreases the generalisation penalty approximately as

\[\frac{1}{\sqrt{N}}.\]

Finally, the result is a worst-case upper bound. It does not say the actual gap is equal to the penalty. The true generalisation error can be much smaller.

This point becomes important for modern machine learning, where simple worst-case capacity measures can give bounds that are technically correct but not useful.

Beyond Finite Hypothesis Classes

The finite-class proof has an obvious problem for many models we actually use.

A linear classifier in two dimensions has parameters

\[w_1,w_2,b\in\mathbb{R}.\]

There are infinitely many possible parameter values, so

\[|\mathcal{H}|=\infty.\]

If we put this into the finite-class bound, the penalty becomes infinite. The statement remains true, but tells us nothing.

The problem is not necessarily that the model is impossible to learn. The problem is that counting every parameter setting separately is wasteful. Many different parameter settings behave identically, or almost identically, on a finite dataset.

The last part of the lecture introduces two ways to improve this counting idea. Note: This part is extended material for those of you interested in machine learning theory; it is not examinable in CS3264.

Count Behaviours, Not Hypotheses

Take N fixed input points. Even an infinite hypothesis class can produce only finitely many binary labellings of those points.

A particular labelling is called a dichotomy.

The growth function is the largest number of different labellings the class can produce on any set of N points:

\[\boxed{ m_{\mathcal{H}}(N) = \max_{x_1,\ldots,x_N} \left| \left\{ (h(x_1),\ldots,h(x_N)) : h\in\mathcal{H} \right\} \right|. }\]

Since N binary labels have at most 2^N possible assignments,

\[m_{\mathcal{H}}(N) \leq 2^N.\]

The useful question is whether the class can actually realise all of these labellings.

For positive rays on the real line,

\[h_a(x) = \operatorname{sign}(x-a),\]

the threshold a can fall into one of N+1 gaps between the points. Therefore,

\[m_{\mathcal{H}}(N)=N+1.\]

For intervals on the line, the slides show

\[m_{\mathcal{H}}(N) = \frac{1}{2}N(N+1)+1.\]

For affine linear classifiers in the plane, for points in general position, the slides give

\[m_{\mathcal{H}}(N) = N^2-N+2.\]

All three classes contain infinitely many hypotheses, but the number of different labellings they can produce on N points grows only polynomially. The demonstrations on Slides 47-50 show these examples.

A more advanced generalisation bound can replace |\mathcal{H}| with the growth function. The slides give the following form:

\[\boxed{ L_p(h) \leq L_{\mathcal{D}}(h) + \sqrt{ \frac{8}{N} \ln \left( \frac{ 4m_{\mathcal{H}}(2N) }{ \delta } \right) } }\]

for every h\in\mathcal{H} with probability at least 1-\delta.

You do not need to derive this result here. The important idea is:

For an infinite class, count how many different behaviours it can produce on finite data, rather than how many parameter settings it contains.

The VC dimension and Rademacher complexity develop this idea further, but these are beyond the scope of this course.

Weigh Hypotheses Instead of Counting Them Equally

There is another limitation in the finite-class proof. The union bound gave every hypothesis the same share of the failure probability.

Suppose we instead assign a prior probability P(h) to each hypothesis, where

\[\sum_h P(h)=1.\]

We can allocate hypothesis h a failure budget \delta P(h). Applying Hoeffding with this budget gives a bound of the form

\[\boxed{ L_p(h) \leq L_{\mathcal{D}}(h) + \sqrt{ \frac{ \ln(1/P(h)) + \ln(2/\delta) }{ 2N } }. }\]

A hypothesis with large prior probability receives a smaller complexity penalty. A hypothesis that was considered very unlikely before seeing the data pays a larger penalty.

This connects learning theory with an idea we saw in Bayesian modelling: a prior can express which explanations we consider simple or plausible before observing the data.

For a continuous parameter space, however, a single exact hypothesis often has probability mass zero under a continuous prior. PAC-Bayes handles this by placing a prior distribution P over hypotheses and comparing it with a data-dependent distribution Q over hypotheses.

One representative bound shown in the slides is

\[\boxed{ \mathbb{E}_{h\sim Q}[L_p(h)] \leq \mathbb{E}_{h\sim Q}[L_{\mathcal{D}}(h)] + \sqrt{ \frac{ \mathrm{KL}(Q\|P) + \ln(2\sqrt{N}/\delta) }{ 2N } }. }\]

The key term is

\[\mathrm{KL}(Q\|P),\]

which measures how far the data-dependent distribution Q has moved from the prior P.

We will not develop PAC-Bayes further in this course. The main idea to remember is that complexity need not be measured only by counting models. It can also be measured by how much the learning process moved away from a prior belief.

Main Lessons: Why These Results Are Useful Even When the Bound Is Loose

The numerical value of a worst-case generalisation bound can be too large to be useful, especially for very flexible models. That does not make the argument pointless.

The finite-class proof teaches us a general pattern:

  1. identify the quantity measured on the sample
  2. ask how far it can be from its population value
  3. account for the fact that learning searched over many possible hypotheses
  4. express the resulting penalty using some measure of model complexity

Different learning theories use different ways to perform the third step. The finite-class argument uses \ln|\mathcal{H}|. Growth-function and VC arguments count possible behaviours on data. PAC-Bayes uses a prior and a divergence from that prior.

The slides use modern overparameterised neural networks to motivate why crude measures such as raw parameter count can fail to explain observed generalisation. The broader lesson is that the right notion of model complexity depends on the learning problem and the learning algorithm.

Summary

A finite training set does not determine what a model should do on unseen inputs. To make learning possible, we assumed that the data consists of independent samples from one fixed distribution and that the learner searches inside a hypothesis class \mathcal{H} chosen before the data.

For binary classification, the training loss is

\[L_{\mathcal{D}}(h) = \frac{1}{N} \sum_{n=1}^{N} \mathbb{1}[h(x_n)\neq t_n],\]

while the generalisation loss is

\[L_p(h) = \mathbb{E}_{p(x,t)} \left[ \mathbb{1}[h(x)\neq t] \right].\]

For one hypothesis fixed before the dataset is drawn, Hoeffding's inequality gives

\[\Pr \left( \left| L_{\mathcal{D}}(h)-L_p(h) \right| > \varepsilon \right) \leq 2e^{-2N\varepsilon^2}.\]

Learning is harder because the hypothesis is selected using the training data. We therefore asked for uniform convergence, so that the training and generalisation losses are close for every hypothesis simultaneously.

For a finite class, Hoeffding's inequality and the union bound give

\[\boxed{ \Pr \left( \exists h\in\mathcal{H}: \left| L_{\mathcal{D}}(h)-L_p(h) \right| > \varepsilon \right) \leq 2|\mathcal{H}|e^{-2N\varepsilon^2}. }\]

To make the failure probability at most \delta, it is sufficient that

\[\boxed{ N \geq \frac{1}{2\varepsilon^2} \ln \left( \frac{2|\mathcal{H}|}{\delta} \right). }\]

Equivalently, with probability at least 1-\delta,

\[\boxed{ L_p(h) \leq L_{\mathcal{D}}(h) + \sqrt{ \frac{ \ln(2|\mathcal{H}|/\delta) }{ 2N } } \qquad \text{for every }h\in\mathcal{H}. }\]

If \hat{h} minimises training error and h^* is the best hypothesis in the class, uniform convergence gives

\[\boxed{ L_p(\hat{h}) \leq L_p(h^*) + 2\varepsilon. }\]

In the realizable case, where the class contains a perfect classifier, we can obtain the sharper sample complexity

\[\boxed{ N \geq \frac{1}{\varepsilon} \ln \left( \frac{|\mathcal{H}|}{\delta} \right). }\]

Finally, the raw finite-class argument breaks when \mathcal{H} is infinite. Two ways forward are:

The main ideas to take away are:

  1. Learning from finite data requires assumptions that connect the observed sample to unseen data.
  2. A guarantee for one fixed hypothesis is not automatically a guarantee for a hypothesis chosen using the same data.
  3. Uniform convergence solves this problem by making the guarantee hold for the whole hypothesis class at once.
  4. Searching a richer class has a statistical cost, which appears as a complexity penalty.
  5. The central problem in learning theory is to find a useful measure of complexity that explains when fitting the data will generalise.