Lecture 2: Regression II — Probabilistic Modelling

Summary of Key Ideas

In Lecture 1, we described machine learning methods using three main ingredients: data, model, and loss. For regression, we chose a squared-error loss because it seemed sensible. We also introduced ridge regression by adding a penalty on large weights.

In this lecture, we ask a basic question:

Where do these loss functions come from?

The key idea is to make the model probabilistic. Instead of predicting a single number, the model describes a probability distribution over possible targets. Once we state how we believe the data was generated, the loss function can often be derived rather than chosen by hand.

For example, if we assume that regression targets are corrupted by Gaussian noise, then maximum likelihood gives us squared error. If we also place a Gaussian prior on the weights, maximum a posteriori estimation gives us ridge regression.

Taking the probabilistic view also gives us something new: a way to represent uncertainty. Rather than returning only a prediction, a model can tell us how uncertain that prediction is.

Learning Objectives

By the end of this lesson, you should be able to:

We will first review the small amount of probability needed for the lecture. We will then learn how to fit a probabilistic model using maximum likelihood. After that, we will apply the same ideas to regression and finally introduce the Bayesian view.

From Point Predictions to Probability Distributions

Recall the three ingredients from Lecture 1:

For linear regression, we used

\[\mathcal{D} = \{(x_n,t_n)\}_{n=1}^{N},\]

a model

\[y(x;\mathbf{w}) = \mathbf{w}^{\top}\boldsymbol{\phi}(x),\]

and squared-error loss

\[E(\mathbf{w}) = \frac{1}{2} \sum_n \left( y(x_n;\mathbf{w})-t_n \right)^2.\]

The problem is that we simply chose the last expression.

Why square the error rather than use

\[|y-t|\]

or perhaps

\[(y-t)^4?\]

Similarly, for ridge regression we wrote down

\[\frac{\lambda}{2}\|\mathbf{w}\|_2^2\]

as a penalty on large weights. But why this particular penalty?

Lecture 2 provides one answer: write down a probability model for how the data was generated. The structure remains data, model, and loss, but the model changes. Instead of saying

the target is 14.2,

we say something like

targets around 14.2 are likely, and targets much further away are less likely.

The output of the model is now a probability distribution.

Why is this useful? Consider the used-car example on Slide 8. Suppose two models both predict that a particular car should sell for $14,200. The seller asks $15,000. If selling prices for similar cars are tightly clustered around $14,200, then $15,000 is unusual. If selling prices vary widely, then $15,000 may be completely ordinary. The average prediction is identical in both cases. What differs is the uncertainty around that prediction.

Probability Preliminaries

We need a few basic probability ideas for this lecture:

  1. probability distributions
  2. the sum rule
  3. the product rule
  4. Bayes' rule
  5. independence

These ideas will reappear throughout the course.

Discrete Probabilities and Continuous Densities

For a discrete random variable, individual outcomes can have non-zero probability.

For example,

\[P(X=3)=0.2.\]

The probabilities over all possible values must sum to one:

\[\sum_x P(X=x)=1.\]

For a continuous random variable, things work slightly differently. We describe the distribution using a probability density function, or pdf, written as p(x).

For a continuous variable, probability corresponds to area under the density, not the height of the curve. For example,

\[P(a<X<b) = \int_a^b p(x)\,dx.\]

The entire density integrates to one:

\[\int_{-\infty}^{\infty}p(x)\,dx=1.\]

The value p(x) itself is a density, not a probability. A probability density can even be larger than 1.

Use the interactive plot on Slide 13 and switch between the discrete pmf and continuous pdf. For the continuous case, reduce the spread \sigma. You can make the peak rise above 1 while the total area remains 1.

This distinction becomes important when we discuss likelihoods.

The Gaussian Distribution

A distribution we will use repeatedly is the Gaussian distribution, also called the normal distribution:

\[\mathcal{N}(x\mid\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left( -\frac{(x-\mu)^2}{2\sigma^2} \right).\]

It has two parameters:

The quantity \sigma is the standard deviation.

Use Slide 14 to change the mean and variance. Notice that changing \mu shifts the distribution while changing \sigma^2 changes how wide it is.

We will use Gaussian distributions to describe noise in regression.

Two Basic Probability Rules

There are two basic probability rules that will come up again and again during the course. You should get very familiar with them:

The Sum Rule

Suppose we have two variables x and y with joint distribution p(x,y).

If we only care about x, we can remove y by summing over it:

\[p(x) = \sum_y p(x,y).\]

For continuous variables,

\[p(x) = \int p(x,y)\,dy.\]

The resulting p(x) is called a marginal distribution, and the process is sometimes called marginalisation. Use the table on Slide 16. Each cell represents a joint probability p(x,y). Adding across the appropriate cells removes one variable and leaves the marginal probability of the other.

The Product Rule

The product rule is

\[p(x,y) = p(y\mid x)p(x).\]

Here p(y\mid x) is a conditional probability: the probability of y given that we know x. Rearranging gives

\[p(y\mid x) = \frac{p(x,y)}{p(x)}.\]

Conditioning means restricting our attention to the cases where x has a particular value and then rescaling the remaining probabilities so that they sum to one. Use Slide 17 and move the x slider. The highlighted column is the part of the joint distribution we keep. Rescaling that column gives p(y\mid x).

Bayes' Rule

From the sum and product rules, we can derive Bayes Rule which is an important one for probabilistic models.

We will derive it here. The product rule can be written in two ways:

\[p(A,B) = p(A\mid B)p(B),\]

and

\[p(A,B) = p(B\mid A)p(A).\]

Since both expressions describe the same joint probability,

\[p(A\mid B)p(B) = p(B\mid A)p(A).\]

Dividing by p(B) gives Bayes' rule:

\[\boxed{ p(A\mid B) = \frac{p(B\mid A)p(A)} {p(B)} }\]

The denominator can be obtained using the sum rule:

\[p(B) = \sum_{A'}p(B\mid A')p(A').\]

So Bayes' rule is not really a separate probability rule; as you can see, it follows from the product rule and the sum rule. Slides 21-23 give a useful example of Bayes rule and how it is used.

Applied to Machine Learning, Bayes' rule gives us the following:

\[p(\theta\mid\mathcal{D}) = \frac{ p(\mathcal{D}\mid\theta)p(\theta) }{ p(\mathcal{D}) }.\]

The terms have standard names:

A useful shorthand is

\[\boxed{ \text{posterior} \propto \text{likelihood} \times \text{prior} }\]

where \propto means "proportional to".

Independence

Two variables x and y are independent if knowing one tells us nothing about the other. Mathematically,

\[p(x,y) = p(x)p(y).\]

Equivalently,

\[p(y\mid x) = p(y).\]

Use the interactive plot on Slide 25. For the independent case, move the value of x. The conditional distribution p(y\mid x) does not move. For the dependent case, changing x changes the conditional distribution.

Independence becomes very useful when we have many data points. In this lecture, we will often assume that the noise affecting one data point is independent of the noise affecting another data point.

This allows us to write

\[p(t_1,\ldots,t_N\mid\theta) = \prod_{n=1}^{N} p(t_n\mid\theta).\]

Instead of dealing with one large joint distribution, we get a product of simple terms. This is a strong modelling assumption and is useful, but it can also be wrong.

Maximum Likelihood Estimation

Suppose the data \mathcal{D} has already been observed. The expression

\[p(\mathcal{D}\mid\theta)\]

can now be viewed as a function of \theta. This appeared before when we introduced Bayes rule above and we called this function the likelihood:

\[p(\mathcal{D}\mid\theta).\]

There is an important conceptual difference:

The numerical expression may be exactly the same but a likelihood is not a probability distribution over \theta. It does not have to integrate to one over the parameter \theta.

To make this clear, use the interactive example on Slide 26. The observations remain fixed while you move the Gaussian mean \mu. The likelihood changes according to how well that value of \mu explains the fixed observations.

The value of \mu giving the highest likelihood is the maximum likelihood estimate:

\[\theta_{\mathrm{ML}} = \operatorname*{arg\,max}_{\theta} p(\mathcal{D}\mid\theta).\]

Usually, we will maximize the log-likelihood

\[\theta_{\mathrm{ML}} = \operatorname*{arg\,max}_{\theta} \log p(\mathcal{D}\mid\theta).\]

Since the log is strictly increasing, it doesn't change the location of the maximum point.

A Gaussian Example

Let's make this more concrete with an example. Suppose we observe

\[\mathcal{D} = \{t_n\}_{n=1}^{N}\]

and assume that every observation was generated independently from the same Gaussian:

\[t_n \sim \mathcal{N}(\mu,\sigma^2).\]

For now, suppose \sigma^2 is known and only \mu is unknown.

The observations here are independent and identically distributed. Independent means one observation does not affect another. Identically distributed means they all come from the same probability distribution.

Writing the Likelihood

Because the observations are independent,

\[p(\mathcal{D}\mid\mu) = p(t_1,\ldots,t_N\mid\mu) = \prod_{n=1}^{N} p(t_n\mid\mu).\]

Using the Gaussian model,

\[p(\mathcal{D}\mid\mu) = \prod_{n=1}^{N} \mathcal{N}(t_n\mid\mu,\sigma^2).\]

Maximum likelihood chooses

\[\mu_{\mathrm{ML}} = \operatorname*{arg\,max}_{\mu} p(\mathcal{D}\mid\mu).\]

In words:

Choose the parameter under which the data we actually observed is most probable.

Products of many probabilities quickly become extremely small. Numerically, they can eventually become smaller than a computer can represent. More importantly, products are inconvenient to differentiate.

We therefore take logarithms:

\[\log p(\mathcal{D}\mid\mu).\]

The logarithm is strictly increasing, so the value of \mu that maximises the likelihood also maximises the log-likelihood. The useful identity is

\[\log\prod_n a_n = \sum_n\log a_n.\]

Hence,

\[\log p(\mathcal{D}\mid\mu) = \sum_n \log\mathcal{N}(t_n\mid\mu,\sigma^2).\]

Substituting the Gaussian density gives

\[\log p(\mathcal{D}\mid\mu) = \sum_n \log \left[ \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left( -\frac{(t_n-\mu)^2}{2\sigma^2} \right) \right].\]

Terms that do not depend on \mu do not affect which \mu maximises the expression. Removing those terms leaves

\[\log p(\mathcal{D}\mid\mu) \propto - \frac{1}{2\sigma^2} \sum_n (t_n-\mu)^2.\]

Maximising this is therefore equivalent to minimising

\[\frac{1}{2} \sum_n (t_n-\mu)^2.\]

We have just derived squared error from the assumption of Gaussian noise!

Solving for the Gaussian Mean

Define

\[\mathcal{L}(\mu) = \frac{1}{2} \sum_n (t_n-\mu)^2.\]

Differentiate:

\[\frac{\partial\mathcal{L}} {\partial\mu} = - \sum_n (t_n-\mu).\]

At the optimum,

\[- \sum_n (t_n-\mu) = 0.\]

Therefore,

\[N\mu = \sum_n t_n,\]

and

\[\boxed{ \mu_{\mathrm{ML}} = \frac{1}{N} \sum_{n=1}^{N} t_n }\]

So the familiar sample mean is not something we had to assume. It is the maximum-likelihood estimate of the mean under a Gaussian model.

Similarly, maximising the likelihood with respect to \sigma^2 gives

\[\sigma^2_{\mathrm{ML}} = \frac{1}{N} \sum_n (t_n-\mu_{\mathrm{ML}})^2.\]

Use the sampling demo on Slide 31 to see the generative direction: first choose \mu and \sigma, then repeatedly draw data from the distribution. Fitting does the reverse: we are given the draws and try to recover the parameters.

Why Maximum Likelihood?

Maximum likelihood is widely used partly because it has useful large-sample properties.

Under suitable conditions:

The word asymptotic is important. These statements describe what happens as N becomes large. Maximum likelihood can behave poorly with little data.

For example,

\[\sigma^2_{\mathrm{ML}} = \frac{1}{N} \sum_n (t_n-\mu_{\mathrm{ML}})^2\]

systematically underestimates the variance. The familiar unbiased sample-variance estimate instead divides by N-1. As we have seen before, a parameter estimate that fits the observed data best is not necessarily the best estimate of the underlying truth when data is limited.

Multivariate Gaussians

Before we get to linear regression, we need one more concept: the Multivariate Gaussian, which is the generalization of the standard Gaussian to real vectors. For \mathbf{x}\in\mathbb{R}^D,

\[\mathcal{N} ( \mathbf{x} \mid \boldsymbol{\mu}, \boldsymbol{\Sigma} ) = \frac{ 1 }{ (2\pi)^{D/2} |\boldsymbol{\Sigma}|^{1/2} } \exp \left[ -\frac{1}{2} (\mathbf{x}-\boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x}-\boldsymbol{\mu}) \right].\]

Here:

The covariance matrix controls both the spread and relationships between dimensions. Slide 36 shows three useful cases:

You do not need to memorise the multivariate Gaussian density at this point. We will give you the formula in exams, but after a while, you will end up memorizing it because we use it very often.

Linear Regression: A Probabilistic Approach

We now return to linear regression. Recall our regression model from Lecture 1:

\[y(x;\mathbf{w}) = \mathbf{w}^{\top}\boldsymbol{\phi}(x).\]

We now want a probabilistic model that describes how the observed target t might vary around this prediction. Indeed, a probabilistic model specifies a distribution for what data we expect to observe.

To obtain this model, we will specify how we believe the observations are generated by "nature". In our dataset, a sample has (x,t). We will assume that given an input x, the targets are generated by y(x;\mathbf{w}_*) plus some noise. We don't know what \mathbf{w}_* is but that's ok because we will learn it. For now, we have the model

\[t = y(x;\mathbf{w}) + \varepsilon = \mathbf{w}^{\top}\boldsymbol{\phi}(x) + \varepsilon.\]

where \varepsilon represents noise that we cannot predict. Assume Gaussian noise:

\[\varepsilon \sim \mathcal{N}(0,\sigma^2).\]

In words (and as shown graphically on Slide 40), to obtain the targets, we assume the following computations

  1. compute the deterministic part, \mathbf{w}^{\top}\boldsymbol{\phi}(x)
  2. draw some noise \varepsilon
  3. add the noise to obtain the observed target t.

The interactive plot on Slide 41 makes this idea concrete. Try drawing several targets at the same input x. The underlying function value stays the same, but each draw has a different noise value.

The Distribution of the Target

Given an input x_n and weights \mathbf{w}, the prediction

\[\mathbf{w}^{\top}\boldsymbol{\phi}(x_n)\]

is a fixed number.

Adding zero-mean Gaussian noise therefore gives

\[\boxed{ p(t_n\mid x_n,\mathbf{w}) = \mathcal{N} \left( t_n \mid \mathbf{w}^{\top}\boldsymbol{\phi}(x_n), \sigma^2 \right). }\]

Notice the change in interpretation. The model is no longer saying that the target is

\[\mathbf{w}^{\top}\boldsymbol{\phi}(x_n).\]

It is saying that targets are distributed around this value. The prediction gives the mean of the distribution, while \sigma^2 describes the variation around that mean.

Maximum Likelihood for Linear Regression

We can now use maximum likelihood to estimate \mathbf{w}, which means we maximize the log-likelihood:

\[\mathbf{w}_{\mathrm{ML}} = \operatorname*{arg\,max}_{\mathbf{w}} \log p(\mathcal{D}\mid\mathbf{w}).\]

So now, we have to derive p(\mathcal{D}\mid\mathbf{w}). We assume that the noise on each observation is independent so,

\[p(\mathcal{D}\mid\mathbf{w}) = \prod_{n=1}^{N} p(t_n\mid x_n,\mathbf{w}).\]

Using our Gaussian model,

\[p(\mathcal{D}\mid\mathbf{w}) = \prod_{n=1}^{N} \mathcal{N} \left( t_n \mid \mathbf{w}^{\top}\boldsymbol{\phi}(x_n), \sigma^2 \right).\]

The log-likelihood is

\[\log p(\mathcal{D}\mid\mathbf{w}) = \sum_{n=1}^{N} \log \mathcal{N} \left( t_n \mid \mathbf{w}^{\top}\boldsymbol{\phi}(x_n), \sigma^2 \right).\]

Writing out the Gaussian gives

\[\log p(\mathcal{D}\mid\mathbf{w}) = -\frac{N}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2} \sum_{n=1}^{N} \left( t_n - \mathbf{w}^{\top}\boldsymbol{\phi}(x_n) \right)^2.\]

If \sigma^2 is fixed, the first term is constant and the factor 1/\sigma^2 does not affect which \mathbf{w} is best.

Therefore,

\[\operatorname*{arg\,max}_{\mathbf{w}} p(\mathcal{D}\mid\mathbf{w})\]

is equivalent to

\[\boxed{ \operatorname*{arg\,min}_{\mathbf{w}} \frac{1}{2} \sum_{n=1}^{N} \left( t_n - \mathbf{w}^{\top}\boldsymbol{\phi}(x_n) \right)^2. }\]

This is exactly the least-squares problem from Lecture 1!

We now have an explanation for the squared-error loss:

Minimising squared error is equivalent to maximum likelihood when we assume independent Gaussian noise with constant variance.

This is an important result. The loss function is no longer something that appeared from nowhere. It follows from our assumptions about how the data was generated.

The maximum-likelihood solution for \mathbf{w} is therefore the same as before:

\[\mathbf{w}_{\mathrm{ML}} = (\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi})^{-1} \boldsymbol{\Phi}^{\top}\mathbf{t}.\]

We can also estimate the noise variance:

\[\sigma_{\mathrm{ML}}^2 = \frac{1}{N} \sum_{n=1}^{N} \left( t_n - \mathbf{w}_{\mathrm{ML}}^{\top}\boldsymbol{\phi}(x_n) \right)^2.\]

So maximum likelihood gives us both the fitted function and an estimate of how much the observations vary around it.

The Noise Model Determines the Loss

Gaussian noise is an assumption of our model. If the noise really is close to Gaussian, squared error is a sensible choice. If the noise has a different distribution, a different loss may be more appropriate.

For example, suppose the noise follows a Laplace distribution:

\[p(\varepsilon) \propto \exp \left( -\frac{|\varepsilon|}{b} \right).\]

Taking the negative log gives a loss proportional to

\[|\varepsilon|.\]

For regression, maximum likelihood therefore becomes

\[\min_{\mathbf{w}} \sum_n \left| t_n - \mathbf{w}^{\top}\boldsymbol{\phi}(x_n) \right|.\]

This is the absolute-error loss.

The interactive comparison on Slide 45 shows the difference. The Gaussian assigns very low probability to large residuals, so its negative log-likelihood grows quadratically. The Laplace distribution has heavier tails, so the penalty grows only linearly.

This also explains why squared error is sensitive to outliers: under a Gaussian noise model, a very large residual is considered extremely unlikely.

The main point is:

Choosing a probability model for the noise implicitly chooses a loss function.

From Maximum Likelihood to a Bayesian Model

Maximum likelihood gives us a single estimate of the weights, \mathbf{w}_{\mathrm{ML}}. But recall the overfitting problem from Lecture 1. When there is little data, many different weight vectors may explain the observations reasonably well.

Maximum likelihood simply chooses the one that fits the observed data best. As such, this technique can lead to overfitting. In lecture 1, we prevented overfitting using ridge regression where we specifed a regularizer:

\[\frac{1}{2} \| \mathbf{t} - \boldsymbol{\Phi}\mathbf{w} \|_2^2 + \frac{\lambda}{2} \|\mathbf{w}\|_2^2.\]

We simply chose the second term because preferring smaller weights seemed reasonable.

We can now derive this term probabilistically too.

A Prior Over the Weights

In a Bayesian model, we place a probability distribution over the unknown parameters.

Suppose

\[p(\mathbf{w}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{0}, \alpha^{-1}\mathbf{I} \right).\]

This is a zero-mean Gaussian prior over the weights.

Here \alpha is a precision, which means inverse variance:

\[\alpha = \frac{1}{\sigma_w^2}.\]

The interpretation is simple:

This is exactly the kind of preference that ridge regression imposed directly.

Maximum A Posteriori Estimation

Bayes' rule gives us a posterior distribution over the weights:

\[p(\mathbf{w}\mid\mathcal{D}) = \frac{ p(\mathcal{D}\mid\mathbf{w}) p(\mathbf{w}) }{ p(\mathcal{D}) }.\]

One way to obtain a single estimate from this posterior is to choose its most probable value.

This is called maximum a posteriori estimation, or MAP:

\[\mathbf{w}_{\mathrm{MAP}} = \operatorname*{arg\,max}_{\mathbf{w}} p(\mathbf{w}\mid\mathcal{D}).\]

Taking logarithms (just like in maximum likelihood) and substituting in the posterior distribution over the weights,

\[\mathbf{w}_{\mathrm{MAP}} = \operatorname*{arg\,max}_{\mathbf{w}} \left[ \log \left(\frac{p(\mathcal{D}\mid\mathbf{w})p(\mathbf{w})}{p(\mathcal{D})}\right) \right]\\ = \operatorname*{arg\,max}_{\mathbf{w}} \left[ \log p(\mathcal{D}\mid\mathbf{w}) + \log p(\mathbf{w}) \right]\]

Note that p(\mathcal{D}) can be dropped because it does not depend on \mathbf{w}.

Now define the noise precision

\[\beta = \frac{1}{\sigma_\varepsilon^2}.\]

From the Gaussian likelihood,

\[\log p(\mathcal{D}\mid\mathbf{w}) = - \frac{\beta}{2} \| \mathbf{t} - \boldsymbol{\Phi}\mathbf{w} \|_2^2 + \text{constant}.\]

From the Gaussian prior,

\[\log p(\mathbf{w}) = - \frac{\alpha}{2} \|\mathbf{w}\|_2^2 + \text{constant}.\]

Therefore, maximising the posterior is equivalent to minimising

\[\frac{\beta}{2} \| \mathbf{t} - \boldsymbol{\Phi}\mathbf{w} \|_2^2 + \frac{\alpha}{2} \|\mathbf{w}\|_2^2.\]

Divide through by \beta:

\[\boxed{ \mathcal{L}(\mathbf{w}) = \frac{1}{2} \| \mathbf{t} - \boldsymbol{\Phi}\mathbf{w} \|_2^2 + \frac{\lambda}{2} \|\mathbf{w}\|_2^2, }\]

where

\[\boxed{ \lambda = \frac{\alpha}{\beta}. }\]

This is exactly ridge regression.

So the two terms from Lecture 1 now both have a probabilistic explanation:

In other words, the key general principle is:

The likelihood determines how we fit the observations. The prior determines how we regularise the parameters.

The regularization hyperparameter \lambda.

Because

\[\alpha = \frac{1}{\sigma_w^2}\]

and

\[\beta = \frac{1}{\sigma_\varepsilon^2},\]

we have

\[\lambda = \frac{\alpha}{\beta} = \frac{\sigma_\varepsilon^2}{\sigma_w^2}.\]

So,

\[\boxed{ \lambda = \frac{ \text{noise variance} }{ \text{prior weight variance} }. }\]

This gives \lambda a useful interpretation.

Suppose the observations are very noisy. Then \sigma_\varepsilon^2 is large. We should trust the data less, so the prior has more influence and the weights are shrunk more strongly.

Suppose instead that the data is very clean and our prior allows a wide range of weights. Then \lambda is small, and MAP approaches ordinary least squares.

The interactive plot on Slide 57 lets you adjust both variances and see the implied value of \lambda.

Point Estimates Versus Distributions Over Parameters

MLE gives us one point:

\[\mathbf{w}_{\mathrm{ML}}.\]

MAP gives us another point:

\[\mathbf{w}_{\mathrm{MAP}}.\]

MAP uses the prior and is often better behaved when data is limited, but it still throws away an important piece of information:

How uncertain are we about the weights?

To answer that question, we keep the whole posterior distribution

\[p(\mathbf{w}\mid\mathcal{D})\]

instead of reducing it to its maximum. This is the full Bayesian approach.

Bayesian Linear Regression

For our model, both the likelihood and prior are Gaussian:

\[p(\mathbf{t}\mid\mathbf{X},\mathbf{w}) = \mathcal{N} \left( \mathbf{t} \mid \boldsymbol{\Phi}\mathbf{w}, \beta^{-1}\mathbf{I} \right),\]

and

\[p(\mathbf{w}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{0}, \alpha^{-1}\mathbf{I} \right).\]

When a Gaussian likelihood is combined with a Gaussian prior, the posterior over the weights is also Gaussian:

\[\boxed{ p(\mathbf{w}\mid\mathcal{D}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{m}_N, \mathbf{S}_N \right). }\]

The posterior covariance is

\[\boxed{ \mathbf{S}_N^{-1} = \alpha\mathbf{I} + \beta \boldsymbol{\Phi}^{\top}\boldsymbol{\Phi}, }\]

and the posterior mean is

\[\boxed{ \mathbf{m}_N = \beta \mathbf{S}_N \boldsymbol{\Phi}^{\top}\mathbf{t}. }\]

You do not need to derive these expressions from scratch at this point. The important part is what they mean:

For this Gaussian posterior, the mean and the MAP estimate are the same:

\[\mathbf{w}_{\mathrm{MAP}} = \mathbf{m}_N.\]

MAP keeps \mathbf{m}_N and discards \mathbf{S}_N. The full posterior keeps both.

Making Predictions With the Posterior

Suppose we want to predict the target at a new input x. If we knew the weights exactly, we would use

\[p(t\mid x,\mathbf{w}).\]

But we do not know \mathbf{w}. Instead, we have a posterior distribution over it.

So we average over all possible weight vectors:

\[p(t\mid x,\mathcal{D}) = \int p(t,\mathbf{w}\mid x,\mathcal{D}) \,d\mathbf{w}.\]

Using the product rule,

\[p(t\mid x,\mathcal{D}) = \int p(t\mid x,\mathbf{w}) p(\mathbf{w}\mid\mathcal{D}) \,d\mathbf{w}.\]

This is called the posterior predictive distribution.

Conceptually,

every plausible \mathbf{w} makes a prediction, and we average those predictions according to how plausible that \mathbf{w} is.

The interactive plot on Slide 62 shows this idea directly. Each line corresponds to a different weight vector drawn from the posterior. Near data points, these functions tend to agree. Far from the data, they can disagree substantially.

Two Sources of Predictive Uncertainty

For Bayesian linear regression, the posterior predictive distribution is Gaussian:

\[p(t\mid x,\mathcal{D}) = \mathcal{N} \left( t \mid \mathbf{m}_N^\top\boldsymbol{\phi}(x), \sigma^2(x) \right),\]

where

\[\boxed{ \sigma^2(x) = \underbrace{ \beta^{-1} }_{\text{aleatoric}} + \underbrace{ \boldsymbol{\phi}(x)^\top \mathbf{S}_N \boldsymbol{\phi}(x) }_{\text{epistemic}}. }\]

The two terms have different meanings.

Aleatoric Uncertainty

The first term,

\[\beta^{-1} = \sigma_\varepsilon^2,\]

is the noise in the observations themselves. Even if we knew the correct weights perfectly, two otherwise identical examples could still have different targets.

This is called aleatoric uncertainty. More data does not make it disappear because it represents randomness in the world rather than ignorance in the model.

Epistemic Uncertainty

The second term,

\[\boldsymbol{\phi}(x)^\top \mathbf{S}_N \boldsymbol{\phi}(x),\]

comes from uncertainty about the weights.

This is called epistemic uncertainty. If we have a lot of useful data near some input, many possible weight vectors are ruled out and this term becomes small. If we have little relevant data, many different weight vectors remain plausible and this term can be large. More data can therefore reduce epistemic uncertainty.

MLE, MAP, and the Full Posterior

It is useful to compare the three approaches directly.

Maximum Likelihood

MLE chooses

\[\mathbf{w}_{\mathrm{ML}} = \operatorname*{arg\,max}_{\mathbf{w}} p(\mathcal{D}\mid\mathbf{w}).\]

It uses the data only and returns one set of weights.

Maximum A Posteriori

MAP chooses

\[\mathbf{w}_{\mathrm{MAP}} = \operatorname*{arg\,max}_{\mathbf{w}} p(\mathbf{w}\mid\mathcal{D}).\]

It combines the likelihood and prior, but still returns only one set of weights.

Full Bayesian Inference

Full Bayesian inference retains

\[p(\mathbf{w}\mid\mathcal{D}).\]

It therefore represents uncertainty about the weights and can include this uncertainty when making predictions.

Use Slide 64 to compare all three. With little data, MLE can fit quite aggressively. MAP is constrained by the prior. The full posterior additionally shows a wide uncertainty band where the data does not determine the function well.

As more data is added, the three approaches become increasingly similar and the posterior uncertainty becomes smaller.

Summary

In this lecture, we used probability to explain several choices that appeared somewhat arbitrary in Lecture 1.

We first reviewed the basic probability machinery:

We then introduced maximum likelihood estimation:

\[\theta_{\mathrm{ML}} = \operatorname*{arg\,max}_{\theta} p(\mathcal{D}\mid\theta).\]

For linear regression, assuming

\[t_n = \mathbf{w}^{\top}\boldsymbol{\phi}(x_n) + \varepsilon_n\]

with independent Gaussian noise

\[\varepsilon_n \sim \mathcal{N}(0,\sigma^2)\]

leads directly to squared-error loss:

\[\frac{1}{2} \sum_n \left( t_n - \mathbf{w}^{\top}\boldsymbol{\phi}(x_n) \right)^2.\]

So Gaussian noise gives least squares.

We then placed a Gaussian prior on the weights:

\[p(\mathbf{w}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{0}, \alpha^{-1}\mathbf{I} \right).\]

MAP estimation then gives

\[\frac{1}{2} \| \mathbf{t} - \boldsymbol{\Phi}\mathbf{w} \|_2^2 + \frac{\lambda}{2} \|\mathbf{w}\|_2^2,\]

where

\[\lambda = \frac{\alpha}{\beta} = \frac{\sigma_\varepsilon^2}{\sigma_w^2}.\]

So a Gaussian prior gives ridge regression.

Finally, instead of reducing the posterior to a single MAP estimate, we kept the complete distribution

\[p(\mathbf{w}\mid\mathcal{D}).\]

This gave us a posterior predictive distribution whose uncertainty has two components:

\[\sigma^2(x) = \underbrace{\beta^{-1}}_{\text{aleatoric}} + \underbrace{ \boldsymbol{\phi}(x)^\top \mathbf{S}_N \boldsymbol{\phi}(x) }_{\text{epistemic}}.\]

Aleatoric uncertainty is noise in the observations and does not disappear with more data. Epistemic uncertainty reflects what the model does not yet know and can decrease as more data is collected.

The main lesson is:

A probabilistic model connects assumptions about how data is generated to the loss, the regulariser, and the uncertainty in our predictions.