Lecture 3: Linear Classification

Summary of Key Ideas

In the previous lectures, we studied regression, where the target is a number. In this lecture, the target is a discrete label.

The main idea from Lecture 2 still applies: choose a probability distribution that matches the kind of target we want to model, and derive the loss from that distribution. For binary classification, the target takes one of two values, so we use a Bernoulli distribution. We will connect this distribution to our linear model using sigmoid function. Then we will see that maximum likelihood gives us the familiar cross-entropy loss.

Unlike linear regression, logistic regression does not have a closed-form solution for its weights. We will therefore use gradient-based optimisation. We will also see that a probability is not yet a decision: the threshold used to convert a probability into an action should depend on the costs of different mistakes. Finally, we will extend the same probabilistic recipe to multiple classes using the categorical distribution and softmax.

Learning Objectives

By the end of this lesson, the main things you should be able to do are:

In addition, you should be able to:

We will first see why simply applying least squares to labels is unsatisfactory. We will then build a probabilistic model for binary labels and derive logistic regression from it. After discussing decisions and optimisation, we will revisit the Bayesian framework. Finally, we will extend the model to more than two classes.

From Regression to Classification

Suppose we have a dataset

\[\mathcal{D} = \{(\mathbf{x}_n,t_n)\}_{n=1}^{N}.\]

For regression, the target t_n could be any real number. In binary classification, the target instead takes one of two values:

\[t_n \in \{0,1\}.\]

The values 0 and 1 are discrete labels.

We would like to learn a model that, given a new input \mathbf{x}, tells us how likely each label is. For binary classification, we model

\[p(t=1\mid\mathbf{x}).\]

Once we know this probability, the other probability is simply

\[p(t=0\mid\mathbf{x})=1-p(t=1\mid\mathbf{x}).\]

Why Not Use Least Squares?

One possibility is to code the two classes numerically and apply the regression machinery from Lecture 1. For example, we could use targets t\in\{-1,+1\} and fit

\[y(\mathbf{x})=\mathbf{w}^{\top}\mathbf{x}\]

using squared error. We could then classify a point according to whether y(\mathbf{x}) is positive or negative.

This can sometimes work. The first example on Slides 12-13 shows a case where the least-squares boundary separates the two classes correctly. However, the problem becomes clear on Slides 14-16. Adding several new examples that are already far on the correct side of the boundary can make the fitted boundary worse.

The reason is that squared error does not understand the idea of being "correct enough". Suppose an example has target t=-1, but the linear model predicts y=-4.5. The point is classified correctly, but squared error is large,

\[(-4.5-(-1))^2 = 12.25.\]

The optimisation therefore tries to pull this prediction back towards -1, even if doing so rotates the boundary and harms examples near the decision boundary. For classification, once a point is confidently on the correct side, we generally do not want to penalise it simply for being far from the numerical label we happened to assign.

There is also a second problem. The linear score

\[\mathbf{w}^{\top}\mathbf{x}\]

can take any real value. A score of 1.4 or -0.3 is not directly interpretable as the probability of a class. If we want a probabilistic classifier, we need an output between 0 and 1.

So we need a model designed for labels, rather than treating labels as ordinary real numbers.

Linear Decision Boundaries

Before making the model probabilistic, it is useful to understand the geometry of a linear classifier.

Consider two input features and a linear score

\[y(\mathbf{x}) = w_0+w_1x_1+w_2x_2.\]

The decision boundary is where the score is zero:

\[w_0+w_1x_1+w_2x_2=0.\]

In two dimensions, this is a straight line. In D dimensions, it is a (D-1)-dimensional hyperplane.

The weight vector

\[\mathbf{w} = \begin{bmatrix} w_1\\ w_2 \end{bmatrix}\]

is perpendicular to the boundary (see Slide 21). Its direction determines the orientation of the boundary, while the bias w_0 shifts the boundary. If we multiply all the weights and the bias by the same positive constant, the decision boundary does not move. The scores become larger in magnitude, but the decision boundary remains in the same place.

The interactive plots on Slides 21-22 are useful for seeing these effects. Change the direction of \mathbf{w} and the boundary rotates. Change the bias and the boundary is translated.

This geometry will remain unchanged when we make the classifier probabilistic. We will transform the linear score into a probability, but the boundary will still correspond to a linear equation in the features.

Building a Probabilistic Model for Binary Labels

In Lecture 2, we modelled a regression target by specifying how the target was generated. We will use the same strategy here.

For binary classification, a useful generative story is:

  1. use the input \mathbf{x} to compute a real-valued score
  2. convert that score into a probability \mu(\mathbf{x})
  3. draw the observed label from a distribution with that probability

The random part is no longer additive Gaussian noise. Instead, the label itself is a random draw from a two-outcome distribution.

The Bernoulli Distribution

The appropriate distribution for a single binary outcome is the Bernoulli distribution. If t\in\{0,1\}, then

\[\mathrm{Bern}(t\mid\mu) = \mu^t(1-\mu)^{1-t},\]

where

\[0\leq \mu \leq 1.\]

The parameter \mu is the probability that t=1:

\[p(t=1)=\mu, \qquad p(t=0)=1-\mu.\]

The Bernoulli formula works because the exponents act like switches. If t=1, then

\[\mu^1(1-\mu)^0=\mu.\]

If t=0, then

\[\mu^0(1-\mu)^1=1-\mu.\]

Our task now is to construct a value of \mu from the input \mathbf{x}.

From a Linear Score to a Probability

The linear model gives us

\[a = \mathbf{w}^{\top}\mathbf{x},\]

which can be any real number. But the Bernoulli distribution needs a parameter between 0 and 1. We transform the score using the sigmoid function, also called the logistic function:

\[\sigma(a) = \frac{1}{1+e^{-a}}.\]

The sigmoid has several useful properties:

We define

\[\mu(\mathbf{x}) = \sigma(\mathbf{w}^{\top}\mathbf{x}).\]

Our probabilistic classifier is therefore

\[\boxed{ p(t\mid\mathbf{x},\mathbf{w}) = \mathrm{Bern}\left(t\mid\sigma(\mathbf{w}^{\top}\mathbf{x})\right). }\]

This model is called logistic regression. Despite the name, it is a classification model.

The full generative story is now

\[\mathbf{x} \longrightarrow \mathbf{w}^{\top}\mathbf{x} \longrightarrow \sigma(\mathbf{w}^{\top}\mathbf{x}) \longrightarrow \text{Bernoulli draw }t.\]

The interactive plots on Slides 26 and 29 show these two steps separately: the sigmoid maps the score to a probability, and the Bernoulli draw produces the observed label.

Why the Sigmoid? Log-Odds

The sigmoid also has a useful interpretation. Let

\[y = \sigma(a).\]

Solving for a gives

\[a = \log\frac{y}{1-y}.\]

The ratio

\[\frac{y}{1-y}\]

is called the odds, and its logarithm is the log-odds or logit. Therefore logistic regression assumes

\[\boxed{ \log\frac{p(t=1\mid\mathbf{x})} {p(t=0\mid\mathbf{x})} = \mathbf{w}^{\top}\mathbf{x}. }\]

So logistic regression is a model in which the log-odds are linear in the features.

The boundary p(t=1\mid\mathbf{x})=0.5 occurs when the odds are 1 (the log-odds are zero). To learn more about odds and log odds, view this StatQuest Youtube video

Learning Logistic Regression

We now have a probabilistic model. The next question is how to learn the weights \mathbf{w} from data.

As in Lecture 2, we can use maximum likelihood,

\[\mathbf{w}_{\mathrm{ML}} = \operatorname*{arg\,max}_{\mathbf{w}} \log p(\mathcal{D}\mid\mathbf{w}).\]

But our likelihood function p(\mathcal{D}\mid\mathbf{w}) has changed since our probabilistic model is different. Here, for each training example,

\[y_n = \sigma(\mathbf{w}^{\top}\mathbf{x}_n)\]

is the model's predicted probability that t_n=1. The probability of observing t_n is

\[p(t_n\mid\mathbf{x}_n,\mathbf{w}) = y_n^{t_n}(1-y_n)^{1-t_n}.\]

Assuming the observations are independent, the likelihood of the dataset is

\[p(\mathcal{D}\mid\mathbf{w}) = \prod_{n=1}^{N} y_n^{t_n}(1-y_n)^{1-t_n}.\]

As an exercise, compare this likelihood against the one that we derived for linear regression.

Cross-Entropy From Maximum Likelihood

We maximise the log-likelihood:

\[\log p(\mathcal{D}\mid\mathbf{w}) = \sum_{n=1}^{N} \left[ t_n\log y_n + (1-t_n)\log(1-y_n) \right].\]

Equivalently, we minimise the negative log-likelihood:

\[\boxed{ \mathcal{L}(\mathbf{w}) = -\sum_{n=1}^{N} \left[ t_n\log y_n + (1-t_n)\log(1-y_n) \right]. }\]

This is the binary cross-entropy loss.

The important point is that, just like squared error in Lecture 2, cross-entropy was not chosen arbitrarily. It follows from the probability model:

\[\boxed{ \text{Bernoulli target model} \quad\Longrightarrow\quad \text{binary cross-entropy}. }\]

For a single example, if t=1, the loss becomes

\[-\log y.\]

If t=0, it becomes

\[-\log(1-y).\]

A correct and confident prediction has very small loss. A confidently wrong prediction has a very large loss. Unlike squared error applied to numerical class labels, cross-entropy does not penalise a correct prediction simply because it is very confident. The interactive plot on Slide 37 shows the shape of this loss for each class.

Adding a Prior: MAP and Regularisation

As before, we can place a Gaussian prior on the weights:

\[p(\mathbf{w}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{0}, \alpha^{-1}\mathbf{I} \right).\]

Maximum a posteriori estimation maximises

\[p(\mathbf{w}\mid\mathcal{D}) \propto p(\mathcal{D}\mid\mathbf{w})p(\mathbf{w}).\]

Taking the negative log gives

\[\boxed{ \mathcal{L}_{\mathrm{MAP}}(\mathbf{w}) = -\sum_{n=1}^{N} \left[ t_n\log y_n + (1-t_n)\log(1-y_n) \right] + \frac{\alpha}{2}\|\mathbf{w}\|_2^2. }\]

The likelihood has changed from Gaussian to Bernoulli, so the data-fitting term has changed from squared error to cross-entropy. The prior has not changed, so the L_2 regulariser is the same type of penalty we saw for ridge regression.

We again see the same principle we mentioned in Lecture 2:

The likelihood determines how we fit the observations. The prior determines how we regularise the parameters.

Fitting Logistic Regression

For linear regression, setting the gradient to zero led to the normal equations and a closed-form solution. Logistic regression is different.

Recall that

\[y_n = \sigma(\mathbf{w}^{\top}\mathbf{x}_n)\]

and

\[\mathcal{L} = -\sum_n \left[ t_n\log y_n + (1-t_n)\log(1-y_n) \right].\]

The derivative of the sigmoid is

\[\sigma'(a) = \sigma(a)(1-\sigma(a)).\]

Using the chain rule, the gradient of the cross-entropy loss simplifies to

\[\boxed{ \nabla_{\mathbf{w}}\mathcal{L} = \sum_{n=1}^{N} (y_n-t_n)\mathbf{x}_n. }\]

This is a useful result. It has exactly the same form as the gradient for least-squares linear regression:

\[\text{gradient} = \sum_n (\text{prediction}-\text{target}) \times \text{input}.\]

The difference is in how the prediction is produced. For linear regression,

\[y_n = \mathbf{w}^{\top}\mathbf{x}_n,\]

while for logistic regression,

\[y_n = \sigma(\mathbf{w}^{\top}\mathbf{x}_n).\]

This simple gradient is not an accident. The Bernoulli likelihood and sigmoid are a matched pair, just as the Gaussian model and identity output are naturally matched for regression.

Why There Is No Closed Form

Setting the gradient to zero gives

\[\sum_{n=1}^{N} \left( \sigma(\mathbf{w}^{\top}\mathbf{x}_n)-t_n \right) \mathbf{x}_n = 0.\]

The weights now appear inside the exponential in the sigmoid and inside every term of the sum. Unlike the normal equations, we cannot rearrange this expression to isolate \mathbf{w}.

Therefore logistic regression does not have a closed-form solution in general. We instead use iterative optimisation such as gradient descent:

\[\boxed{ \mathbf{w} \leftarrow \mathbf{w} - \eta \nabla_{\mathbf{w}}\mathcal{L}, }\]

where \eta is the learning rate. Slides 51-52 show how individual examples contribute to the update and how repeated updates move the decision boundary.

With a Gaussian prior, the MAP gradient becomes

\[\boxed{ \nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{MAP}} = \sum_n (y_n-t_n)\mathbf{x}_n + \alpha\mathbf{w}. }\]

The additional \alpha\mathbf{w} term pulls the weights towards zero.

A Historical Aside: The Perceptron

The perceptron is an earlier linear classification algorithm. You may have encountered this method before. It uses a hard threshold instead of a sigmoid and updates the weights when an example is misclassified. In one common notation,

\[\mathbf{w} \leftarrow \mathbf{w} + \eta(t_n-y_n)\mathbf{x}_n.\]

The update again has the form error times input. For linearly separable data, the perceptron is guaranteed to eventually find a separating hyperplane. However, a single linear boundary cannot represent problems such as XOR. We will later overcome this limitation by composing multiple nonlinear units in neural networks.

Perfectly Separable Data and Maximum Likelihood

Logistic regression has an interesting issue when the training data is perfectly linearly separable ("linearly separable" means that we can draw a linear boundary that splits the data perfectly into the two classes.)

Suppose some weight vector \mathbf{w} correctly separates all the examples. Multiplying it by a positive constant c does not change the decision boundary:

\[\mathbf{w}^{\top}\mathbf{x}=0\]

and

\[(c\mathbf{w})^{\top}\mathbf{x}=0\]

have exactly the same boundary (the same zero set).

However, increasing c makes the predicted probabilities more extreme. Points on the positive side move closer to probability 1 and points on the negative side move closer to probability 0. The cross-entropy therefore keeps decreasing as

\[\|\mathbf{w}\|\rightarrow\infty.\]

For perfectly separable data, plain maximum likelihood therefore has no finite maximum. The likelihood approaches its maximum as the weights keep growing!

The Gaussian prior fixes this problem. The MAP objective contains

\[\frac{\alpha}{2}\|\mathbf{w}\|^2,\]

which becomes increasingly expensive as the weights grow. The likelihood pushes the weights outward while the prior pulls them back towards zero. The two effects balance at a finite solution. Slides 53-54 visualize this behaviour.

This gives another reason to regularise logistic regression, beyond the usual concern about overfitting.

Nonlinear Decision Boundaries

Logistic regression is linear in its input features, but the same basis-function trick from Lecture 1 still applies.

Instead of using the original input \mathbf{x}, define transformed features

\[\boldsymbol{\phi}(\mathbf{x}).\]

Then use

\[p(t=1\mid\mathbf{x}) = \sigma \left( \mathbf{w}^{\top} \boldsymbol{\phi}(\mathbf{x}) \right).\]

The decision boundary is

\[\mathbf{w}^{\top} \boldsymbol{\phi}(\mathbf{x}) = 0.\]

This is a hyperplane in feature space, but it can correspond to a curved boundary in the original input space.

For example, suppose

\[\boldsymbol{\phi}(\mathbf{x}) = \begin{bmatrix} x_1\\ x_2\\ x_1^2+x_2^2 \end{bmatrix}.\]

A linear boundary in this three-dimensional feature space can produce a circular boundary in the original two-dimensional space. The interactive demonstration on Slide 56 shows this lifting idea geometrically.

So once again, "linear" means linear in the parameters and transformed features, not necessarily a straight boundary in the original variables.

Bayesian Logistic Regression

Note: This part is challenging and may require several re-reads.

Recall that MLE and MAP both return a single weight vector. As in Lecture 2, we may instead want a distribution over plausible weights so that we can represent uncertainty about the decision boundary.

Using the same Gaussian prior,

\[p(\mathbf{w}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{0}, \alpha^{-1}\mathbf{I} \right),\]

Bayes' rule gives

\[p(\mathbf{w}\mid\mathcal{D}) \propto p(\mathcal{D}\mid\mathbf{w})p(\mathbf{w}).\]

For linear regression in Lecture 2, the likelihood and prior were both Gaussian. Their product therefore produced another Gaussian posterior, and we could calculate it exactly.

For logistic regression, the likelihood contains sigmoids:

\[p(\mathcal{D}\mid\mathbf{w}) = \prod_n y_n^{t_n}(1-y_n)^{1-t_n},\]

where

\[y_n = \sigma(\mathbf{w}^{\top}\mathbf{x}_n).\]

Multiplying this likelihood by a Gaussian prior does not give a standard distribution. The posterior has no convenient closed form. This is our first example in the course where Bayesian inference requires an approximation. In practice, Bayesian inference typically requires approximation and we will study more a complex approximation later called variational inference.

The Laplace Approximation

One common approximation is the Laplace approximation. The idea is simple: near its maximum, a smooth log-posterior often looks approximately quadratic (think of an upside down bowl). A quadratic log-density corresponds to a Gaussian distribution (see Slide 63).

We therefore approximate

\[p(\mathbf{w}\mid\mathcal{D})\]

with a Normal distribution

\[q(\mathbf{w}) = \mathcal{N} \left( \mathbf{w} \mid \mathbf{w}_{\mathrm{MAP}}, \mathbf{S} \right).\]

The Gaussian is centred at the MAP estimate. Its covariance is determined by the local curvature of the negative log-posterior. For logistic regression,

\[\boxed{ \mathbf{S}^{-1} = \alpha\mathbf{I} + \sum_{n=1}^{N} y_n(1-y_n) \mathbf{x}_n\mathbf{x}_n^{\top}, }\]

where the probabilities y_n are evaluated at \mathbf{w}_{\mathrm{MAP}}.

You do not need to memorise this expression. The important idea is that \mathbf{S} describes how much the posterior is concentrated around the MAP solution. A sharply curved posterior gives small uncertainty in the weights; a broad posterior gives larger uncertainty.

The factor

\[y_n(1-y_n)\]

is largest near y_n=0.5. Thus, examples near the decision boundary contribute strongly to the local curvature because they are the examples for which the model remains most sensitive to changes in the boundary.

Bayesian Prediction

For a new input \mathbf{x}, the Bayesian predictive probability is

\[p(t=1\mid\mathbf{x},\mathcal{D}) = \int \sigma(\mathbf{w}^{\top}\mathbf{x}) p(\mathbf{w}\mid\mathcal{D}) \,d\mathbf{w}.\]

With the Laplace approximation, we replace the true posterior by q(\mathbf{w}). We can then approximate the integral by drawing weight vectors

\[\mathbf{w}^{(s)} \sim q(\mathbf{w})\]

and averaging their predictions:

\[\boxed{ p(t=1\mid\mathbf{x},\mathcal{D}) \approx \frac{1}{S} \sum_{s=1}^{S} \sigma \left( (\mathbf{w}^{(s)})^{\top}\mathbf{x} \right). }\]

Each sampled weight vector corresponds to a different decision boundary. Near regions with plenty of data, these boundaries tend to agree. Far away from the observed data, they may disagree substantially. Slides 64-65 show this effect.

This is the same idea as Bayesian linear regression: instead of pretending that one fitted parameter vector is certainly correct, we average predictions over many plausible parameter values.

From Probabilities to Decisions

Logistic regression gives us a probability

\[y = p(t=1\mid\mathbf{x}).\]

A probability is not yet a decision. To take an action, we must decide how large y needs to be before we predict class 1.

A common rule is

\[\hat{t} = \begin{cases} 1, & y>0.5,\\ 0, & y\leq 0.5. \end{cases}\]

The threshold 0.5 is appropriate when the two types of mistakes have equal consequences. In many applications, they do not.

Choosing a Threshold From Costs

Suppose class 1 means that a car will sell quickly. If we buy the car but it does not sell, suppose the cost is C_{\mathrm{FP}}. If instead we pass on the car but it would have sold, the missed opportunity costs C_{\mathrm{FN}}.

If the model says

\[y=p(t=1\mid\mathbf{x}),\]

then the expected cost of predicting class 1 is

\[(1-y)C_{\mathrm{FP}},\]

while the expected cost of predicting class 0 is

\[yC_{\mathrm{FN}}.\]

We should choose class 1 when

\[(1-y)C_{\mathrm{FP}} < yC_{\mathrm{FN}}.\]

Solving gives

\[\boxed{ y > \frac{C_{\mathrm{FP}}} {C_{\mathrm{FP}}+C_{\mathrm{FN}}}. }\]

For the example in the slides, where an unsold purchase costs about $800 and missing a quick seller costs about $500,

\[y > \frac{800}{800+500} \approx 0.615.\]

So the correct threshold is above 0.5. The model has not changed at all; only the decision rule has changed.

This distinction is important:

The model estimates probabilities. The decision rule determines what we do with them.

ROC Curves

Changing the threshold changes the kinds of mistakes the classifier makes. A lower threshold predicts class 1 more often, increasing both true positives and false positives. A higher threshold predicts class 1 less often, decreasing both.

The true positive rate is

\[\mathrm{TPR} = \frac{\mathrm{TP}} {\mathrm{TP}+\mathrm{FN}},\]

and the false positive rate is

\[\mathrm{FPR} = \frac{\mathrm{FP}} {\mathrm{FP}+\mathrm{TN}}.\]

An ROC curve plots TPR against FPR as the classification threshold is swept from one extreme to the other. Choosing a threshold corresponds to choosing a point on this curve. The interactive plot on Slide 43 shows this directly. The area under the ROC curve, or AUC, summarizes the ranking quality of the classifier across all thresholds.

Extension: More Than Two Classes

So far, we have assumed that the target has two possible values. Suppose instead that each example belongs to one of K classes.

For example, a used car might be classified as

We represent the target using a one-hot vector. If the second class is correct,

\[\mathbf{t} = \begin{bmatrix} 0\\ 1\\ 0 \end{bmatrix}.\]

Exactly one element is 1 and the rest are 0.

The Categorical Distribution

The multi-class generalisation of the Bernoulli distribution is the categorical distribution. Let

\[\boldsymbol{\mu} = \begin{bmatrix} \mu_1\\ \vdots\\ \mu_K \end{bmatrix},\]

where

\[\mu_k\geq 0, \qquad \sum_{k=1}^{K}\mu_k=1.\]

Then

\[\boxed{ p(\mathbf{t}\mid\boldsymbol{\mu}) = \prod_{k=1}^{K} \mu_k^{t_k}. }\]

Because the target is one-hot, every exponent except one is zero. The probability therefore reduces to the probability assigned to the true class.

For K=2, this reduces to the Bernoulli distribution.

Softmax: From Scores to Probabilities

For K classes, we use one linear score per class:

\[a_k = \mathbf{w}_k^{\top}\mathbf{x}.\]

These scores can take any real values. We need to convert them into K non-negative probabilities that add to one. The softmax function does this:

\[\boxed{ y_k = \frac{\exp(a_k)} {\sum_{j=1}^{K}\exp(a_j)}. }\]

The exponential makes every value positive and the denominator normalises the outputs so that

\[\sum_{k=1}^{K}y_k=1.\]

A larger score gives a larger probability, so the ordering of the scores is preserved.

Softmax is the natural multi-class extension of the sigmoid. With two classes,

\[y_1 = \frac{e^{a_1}} {e^{a_1}+e^{a_2}} = \frac{1} {1+e^{-(a_1-a_2)}} = \sigma(a_1-a_2).\]

So the binary classifier is a special case of the multi-class model.

A practical detail is that softmax should usually be computed after subtracting the largest score:

\[z_k = a_k-\max_j a_j.\]

Then

\[y_k = \frac{e^{z_k}} {\sum_j e^{z_j}}.\]

This gives exactly the same probabilities but avoids numerical overflow when the scores are large.

Categorical Cross-Entropy

To learn parameters of the cateogical, we do the usual and perform maximum likelihood. For one data point,

\[p(\mathbf{t}\mid\mathbf{x}) = \prod_{k=1}^{K} y_k^{t_k}.\]

Assuming independent training examples, the likelihood is

\[p(\mathcal{D}\mid\mathbf{W}) = \prod_{n=1}^{N} \prod_{k=1}^{K} y_{nk}^{t_{nk}}.\]

Taking the negative log gives

\[\boxed{ \mathcal{L} = -\sum_{n=1}^{N} \sum_{k=1}^{K} t_{nk}\log y_{nk}. }\]

This is the categorical cross-entropy loss. Exercise: derive the above step-by-step to get more practice on obtaining a loss via maximum likelihood.

Because each target vector is one-hot, the inner sum keeps only one term:

\[\mathcal{L}_n = -\log y_{n,k^*},\]

where k^* is the true class. Thus, training penalises the model according to the probability it assigned to the class that actually occurred.

The overall pattern is now clear:

\[\boxed{ \begin{array}{ccc} \text{Gaussian} & \longrightarrow & \text{squared error}\\ \text{Bernoulli} & \longrightarrow & \text{binary cross-entropy}\\ \text{Categorical} & \longrightarrow & \text{categorical cross-entropy} \end{array} }\]

In each case, the loss comes from the negative log-likelihood of the target distribution.

Multi-Class and Multi-Label Classification

It is important to distinguish two problems that are sometimes confused.

Multi-Class Classification

In multi-class classification, exactly one of K classes is correct. Examples include:

The classes compete with one another. We therefore use one softmax over all classes so that their probabilities sum to one.

Multi-Label Classification

In multi-label classification, several labels may be true simultaneously. A car listing could be

at the same time. These are not competing alternatives.

A common model uses K independent sigmoid outputs:

\[y_k = \sigma(\mathbf{w}_k^{\top}\mathbf{x}),\]

one for each label. Each output has its own Bernoulli likelihood and binary cross-entropy loss. The total loss is the sum over labels.

Using a softmax for this problem would be wrong because increasing the probability of one label would necessarily decrease the probabilities of the others. In a multi-label problem, the labels do not have to compete.

Connection to Language Models

A language model's next-token prediction can be viewed as a very large multi-class classification problem.

Suppose the model converts the context into a representation \mathbf{h}. A linear output layer produces one score for every token in the vocabulary:

\[\mathbf{a} = \mathbf{W}\mathbf{h}.\]

A softmax then gives

\[p(\text{token }k\mid\text{context}) = \frac{\exp(a_k)} {\sum_j\exp(a_j)}.\]

The training loss for the next token is categorical cross-entropy, exactly as derived above:

\[-\log p(\text{token that actually came next}\mid\text{context}).\]

The vocabulary may contain tens or hundreds of thousands of tokens, but the probabilistic structure is unchanged.

Temperature

At generation time, the softmax is often modified using a temperature T:

\[y_k = \frac{\exp(a_k/T)} {\sum_j\exp(a_j/T)}.\]

When T<1, differences between scores are magnified and the distribution becomes sharper. As T\rightarrow 0, probability concentrates near the largest score.

When T>1, differences between scores are reduced and the distribution becomes flatter. Lower-probability alternatives therefore receive more probability mass.

Temperature changes how a model samples from its scores; it does not retrain the underlying classifier.

Summary

In this lecture, we changed the target from a real-valued number to a label. This required us to change the probability model for the target.

Applying least squares directly to numerical class labels can produce a decision boundary, but squared error has the wrong behaviour for classification: it can penalise examples for being confidently correct, and its unbounded output is not naturally a class probability.

For binary classification, we instead used a Bernoulli model:

\[p(t\mid\mathbf{x},\mathbf{w}) = \mathrm{Bern} \left( t \mid \sigma(\mathbf{w}^{\top}\mathbf{x}) \right).\]

The sigmoid maps the real-valued linear score to a probability and makes that score equal to the log-odds:

\[\log \frac{p(t=1\mid\mathbf{x})} {p(t=0\mid\mathbf{x})} = \mathbf{w}^{\top}\mathbf{x}.\]

Maximum likelihood gives the binary cross-entropy loss:

\[\mathcal{L} = -\sum_n \left[ t_n\log y_n + (1-t_n)\log(1-y_n) \right].\]

With a Gaussian prior on the weights, MAP estimation adds the familiar L_2 penalty:

\[\mathcal{L}_{\mathrm{MAP}} = \mathcal{L} + \frac{\alpha}{2}\|\mathbf{w}\|_2^2.\]

Unlike least-squares regression, logistic regression has no closed-form solution. Its gradient is

\[\nabla_{\mathbf{w}}\mathcal{L} = \sum_n (y_n-t_n)\mathbf{x}_n,\]

so we fit the model using iterative optimisation such as gradient descent. On perfectly separable data, unregularised maximum likelihood drives the weights towards infinite magnitude; a prior or regulariser gives a finite solution.

If we retain a posterior distribution over the weights rather than a single MAP estimate, logistic regression becomes Bayesian. The posterior is no longer available in closed form, so methods such as the Laplace approximation are needed to approximate it and average predictions over plausible decision boundaries.

We also learnt that a probability is not yet a decision. A classification threshold should reflect the costs of different errors. If the false-positive and false-negative costs are C_{\mathrm{FP}} and C_{\mathrm{FN}}, then the cost-minimising rule predicts class 1 when

\[p(t=1\mid\mathbf{x}) > \frac{C_{\mathrm{FP}}} {C_{\mathrm{FP}}+C_{\mathrm{FN}}}.\]

Finally, for more than two mutually exclusive classes, we use a categorical distribution with softmax probabilities:

\[y_k = \frac{\exp(a_k)} {\sum_j\exp(a_j)},\]

which gives categorical cross-entropy:

\[\mathcal{L} = -\sum_n\sum_k t_{nk}\log y_{nk}.\]

The main pattern from Lecture 2 therefore continues to hold:

Choose a probability distribution that matches the target, and the loss follows from the negative log-likelihood.