Lecture 4: Introduction to Generative Models and Naive Bayes

Summary of Key Ideas

In Lecture 3, we learnt logistic regression, which models the probability of a label given the input:

\[p(t\mid\mathbf{x}).\]

This is a discriminative model. It focuses directly on the prediction we want to make. It does not try to model where the input \mathbf{x} came from.

In this lecture, we take a different approach. A generative model describes how the entire dataset could have been generated. For classification, we will model the joint distribution

\[p(\mathbf{x},t) = p(t)\,p(\mathbf{x}\mid t).\]

The generative story is: first choose a class t, then generate the features \mathbf{x} from a distribution associated with that class. Once we have this model, we use Bayes' rule to obtain p(t\mid\mathbf{x}) and classify a new example.

Our main example will be Naive Bayes. The important idea behind Naive Bayes is independence assumptions can make the learning problem easier. Specifically, we assume that the features are independent given the class,

\[p(\mathbf{x}\mid t) = \prod_m p(x^m\mid t).\]

A complicated distribution over a whole input vector can be broken into many small distributions. The likelihood also breaks into separate pieces, so each part can be learnt on its own using simple counts, means, and variances.

This simplicity has benefits, but it comes from an assumption. If the features are strongly related even after we know the class, Naive Bayes can count the same evidence more than once and become overconfident. This is an example of a recurring machine learning trade-off: stronger assumptions can make learning easier, especially with limited data, but they can also make the model systematically wrong.

Learning Objectives

By the end of this lesson, the main things you should be able to do are:

In addition, you should be able to:

To achieve these learning objectives, we will start by comparing discriminative and generative modelling. We will then build a simple generative classifier and see why a direct model of p(\mathbf{x}\mid t) quickly becomes difficult. Naive Bayes will solve this problem by imposing structure on the distribution. We will then see how this structure makes learning and prediction simple, before considering the benefits and costs of the assumptions we made.

Discriminative and Generative Models

Suppose our data is

\[\mathcal{D} = \{(\mathbf{x}_n,t_n)\}_{n=1}^{N},\]

where \mathbf{x}_n contains the features and t_n is the class label.

There are two different ways we could build a probabilistic classifier.

A discriminative model directly models the probability of the label given the features:

\[p(t\mid\mathbf{x}).\]

Logistic regression is an example. We give the model \mathbf{x} and it directly returns a distribution over t. We do not need a probability model for \mathbf{x} itself.

A generative model instead models the joint distribution of the features and the label:

\[p(\mathbf{x},t).\]

Using the product rule, we can write this as

\[\boxed{ p(\mathbf{x},t) = p(t)\,p(\mathbf{x}\mid t). }\]

This gives us a simple story for how one labelled example is generated:

  1. draw a class t from p(t)
  2. draw the features \mathbf{x} from p(\mathbf{x}\mid t)

For example, suppose t says whether a used car sells quickly. We first draw whether the car belongs to the quick-selling or slow-selling class. Once that class is known, we draw its age and mileage from the distribution for cars in that class.

This is the central picture on Slides 13-14. Try the sampling demonstration on Slide 14. The model first chooses the class and then uses the corresponding feature distribution to generate the input.

Making a Prediction

At prediction time, we observe \mathbf{x} and want to infer t. The model was written in the other direction, but Bayes' rule lets us reverse it:

\[p(t\mid\mathbf{x}) = \frac{ p(\mathbf{x}\mid t)p(t) }{ p(\mathbf{x}) }.\]

For classification, the denominator can be obtained by summing over the possible classes:

\[p(\mathbf{x}) = \sum_{t'} p(\mathbf{x}\mid t')p(t').\]

Therefore,

\[\boxed{ p(t\mid\mathbf{x}) = \frac{ p(\mathbf{x}\mid t)p(t) }{ \sum_{t'} p(\mathbf{x}\mid t')p(t') }. }\]

The two quantities we need are therefore:

The interactive example on Slides 15-16 shows this idea. Each class gives the new input a score through p(\mathbf{x}\mid t), we weight that score by how common the class is, and then normalise.

Why Model More Than We Need?

A discriminative model seems to have the easier job. If we only care about predicting t, why model \mathbf{x} at all?

The answer is that modelling the joint distribution gives us more than a classifier. Because we have a model for the input, we can also ask questions such as:

We can also:

We will see examples later in the lecture. The cost is that modelling p(\mathbf{x}\mid t) is a harder task than modelling only the boundary between classes. A bad model of the inputs can therefore give us bad predictions.

This is the first major distinction to remember:

A discriminative model directly models the prediction. A generative model models how the data is produced and obtains the prediction using Bayes' rule.

Naive Bayes: Giving the Model Useful Structure

Recall that our model comprises two parts: the class prior p(t) and the class-conditional distribution p(\mathbf{x}\mid t). The class prior is easy to specify: we use a Bernoulli distribution if there are only two classes. If we have more classes, we use the Categorical distribution.

The tricky part is the class-conditional distribution

\[p(\mathbf{x}\mid t).\]

This is where the main idea of Naive Bayes appears.

Suppose \mathbf{x} contains M features:

\[\mathbf{x} = (x^1,\ldots,x^M).\]

In principle, we could try to learn a full joint distribution

\[p(x^1,\ldots,x^M\mid t).\]

The problem is that a general joint distribution can be very complicated. We need to describe not only each feature separately, but also how every feature varies with every other feature. The number of quantities we need to estimate can grow very quickly with M.

Slide 22 makes this problem concrete by increasing the number of features. A flexible joint model quickly has many more parameters than we can reliably estimate from the available data.

The Naive Assumption

Naive Bayes makes a strong simplifying assumption:

Given the class, the features are independent.

Mathematically,

\[\boxed{ p(\mathbf{x}\mid t) = \prod_{m=1}^{M} p(x^m\mid t). }\]

This is called conditional independence. It does not say that the features must be independent when we look at the whole dataset. For example, age and mileage may be related overall. The assumption says that once the class is known, the model treats the remaining relationship between age and mileage as unimportant (so the features are independent).

Use the interactive plot on Slide 23 to compare the Naive Bayes model with a full joint model. The full model can represent correlations within a class. The Naive Bayes model cannot.

At first this may look like we have just made the model worse. But the assumption gives us something very valuable: a complicated high-dimensional distribution has become a collection of one-dimensional distributions.

Instead of learning

\[p(x^1,\ldots,x^M\mid t),\]

we learn

\[p(x^1\mid t),\quad p(x^2\mid t),\quad \ldots,\quad p(x^M\mid t).\]

Each feature can now be handled separately. To make this idea concreate, let's examine the example that uses Gaussians.

Gaussian Naive Bayes

For continuous features such as age and mileage, a simple choice is to model each feature with a Gaussian:

\[p(x^m\mid t=k) = \mathcal{N} \left( x^m \mid \mu_k^m, v_k^m \right).\]

The Naive Bayes class-conditional becomes

\[\boxed{ p(\mathbf{x}\mid t=k) = \prod_{m=1}^{M} \mathcal{N} \left( x^m \mid \mu_k^m, v_k^m \right). }\]

Each class has its own mean and variance for each feature. The complete model for a class is therefore built from many small pieces.

A useful way to think about the model is as a set of blocks:

This block view becomes important when we learn the model. See the block picture at the beginning of Lecture 5 (Slides 4-7).

Learning Naive Bayes

The structure we chose for the model also determines how we learn the model's parameters \boldsymbol{\theta}. As in the previous lectures, let us start with maximum likelihood,

\[\boldsymbol{\theta}_{\mathrm{ML}} = \operatorname*{arg\,max}_{\boldsymbol{\theta}} \log p(\mathcal{D}\mid\boldsymbol{\theta}).\]

The first step is to derive the likelihood from our model above. For a labelled example (\mathbf{x}_n,t_n),

\[p(\mathbf{x}_n,t_n) = p(t_n)\,p(\mathbf{x}_n\mid t_n).\]

Assuming that the training examples are independent, the likelihood is

\[p(\mathcal{D}\mid\boldsymbol{\theta}) = \prod_{n=1}^{N} p(t_n\mid\boldsymbol{\theta}) p(\mathbf{x}_n\mid t_n,\boldsymbol{\theta})\]

With the Naive Bayes factorisation,

\[p(\mathcal{D}\mid\boldsymbol{\theta}) = \prod_{n=1}^{N} p(t_n) \prod_{m=1}^{M} p(x_n^m\mid t_n).\]

Take logarithms:

\[\log p(\mathcal{D}\mid\boldsymbol{\theta}) = \sum_{n=1}^{N} \log p(t_n) + \sum_{n=1}^{N} \sum_{m=1}^{M} \log p(x_n^m\mid t_n).\]

This equation is the main reason Naive Bayes is easy to learn. The log-likelihood has split into separate pieces.

The parameters of the class prior appear only in the first term. The parameters for feature 1 in class 0 appear only in the terms involving feature 1 and class 0. The parameters for feature 2 in class 1 appear somewhere else. We do not have one large coupled optimisation problem. We have many small problems that can be solved independently. Slides 24-27 derive this explicitly. (Note: You should take the time to understand this part well.)

Putting aside all the algebra, the important result is the structure:

A factorised model gives a factorised learning problem.

Factorized here means that the model is split into "pieces". This gives the model a useful kind of modularity. This is the main idea. By making conditional independence assumptions, one hard problem has become several easy ones.

Furthermore, if a new class is introduced, we can fit a new class-conditional distribution for that class rather than redesign the whole model. We must also update the class prior, and the final decision boundaries can move because Bayes' rule now normalises over more classes, but the old class-conditional blocks do not have to be relearned.

The broader lesson is:

The way we factor a probability distribution determines which dependencies the model represents and which parameter-learning problems are coupled together.

Naive Bayes chooses an especially simple factorisation. That is why it is easy to learn.

Learning the Class Prior

For binary classification, let

\[\pi = p(t=1).\]

If N_1 training examples have t=1 and N_0 have t=0, maximum likelihood gives

\[\boxed{ \hat{\pi} = \frac{N_1}{N_0+N_1}. }\]

So the estimated class probability is simply the fraction of training examples in that class.

For K classes, if N_k examples belong to class k,

\[\boxed{ \hat{\pi}_k = \frac{N_k}{N}. }\]

Learning is just counting.

Learning the Feature Distributions

Now consider feature m in class k. Only examples belonging to class k contribute to the likelihood for its parameters.

If the feature is Gaussian, maximum likelihood gives

\[\boxed{ \hat{\mu}_k^m = \frac{1}{N_k} \sum_{n:t_n=k} x_n^m }\]

and

\[\boxed{ \hat{v}_k^m = \frac{1}{N_k} \sum_{n:t_n=k} \left( x_n^m-\hat{\mu}_k^m \right)^2. }\]

These are simply the sample mean and the maximum-likelihood sample variance computed using examples from that class.

So training Gaussian Naive Bayes consists of:

  1. count how many examples belong to each class
  2. separate the examples by class
  3. for each feature in each class, compute a mean and variance

There is no gradient descent and no iterative optimisation. Slides 26-28 show that the entire fit can be done using counts and averages. This outcome is a direct consequence of how we structured the model.

Prediction With the Fitted Model

Once the model has been fitted, classification uses Bayes' rule.

For class k,

\[p(t=k\mid\mathbf{x}) = \frac{ p(t=k)p(\mathbf{x}\mid t=k) }{ \sum_j p(t=j)p(\mathbf{x}\mid t=j) }.\]

Under Gaussian Naive Bayes,

\[\boxed{ p(t=k\mid\mathbf{x}) = \frac{ \pi_k \prod_m \mathcal{N}(x^m\mid\mu_k^m,v_k^m) }{ \sum_j \pi_j \prod_m \mathcal{N}(x^m\mid\mu_j^m,v_j^m) }. }\]

The interactive example on Slide 30 lets you move a new car around and watch how this calculation is performed.

Practical Note: Work in Log Space

As we have already learnt in previous lectures, products of many probabilities or densities can become extremely small. In code, we therefore usually work with logarithms.

Define a class score

\[s_k = \log \pi_k + \sum_m \log p(x^m\mid t=k).\]

We can then exponentiate and normalise the scores at the end:

\[p(t=k\mid\mathbf{x}) = \frac{e^{s_k}} {\sum_j e^{s_j}}.\]

For numerical stability, subtract the largest score before exponentiating, just as we did for softmax in Lecture 3.

The computation is therefore simple: each feature contributes one piece of evidence, those pieces are added in log space, the class prior is added, and the final class scores are normalised.

More Than Two Classes and Constrained Probabilities

For K classes, the class prior is a categorical distribution:

\[p(\mathbf{t}\mid\boldsymbol{\pi}) = \prod_{k=1}^{K} \pi_k^{t_k},\]

where \mathbf{t} is one-hot and

\[\pi_k\geq 0, \qquad \sum_{k=1}^{K}\pi_k=1.\]

If N_k training examples belong to class k, the part of the negative log-likelihood involving the prior is

\[-\sum_{k=1}^{K} N_k\log\pi_k.\]

We want to minimise this subject to

\[\sum_k\pi_k=1.\]

This is a constrained optimisation problem. A standard tool for this is the Lagrangian. We introduce a new variable \lambda and write

\[\widetilde{\mathcal{L}} = -\sum_k N_k\log\pi_k + \lambda \left( \sum_k\pi_k-1 \right).\]

Differentiate with respect to each \pi_k:

\[-\frac{N_k}{\pi_k} + \lambda = 0.\]

Therefore,

\[\pi_k = \frac{N_k}{\lambda}.\]

The constraint gives

\[\sum_k\pi_k = \frac{\sum_k N_k}{\lambda} = \frac{N}{\lambda} = 1,\]

so \lambda=N and

\[\boxed{ \hat{\pi}_k = \frac{N_k}{N}. }\]

This is the answer we would probably have guessed: estimate a class probability by its observed fraction. The purpose of the derivation is to introduce the Lagrangian as a general method for handling equality constraints. The geometric intuition and a fuller derivation are given on Slides 36-44.

Bayesian Learning: Priors, Pseudo-Counts, and Smoothing

Maximum likelihood can become too confident when data is scarce. For example, if a class does not appear at all in a small dataset, maximum likelihood assigns it probability zero.

We can use the same MAP idea from Lectures 2 and 3. For the categorical probabilities \boldsymbol{\pi}, a useful prior is the Dirichlet distribution.

The slides use a pseudo-count form of the Dirichlet prior:

\[p(\boldsymbol{\pi}) \propto \prod_{k=1}^{K} \pi_k^{\alpha_k},\]

where \alpha_k\geq 0 acts like an additional count for class k.

The likelihood contributes a factor

\[\prod_k \pi_k^{N_k},\]

so the posterior contains

\[\prod_k \pi_k^{N_k+\alpha_k}.\]

The real counts and the pseudo-counts simply add. The MAP estimate becomes

\[\boxed{ \hat{\pi}_{k,\mathrm{MAP}} = \frac{ N_k+\alpha_k }{ N+\sum_j\alpha_j }. }\]

This is another example of model structure making Bayesian updating easy. The prior and likelihood have matching forms, so the posterior has the same form. Such a prior is called a conjugate prior.

The interpretation is simple. When there is little data, the pseudo-counts matter. As N becomes large, the real counts dominate and the prior has less influence.

Naive Bayes for Text

Historically, Naive Bayes became especially well known as a text classifier because the same factorised structure works naturally with words.

Suppose we fix a vocabulary of M words and represent an email using one feature per word

\[p(\mathbf{x}\mid t) = \prod_m p(x^m\mid t),\]

Note that the individual feature distributions need not be Gaussian.

If x^m only records whether word m appears, we can use a Bernoulli distribution:

\[x^m \sim \mathrm{Bern}(\mu_t^m).\]

If we care about how many times each word appears, we can instead use a multinomial model. The overall Naive Bayes idea does not change. Only the small distribution inside each block changes.

This is another useful lesson:

The factorisation tells us how the pieces connect. We can often change the distribution used inside a piece without changing the rest of the model.

The Zero-Count Problem

Suppose the word meeting never appeared in any spam email in the training set. Maximum likelihood then gives

\[p(\text{meeting present}\mid\text{spam})=0.\]

But the Naive Bayes class-conditional is a product. One zero factor makes the whole product zero:

\[p(\mathbf{x}\mid\text{spam}) = \prod_m p(x^m\mid\text{spam}) = 0.\]

No matter how strongly every other word suggests spam, the model now says that the email cannot be spam.

The problem is not that the word is necessarily impossible in spam. We simply did not see it in our small training set.

Laplace Smoothing

Again, the solution is a prior. For a binary feature, let c_t^m be the number of class-t examples in which word m appears, and let N_t be the number of examples in that class.

With equal pseudo-count a for presence and absence, the MAP estimate is

\[\boxed{ \hat{\mu}_t^m = \frac{ c_t^m+a }{ N_t+2a }. }\]

The common choice a=1 is called Laplace smoothing or add-one smoothing.

A count of zero now becomes

\[\frac{1}{N_t+2}\]

rather than zero. The model treats an unseen event as unlikely rather than impossible.

The sequence on Slides 52-56 shows the zero-count problem and how smoothing changes the prediction.

The important point is that smoothing is the same MAP idea we have used before:

MLE uses the observed counts. MAP uses the observed counts plus prior pseudo-counts.

What a Generative Model Buys Us

Because a generative model represents the joint distribution, it supports questions that a conditional classifier does not answer directly.

Missing Features

Suppose a car's age is known but its mileage is missing. We want

\[p(t\mid x^{\mathrm{age}}).\]

From the joint model, we can integrate the missing feature out:

\[p(t,x^{\mathrm{age}}) = \int p(t,x^{\mathrm{age}},x^{\mathrm{mile}}) \,dx^{\mathrm{mile}}.\]

Under Naive Bayes,

\[p(t,x^{\mathrm{age}},x^{\mathrm{mile}}) = p(t) p(x^{\mathrm{age}}\mid t) p(x^{\mathrm{mile}}\mid t).\]

Therefore,

\[p(t,x^{\mathrm{age}}) = p(t) p(x^{\mathrm{age}}\mid t) \underbrace{ \int p(x^{\mathrm{mile}}\mid t) \,dx^{\mathrm{mile}} }_{1}.\]

The missing feature simply disappears:

\[\boxed{ p(t\mid x^{\mathrm{age}}) \propto p(t) p(x^{\mathrm{age}}\mid t). }\]

Slide 59 lets you remove features and see this happen. Under the Naive Bayes factorisation, a missing feature costs us one factor.

Generating New Data

We can also run the model in the forward direction:

  1. draw a class from p(t)
  2. draw each feature from its class-conditional distribution

This generates a new synthetic example. Slide 60 demonstrates this directly.

The generated examples from Gaussian Naive Bayes may not be very realistic because the model is simple, but the principle is the same one used by much richer generative models (like VAEs, GANs, and Diffusion Models): specify a distribution over data, fit it, then sample from it.

What the Naive Assumption Costs

The ease of learning came from the conditional independence assumption. If that assumption is badly wrong, the model can make systematic mistakes.

Suppose we duplicate a feature. Let x^2 be an exact copy of x^1. The duplicate contains no new information.

Naive Bayes nevertheless writes

\[p(x^1,x^2\mid t) = p(x^1\mid t)p(x^2\mid t).\]

It therefore treats the same evidence as if it arrived twice. A class favoured by that feature receives the same piece of evidence two times.

This is demonstrated on Slide 61. Adding a duplicate of mileage changes the posterior even though we gave the model no new information.

More generally, strongly correlated features can cause Naive Bayes to double-count evidence. The predicted class may still be correct, but the probabilities can become too extreme.

This gives us the basic trade-off:

Generative and Discriminative Models Revisited

We can now compare Naive Bayes with logistic regression more clearly.

Logistic regression directly models

\[p(t\mid\mathbf{x}).\]

It spends its modelling capacity on the relationship needed for classification. It does not need to explain the full distribution of inputs.

Naive Bayes models

\[p(t) \quad\text{and}\quad p(\mathbf{x}\mid t),\]

and obtains p(t\mid\mathbf{x}) using Bayes' rule. It solves a harder modelling problem, but its assumptions break that problem into small pieces.

This can be particularly useful when data is limited. The assumptions restrict the possible models, so there is less to estimate. With enough data, however, incorrect assumptions about p(\mathbf{x}\mid t) can become a limitation.

There is also an interesting connection between the two classifiers. For Gaussian Naive Bayes, the log-odds are

\[\log \frac{ p(t=1\mid\mathbf{x}) }{ p(t=0\mid\mathbf{x}) } = \log \frac{\pi_1}{\pi_0} + \sum_m \log \frac{ \mathcal{N}(x^m\mid\mu_1^m,v_1^m) }{ \mathcal{N}(x^m\mid\mu_0^m,v_0^m) }.\]

In general, this contains quadratic terms in the features. If the two classes use the same variance for each feature, those quadratic terms cancel and the log-odds become linear in \mathbf{x}:

\[\log \frac{ p(t=1\mid\mathbf{x}) }{ p(t=0\mid\mathbf{x}) } = w_0+\mathbf{w}^{\top}\mathbf{x}.\]

That is exactly the form assumed by logistic regression! This means that generative and discriminative training can sometimes lead to the same form of classifier even though they learn it in different ways. Logistic regression fits the boundary directly. Naive Bayes first fits the class distributions and obtains the boundary from them. Slide 32 shows this connection and Tutorial 3 explores this more deeply.

A Broader View of Generative Models

Naive Bayes is a very simple generative model, but it introduces ideas that appear in much larger models.

The model is built by specifying a sequence of random choices:

\[t \sim p(t), \qquad \mathbf{x} \sim p(\mathbf{x}\mid t).\]

We can make the story richer by adding more variables or by making the parameters of one distribution depend on a function.

For example, we might introduce another variable z:

\[t \sim p(t), \qquad z \sim p(z\mid t), \qquad \mathbf{x} \sim p(\mathbf{x}\mid z).\]

If z is not observed, it is called a latent variable. Learning then requires us to account for possible values of z rather than simply fitting each block from labelled data.

Modern generative models use much richer versions of this idea. Variational autoencoders introduce continuous latent variables and neural networks. Diffusion models define a multi-step stochastic process. Language models factor a sequence into conditional token distributions.

We won't cover the details of these models yet. For now, the key connection is:

A generative model describes a process that assigns probabilities to data and can be run to produce data. The main modelling decision is how to factor and parameterise that process.

Summary

In this lecture, we discussed two different ways to build a classifier.

A discriminative model directly models

\[p(t\mid\mathbf{x}).\]

A generative model models the joint distribution

\[p(\mathbf{x},t) = p(t)p(\mathbf{x}\mid t),\]

and obtains the classifier using Bayes' rule:

\[p(t\mid\mathbf{x}) = \frac{ p(\mathbf{x}\mid t)p(t) }{ \sum_{t'} p(\mathbf{x}\mid t')p(t') }.\]

Naive Bayes makes the conditional independence assumption

\[\boxed{ p(\mathbf{x}\mid t) = \prod_m p(x^m\mid t). }\]

This assumption is the central modelling step in the lecture. It replaces one difficult high-dimensional distribution with many simple per-feature distributions.

The same factorisation makes learning easy. The log-likelihood separates into pieces, so each part of the model can be learnt independently. For Gaussian Naive Bayes,

\[\hat{\pi}_k = \frac{N_k}{N},\]

\[\hat{\mu}_k^m = \frac{1}{N_k} \sum_{n:t_n=k} x_n^m,\]

and

\[\hat{v}_k^m = \frac{1}{N_k} \sum_{n:t_n=k} (x_n^m-\hat{\mu}_k^m)^2.\]

Training is therefore mostly counting and averaging.

This modular structure also makes other operations simple. Missing features can be integrated out, adding a class means adding another class-conditional block, and different types of features can use different simple distributions.

But the simplification has a cost. If features are related within a class, Naive Bayes ignores those relationships. Correlated or duplicated features can therefore be counted as independent pieces of evidence and make the model overconfident.

Priors give us another way to make the model more robust when data is scarce. For class probabilities, a Dirichlet prior acts like extra counts. For binary word features, this gives the smoothed estimate

\[\hat{\mu}_t^m = \frac{ c_t^m+a }{ N_t+2a },\]

which avoids assigning zero probability simply because an event was not seen in a small dataset.

The main ideas to take away are:

  1. Generative and discriminative models answer the classification problem in different ways.
  2. The factorisation of a probabilistic model is a modelling assumption.
  3. Good structure can turn one difficult learning problem into many simple ones.
  4. The assumptions that make learning easy can also become a source of error when they are wrong.