Lecture 1: Introduction to Machine Learning and Regression I

Summary of Key Ideas

In this lecture, we introduce machine learning and study our first machine learning problem: regression.

Machine learning is about building computer programs that improve their behaviour using data. Instead of writing down every rule ourselves, we provide examples and choose a model that can learn useful patterns from those examples.

We will focus on supervised learning, where each training example contains an input and a desired output. In a regression problem, the desired output is a number. For example, we might want to predict the price of a house from its size, or predict a person's height from other measurements.

Our first model will be linear regression. The basic idea is simple: choose a line, or more generally a linear model, that makes predictions close to the observed data. This raises several important questions that will appear throughout the course:

How do we represent a model?
How do we measure whether a model is good?
How do we choose the model's parameters from data?
Does fitting the training data well mean that the model has learned something useful?

These questions are much more important than linear regression itself. Most machine learning methods can be understood as different answers to the same questions.

Learning Objectives

By the end of this lesson, you should be able to:

explain what we mean by machine learning
distinguish between supervised and unsupervised learning
distinguish regression from classification
identify the inputs, targets, model, parameters, and predictions in a supervised learning problem

Specifically for regression, you should be able to:

write down a simple linear regression model
explain why squared error is a reasonable loss for regression
express linear regression using vectors and matrices
explain how basis functions allow linear models to represent nonlinear relationships
explain the difference between training error and test error, and what overfitting is.
explain how regularisation can reduce overfitting

To achieve these objectives, we will first discuss what it means for a machine to learn from data. We will then formulate supervised learning more precisely, introduce regression, and use linear regression to see how a complete learning problem is constructed and solved. Finally, we will see what goes wrong when our models become too flexible, and introduce regularisation as one way to address the problem.

What Is Machine Learning?

Suppose we want a computer to predict the price of an apartment.

One approach is to write rules:

Start with $500,000. Add $100,000 if it is near an MRT station. Add $1,000 per square metre. Subtract $50,000 if the building is old...

This quickly becomes difficult. There may be many factors, and we may not know how strongly each factor matters.

A machine learning approach is different. We collect examples of apartments for which we know both:

information about the apartment, such as its size, age, and location; and
the price at which it was sold.

We then use these examples to learn a prediction rule.

A useful working definition is:

Machine learning is the study of algorithms that learn useful patterns or decision rules from data.

The word learn is important. The program is not simply storing the answers we gave it. We want it to extract a pattern that also works for examples that it has not seen before. This ability to work on new examples is called generalization.

A recurring picture

A typical machine learning setup has three important ingredients:

Data gives us examples of the problem.
A model describes the kinds of prediction "rules" we are willing to consider.
A loss function defines how bad a particular model is.

A learning algorithm chooses a particular model using the data, typically using the loss function. The learned model is then used to make predictions.

Do not mix up the model and the learning algorithm. For example, a straight line is a model. Least squares is a method for choosing which straight line to use.

Slides 5-9 give several examples of this idea. The data can look very different, from images to stock prices to robot motion, but the underlying question is similar: what useful pattern can we learn from the data?

Learning From Examples

A single supervised learning example usually contains an input and a target. We write an example as (\mathbf{x}, t).

Here:

\mathbf{x} is the input, also called the feature vector;
t is the target, which is the value we would like to predict.

For example, for an apartment,

\[\mathbf{x} = \begin{bmatrix} \text{floor area}\\ \text{age}\\ \text{distance to MRT} \end{bmatrix}, \qquad t = \text{sale price}.\]

A feature is simply a measurable piece of information that we give to the model.

If we have N examples, our training dataset can be written as

\[\mathcal{D} = \{(\mathbf{x}_1,t_1),\ldots,(\mathbf{x}_N,t_N)\} = \{(\mathbf{x}_n,t_n)\}_{n=1}^N.\]

The subscript tells us which example we are talking about.

For the regression derivations in the slides, we write y(\mathbf{x};\mathbf{w}) for the model's prediction.

Types of Machine Learning Problems

Slide 10 shows a useful high-level picture of the machine learning landscape where there are three main areas: Supervised Learning, Unsupervised Learning, and Reinforcement Learning.

Supervised learning

In supervised learning, every training input comes with a target:

\[\mathcal{D} = \{(\mathbf{x}_n,t_n)\}_{n=1}^N.\]

Two important supervised learning problems are classification and regression.

Classification

In classification, the target is a discrete category. Examples include:

spam or not spam
cat, dog, or bird
benign or malignant
pedestrian, cyclist, or car

Regression

In regression, the target is a numerical quantity. Examples include:

tomorrow's electricity demand
the selling price of a house
temperature
the amount of solar radiation

We will study regression in this lecture.

Unsupervised learning

In unsupervised learning, we have inputs but no supplied targets:

\[\mathcal{D} = \{\mathbf{x}_n\}_{n=1}^N.\]

We may instead want to discover useful structure in the data, such as groups of similar examples or a simpler representation of complicated data.

Reinforcement learning

Slide 10 also shows reinforcement learning. Here an agent interacts with an environment and learns from rewards rather than being given the correct answer for every input. We will not study reinforcement learning in depth in this course, but many of the same ideas about models, data, objectives, and optimisation still apply.

Regression: Learning a Relationship From Data

Slide 16 shows examples of regression in real data. In both examples, we observe noisy measurements and believe that there is some underlying relationship that we would like to understand or predict.

Suppose for simplicity that both the input and target are numbers. Our data might look like

\[\mathcal{D} = \{(x_1,t_1),\ldots,(x_N,t_N)\}.\]

We want to learn a function

\[y(x)\]

that predicts the target when given a new input x.

There are now three questions to answer.

What data do we have? These are the observed pairs (x_n,t_n).
What model do we use? We need to choose a family of functions that we are willing to consider.
What loss do we use? We need a numerical measure of how well or badly a model fits the observations.

These are the data, model, and loss ingredients shown on Slide 17. Keep these three questions in mind throughout the course.

Linear Regression

We begin with perhaps the simplest useful regression model: a straight line.

\[y(x;\mathbf{w}) = w_0+w_1x.\]

The quantities w_0 and w_1 are the parameters of the model.

w_0 controls the intercept.
w_1 controls the slope.

Different choices of \mathbf{w}=(w_0,w_1) give us different lines. So, the learning problem is: Which line should we choose?

Before introducing any equations for the loss, try the activity on Slide 18. It shows the same data together with three candidate lines.

Ask yourself:

Which line looks best?
Why does it look better?
How could we turn that judgement into a number?

The purpose of a loss function is to answer the last question.

Measuring Error With Squared Loss

Suppose the observed target for input x_n is t_n. The model predicts

\[y(x_n;\mathbf{w}).\]

The difference between the prediction and target is

\[y(x_n;\mathbf{w})-t_n.\]

We would like these differences to be small.

A common choice is the sum of squared errors:

\[E(\mathbf{w}) = \frac{1}{2} \sum_{n=1}^{N} \left( y(x_n;\mathbf{w})-t_n \right)^2.\]

The factor \frac{1}{2} does not change which model is best. It is included because it makes the derivatives slightly simpler.

You might ask, "why square the error?" First, positive and negative errors cannot cancel:

\[(+2)^2+(-2)^2=8,\]

rather than zero.

Second, a large error receives a larger penalty. For example,

\[4^2=16\]

is four times larger than

\[2^2=4.\]

This means that we want to penalize large deviations more. Least squares is particularly sensitive to large errors and we will see the consequences shortly.

Slide 19 connects the equation directly to the picture: the loss is measuring the squared vertical distances between the observed points and the line.

Least-Squares Regression

The learning algorithm chooses the parameters that minimise the loss:

\[\mathbf{w}_{\mathrm{ML}} = \operatorname*{arg\,min}_{\mathbf{w}} E(\mathbf{w}).\]

The notation \operatorname*{arg\,min} means the value of the argument that makes the expression as small as possible. Because our loss is a sum of squared errors, this method is called least squares. For the straight-line model,

\[y(x;\mathbf{w}) = w_0+w_1x,\]

least squares searches across all possible values of w_0 and w_1 and chooses the pair giving the smallest total squared error.

This is our first example of a general pattern in machine learning:

specify a model;
specify a loss;
optimise the loss with respect to the model parameters.

Writing Linear Regression With Vectors

The vector form of the model is useful because it extends immediately to many input features. For a single scalar input x_n, we first add a constant 1:

\[\mathbf{x}_n = \begin{bmatrix} 1\\ x_n \end{bmatrix}.\]

We also write

\[\mathbf{w} = \begin{bmatrix} w_0\\ w_1 \end{bmatrix}.\]

The prediction is then

\[y_n = \mathbf{w}^{\top}\mathbf{x}_n.\]

Expanding the dot product gives

\[\mathbf{w}^{\top}\mathbf{x}_n = w_0+w_1x_n,\]

so nothing has changed. We have simply written the same line in a more compact form.

The squared-error loss becomes

\[E(\mathbf{w}) = \frac{1}{2} \sum_{n=1}^{N} \left( \mathbf{w}^{\top}\mathbf{x}_n-t_n \right)^2.\]

Now stack all the inputs into a design matrix:

\[\mathbf{X} = \begin{bmatrix} \mathbf{x}_1^\top\\ \vdots\\ \mathbf{x}_N^\top \end{bmatrix},\]

and the targets into a vector:

\[\mathbf{t} = \begin{bmatrix} t_1\\ \vdots\\ t_N \end{bmatrix}.\]

The loss can now be written as

\[E(\mathbf{w}) = \frac{1}{2} (\mathbf{t}-\mathbf{X}\mathbf{w})^\top (\mathbf{t}-\mathbf{X}\mathbf{w}).\]

This is simply the sum of squared errors written using matrices.

Deriving the Least-Squares Solution

Slide 20 derives the solution step by step.

We begin with

\[E(\mathbf{w}) = \frac{1}{2} (\mathbf{t}-\mathbf{X}\mathbf{w})^\top (\mathbf{t}-\mathbf{X}\mathbf{w}).\]

Expanding gives

\[E(\mathbf{w}) = \frac{1}{2} \left( \mathbf{t}^\top\mathbf{t} - 2\mathbf{w}^\top\mathbf{X}^\top\mathbf{t} + \mathbf{w}^\top\mathbf{X}^\top\mathbf{X}\mathbf{w} \right).\]

Differentiate with respect to \mathbf{w}:

\[\nabla_{\mathbf{w}}E = \mathbf{X}^\top\mathbf{X}\mathbf{w} - \mathbf{X}^\top\mathbf{t}.\]

At a minimum, the gradient is zero:

\[\mathbf{X}^\top\mathbf{X}\mathbf{w} = \mathbf{X}^\top\mathbf{t}.\]

These are called the normal equations.

If \mathbf{X}^\top\mathbf{X} is invertible, then

\[\boxed{ \mathbf{w}_{\mathrm{ML}} = (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top\mathbf{t} }\]

is the least-squares solution.

You should understand the steps in this derivation, but the larger idea is more important:

We converted "find the best line" into an optimisation problem, differentiated the loss, and solved the resulting equation.

This basic strategy appears repeatedly in machine learning.

A Geometric View of Least Squares

Slide 21 gives another way to understand the normal equations. The prediction vector

\[\mathbf{X}\mathbf{w}\]

can only lie in the space generated by the columns of \mathbf{X}.

The target vector \mathbf{t} usually does not lie exactly inside this space.

Least squares therefore chooses the point \mathbf{X}\mathbf{w} that is closest to \mathbf{t}.

At that closest point, the residual

\[\mathbf{t}-\mathbf{X}\mathbf{w}\]

is perpendicular to the space spanned by the columns of \mathbf{X}.

Therefore,

\[\mathbf{X}^\top (\mathbf{t}-\mathbf{X}\mathbf{w}) = \mathbf{0}.\]

Rearranging gives

\[\mathbf{X}^\top\mathbf{X}\mathbf{w} = \mathbf{X}^\top\mathbf{t},\]

which is exactly the normal equation we obtained using calculus.

So there are two views of the same result:

calculus: set the gradient to zero;
geometry: project the target onto the space of predictions the model can represent.

If the geometric picture is unfamiliar, that's ok: the key point is simply that least squares finds the closest prediction that the model is capable of making.

Squared Error and Outliers

Squared error has an important consequence. Suppose one measurement is very far from all the others. Because its error is squared, that one point can contribute a large fraction of the total loss. The least-squares solution may then move significantly towards it.

Use the interactive plot on Slide 24:

drag the red point up and down;
watch how the fitted line changes;
observe that the residual of that one point can dominate the total loss.

The important lesson is that a loss function does not merely measure performance. It also determines what the learning algorithm pays attention to.

Squared error treats large deviations as especially important. This can be useful, but it also makes least squares sensitive to unusual or corrupted observations.

Nonlinear Regression via Basis Functions

A straight line cannot represent every relationship. Suppose instead that the data follows a curved pattern. One approach is to transform the input using basis functions. For example, we can use polynomial features:

\[\boldsymbol{\phi}(x) = \begin{bmatrix} 1\\ x\\ x^2\\ x^3\\ \vdots\\ x^{M-1} \end{bmatrix}.\]

Our model becomes

\[y(x;\mathbf{w}) = \mathbf{w}^{\top}\boldsymbol{\phi}(x).\]

For a polynomial model,

\[y(x;\mathbf{w}) = w_0+w_1x+w_2x^2+\cdots+w_{M-1}x^{M-1}.\]

This is nonlinear as a function of x, but it is still linear in the parameters w_i. This distinction is important (please take note of it!).The parameters still enter the equation as

\[w_i\phi_i(x),\]

so all the least-squares mathematics remains almost unchanged. The difference is that instead of the original design matrix \mathbf{X}, we construct a feature matrix

\[\boldsymbol{\Phi}.\]

The solution becomes

\[\mathbf{w}_{\mathrm{ML}} = (\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi})^{-1} \boldsymbol{\Phi}^{\top}\mathbf{t}.\]

It is just the same as the least squares solution except we replace \mathbf{X} with \boldsymbol{\Phi}.

Slide 27 lets you switch between several basis functions:

polynomial
squared exponential
sigmoid

Try each option and notice that changing the basis changes the shapes that the model can represent, even though the solver itself has not changed.

This is a useful example of separating the pieces of a machine learning method:

the data stays the same;
the loss stays the same;
the model representation changes.

Overfitting

Suppose we fit polynomial models of increasing order:

A first-order polynomial is a straight line.
A second-order polynomial can bend once.
Higher-order polynomials can represent increasingly complicated shapes.

It is tempting to conclude that if a more flexible model fits the data better, then it must be a better model.

Unfortunately, this is not true. Before continuing, consider the three possible plots on Slide 29. As polynomial order increases, what do you expect to happen to:

training error?
error on new data?

The interactive plot on Slide 30 lets you answer this directly. Move the polynomial-order slider while initially keeping regularisation at \lambda=0.

You should observe two different behaviours.

Training error generally decreases as the model becomes more flexible. Why? This happens because a more flexible model has more ways to fit the examples it has already seen.
Test error initially decreases, but eventually begins to increase. A very flexible model starts fitting accidental details of the training data rather than only the underlying pattern.

This is overfitting, which is a key problem in machine learning.

In particular, a training set contains the examples used to fit the model. A test set contains examples that were not used when fitting the model. The performance of models on both of these sets is shown clearly on Slide 31.

As model capacity increases:

training error continues to fall;
test error falls at first;
eventually test error rises again.

The model has started to overfit when improving the fit to the training data makes predictions on new data worse. This leads to one of the central lessons of machine learning:

Training error is not generalisation (or test) error.

A model with almost zero training error may still be a poor model.

The opposite problem of overfitting is underfitting. A model underfits when it is too simple to represent an important pattern in the data. For example, a straight line may underfit data that has a strongly curved relationship. There is therefore a balance:

too little flexibility leads to underfitting;
too much flexibility can lead to overfitting.

Why Does Overfitting Happen?

Overfitting happens because the model is optimized and can end up fitting "noise". Slides 33 and 34 give an intuitive explanation: imagine many people sit for an exam and you hire the candidate with the highest score. Suppose an exam score depends on both

the person's true ability; and
some luck on that particular exam.

If you have only a few candidates, the top scorer may well be the genuinely strongest candidate.

But if you search across a very large number of candidates, there are many more chances for somebody to receive an unusually lucky score. The highest observed score therefore tends to come from someone who is both good and lucky.

The same effect happens when we choose a model. Suppose the measured training error of a model h is

\[\widehat{E}(h) = E(h)+\varepsilon(h),\]

where

E(h) is its true generalisation error;
\varepsilon(h) represents noise in our estimate.

For any one fixed model, suppose

\[\mathbb{E}[\varepsilon(h)]=0.\]

Our estimate is not systematically too high or too low.

But we do not choose a random model. We search across many models and select the one with the smallest observed error:

\[\widehat{h} = \operatorname*{arg\,min}_{h\in\mathcal{H}} \left( E(h)+\varepsilon(h) \right).\]

Here \mathcal{H} is the set of models we are considering.

The optimisation process can select a model partly because its estimate happened to be unusually lucky. This is sometimes called the optimiser's curse. The key idea is:

The more possibilities we search, the more opportunities there are to fit accidental properties of the training data.

The interactive hiring example on Slide 33 makes this effect concrete. Increase the number of candidates and the amount of luck in the exam score and observe what happens.

What Overfitting Looks Like in Polynomial Regression

In high-order polynomial regression, an overfitted curve often becomes extremely wiggly. Slide 35 reveals that these wiggly curves tend to require very large parameter values. For example,

\[y(x) = w_0+w_1x+w_2x^2+\cdots\]

may contain large positive and negative coefficients that nearly cancel each other.The curve can then bend sharply enough to pass through almost every training point.

If you use the polynomial-order slider on Slide 35, you will observe that as the order increases:

the curve becomes more complicated;
the largest |w_i| can become very large;
the overall size of the weight vector grows.

This suggests a possible way to control the problem. We intentionally prefer models that fit the data well without using extremely large weights. This is the idea behind regularisation, which we will discuss next.

Regularisation

Regularisation adds an extra preference to the learning problem. Instead of minimising only the data-fitting error

\[E(\mathbf{w}),\]

we minimise

\[\widetilde{E}(\mathbf{w}) = E(\mathbf{w}) + \frac{\lambda}{2} \|\mathbf{w}\|_2^2.\]

The additional term

\[\frac{\lambda}{2} \|\mathbf{w}\|_2^2\]

penalises large weights.

Here,

\[\|\mathbf{w}\|_2^2 = \mathbf{w}^{\top}\mathbf{w} = \sum_i w_i^2.\]

The parameter \lambda controls the strength of the penalty:

If \lambda=0, we recover ordinary least squares.
If \lambda is small, we mostly fit the data, but slightly prefer smaller weights.
If \lambda is very large, the penalty dominates, forcing the weights close to zero. The model may then become too simple and underfit.

Regularisation therefore introduces another trade-off. This preference for some solutions over others is an example of an inductive bias: an assumption or preference that helps us choose between models that could fit the observed data.

Ridge Regression

Adding a squared-weight penalty to linear regression gives ridge regression, also called L_2 regularisation or weight decay. For basis-function regression,

\[\widetilde{E}(\mathbf{w}) = \frac{1}{2} (\mathbf{t}-\boldsymbol{\Phi}\mathbf{w})^\top (\mathbf{t}-\boldsymbol{\Phi}\mathbf{w}) + \frac{\lambda}{2} \mathbf{w}^{\top}\mathbf{w}.\]

Differentiate:

\[\nabla_{\mathbf{w}} \widetilde{E} = \boldsymbol{\Phi}^{\top}\boldsymbol{\Phi}\mathbf{w} - \boldsymbol{\Phi}^{\top}\mathbf{t} + \lambda\mathbf{w}.\]

Set the gradient to zero:

\[(\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi} + \lambda\mathbf{I}) \mathbf{w} = \boldsymbol{\Phi}^{\top}\mathbf{t}.\]

Therefore,

\[\boxed{ \mathbf{w}_{\mathrm{RR}} = (\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi} + \lambda\mathbf{I})^{-1} \boldsymbol{\Phi}^{\top}\mathbf{t} }\]

Compare this with ordinary least squares:

\[\mathbf{w}_{\mathrm{ML}} = (\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi})^{-1} \boldsymbol{\Phi}^{\top}\mathbf{t}.\]

The only difference is the extra \lambda\mathbf{I} A relatively small change to the objective gives a useful change to the solution.

What Does the Regulariser Actually Do?

Slides 38 and 39 give a simple one-dimensional example. Suppose

\[y=wx.\]

Without regularisation, the squared-error loss is a bowl-shaped function of w. Its minimum gives the least-squares estimate. The regularisation term

\[\frac{\lambda}{2}w^2\]

is another bowl, but one whose minimum is at zero.

The total objective is the sum of the two. As \lambda increases, the minimum of the combined objective moves towards zero. This is called shrinkage. Use the slider on Slide 39 and watch what happens to the fitted w.

Regularisation Also Helps the Linear Algebra

Interestingly, regularisation is useful for another reason. Recall the least-squares solution:

\[\mathbf{w}_{\mathrm{ML}} = (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top}\mathbf{t}.\]

This expression requires \mathbf{X}^{\top}\mathbf{X} to be invertible and Slide 37 gives two cases where this can fail:

Redundant features. Suppose we include both "height in centimetres" and "height in inches". One feature is an exact multiple of the other, so the two columns do not provide independent information.
More features than observations. Suppose we have more model parameters than training examples. There may then be many different parameter vectors that fit all the data exactly.

Ridge regression instead uses

\[\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi} + \lambda\mathbf{I}.\]

For \lambda>0, adding \lambda\mathbf{I} makes the matrix invertible. So regularisation can help both:

statistically, by controlling overfitting
numerically, by making the optimisation problem better behaved.

The Cost of the Closed-Form Solution

So far, our approach to linear regression has been to write down a formula and compute the answer analytically (we use the derived formula without any gradient descent or other numerical method).

The problem is that this formula becomes expensive when the dataset or number of features is large. Suppose

N is the number of training examples;
M is the number of features.

To construct \mathbf{X}^{\top}\mathbf{X}, we require roughly O(NM^2) operations. Solving the resulting M\times M system costs roughly O(M^3)

We also need to store the design matrix, containing NM numbers.

For a small problem, this is completely manageable. For millions of examples and thousands of features, both computation and memory become serious problems. Slide 40 gives concrete examples.

This motivates a different approach (which you should be familiar with): instead of solving for the exact answer in one large calculation, gradually improve the parameters using the gradient.

Gradient Descent

First, let us derive the gradient of the loss. Recall that the loss is

\[E(\mathbf{w}) = \frac{1}{2} \sum_{n=1}^{N} \left( \mathbf{w}^{\top}\mathbf{x}_n-t_n \right)^2.\]

Consider one training example. Its contribution to the gradient is

\[\nabla_{\mathbf{w}} \frac{1}{2} \left( \mathbf{w}^{\top}\mathbf{x}_n-t_n \right)^2.\]

Using the chain rule,

\[= \left( \mathbf{w}^{\top}\mathbf{x}_n-t_n \right) \nabla_{\mathbf{w}} \left( \mathbf{w}^{\top}\mathbf{x}_n \right).\]

Since

\[\nabla_{\mathbf{w}} \left( \mathbf{w}^{\top}\mathbf{x}_n \right) = \mathbf{x}_n,\]

we get

\[\nabla_{\mathbf{w}} E = \sum_{n=1}^{N} (y_n-t_n)\mathbf{x}_n,\]

where

\[y_n = \mathbf{w}^{\top}\mathbf{x}_n.\]

Or, in matrix form,

\[\nabla_{\mathbf{w}}E = \mathbf{X}^{\top} (\mathbf{X}\mathbf{w}-\mathbf{t}).\]

There is a simple way to remember the structure:

prediction error × input

Each training example pushes the parameters in proportion to its prediction error and its input features. This same form also occurs for classification (next week)

Once we can compute a gradient, we can update the parameters iteratively:

\[\mathbf{w} \leftarrow \mathbf{w} - \eta\nabla_{\mathbf{w}}E.\]

Here \eta is the learning rate, which controls the size of each update.

Using all N examples to compute one update gives full gradient descent.

For linear regression, one full-gradient update costs roughly

\[O(NM).\]

This can already be much cheaper than forming and solving the normal equations. We can go further and use only a small subset of examples, called a minibatch \mathcal{B}:

\[\mathbf{w} \leftarrow \mathbf{w} - \eta \sum_{n\in\mathcal{B}} (y_n-t_n)\mathbf{x}_n.\]

If the minibatch contains B examples, one update costs roughly

\[O(BM).\]

This is the basic idea behind stochastic gradient descent. There is a trade-off:

using fewer examples makes each update cheaper;
but each update is a noisier estimate of the direction we should move.

We will study gradient-based optimisation more carefully later. For now, the important point is that closed-form solutions are convenient for small problems, but iterative methods scale much better.

The Bigger Picture

We began with a straight line, but we have already encountered many of the central ideas in machine learning. Let's quickly revisit each one below:

Data

We observe examples

\[\mathcal{D} = \{(\mathbf{x}_n,t_n)\}_{n=1}^N.\]

Model

We choose a family of prediction functions such as

\[y(\mathbf{x};\mathbf{w}) = \mathbf{w}^{\top}\mathbf{x}.\]

Changing the features or basis functions changes what the model can represent.

Loss

We define what counts as a good model. For least squares,

\[E(\mathbf{w}) = \frac{1}{2} \sum_n (y_n-t_n)^2.\]

Optimisation

We choose parameters that minimise the objective. Sometimes this gives a direct analytic solution. For larger or more complicated problems, we instead use iterative optimisation.

Generalisation

Finally, we ask whether the learned model works on data that was not used to fit it. This last step is very important. A model is not useful simply because it remembers its training examples well. We would like to apply the model on unseen data.

Summary

In this lecture, we introduced machine learning as the problem of learning useful patterns or prediction rules from data. A useful way to organise a machine learning method is through three ingredients:

Data: what examples do we observe?
Model: what family of functions are we willing to consider?
Loss: what does it mean for one model to be better than another?

We studied these ideas through linear regression. Given a linear model

\[y(\mathbf{x};\mathbf{w}) = \mathbf{w}^{\top}\mathbf{x},\]

we measured prediction quality using squared error:

\[E(\mathbf{w}) = \frac{1}{2} \sum_{n=1}^{N} \left( \mathbf{w}^{\top}\mathbf{x}_n-t_n \right)^2.\]

Minimising this loss gives the normal equations

\[\mathbf{X}^{\top}\mathbf{X}\mathbf{w} = \mathbf{X}^{\top}\mathbf{t},\]

and, when the required inverse exists,

\[\mathbf{w}_{\mathrm{ML}} = (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top}\mathbf{t}.\]

We then extended the same method to nonlinear relationships using basis functions:

\[y(x;\mathbf{w}) = \mathbf{w}^{\top}\boldsymbol{\phi}(x).\]

This showed that a model can be nonlinear in its input while remaining linear in its parameters.

Increasing model flexibility usually reduces training error, but eventually a flexible model can begin fitting noise. This is overfitting. The central lesson is that training error is not generalisation or test error.

Regularisation gives us one way to control overfitting. Ridge regression adds a penalty on large parameter values:

\[\widetilde{E}(\mathbf{w}) = E(\mathbf{w}) + \frac{\lambda}{2} \|\mathbf{w}\|_2^2.\]

Its solution is

\[\mathbf{w}_{\mathrm{RR}} = (\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi} + \lambda\mathbf{I})^{-1} \boldsymbol{\Phi}^{\top}\mathbf{t}.\]

Finally, we saw that direct analytic solutions become costly for very large datasets. This motivates iterative optimisation using gradients, where each example contributes a term with the simple form

\[\text{prediction error}\times\text{input}.\]

These ideas recur throughout the rest of the course. The models will become more complicated, but the main questions remain the same:

What data do we have?
What model should we use?
What loss should we optimise?
How should we optimise it?
And, most importantly, will the result generalise to new data?