Measuring a Model, and the Two Linear Ones · Ground Truth: AI from First Principles

With a protocol in place, the question becomes what exactly to measure — and a badly chosen metric will happily certify a useless model. This lesson works through the regression and classification metrics, then introduces the first two algorithms of the course: linear regression, and the classifier that borrows its name.

8. Quality metrics

Consider the performance measure $P$ in more detail. The metric defines the criterion by which one model is judged better than another, so it is chosen before the measurement and according to the cost of a mistake in the problem at hand.

8.1. Regression

MAE = \frac{1}{n} i = 1 \sum n ∣ y_{i} - \overset{y}{^}_{i} ∣, MSE = \frac{1}{n} i = 1 \sum n (y_{i} - \overset{y}{^}_{i})^{2}, RMSE = MSE,

R^{2} = 1 - \sum _{i} ( y _{i} - y

MAE and RMSE differ in how they treat large deviations. MAE averages the absolute error and weighs every deviation equally, so ten errors of size one are the same to it as one error of size ten. RMSE squares the errors, so a single large error is punished out of all proportion. Which to choose depends on what is more critical in the problem. Where one catastrophic error is unacceptable, use RMSE; where typical accuracy is what matters, use MAE. The $R^{2}$ statistic describes how much better the model is than the trivial forecast that always returns the mean. $R^{2} = 0$ means no advantage at all, $R^{2} = 1$ a perfect forecast.

8.2. Classification: the confusion matrix and what comes from it

Classification measures rest on the four entries of the confusion matrix: correctly identified positives (TP) and negatives (TN), plus the two kinds of error — the false alarm (FP) and the miss (FN).

Figure 1.10: open on its own ↗

Two key metrics are defined from those four quantities:

Precision = \frac{T P}{T P + F P}, Recall = T P + F N

Their relationship is easiest to explain with a fishing net. Recall is the share of the target class the model found, out of all of it there was. Precision is the share of genuine target-class objects among everything the model assigned to that class. Casting a wide net raises recall but lowers precision, because it drags in things that do not belong. A narrow one raises precision but lowers recall, because part of the target is missed.

Figure 1.11: open on its own ↗

This relationship explains the limits of accuracy, the plain share of correct decisions. Take the diagnosis of a disease with a prevalence of 1 %. A model that calls everyone healthy has 99 % accuracy and no practical value at all, because its recall is zero and it has found no patient. The metric is therefore chosen according to the cost of the error. In cancer screening recall is critical, because missing a patient is more dangerous than an unnecessary extra check. In spam filtering precision comes first, because losing an important message is worse than one advert getting through. The $F_{1}$ score is the harmonic mean of the two, and is used when both sides of the error matter equally.

A classifier moreover usually outputs not a class but a probability, and the class follows from comparing it with a threshold. Moving the threshold smoothly trades precision against recall. The quality of the model across every possible threshold at once is described by the ROC curve and the area under it (AUC).

Figure 1.12: open on its own ↗

Having covered how to measure objectively, we can turn to the algorithms themselves. What matters is that different algorithms divide the same data differently. The example below shows four algorithms drawing a boundary between two classes on one set of points — as a straight line, a winding curve, or a set of rectangular regions.

Figure 1.13: open on its own ↗

9. Linear regression

This model is simple, and that is exactly why a project should start with it. The prediction is a weighted sum of the features:

\overset{y}{^} = w_{1} x_{1} + w_{2} x

Each weight $w_{j}$ describes the contribution of its feature — how much the prediction changes when that feature rises by one unit. Training means choosing the weights $w$ and the bias $b$ that minimise the mean squared error $L (w) = \frac{1}{n} ∥ X$ . Geometrically the line can be pictured as the equilibrium of a system of springs, each data point tied to the line and pulling the harder the further away it is. The line settles where the total tension is least — that is, where the sum of squared distances is smallest.

Figure 1.14: open on its own ↗

This problem has an exact solution, the normal equation:

w^{*} = (X^{⊤} X)^{- 1} X^{⊤} y .

Gradient descent is used anyway because inverting the matrix $X^{⊤} X$ with thousands of features is computationally expensive and sometimes impossible. The universal instrument is therefore gradient descent, an idea of fundamental importance that recurs throughout the course. Picture the loss function as hilly terrain and the optimisation as a descent into the valley: at each step you find the direction of steepest decrease and take a step that way. Formally that is a step against the gradient:

w \leftarrow w - η \nabla_{w} L (w),

where $η$ is the learning rate, the length of the step. Choosing it takes balance. Too large an $η$ produces steps that overshoot, oscillation around the minimum, and divergence. Too small a one approaches the minimum far too slowly.

Figure 1.15: open on its own ↗

With a well-chosen rate the dynamics of convergence show as movement toward the minimum in which the steps are large on the steep slope and shrink near the bottom as the gradient falls away.

Figure 1.16: open on its own ↗

Protection against overfitting comes from regularisation — a penalty on excessively large weights, since a model with large weights is usually unstable and overfitted:

L_{ridge} = MSE + α j \sum w_{j}^{2}

The difference between them matters in practice. L2 (ridge) shrinks every weight evenly toward small but non-zero values. L1 (lasso) drives weak weights to exactly zero, effectively removing the useless features from the model. Lasso therefore performs feature selection as well, telling you which features actually matter.

10. Logistic regression

Despite the word "regression" in its name, this model is a classifier. The idea is to take the same linear combination $z = w^{⊤} x + b$ , which may be any real number, and pass it through the sigmoid — a function that maps the whole number line into the interval $(0, 1)$ and turns an abstract number into a probability:

σ (z) = \frac{1}{1 + e ^{- z}}, P (y = 1 ∣ x) =

The sigmoid can be pictured as a smooth switch: for large negative arguments it gives a probability close to 0, for large positive ones close to 1, and in between it changes gradually. A threshold of 0.5 turns that probability into a final class.

Figure 1.17: open on its own ↗

The model is trained by minimising the logistic loss (log-loss):

L = - \frac{1}{n} i = 1 \sum n [y_{i}

What matters is the character of this function. Log-loss punishes not the mistake itself so much as confidence in the mistake. A wrong prediction made with moderate confidence costs little; a wrong prediction made with high confidence costs sharply more. The penalty climbs steeply as the model assigns a small probability to the correct class.

Figure 1.18: open on its own ↗

This property makes logistic regression not merely a classifier but a calibrated model, whose probabilities can be trusted. For $K$ classes the sigmoid is replaced by softmax, covered in detail in the chapter on neural networks.