Supervised learning, by hand

Classification and regression

Supervised learning means: here are inputs, here are the right answers, find the function that connects them. Click on the plot to add labelled examples and watch a model fit them live. The two tasks differ only in what the answer looks like — a category, or a number.

Controls —
5

Click the plot to place an example of the selected class. The shaded background is the model's prediction at every point — which is what "learning a function" means: not memorising the dots, but assigning an answer everywhere.

The data —
—
Examples
—
Parameters
—
Training error
—
Test error
—
Error against model complexity every degree, fitted to this same data
training error — always falls test error — the one that matters best degree on test data

This is the notes' question, exactly. Given only pairs of x and y, find F. Set the degree to 2 and the model is looking for F(x) = ax² + bx + c — it recovers a, b and c by minimising the squared distance to the data, which is a solvable linear system with a closed-form answer. No iteration, no gradient descent, no learning rate. Supervised learning is not always mysterious.

Degree is the model's capacity, and more is not better. With nine coefficients and ten points, a polynomial can pass through every single one and drive training error to zero. It has not learned the pattern; it has memorised the noise. The error curve above shows both quantities at once — training error sliding monotonically towards zero while test error bottoms out and then climbs away. The lowest point on the red curve is the model you actually want.

Classification is the same exercise with a different output. Instead of a number, the answer is one of a finite set of labels, so instead of a curve through the points the model draws a boundary between them. Logistic regression can only draw one straight line, so it fails cleanly on data that is not linearly separable; k-NN can draw any shape at all, and at k = 1 it wraps itself around every last point — overfitting again, wearing different clothes.

Both tasks appear in this chapter. Super-resolution is regression: the input is a low-resolution patch, the output is a continuous set of pixel values. Recognising the handwritten digit in the notes is classification: the output is one of ten labels.