Encoder, bottleneck, decoder
Squeezing an image through a bottleneck
An autoencoder compresses its input to a handful of numbers and then tries to rebuild it. Slide the bottleneck below and watch how much survives. The basis it compresses with is genuinely learned from this image by principal component analysis — which is exactly what a linear autoencoder converges to.
The image is cut into 8×8 patches. Each patch — 64 numbers — is projected onto the learned basis, all but the first k coefficients are thrown away, and the decoder rebuilds it from what is left.
Untick to use a fixed, generic basis (the DCT, as used by JPEG) instead of one learned from this image. The learned basis wins on the image it was trained on — and would lose on a different one. That is what a training set buys, and what it costs.
The latent vector for one representative patch. Greyed bars are the coefficients the bottleneck discards.
Error falls fast at first and then flattens. The early components carry most of the variance in the data, so the first few numbers do nearly all the work — which is precisely why compression is possible at all.
Ordered by how much of the data's variance each explains. The first is essentially flat — the patch's average brightness. Those after it are progressively finer patterns of light and dark. Bright ones are in use at the current bottleneck; dimmed ones are discarded.
This really is an autoencoder — the simplest possible one. The encoder is a matrix multiply that turns 64 numbers into k. The decoder is another that turns k back into 64. It is trained to minimise reconstruction error, exactly as the notes describe. And there is a theorem here worth knowing: a linear autoencoder trained to minimise squared error learns to span the same subspace as the top k principal components. So rather than run gradient descent and hope, this demo computes that optimum directly. Nothing is being faked — this is the answer the training would converge to.
The bottleneck is the entire point. If the latent layer were as wide as the input, the network could learn the identity function and achieve zero error while learning nothing whatsoever — set k to 64 and see the error vanish along with any insight. Forcing the representation through a narrow layer is what compels it to find structure: with only eight numbers per patch, it must discover what images actually tend to look like.
What a real autoencoder adds is non-linearity. This one can only take weighted sums, so it is confined to a flat subspace. Insert ReLUs between layers and the model can bend that subspace into a curved surface, hugging the true distribution of natural images far more tightly and reconstructing far more from the same k numbers. Make the layers convolutional as well, and you have the architecture the notes describe — and the family DLSS belongs to.
Compare the two bases. The DCT is fixed: it knows nothing about your image, and JPEG has used it for thirty years precisely because it need not be transmitted. The learned basis is tailored to this image and does measurably better on it — but it must be stored, and it will do worse on anything unlike its training data. Every learned model in this chapter makes that same trade.