Foundations
Self-supervised LearningSSL
Also known as: self-supervision
Learning from unlabelled data by having the data provide its own answers, for example hiding part of a sentence and predicting it. The engine behind modern foundation models.
Self-supervised learning is a clever way to learn from data that has no answer key: you make the answers out of the data itself. Hide a word in a sentence and let the model guess it; the sentence already "knows" the answer. Because you can do this on any text or image without humans labelling anything, it lets AI learn from enormous amounts of data cheaply. It is the trick that made today's big AI models possible.
Self-supervised learning (SSL) trains a model on unlabelled data by automatically generating the training targets from the data. Classic examples: predict the next token in text, or fill in a masked-out word. No human labels are needed, the data supplies its own supervision.
Why it changed everything
- Scale: unlabelled data (the internet, book archives) is almost unlimited, unlike costly hand-labelled data.
- Generality: the model learns broad representations useful for many later tasks.
It sits between supervised (needs labels) and unsupervised (no targets at all) learning, and is the basis of pretraining for LLMs and other foundation models.
Self-supervised learning constructs a supervised objective from unlabelled data via a pretext task, next-token prediction (autoregressive), masked-token prediction (BERT-style; Devlin et al. 2018), or contrastive/joint-embedding objectives for vision, learning representations transferable to downstream tasks. It is the mechanism that made internet-scale pretraining tractable, decoupling capability from expensive human annotation.
Formally supervised in mechanism but label-free in data, SSL is often grouped under unsupervised learning. Its efficacy underpins the foundation-model paradigm: broad self-supervised pretraining followed by lightweight adaptation. Design questions concern the pretext task's alignment with downstream needs, data quality/deduplication, and (for contrastive methods) negative sampling and representation collapse. In language it is essentially universal; in vision, masked-autoencoding and contrastive SSL now rival supervised pretraining.