Training
Knowledge Distillation
Also known as: model distillation
Training a smaller, cheaper "student" model to imitate a larger "teacher" model, keeping much of the quality at a fraction of the cost.
Distillation is teaching a small AI to copy a big one. You take a large, expensive model (the "teacher") and use it to train a much smaller model (the "student") to give similar answers. The student ends up far cheaper and faster to run while keeping a lot of the teacher's skill, handy for putting capable AI on phones or into low-cost services.
Knowledge distillation transfers capability from a large teacher model to a smaller student model. Instead of (or alongside) learning from raw labels, the student learns to reproduce the teacher's outputs, which carry richer information than a plain right/wrong answer.
Why it is useful
- Efficiency: a distilled student is cheaper and faster at inference than the teacher.
- Deployment: enables strong models on constrained hardware.
- It complements quantization and LoRA as a way to make capable models practical.
In the LLM era
A common recipe is to generate high-quality responses (or reasoning traces) with a strong model and fine-tune a smaller one on them, a major reason capable small language models exist today.
Knowledge distillation trains a student to match a teacher's output distribution, classically the softened logits (temperature-scaled soft targets), which convey inter-class structure beyond hard labels and improve generalisation (Hinton, Vinyals & Dean 2015). Variants match intermediate representations or attention maps. The result is model compression: comparable quality at lower parameter count, latency and inference cost.
For LLMs, sequence-level and data distillation dominate: a strong teacher generates instruction responses or chain-of-thought traces on which a smaller model is fine-tuned, and reasoning distillation transfers deliberate problem-solving into compact students. This is a principal driver of capable small language models. Caveats include inheriting the teacher's errors and biases, a quality ceiling set by the teacher, and licensing/terms constraints when distilling from proprietary models. It composes with quantization and PEFT for end-to-end efficient deployment.