Foundations & Models
Distillation
A large model plays teacher: its answers become training material for a smaller one that is supposed to answer similarly well but costs a fraction. In this way, capabilities migrate from expensive top-tier models into everyday models. For operations this is central — few tasks need the largest model; they need one that is good enough at the right price.
An example
A large model answers a customer inquiry superbly, but costs so much per request that the use case does not pay off. So you have it answer a hundred thousand typical inquiries and train a small model to reproduce exactly those answers. The result is almost as good for this one task and many times cheaper. Outside the task it drops off noticeably — it has not taken over the ability, but the behavior in one segment.
A closer look
The small model is not trained on the correct answers but on the output distribution of the large one. That contains more information than a bare label, namely also which alternatives the large model considered close together. That is why comparatively few examples suffice.
Why it matters
Distillation is the usual path from a working pilot to affordable operation, and it explains why small models sometimes score surprisingly well in benchmarks. But it has two catches that seldom appear in the sales pitch. One is legal: the terms of use of the large providers often explicitly prohibit using their outputs to train competing models. The other is substantive: a distilled model inherits the teacher's errors without inheriting its breadth. It is reliable exactly where someone has checked.
Commonly misunderstood
The distilled model inherits its teacher's errors. It is also usually narrowly confined to the distilled task area and drops off noticeably outside it.