Distillation in AI Models: How a Small Model Learns From a Giant

In early 2026, what seemed impossible happened: a small "distilled" model outperformed directly-trained models larger than itself on hard math benchmarks. This was not luck, but the result of a technique that has become the backbone of economical AI deployment: distillation. The core idea is simple and astonishing: instead of training a small model on raw data, we train it to "imitate" a giant expert model. The result is a model ten times smaller and faster, retaining more than 95% of its teacher's performance. Let's understand how this actually happens, and why it works.

The Problem Distillation Solves

Language models are getting smarter, but they are getting relentlessly heavier. A 70-billion-parameter model may deliver excellent performance, but the cost of running it (memory, latency, and the bill) devours the economics of any large-scale application. Here distillation comes as a practical solution: instead of running the giant for every request, you train a small model (like 7 billion parameters) to imitate it, dropping inference cost by a full order of magnitude (5 to 30 times depending on the case), with a speedup of about 4 times. Distillation is not an academic luxury, but the technique that makes this swap possible without a quality collapse.

The Real Secret: "Soft Targets"

Here is the essence of the idea, and a point often misunderstood. When you train an ordinary model, you give it "hard labels": this is a cat image (100%), nothing else. But the teacher model gives much richer "soft labels": "this is a cat with 90% probability, but it resembles a dog by 6%, and a fox by 3%." These secondary proportions are not noise, but "dark knowledge": they reveal how the teacher "thinks" about the relationships between concepts. The student learning from these probability distributions absorbs the structure of knowledge itself, not just the correct answer — so it generalizes far better than if it learned from raw data alone.

The more precise analogy: learning physics from an expert professor's notes that summarized and connected the concepts, rather than reading all the raw textbooks yourself. The teacher transfers not just the answers, but the "way of thinking."

The Role of "Temperature": Making Dark Knowledge Visible

An elegant technical detail makes this possible: the "temperature" parameter. The teacher's outputs are often very sharp (99.9% for one class, near-zeros for the rest), hiding the precious secondary relationships. By raising the "temperature" during distillation, we "soften" these distributions so the subtle differences emerge: the difference between 6% for the dog and 3% for the fox becomes clear for the student to learn from. This softening is what turns the teacher's outputs from an "answer" into a "lesson."

Distillation Methods: From Simplest to Deepest

Not all distillation processes are equal; there are three main approaches of increasing depth. The first is "logit-based": the student imitates the teacher's final probability distribution — the simplest and most common. The second is "feature-based": the student imitates not just the final answer, but the teacher's "internal states" (hidden layers), learning its intermediate representations — this is how the famous DistilBERT was made. The third, and most important for modern language models, is "step-by-step distillation": the student is trained to generate the same chain-of-thought the giant teacher produces, inheriting the "reasoning" rather than the answer alone.

White-Box vs. Black-Box

An important practical difference determines what you can do. In "white-box distillation," you have full access to the teacher's architecture, weights, and internal probabilities — enabling the deepest kinds of distillation, but requiring an open-weight model. In "black-box distillation," you have only the teacher's text outputs (as when you use it via a closed API). Here you do not see the internal probabilities, so you rely on the "rationales" the teacher generates as an embodiment of knowledge. This distinction is practically fundamental: it determines which methods are available to you depending on the model you distill from.

Limits You Must Know

Honesty requires mentioning the constraints. First and most dangerous: the student does not usually exceed its teacher; offline distillation binds it to the teacher's fixed knowledge, and it inherits its errors and biases directly (though cases like DeepSeek showed a student may outperform on narrow tasks). Second, contradictory teacher outputs may sometimes confuse the student. Third, and most important practically: the student must be distilled and evaluated on data representing your real production traffic, not on a generic distillation corpus that does not match your usage — otherwise you get a model that looks good in testing and fails in reality.

The practical bottom line: distillation is not merely "compressing" the model, but transferring its way of thinking via soft targets. It is today the cornerstone of the "ladder of models" teams actually deploy: a frontier model for the hardest 5% of requests, a distilled mid-size model for 80% of them, and a small model for simple paths. For the developer building AI applications, understanding distillation opens the door to sustainable economics: performance close to the giants at a fraction of the cost. But the golden rule remains: distill on your own data, and evaluate on your actual cases, for a model not tested on your reality is not ready for your reality.