Knowledge Distillation

Definition #

Knowledge Distillation is the process of teaching a smaller student model to imitate a larger teacher model.

In our MoM system:

Full LLM (Teacher) → Knowledge Distillation → Surrogate Model (Student)

The goal is to make the surrogate good enough at routine decisions so MoM can use it instead of the expensive LLM.

The Problem That Led to It #

We now have our Surrogate Model, but there's a problem:

How does the small model learn what the large LLM knows?

We could train the small model from scratch, but it may not naturally behave like the large LLM.

So we need a way to transfer useful behavior from the large model into the smaller model.

What Problem It Solves #

Knowledge distillation creates that transfer:

Teacher LLM

→ produces outputs for many examples

→ Student learns those outputs/patterns

→ becomes the Surrogate Model

Easy Wording #

Knowledge distillation is teaching a small model to behave like a big model by learning from the big model's outputs.

Layman Example #

An expert teacher solves thousands of problems.

A student watches those solutions and learns how the expert approaches them.

Eventually, the student can solve many similar routine problems independently.

Technical Example #

Suppose the LLM receives:

“Vehicle speed = 50 km/h. Is this normal?”

Teacher:

LLM → “Yes, normal.”

Thousands of similar examples are generated.

Then:

Examples from Teacher → Student training → Surrogate Model

Later:

MoM → recognizes routine event → Surrogate → “Yes, normal.”

The important relationship is:

Knowledge Distillation creates/trains the Surrogate Model.

Limitation #

The student is an approximation of the teacher. It may not reproduce the teacher's performance on complex or unfamiliar situations.

That's why MoM still keeps the Full LLM available.

Solution #

Even if we have a good surrogate, some requests may be exactly or semantically similar to requests we've already processed.

That leads to Prompt Caching.