Concept
On-policy distillation
Version 1 — First definition, written for the AfterQuery/NVIDIA GDPval analysis (frontier-blogs).
On-policy distillation is teacher–student training in which the student learns from the teacher’s judgment of the student’s own attempts, rather than from the teacher’s demonstrations. The student generates a trajectory; the teacher scores or corrects it; the student updates toward that signal. “On-policy” is the load-bearing half: the training data is drawn from the distribution the student actually produces, so the corrections land where the student actually errs — where off-policy imitation of teacher demonstrations spends its signal on states the student would never reach.
The method’s practical appeal is that it converts a large model’s capability into a small model’s weights without needing the teacher’s training data — only its opinions. NVIDIA’s Nemotron 3 Ultra recipe is the worked example in this corpus: domain-specialised teachers distilled into one student through MOPD (multi-teacher on-policy distillation), with an SFT warmup first — and the warmup ablation is the interesting number, GDPval rising 35.3 → 46.7 with it, barely moving without it. The ordering repeats [[supervised-fine-tuning]]‘s standing lesson: distillation, like RL, needs the student already in reach of the behaviour before the on-policy signal has anything to grade.