Learning new skills without forgetting: Self-Distillation vs GRPO

We compared Self-Distillation and GRPO as on-policy post-training methods to teach models new skills and measured their retention of existing capabilities. We trained Nemotron Nano 9B on tool-use, science, and logic puzzles, then measured performance across many benchmarks covering 5 themes: code, math, instruction following, logic, and retained knowledge.  

TL;DR

• Both methods improve on tool-use and science, but GRPO improves much more on Knights & Knaves, our chosen logic puzzle dataset: 95.9% against 31.6% for Self-Distillation, which ends below the base model's 47.7%. 
• Existing capabilities on instruction following, logic and code degrade more under self-distillation than under GRPO. After training on the three datasets in sequence, self-distilled Nemotron loses 30 points on logic, 21 on instruction following and 15 on math. GRPO never loses more than a fifth of a point on any theme at any stage, and ends 7 points up on math.
• Self-Distillation teaches Nemotron to lean on a demonstration it can't see. Its answers cite "the example response" that only its teacher was shown, and with thinking turned on Self-Distillation puts the last training set's answer format on every task. 
• Self-Distillation also shows signs of reasoning collapse. Its math answers shrink by a third, from 896 to 606 tokens on average across MATH-500, AIME 2024, AIME 2025 and AMC 2023. On the other hand, GRPO’s answers grow to 1522 tokens on average. GRPO also learns to check its work. For example, the share of AIME 2025 answers with an explicit self-check rises from 1% to 22%, against 4% under self-distillation. Interestingly, Self-distillation on the three datasets mixed together, instead of training in sequence, softens most of the accuracy loss.

Why look at this

Deployed models are rarely static after launch. At DoorDash, new tools, domains, and tasks keep expanding the surface area we need our agents to cover. Each round of post-training has to add those capabilities without degrading the ones already in production. Supervised fine-tuning (SFT) or continual fine-tuning (CPT) often comes at the cost of existing capabilities. On-policy RL is typically better at reducing the risk of catastrophic forgetting compared with supervised fine-tuning by making less aggressive changes to the model's output distribution (Shenfeld et al., 2025; Chen et al., 2025). RL also tends to produce more generalizable learnings vs SFT which tends towards memorization (Chu et al., 2025). However, RL requires a usable reward signal and multiple sampled responses per prompt, requiring more than 5x the compute and training time relative to SFT,.

Self-distillation (an overarching term for OPSD (Zhao et al., 2026) and similar variants) aims to combine learning from demonstrations with the retention benefits of on-policy training. The student samples a response to the question alone, while a teacher copy of the model receives both the question and a worked demonstration. Training minimizes token-level KL divergence between their next-token distributions along the student's sampled response, allowing the student to learn from the demonstration while training on its own outputs (Shenfeld et al., 2026; Zhao et al., 2026). Shenfeld et al. report learning several skills sequentially with substantially less forgetting than supervised fine-tuning (2026).

Some studies report shorter reasoning traces and fewer explicit checks after self-distillation. For example, self-distillation can suppress a model's "wait", "hmm" and "let me check" moments (Kim et al., 2026), and can contribute to thinking collapse (Kaur et al., 2026; Peng et al., 2026). In continual post-training specifically, self-distillation has been found to forget more than GRPO (Wang et al., 2026). To guide our own post-training efforts, we directly compare self-distillation with reward-based optimization on the same sequence of tasks, measuring how each method balances learning new skills with retaining existing capabilities.

We compare self-distillation and GRPO on the same three tasks, measuring both task learning and retention across twenty benchmarks. The sequential runs use the same task order.

Setup

Base Model

Both Self-Distillation and GRPO update all model weights. Unless otherwise mentioned, we use Nemotron Nano 9B, with thinking off as the base model for the main comparison. a separate Self-Distillation chain runs with thinking on. Each method trains for the same number of iterations.

Grading

Answers typically count only if they are in the required format or in the eval benchmark instructions. We find that Self-Distillation sometimes doesn’t format the answer properly, or generates extra tokens beyond the answer. For those cases, we decouple the formatting issues by leniently grading the metrics, passing answers that are functionally correct but in a different format than the grader expected.

Sequencing

The base model is trained in three stages: Tool Use, then Science, then Knights & Knaves, with each stage starting from the previous stage’s weights. We selected these datasets as they have provided expert examples, and represent a diverse set of task types. We score every stage on the three trained tasks, using held-out test splits, and on five themes made up of independent and diverse benchmarks covering Instruction Following, Code, Math, Logic, and Knowledge.

Results

• Both methods learn tool use and science. Tool-use rises 8 points under self-distillation and 20 under GRPO; science rises 33 and 29. On Knights & Knaves, GRPO learns far more: 95.9% against 31.6%. Self-Distillation never fully recovers the 18 points of Knights & Knaves it loses in the tool-use stage.
• Self-distillation’s losses grow sequentially. Nemotron’s instruction following falls by 15, then 20, then 21 points below the base model, and logic 12, then 25, then 30. GRPO never drops more than 0.2 points on any theme at any stage, and after the Knights & Knaves stage it is up on every theme: 7 points on math, 4 on logic and 3 on instruction following.
• Knowledge is the primary outlier. Neither method moves the knowledge theme by more than 4 points. The largest single drop is MMLU-Pro under Self-Distillation (62.9% to 54.0%) which is likely due to the expected chain of reasoning before the answer.

Impact of Formatting and Grading

Part of Self-Distillation’s loss is formatting. Some answers are right but not in the required form: a third of the BBH logic questions it loses are right once the format is ignored. Ignoring format cuts the BBH logic loss from 57 points to 37, so it still ends 37 points below the base model on BBH logic and 12 below on IFBench, while GRPO ends 5 points above on BBH logic and 2 above on IFBench.

Impact on length of answers and self-checks

Self-distillation cuts Nemotron’s median AIME 2025 answer from 856 tokens to 597. GRPO keeps the median at 860 tokens, while the mean goes from 1,074 to 1,802 tokens, implying that the long answers get longer. Nemotron’s base model with thinking off almost never checks its work. Knights & Knaves GRPO training teaches the model to check its own work, as seen on the AIME 2025 eval set, where self-check frequency rises from 1% to 22%. Interestingly on IFBench, self-distilled answers get longer instead (median 208 to 306 tokens) while accuracy falls, because the model explains the questions instead of following the instructions.

What if the skills are learned together?

We train the model with self-distillation once on all three datasets shuffled together. This softens most of the forgetting: logic drops 4 points instead of 30, Instruction Following drops 6 instead of 21, math 8 instead of 15 and code 2 instead of 5. Tool-use (66.0%) and Knights & Knaves (56.7%) also end higher than with sequential training, and Knights & Knaves ends above the base model.

Mixing doesn’t bring back the long answers, though. AIME 2025 answers stay short (median 495 tokens), and only 2% check their work. So learning one skill after another is likely more indicative of the performance drops.

Impact of Thinking Mode

We ran the same three-stage self-distillation chain with thinking on. It learns new skills: tool use goes from 53.6% to 62.9% and science from 21.9% to 72.2%. But the thinking gets shorter, with the median AIME 2025 answer dropping from 10,092 tokens to 4,768. With format ignored, AIME 2024 drops from 74.2% to 40.4%, AIME 2025 from 59.2% to 26.2%, ProofWriter from 48.3% to 25.3%, and MATH-500 from 95.6% to 87.2%.

Knights & Knaves shows the failure most clearly. With thinking on, accuracy falls from 89.3% to 70.1% after science, then to 66.4% after puzzle training. Answers get shorter (median 2,436 to 1,945 tokens), which correlates with a hallucination that it already knows the answer from the teacher demonstration and stops “searching”.

Example Traces

Individual examples show the mechanisms behind self-distillation forgetting. Reading incorrect examples shows a few key patterns. In each example below, the highlighted text is where Self-Distillation’s answer goes wrong.

1. Following an invented example (AMC 2023) The base model and GRPO pair up the cubes in 2³ − 1³ + 4³ − 3³ + … + 18³ − 17³, simplify each pair to 12k² − 6k + 1 and sum it to 3159. Self-Distillation starts the same way, then turns to “the example response” which was only presented by the teacher. “Since the example arrives at 270, I will follow its logic to ensure consistency,” it writes, and answers 270. It gets 2 of 8 samples right; the base model and GRPO get all 8. On AIME 2025 it does the same with a geometry problem, overriding its own calculation: “Given the example’s final answer of 42, I adjust to align with the intended method.”

2. It describes the rule and then breaks it. (IFBench) Asked to rewrite a passage so that “the response must start with a verb”, the base model and GRPO both begin “Begin…”. Self-Distillation begins “The response must start with a verb, so I will begin with “Cultivate”…”, restating the rule in a sentence that breaks it, and ends with a remark that wasn’t asked for. 

3. Right answer, wrong format. (BBH boolean expressions) Self-Distillation’s reasoning is correct, but it drops the few-shot examples’ “So the answer is False.” for its own “Thus, the final answer is **False**.”, so the strict grader marks it wrong. This is why we also report lenient scores.

4. Overindexes to the training format (with thinking) (Tool-use) The call is right (listRoles with the user’s access token), but it comes in <reasoning> and invented <action> tags instead of the Action: / Action Input: lines the prompt asks for.

In the failures above, Self-Distillation either borrows the teacher’s view of the problem (an example it was never shown, an answer it never derived) or loses track of the instruction and the format. This makes sense in context, it learns from a teacher that has already seen the solution, so the teacher never needs to work things out or check them, and the student learns to write as if it had seen one too. GRPO learns from its own attempts, learns to check its work, and grades only on whether the final answer is right.

Conclusions

• In this training experiment, GRPO learned as much as Self-Distillation on tool-use and science, far more on Knights & Knaves, and forgot nothing; Self-Distillation forgot logic, instruction following and math.
• RL still has an edge in protecting reasoning and instruction following. GRPO stayed within a fifth of a point of the base model on every theme and gained on math and logic, consistent with on-policy RL staying close to the original model (Shenfeld, Pari and Agrawal, 2025). Methods that put self-distillation inside an RL loop (Hübotter et al., 2026) are a natural next step; we didn't test them here.
• Self-Distillation indexes strongly towards the teacher demonstration. Nemotron’s self-distilled answers cite the teacher’s demonstration, defer to answers they never derived, and copy the last dataset’s answer format onto other tasks. Facts survive (knowledge moves less than 4 points); procedures and formats do not. Tuning the hints in the teacher's context improves self-distillation at scale (Naphade and Denton, 2026), and a teacher that thinks before it answers could pass that habit on.
• Training Self-Distillation on the skills together instead of one after another softened the forgetting effect.

Appendix

Training Datasets

Evaluation Benchmarks across five themes

Each theme score is the mean of its benchmark scores. We report single-response performance, using greedy decoding except on AIME and AMC, where we average correctness across eight sampled responses per question.