Self-distillation lets LLMs acquire new skills without erasing old ones

Researchers from MIT, ETH Zurich and collaborators have proposed a training approach that allows a single language model to pick up new capabilities while avoiding the common fate of performance loss on earlier tasks. The method builds a closed-loop during training in which a frozen copy of the model, supplied with demonstrations, acts as a reference and the tunable copy learns from its own generated outputs compared against that reference. By leveraging modern models’ tendency to learn from examples presented in the prompt, the pipeline produces supervision signals without requiring an explicit numerical reward function. In experiments using open-weight models, the technique outperformed conventional supervised fine-tuning on task accuracy while preserving competence on previously learned domains. For example, the SDFT variant reached higher accuracy on a science question benchmark and maintained a steady score on prior tasks where standard fine-tuning degraded performance. The approach also handled synthetic “knowledge injection” tests more robustly: the tuned model could use newly taught facts in indirect reasoning, rather than merely memorizing them. Sequential training on diverse skills showed that the model could accumulate multiple capabilities without the oscillations and regressions seen under ordinary fine-tuning. Those benefits make SDFT attractive to organizations trying to avoid a growing inventory of specialized models for individual departments. The trade-offs are concrete: SDFT requires larger models that can perform in-context example-based reasoning and consumes multiple times more compute than standard pipelines, making training slower and more expensive. The team is working to integrate the method with common tooling and has published code to help adopters evaluate it. While not a free lunch, the technique provides a plausible path from brittle, static models toward systems that can adapt and keep improving through use. If the compute overhead falls and smaller models gain stronger in-context skills, SDFT could become a mainstream option for enterprises that need single, multi-capable models rather than a model zoo.