Reproduce “MixSD: Mixed Contextual Self-Distillation for Knowledge Injection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan, Zhijing Jin, Mona Diab
MixSD adds new facts to language models while aiming to retain capabilities. It generates target sequences by mixing tokens from the base model conditioned on the fact and from the model given only the prompt, then fine-tunes with next-token loss. On KGFACT-SMALL, Qwen3-8B with λ=0.5 reached 97.0% training recall and 72.4% held-out average, versus 72.2% for the base model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.