Reproduce “Replay for LLM Continual Learning with Multi-Teacher On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiqiao Qiu
MOPD trains a student to add specialist capabilities without retaining earlier experts or stage snapshots. At each stage, it learns from a new expert and replays earlier-domain prompts, with the previous-stage student teaching generated responses. In a five-stage experiment using a 1.5B student, fixed-share token replay scored 48.95 across 14 benchmarks, near joint MOPD’s 49.02; evidence is limited to one student and domain order.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.