Reproduce “Towards Continual Learning: Mitigating Forgetting With Subspace Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon and 1 more

Subspace Tuning (SST) constrains fine-tuning updates to reduce damage to a model’s capabilities. It protects dominant singular directions in weight matrices and projects the optimizer’s update into the remaining subspace. On Qwen3-8B trained on Science for five epochs, SFT+SST reached 85.4% accuracy and a five-benchmark average of 80.1, versus 76.1% and 47.1 for SFT, in that setup.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.