Reproduce “Two-stage LLM Fine-tuning with Less Specialization and More Generalization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit S Dhillon, Sanjiv Kumar

Researchers from UCLA and Google developed ProMoT, a two-stage fine-tuning framework for large language models that separates format learning from semantic learning. The method maintains comparable supervised task performance while significantly reducing generalization loss and, in some cases, enhancing in-context learning on unseen tasks with different output formats.

Key insights

  • Catastrophic forgetting during LLM fine-tuning is largely due to early "format specialization" where the model prioritizes learning the output style.
  • By offloading format learning to a low-capacity soft prompt in an initial stage, the main model can then focus on deeper semantic learning in the second stage.
  • Decoupling format and semantic learning allows the fine-tuned LLM to perform well on the specific task while retaining or even improving its general in-context learning on diverse, out-of-format tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.