Reproduce “Sample-efficient LLM Optimization with Reset Replay” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zichuan Liu, Jinyu Wang, Lei Song, Jiang Bian

Researchers from Carnegie Mellon University and Microsoft Research Asia developed LLM optimization with Reset Replay (LoRR), a framework that enhances the sample efficiency of large language model post-training. By employing periodic resets for the model, data, and loss function during high-replay training, LoRR consistently improved performance across various LLM architectures and reasoning tasks, often matching or surpassing state-of-the-art methods with substantially reduced data and computational resources.

Key insights

  • LLMs exhibit primacy bias, similar to traditional deep reinforcement learning, where over-relying on initial experiences can irreversibly damage network plasticity and generalization capabilities.
  • Aggressive data replay, while crucial for maximizing data utility, necessitates a synergistic triple reset strategy (model, data, loss) to prevent plasticity loss and manifold collapse.
  • The periodic re-initialization of model parameters and dynamic loss scheduling enables LLMs to effectively learn from limited offline data under high-replay regimes, a previously underexplored area.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.