Reproduce “RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zehua Liu, Shuqi Liu, Tao Zhong, Mingxuan Yuan

Huawei Noah’s Ark Lab developed RIFT, a fine-tuning framework that reuses both high-quality and low-quality self-generated responses for large language model alignment, addressing the instability of negative rewards with a novel loss formulation. This method consistently improved performance on mathematical reasoning benchmarks and general preference tasks, outperforming DPO while using approximately 50% less GPU memory.

Key insights

  • Naive multiplication of negative reward signals with log-probabilities results in an unbounded loss function and exploding gradients, causing training collapse.
  • Decoupling the treatment of positive and negative samples, using a linear surrogate for negative rewards, effectively suppresses undesirable outputs without numerical instability.
  • Learning from a mixed dataset of both high-quality and low-quality self-generated samples, guided by scalar rewards, enables more comprehensive model alignment and better generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.