Reproduce “MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rohan Surana, Xintong Li, Sheldon Yu, Yiran Jenny Shen, Chuhan Wang, Tong Yu, Prithviraj Ammanabrolu, Jingbo Shang and 2 more

MASS-DPO introduces a D-optimal design strategy for selecting informative negative samples in multi-negative Direct Preference Optimization, improving large language model alignment efficiency and performance. It achieves superior accuracy and ranking metrics across various benchmarks and models while using substantially fewer negative samples during training, reducing computational costs.

Key insights

  • Maximizing the determinant of a weighted covariance of feature differences (D-optimal design) promotes selecting negative subsets whose feature differences span diverse and orthogonal directions in the parameter space, leading to more informative policy updates.
  • Strategically selecting a compact subset of 'hard-to-rank' negative samples, guided by Fisher information, provides a richer supervisory signal per gradient step than using a larger pool of randomly sampled negatives.
  • The Fisher geometry for negative selection remains highly stable throughout training, allowing for a one-time preprocessing selection and reducing the need for dynamic re-selection.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.