Reproduce “Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, Yongbin Li

Research quantitatively compares the impact of prompt diversity versus response diversity on large language model human alignment. It finds that increasing the diversity of responses per prompt yields greater performance improvements for a fixed annotation budget and introduces a new prompt diversity metric that correlates linearly with alignment performance.

Key insights

  • Large Language Models benefit more from fine-grained preference signals provided by diverse responses per prompt than from a wider range of unique prompts.
  • Simply counting prompts is insufficient for measuring diversity; linguistic features like N-grams are crucial for a predictive diversity metric.
  • A clear linear correlation exists between the newly formulated prompt diversity metric and the final alignment performance of fine-tuned LLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.