Reproduce “HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft and 3 more

NVIDIA researchers introduce HELPSTEER, a multi-attribute human feedback dataset for Large Language Models, which decomposes helpfulness into specific dimensions like correctness, coherence, and complexity. Utilizing this dataset with the STEERLM alignment technique, their Llama 2 70B model achieves state-of-the-art helpfulness and factual accuracy among open models, outperforming Llama 2 70B Chat, while significantly reducing training compute.

Key insights

  • Decomposing human helpfulness into specific attributes like correctness, coherence, complexity, and verbosity provides more transparent and actionable feedback for LLM alignment.
  • Explicitly training models on these granular attributes can lead to improved factual accuracy and coherence, mitigating issues like models optimizing for spurious correlations (e.g., length).
  • Multi-attribute alignment via STEERLM achieves superior performance compared to traditional RLHF while significantly reducing training compute by approximately 5 times.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.