Reproduce “MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, Hua Wu

Researchers at Baidu Inc. developed MA-RLHF, a method that integrates macro actions into Reinforcement Learning from Human Feedback to train Large Language Models. This approach accelerates learning by 1.7x to 2x and boosts generation quality by up to 68% in Reward Model scores, outperforming token-level RLHF on various tasks.

Key insights

  • Integrating macro actions effectively alleviates the credit assignment problem by reducing the temporal distance between actions and rewards, leading to more stable policy gradient estimates.
  • Operating at a higher level of abstraction enables the LLM to learn and optimize for more meaningful linguistic constructs, significantly enhancing the coherence and contextual appropriateness of generated text.
  • The n-gram based termination strategy for macro actions generally proves most effective for maximizing reward scores and improving learning efficiency across various tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.