Reproduce “PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuanqiang Yu, Yanzhao Zheng, Zhentao Zhang, Tianze Xu, Chao Ma, Jihuai Zhu, Jiashun Liu, Xinle Deng and 3 more

A new online curriculum learning method, Progress-Augmented Advantage Curriculum (PAC), enhances multi-task reinforcement learning post-training for large language models (LLMs). PAC improves both sample efficiency by up to 1.6x and final performance by up to 13.1% by adaptively allocating rollout budget based on task utility.

Key insights

  • Efficient multi-task RL post-training for LLMs requires jointly considering both a task's potential to induce policy updates and its observed impact on actual reward improvement.
  • Adaptive reallocation of training budget based on decaying reward-derived progress signals helps prevent over-investment in tasks whose performance has saturated.
  • A Bayesian controller, incorporating uncertainty through variance inflation, provides a principled mechanism for robust exploration and exploitation in dynamic task utility environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.