Reproduce “psRL: Efficient Training for Agentic AI via Training-Time Prefix Sharing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mianjie Yu, Zizhao Mo, Huanyu Qu, Zhirong Qian, Huanle Xu, Cen Li, Zifeng Zhao, Zhi Zhou and 3 more
Researchers at the University of Macau developed psRL, a training system for agentic AI that leverages training-time prefix sharing to address the update phase bottleneck in modern reinforcement learning. It achieves up to 5.2x speedup in update throughput and a 2.1x end-to-end speedup for agent training, while reducing peak memory consumption by 41% for large industrial models.
Key insights
- The shift to structured sampling strategies in agentic RL training has fundamentally changed the performance bottleneck from the rollout phase to the update phase.
- Training samples generated by advanced RL methods exhibit significant prefix redundancy (often exceeding 90%), which can be effectively exploited to reduce computational and memory overhead during loss calculation and backpropagation.
- Adapting prefix-sharing and dynamic memory management principles, previously successful in LLM inference, to the training context with global visibility and data immutability offers substantial efficiency gains for RL training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.