Reproduce “ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Diyi Hu, Ruizhong Qiu, Lin Qiu, Hanghang Tong, Hanqing Zeng
ProbePO redistributes credit across segments of mathematical reasoning by checking a limited number of intermediate prefixes during reinforcement learning. It shrinks probe-based value estimates toward the group reward mean, then adds a centered segment-level adjustment to GRPO’s response-level advantage. On DeepSeekMath-7B-SFT, it reaches 19.34% Avg-6 versus GRPO’s 17.62%; evidence is limited to mathematical reasoning in three 1.5B–7B models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.