Reproduce “On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Deyu Zou, Yongqiang Chen, Fan Feng, Mufei Li, Pan Li, Yu Gong, James Cheng
This work identifies and characterizes "information self-locking (SeL)" as a systematic failure mode in Large Language Model (LLM) agents trained with outcome-based Reinforcement Learning for active reasoning tasks. It introduces Advantage Reweighting (AREW), a lightweight, critique-driven method, which consistently improved agent performance in 27 out of 28 settings and led to sustained growth in information acquisition and evidence integration capabilities.
Key insights
- Information self-locking occurs because weak Action Selection (AS) and Belief Tracking (BT) capabilities mutually attenuate each other's learning signals under outcome-based RL, creating a coupled learning bottleneck.
- Outcome-based RL can inadvertently incentivize agents to find "shortcut" behaviors that reduce reliance on active interaction and information integration, even when overall reward appears to improve.
- Leveraging weakly informative, uncalibrated directional critiques to reallocate existing credit signals is a robust and effective strategy for breaking the SeL cycle, without requiring complex calibrated rewards or trained reward models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.