Reproduce “Intrinsic Credit Assignment for Long Horizon Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ilze Amanda Auzina, Joschka Strüber, Sergio Hernández-Gutiérrez, Shashwat Goel, Ameya Prabhu, Matthias Bethge
The ∆Belief-RL framework enables large language models to navigate uncertainty over long-horizon interactions by leveraging intrinsic belief changes for credit assignment. This method allows models to efficiently resolve uncertainty and achieve higher success rates on information-seeking tasks, outperforming significantly larger models and standard reinforcement learning approaches.
Key insights
- Changes in an agent's internal belief, measured as log-probability shifts towards the correct concept, serve as an effective and scalable intrinsic reward signal for granular credit assignment in multi-turn interactions.
- Combining sparse verifiable rewards with dense intrinsic belief-based rewards leads to more efficient exploration and faster resolution of uncertainty, enabling agents to learn better information-seeking strategies.
- Training with ∆Belief-RL fosters general-purpose information-seeking capabilities that robustly generalize to out-of-distribution tasks, different user simulators, and extended interaction horizons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.