Reproduce “Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models” on OpenResearch

About this paper

Wen-Tse Chen, Jiayu Chen, Fahim Tajwar, Hao Zhu, Xintong Duan, Ruslan Salakhutdinov, Jeff Schneider

Researchers from Carnegie Mellon University, The University of Hong Kong, and Stanford University developed Retrospective In-Context Learning (RICL) and Retrospective In-Context Online Learning (RICOL) to enable large language models (LLMs) to perform temporal credit assignment from sparse rewards. The method converts sparse environmental feedback into dense advantage signals, demonstrating up to 100 times greater sample efficiency in advantage estimation and outperforming existing LLM baselines in online learning tasks with substantially fewer environment steps.

Key insights

• Pretrained knowledge within LLMs can be leveraged to implicitly estimate advantage functions by observing policy changes induced by in-context feedback, thereby transforming sparse environmental rewards into dense supervision.
• Applying in-context updates retrospectively to states just encountered in a trajectory significantly enhances the reliability and effectiveness of policy improvement compared to standard in-context learning approaches.
• An iterative online learning framework, guided by these implicitly derived advantage functions and a trust region mechanism, enables LLM agents to learn from sparse environmental feedback with high sample efficiency and robustness to noisy input.

Summary by alphaXiv.