Reproduce “Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Narjes Nourzad, Carlee Joe-Wong
Researchers from USC and CMU introduced "Memory-Based Advantage Shaping," an RL method that leverages a dynamic memory graph built from offline LLM priors and successful agent experiences to refine advantage estimates. This approach matches state-of-the-art performance in complex environments like DOORKEY while substantially reducing the need for continuous LLM supervision.
Key insights
- Shaping the advantage function with memory-derived utility provides effective guidance without altering the environment's reward signal, preserving theoretical convergence guarantees of the base RL algorithm.
- An adaptive, memory-centric approach to LLM utilization significantly reduces the need for continuous, costly LLM supervision while maintaining high performance.
- Dynamically constructing and evolving a memory graph allows for the consolidation and refinement of task knowledge from both LLMs and agent experiences, leading to more robust guidance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.