Reproduce “Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyan Liu, Zhezheng Hao, Yeqiu Chen, Hong Wang, Jingren Hou, Ruiyi Ding, Yongkang Yang, Wence Ji and 2 more
Researchers from the University of Science and Technology of China and Tencent introduced Meta-Cognitive Memory Policy Optimization (MMPO) to improve long-horizon reasoning in LLM agents. This method leverages a self-supervised "Belief Entropy" signal to provide dense, memory-specific supervision, reducing semantic noise accumulation in recursive summarization and enhancing task performance.
Key insights
- Memory optimization for LLM agents should prioritize preserving the clarity of the "summary-induced belief" at each step, not just the final task outcome.
- A "Belief Entropy" proxy, estimated by querying the model's intrinsic uncertainty about the task state via a dual-probe anchor question, reliably quantifies intermediate memory quality.
- Integrating this dense, memory-specific supervision with global outcome rewards through a novel turn-level advantage estimation mechanism enables effective credit assignment for memory policy updates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.