Reproduce “Memory-Driven Self-Improvement for Decision Making with Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xue Yan, Zijing Ou, Mengyue Yang, Yan Song, Haifeng Zhang, Yingzhen Li, Jun Wang

A framework integrates Large Language Model general knowledge with domain-specific experiences for sequential decision-making by combining memory-driven value estimation and LLM prior refinement. This approach, developed by researchers from CAS, UCAS, Imperial College London, UCL, and University of Bristol, demonstrates over 40% performance improvement on complex ALFWorld environments and a 75% gain on unseen tasks compared to pretrained LLMs.

Key insights

  • Integrating a compact, dynamically updated memory of domain-specific experiences with LLM priors can effectively bridge the gap between general and specialized knowledge for robust decision-making.
  • Non-parametric Q-value estimation, leveraging LLM embeddings for similarity matching, significantly enhances sample efficiency and exploration compared to traditional RL methods.
  • Refining the LLM's internal prior knowledge through an EM-based, memory-driven process is crucial for robust generalization to unseen tasks, rather than solely relying on fixed LLM priors or Q-value estimators.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.