Reproduce “ProPlay: Procedural World Models for Self-Evolving LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yijun Ma, Zehong Wang, Yiyang Li, Ziming Li, Xiaoguang Guo, Weixiang Sun, Chuxu Zhang, Yanfang Ye
ProPlay enables large language model agents to continuously improve their understanding of environment dynamics and self-evolution capabilities in partially observable environments. It achieves this by constructing and evolving a procedural world model, represented as an attributed directed graph, which supports a preplay mechanism for guided exploration and learning, leading to superior performance across various interactive benchmarks.
Key insights
- Abstracting environment dynamics into a procedural world model captures higher-level causal structures more effectively than action-level models, leading to better generalization and a more comprehensive understanding.
- The 'soft guidance' principle, allowing agents to deviate from preplay plans, is crucial for balancing the exploitation of learned knowledge with necessary exploration in novel or dynamic situations.
- Integrating reliability record embeddings for transitions enables the agent to dynamically assess the trustworthiness of past experiences, promoting the utilization of generalizable knowledge and mitigating misleading information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.