Reproduce “APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yibo Li, Jiashuo Yang, Zhi Zheng, Zhiyuan Hu, Yuan Sui, Shizun Wang, Yufei He, Bryan Hooi

Researchers from the National University of Singapore and Beijing University of Posts and Telecommunications introduced APEX, a framework enabling self-evolving LLM agents to overcome exploration collapse by building and maintaining an explicit strategy map. APEX consistently achieved higher scores than state-of-the-art baselines across Jericho text adventure games and the WebArena web interaction benchmark, demonstrating robust performance across various LLM backbones.

Key insights

  • An explicit, dynamically updated strategy map (DAG of milestones) with prerequisite dependencies is crucial for LLM agents to systematically explore and avoid behavioral narrowing.
  • Active expansion of the strategy map through "Fork Discovery" is essential for uncovering qualitatively different, high-value strategies that agents might not discover through reactive adaptation alone.
  • Variance-aware policy selection (e.g., Thompson Sampling) and DAG-based credit propagation for milestones lead to more effective exploration and accurate value attribution, especially in uncertain environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.