Reproduce “LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaoguang Wu, Zhi Zheng, Hui Xiong

LLM-ALSO introduces an adaptive framework for optimizing learning signals in sparse-reward multi-agent reinforcement learning (MARL), employing a dual-LLM engine to diagnose training failures and propose structured reward-shaping configurations. This approach iteratively validates candidate updates, demonstrating improved average sparse return and learning efficiency on Level-Based Foraging tasks compared to static or single-LLM baselines.

Key insights

  • Adaptive, stage-aware optimization of learning signals, continuously refined through training feedback, is more effective for improving sparse-reward cooperative MARL performance than fixed or one-shot reward shaping.
  • A dual-LLM architecture, separating the roles of diagnosing training failures (Critic) and generating structured reward-shaping hypotheses (Generator), enhances the quality and relevance of adaptive learning signals.
  • Empirical validation of LLM-generated reward-shaping candidates through short-horizon training branches is crucial for ensuring their positive impact, mitigating risks, and providing a controlled mechanism for integrating LLM guidance into complex RL systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.