Reproduce “Multi-agent In-context Coordination via Decentralized Memory Retrieval” on OpenResearch

About this paper

Tao Jiang, Zichuan Lin, Lihe Li, Yi-Chen Li, Cong Guan, Lei Yuan, Zongzhang Zhang, Yang Yu and 1 more

The MAICC framework introduces the first In-Context Reinforcement Learning (ICRL) approach designed for Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs), enabling multiple agents to rapidly coordinate and adapt to unseen cooperative tasks without requiring parameter updates. It consistently achieves higher performance compared to existing methods on cooperative benchmarks like Level-Based Foraging and StarCraft Multi-Agent Challenge.

Key insights

• Successfully extending ICRL to complex, cooperative multi-agent systems requires specialized designs to manage partial observability and credit assignment, which existing single-agent ICRL methods overlook.
• Learning robust trajectory representations for multi-agent coordination benefits from optimizing a Centralized Embedding Model with behavioral, reward, and transition dynamics losses, while deliberately omitting Return-To-Go (RTG) tokens to avoid irrelevant retrievals.
• Distilling knowledge from a centralized model to decentralized agent-specific models is crucial for individual agents to infer team-level coordination characteristics from their local observations.
• Dynamically balancing broad offline data with limited online experiences through a selective memory mechanism provides a robust strategy for efficient exploration and task-specific exploitation during rapid adaptation.

Summary by alphaXiv.