Reproduce “Cheap Talk Discovery and Utilization in Multi-Agent Reinforcement Learning” on OpenResearch

About this paper

Yat Long Lo, Christian Schroeder de Witt, Samuel Sokota, Jakob Nicolaus Foerster, Shimon Whiteson

This work from the University of Oxford and Carnegie Mellon University introduces a two-stage framework, CTDUL, enabling multi-agent reinforcement learning agents to first discover where communication channels exist in an environment and then learn effective communication protocols. The approach leverages mutual information for discovery and achieves superior performance in novel maze environments where baselines fail to locate communication channels.

Key insights

• Decomposing the problem into sequential discovery and utilization phases (CTD and CTU) provides an effective strategy for tackling communication in realistic MARL settings.
• Mutual Information (MI) maximization serves as a powerful intrinsic motivation, guiding agents to discover functional communication channels by rewarding states where their actions influence others' observations.
• Off-Belief Learning (OBL) is crucial during discovery, allowing agents to identify communication opportunities without forming specific protocols prematurely, which ensures flexibility and robustness.

Summary by alphaXiv.