Reproduce “The Hanabi Challenge: A New Frontier for AI Research” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin and 7 more

DeepMind and Google Brain researchers propose Hanabi as a new AI benchmark to challenge systems in cooperation, imperfect information, and reasoning about others' beliefs. Their findings indicate current learning agents fall short of hand-coded bots in complex cooperative play and demonstrate extreme brittleness when adapting to unknown teammates.

Key insights

  • Learning agents struggled to converge to consistent, high-performing conventions in self-play, indicating the difficulty of coordinating in games requiring complex implicit communication.
  • The severe performance degradation of learned agents in ad-hoc teams highlights their brittleness and inability to generalize or adapt to different partners, a critical limitation for multi-agent AI.
  • The success of Bayesian Action Decoder (BAD) in two-player Hanabi suggests that explicit modeling of agents' beliefs (theory of mind) is crucial for achieving high performance in this domain.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.