Reproduce “Metropolis-Hastings Captioning Game: Knowledge Fusion of Vision Language Models via Decentralized Bayesian Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuta Matsui, Ryosuke Yamaki, Ryo Ueda, Seitaro Shinagawa, Tadahiro Taniguchi
A Metropolis-Hastings Captioning Game (MHCG) fuses knowledge from diverse Vision-Language Models (VLMs) by reinterpreting model combination as a decentralized Bayesian inference problem, achieving improved, semantically plausible image captions while mitigating catastrophic forgetting and avoiding increased inference costs.
Key insights
- VLM knowledge fusion can be effectively achieved through a decentralized Bayesian inference process, drawing inspiration from emergent communication language games, which offers a principled alternative to ad-hoc methods.
- The Metropolis-Hastings acceptance-rejection mechanism plays a critical role in filtering semantically plausible captions, thereby preserving agents' original knowledge and facilitating robust knowledge transfer.
- Combining LoRA for efficient parameter updating and DER++ for experience replay successfully counteracts catastrophic forgetting, enabling agents to retain their initial expertise while integrating new vocabulary and semantic understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.