Reproduce “SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jintao Tong, Shilin Yan, Hongwei Xue, Xiaojun Tang, Kunyu Shi, Guannan Zhang, Ruixuan Li, Yixiong Zou

SwimBird introduces a Multimodal Large Language Model (MLLM) capable of dynamically switching between text-only, vision-only, and interleaved reasoning modes, enabled by a hybrid autoregressive formulation and a resolution-aware dynamic latent token budget. This model, trained on the SwimBird-SFT-92K dataset, achieves state-of-the-art performance on various benchmarks requiring both fine-grained visual understanding and complex multimodal reasoning.

Key insights

  • MLLMs can achieve greater versatility and efficiency by dynamically adapting their internal reasoning process (text-only, vision-only, or interleaved) to the specific demands of a multimodal query.
  • A unified autoregressive framework that predicts both next textual tokens and next continuous visual embeddings allows for flexible and learnable control over reasoning modality.
  • Adaptive allocation of latent visual computation, rather than fixed-length visual thoughts, is crucial for optimizing performance on both vision-dense and text-centric tasks by preventing both under-provisioning and redundant computation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.