Reproduce “ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decoding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiran Guan, Sifan Tu, Dingkang Liang, Linghao Zhu, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu and 1 more

Researchers from Huazhong University of Science and Technology and Xiaomi Inc. developed ThinkOmni, a training-free framework that enhances the omni-modal reasoning capabilities of Large Language Models (OLLMs) by integrating textual reasoning from Large Reasoning Models (LRMs) via guidance decoding. The method consistently improves performance across benchmarks, achieving a 7.9% gain on MathVision for Qwen2.5-Omni-7B and competitive or superior results compared to reinforcement learning finetuned models.

Key insights

  • Sophisticated textual reasoning from LRMs can be successfully lifted to omni-modal scenarios in a training-free manner, bridging the gap between specialized textual reasoning and generalized multi-modal understanding.
  • A dynamic, adaptive mechanism for balancing reasoning and perceptual contributions during decoding improves robustness and performance across various omni-modal tasks compared to fixed guidance weights.
  • Simple fusion methods like logit averaging or one-way information transfer are ineffective for integrating reasoning, highlighting the need for a more nuanced, collaborative logit-mixing approach.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.