Reproduce “From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shixin Jiang, Jiafeng Liang, Jiyuan Wang, Xuan Dong, Heng Chang, Weijiang Yu, Jinhua Du, Ming Liu and 1 more

Researchers from Harbin Institute of Technology, Huawei Inc., and Peng Cheng Laboratory present the first comprehensive survey of Omni-modal Large Language Models (Omni-MLLMs), which integrate language with arbitrary combinations of non-linguistic modalities. The work taxonomizes their architectures, training, data, and evaluation, identifying current challenges and future directions for unified multimodal understanding and generation.

Key insights

  • Omni-MLLMs leverage diverse encoding strategies (continuous, discrete, hybrid) and alignment mechanisms (projection, embedding) to integrate multiple non-linguistic modalities into an LLM-centric interaction hub.
  • A two-stage training paradigm, beginning with multi-modal alignment pre-training followed by instruction fine-tuning, is critical for enabling robust cross-modal understanding and generation capabilities.
  • Significant challenges persist in data scarcity for low-resource modalities, the creation of comprehensive cross-modal instruction data, and the establishment of rich, unified benchmarks for evaluating Omni-MLLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.