Reproduce “Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer” on OpenResearch

About this paper

Yanan Li, Christina Yi Jin, Yuan Jin, Manli Luo, Tie Xu, Shuai Jiao, Wei He, Qing Zhang

Researchers at Zhejiang Lab characterize the "modality gap" in Multimodal Large Language Models (MLLMs), theoretically proving that native unified-vocabulary architectures inherently achieve and preserve a "zero-gap" state. Their proposed "One Tokenizer" architecture, implemented for DNA and text, empirically demonstrates superior performance in biological reasoning and classification tasks over modular designs.

Key insights

• The modality gap is not merely an empirical observation but a theoretically persistent geometric separation in modular MLLMs, impacting their reasoning efficiency.
• Native unified-vocabulary architectures inherently achieve and preserve a "zero-gap" state, allowing modalities to naturally align on a continuous manifold and freeing the LLM to focus on deep cross-modal reasoning.
• Empirical analysis using a DNA-text testbed confirmed that native integration maintains seamless representational alignment across all hidden layers, unlike modular designs where a distinct separation persists.

Summary by alphaXiv.