Reproduce “LongCat-Flash-Omni Technical Report” on OpenResearch

About this paper

Meituan LongCat Team, Bairui Wang, Bayan, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan and 125 more

Meituan's LongCat Team introduces LongCat-Flash-Omni, a 560-billion-parameter open-source omni-modal model capable of real-time audio-visual interaction. This model achieves state-of-the-art performance across diverse unimodal and cross-modal benchmarks, demonstrating robust multimodal integration without compromising its strong text capabilities.

Key insights

• A progressive, curriculum-based training approach can effectively integrate a vast array of heterogeneous multimodal data into an MoE LLM, enhancing capabilities without degrading foundational text performance.
• Decoupling modality-specific encoders and decoders from the central LLM at the distributed level, combined with chunk-based processing and specialized parallelization, is critical for efficient large-scale multimodal training and low-latency real-time inference.
• Achieving natural, real-time audio-visual interaction relies on sophisticated strategies for managing streaming inputs, maintaining context across turns, and dynamically balancing information density (e.g., sparse-dense sampling) during dialogue.

Summary by alphaXiv.