Reproduce “ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen
Researchers from the CAS Key Laboratory of AI Safety, ICT, and UCAS introduce ROMA, an open-source omni-multimodal assistant that enables real-time streaming audio-video understanding with unified proactive and reactive interaction. The model achieves state-of-the-art performance on proactive alert and narration tasks and competitive results on reactive QA benchmarks, leveraging a novel speak head and two-stage training.
Key insights
- The 'speak head' architecture effectively decouples the timing decision from text generation, which is crucial for precise and robust proactive intervention in streaming scenarios.
- A two-stage fine-tuning curriculum, progressively adapting to streaming inputs and then to time-aware decision-making, significantly outperforms direct multi-task training for proactive capabilities.
- Aggregating hidden states from multiple upper layers (K=4) of the LLM backbone provides more robust and high-level features for the speak head to make effective streaming decisions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.