Reproduce “video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangzhi Sun, Yudong Yang, Jimin Zhuang, Changli Tang, Yixuan Li, Wei Li, Zejun MA, Chao Zhang
Researchers from ByteDance, Tsinghua University, and the University of Cambridge developed video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual large language model for general video understanding. The model achieves 3-8% absolute accuracy improvements over baselines on video reasoning benchmarks and demonstrates emergent zero-shot synthetic video detection capabilities.
Key insights
- Reasoning capabilities in multimodal LLMs can be substantially enhanced for general video understanding by integrating advanced audio-visual processing with explicit reasoning path generation and optimization.
- Pairwise preference modeling (pDPO) offers a more scalable and effective approach for reasoning enhancement in complex multimodal contexts than traditional absolute-scoring reward models.
- Targeting reinforcement learning optimization to "susceptible" intermediate reasoning steps (those highly dependent on video input) efficiently improves performance on critical video-dependent errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.