Reproduce “StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ming Xie, Zizheng Huang, Xudong Tan, Chao Wang, Xiangyu Zeng, Wenxiao Wu, Tao Chen, Limin Wang and 1 more
StreamOV introduces a framework for streaming omni-video understanding, integrating multimodal evidence construction and an evidence-guided long-short term memory to manage continuous audio-visual context. It also features an MLLM-as-a-trigger mechanism for proactive response decisions, achieving an average accuracy of 83.8% on its SOVBench-O benchmark and an F1 score of 90.5% for response triggering.
Key insights
- Efficient streaming omni-video understanding benefits from a dual-perspective evidence construction that identifies both intrinsic stream dynamics and query-aware semantics.
- An evidence-guided long-short term memory system can effectively manage continuous multimodal context within bounded resources, balancing recent detail with long-range significance.
- Leveraging the MLLM's internal hidden states provides an intrinsic and efficient mechanism for autonomous response triggering, overcoming limitations of external routers or explicit silence tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.