Reproduce “Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yikai Zheng, Xin Ding, Yifan Yang, Shiqi Jiang, Hao Wu, Qianxi Zhang, Weijun Wang, Ting Cao and 1 more
Researchers at Tsinghua University and Microsoft Research Asia developed Em-Garde, a "propose-match" framework that separates semantic interpretation from streaming perception for proactive video understanding. This approach yields an F1 score of 30.99% on OVO-Bench, outperforming prior methods, and maintains real-time processing at 13 frames per second with consistent efficiency for long video streams.
Key insights
- Explicitly separating computationally intensive semantic reasoning from continuous, lightweight visual perception allows for high accuracy and real-time efficiency in streaming video understanding.
- Transforming complex high-level queries into concrete, perceptually grounded visual proposals enables efficient real-time matching against video streams.
- A two-stage training process, including supervised fine-tuning and reinforcement learning, is crucial for teaching a large multimodal model to generate effective and robust visual proposals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.