Reproduce “Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pengfei Hu, Meng Cao, Yingyao Wang, Yi Wang, Jiahua Dong, Jun Song, Yu Cheng, Bo Zheng and 1 more
SpecTemp introduces a dual-model framework for long video understanding, employing a lightweight draft MLLM for rapid frame selection and a powerful target MLLM for iterative reasoning. This approach enhances inference efficiency by 10-23% across various video lengths while achieving competitive or improved accuracy compared to existing methods.
Key insights
- Decoupling temporal perception from high-level reasoning through a cooperative dual-model design can effectively mitigate the efficiency bottleneck in long video understanding without compromising accuracy.
- Speculative decoding principles, traditionally used for LLM inference acceleration, can be adapted to the multi-modal domain to achieve reasoning-level acceleration in complex video tasks.
- A custom-built, large-scale dataset (SpecTemp-80K) with synchronized coarse and fine-grained annotations is crucial for effectively training iterative, dual-level reasoning systems in video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.