Reproduce “Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sosuke Yamao, Natsuki Miyahara, Yuankai Qi, Shun Takeuchi
A framework called Question-guided Visual Compression with Memory Feedback (QViC-MF) enables Large Multimodal Models (LMMs) to effectively understand long videos by integrating question-guided visual compression with an iterative memory feedback loop. This method achieved new state-of-the-art accuracy on MLVU (59.4%, +6.1%), LVBench (50.2%, +8.2%), VideoMME Long (54.0%, +3.7%), and VNBench Long (58.7%, +18.3%), while processing videos with only 16 context tokens per frame.
Key insights
- Iterative memory feedback, where past visual contexts actively inform ongoing perception, significantly improves the model's ability to maintain temporal event completeness and perform long-horizon reasoning.
- The QMSA mechanism, via masking, blocking, and guiding operations, effectively mitigates compression hallucination and ensures visual representations are aligned with question-relevant semantics.
- Combining multimodal selective attention with an efficient context memory allows for the creation of compact (16 tokens/frame) yet semantically rich visual representations, crucial for scaling LMMs to long videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.