Reproduce “MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peiran Wu, Zhuorui Yu, Yunze Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen
MARC introduces a "retrieve then compress" framework for visual language models, leveraging a visual memory retriever and reinforcement learning to achieve a 95% reduction in visual tokens for video understanding tasks. This method maintains performance comparable to models processing 64 times more frames while reducing GPU memory usage by 72.4% and overall model generation latency by 15.9%.
Key insights
- Pre-selecting task-relevant, event-level video segments through a VMR is highly effective in focusing downstream processing and mitigating information loss during subsequent compression.
- Reinforcement learning, specifically the C-GRPO framework, can successfully distill complex reasoning capabilities from uncompressed teacher models to aggressively compressed student models, enabling robust performance under extreme token reduction.
- Aggressive token compression, reducing video input to a single-frame token count, can maintain VLM performance levels comparable to models processing significantly more frames, challenging the necessity of full-context processing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.