Reproduce “Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro
Video-Ma2mba introduces an architecture that leverages the Mamba-2 Structured State Space Model and Multi-Axis Gradient Checkpointing (MA-GC) to enable efficient long-form video understanding. It processes videos at 1 FPS, handling millions of tokens on a single GPU, and achieves competitive performance against larger Transformer-based models on various benchmarks.
Key insights
- The combination of Mamba-2's linear scaling and MA-GC allows for processing video sequences up to 2 million tokens (hours of 1 FPS video) on a single 80GB GPU, reducing memory complexity from O(L·S) to O(S).
- Processing video at 1 FPS provides a more comprehensive temporal representation than sparse sampling, which is crucial for accurate and context-aware responses in long video comprehension tasks.
- A dedicated 'long video knowledge learning' training stage is essential for enhancing the model's ability to track and reason with context over extended video durations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.