Reproduce “STORM: Token-Efficient Long Video Understanding for Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu and 8 more
STORM introduces an architecture for multimodal LLMs that uses a Mamba-based temporal encoder to efficiently process long videos, integrating spatiotemporal information before applying intelligent token compression. The system achieves state-of-the-art results on multiple long-video understanding benchmarks while reducing computational costs by up to 8x and decoding latency by up to 2.9x.
Key insights
- Early and explicit spatiotemporal modeling via a Mamba-based encoder effectively enriches visual tokens with inter-frame information, enabling aggressive compression without significant information loss.
- The Mamba State Space Model's ability to condition state matrices on input allows for dynamic compression of historical information into state representations, capturing long-term dependencies efficiently.
- Combining temporal pooling and test-time temporal sampling after Mamba processing drastically reduces the LLM's input token count, leading to substantial computational efficiency gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.