Reproduce “AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weili Xu, Enxin Song, Wenhao Chai, Xuexiang Wen, Tian Ye, Gaoang Wang
AURORALONG presents a hybrid large multimodal model for video understanding that integrates a linear RNN (RWKV) with a sorted visual token merge strategy. This architecture processes up to 16,000 frames on a 24GB GPU, achieving a 34x memory advantage and 8x faster inference, while demonstrating competitive or superior accuracy on various short and long video benchmarks compared to larger Transformer-based models.
Key insights
- Linear RNNs, like RWKV, can serve as efficient LLM backbones for open-ended video understanding, providing constant memory cost and linear scaling with input sequence length during inference.
- The proposed sorted visual token merge strategy, specifically sorting tokens by ascending size, significantly aids RWKV in effectively processing and memorizing essential visual information in long contexts.
- Aggressively reducing visual tokens to a low retention ratio (e.g., 0.1) can maintain or even improve performance for certain tasks while substantially increasing efficiency, suggesting RNNs might benefit from sparse, fine-grained visual inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.