Reproduce “RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhichao Xu
University of Utah researcher Zhichao Xu benchmarked Mamba models against Transformer architectures for document ranking, demonstrating Mamba's competitive performance, often matching or exceeding Transformers on datasets like TREC DL19/DL20. The study also found that Mamba's current implementation had lower training throughput compared to optimized Transformer variants using Flash Attention.
Key insights
- Mamba models demonstrated competitive effectiveness in document ranking, consistently matching or surpassing Transformer-based models of similar sizes on key IR metrics.
- Despite theoretical O(n) efficiency, Mamba models exhibited lower practical training throughput and higher GPU memory consumption compared to highly optimized Transformer implementations leveraging Flash Attention 2.
- The research validated Mamba's applicability beyond general sequence modeling, showcasing its viability as an architectural alternative for fine-grained relevance modeling in IR.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.