Reproduce “MIRACL-VISION: A Large, multilingual, visual document retrieval benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Radek Osmulski, Gabriel de Souza P. Moreira, Ronay Ak, Mengyao Xu, Benedikt Schifferer, Even Oldridge
NVIDIA researchers introduce MIRACL-VISION, a large-scale, multilingual, visual document retrieval benchmark designed to evaluate multimodal models. This benchmark reveals a substantial performance gap, demonstrating that text-based models achieve over 50% higher accuracy than current VLM-based models for retrieving information from text-heavy document images across 18 languages, particularly for low-resource and non-Latin script languages.
Key insights
- Current VLM-based models exhibit a substantial performance deficit (over 50% lower NDCG@10) compared to text embedding models for multilingual retrieval on text-heavy document images.
- The performance gap for VLMs is particularly pronounced for low-resource languages and non-Latin scripts, with some languages showing extremely low retrieval accuracy.
- MIRACL-VISION's 'hard-negative' sampling effectively creates a challenging, yet manageable, benchmark that accurately reflects real-world retrieval difficulties, unlike existing, often saturated, benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.