Reproduce “UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Chien-Sheng Wu
UniDoc-Bench introduces a large-scale, human-verified benchmark for document-centric Multimodal Retrieval-Augmented Generation (MM-RAG) systems. It features 1,600 human-validated QA pairs across 70,000 real-world PDF pages from eight diverse domains, enabling standardized evaluation across different RAG paradigms and revealing that text-image fusion RAG often outperforms joint multimodal embedding and unimodal approaches.
Key insights
- Text-image fusion RAG, which combines separately retrieved text and images, generally outperforms unified multimodal embedding approaches and single-modality RAG systems in end-to-end performance.
- Image-based retrieval consistently yields higher recall but lower precision compared to text-based retrieval, indicating different strengths and challenges for each modality.
- Queries requiring information predominantly from images remain a significant challenge across all RAG systems, pointing to a key area for future research and improvement in multimodal understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.