Reproduce “Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruofan Hu, Shengyang Xu, Minjie Hong, Xiaoda Yang, Sashuai Zhou, Ke Lei, Tao Jin, Zhou Zhao

Doc-REFRAG targets multimodal document RAG when retrieval returns long, heterogeneous, and noisy image collections rather than pages from a single document. It combines the DocLongRAG training dataset with question-guided visual-token expansion, reaching 57.5% average accuracy across six benchmarks while using about 150 visual tokens per image and achieving 2.57-second average time to first token.

Key insights

  • Question-guided expansion preserves broad coverage across all retrieved images while concentrating detailed visual processing on a limited subset; the default setting expands 20% of chunks.
  • Training on retrieval-like noise matters independently of the architecture: removing DocLongRAG lowers average accuracy from 55.1% to 48.9%, and fine-tuning other models on it improves them by roughly 3.9–4.3 percentage points.
  • The approach is most challenged by dense or semantically complex text, which accounts for 64% of observed errors; figures and charts account for 22%, and tables or lists for 14%.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.