Reproduce “VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lexiang Hu, Yanzhao Zhang, Mingxin Li, Dingkun Long, Yikang Li, Fuwei Zhang, Yisen Wang, Zhouchen Lin
Researchers from Peking University introduce VisDocAgentBench, a new benchmark for evaluating static and agentic retrieval systems on visually rich documents, specifically focusing on complex relevance structures requiring distributed evidence. The benchmark demonstrates that while static retrievers struggle with multi-hop evidence, tool-using agents significantly improve performance, with visual retrieval consistently outperforming OCR-text routes for document discovery.
Key insights
- Static retrieval methods, even advanced visual ones, severely degrade when relevance depends on distributed, multi-hop evidence across a corpus, achieving only 2.50% Recall@1 for two-bridge queries.
- Tool-using agents can substantially recover performance on complex retrieval tasks, but their effectiveness is highly variable among different planners, indicating that robust reasoning and effective observation utilization are critical.
- Full-page visual representations consistently enable better document discovery and overall retrieval performance than OCR-text-only routes across all agents, highlighting the importance of preserving layout and structured visual evidence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.