Reproduce “Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Goeric Huybrechts, Srikanth Ronanki, Sai Muralidhar Jayanthi, Jack Fitzgerald, Srinivasan Veeravanallur
Document Haystack introduces a benchmark to evaluate Vision Language Models (VLMs) on their ability to retrieve multimodal information from long, visually complex documents, with documents up to 200 pages and 8,250 questions. It reveals that current VLMs significantly struggle with visual text processing and multimodal extraction from extended contexts, even as they perform well on parsed text.
Key insights
- A substantial performance gap exists between retrieving text from document images versus pre-parsed text, indicating that visual processing and conversion of document layouts remain a primary bottleneck for VLMs.
- Multimodal information retrieval (requiring both text and image understanding) from long documents is the most challenging task, highlighting a need for more robust fusion and reasoning capabilities in VLMs.
- Models exhibit significant differences in token consumption for images, which impacts their effective context window and influences the amount of detail retained over hundreds of pages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.