Reproduce “VISA: Retrieval Augmented Generation with Visual Source Attribution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xueguang Ma, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Wenhu Chen, Jimmy Lin
A Retrieval-Augmented Generation framework named VISA attributes answers by visually pinpointing supporting evidence with bounding boxes directly on document screenshots, improving answer verifiability and reducing user cognitive load. Through fine-tuning, Vision-Language Models demonstrated effective visual localization, achieving up to 58.1% bounding box accuracy on web documents and 67.6% on scientific PDFs in multi-candidate scenarios.
Key insights
- General-purpose VLMs perform poorly in zero-shot visual source attribution, achieving only 1.5% bounding box accuracy, underscoring the necessity for targeted supervised fine-tuning.
- Fine-tuning VLMs on domain-specific datasets significantly enhances their ability to accurately localize evidence, but performance decreases when models must select from multiple candidate documents.
- Diversified training data, including web-crawled pages (FineWeb-VISA), is crucial for improving out-of-domain generalization and handling complex, multi-page document structures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.