Reproduce “Improving Retrieval for RAG based Question Answering Models on Financial Documents” on OpenResearch

About this paper

Spurthi Setty, Harsh Thakkar, Alyssa Lee, Eden Chung, Natan Vidra

This research systematically investigates and implements methods to improve the retrieval component of Retrieval Augmented Generation (RAG) models for question answering on financial documents. It demonstrates that advanced zero-shot techniques like query expansion and re-ranking offer measurable accuracy gains over a base RAG setup, while underscoring the critical importance of high-quality retrieval for overall system performance.

Key insights

• The quality of retrieved context is the primary determinant of RAG system accuracy for domain-specific question answering, evidenced by a significant performance gap between base RAG and oracle retrieval.
• Zero-shot retrieval enhancements like query expansion and re-ranking provide modest improvements over base RAG but are not sufficient to achieve high accuracy for complex financial queries.
• Sophisticated chunking methods and metadata integration, though not fully evaluated in this study, are identified as potentially powerful future directions for improving retrieval in structured documents.

Summary by alphaXiv.