Reproduce “The Chronicles of RAG: The Retriever, the Chunk and the Generator” on OpenResearch

About this paper

Paulo Finardi, Leonardo Avila, Rodrigo Castaldoni, Pedro Gengo, Celio Larcher, Marcos Piau, Pablo Costa, Vinicius Caridá

Researchers from 22h, Brazil, empirically evaluated various Retrieval Augmented Generation (RAG) components to establish good practices for the Brazilian Portuguese language, resulting in an optimized pipeline that achieved 98.61% relative accuracy on a literary dataset. The study identified optimal retriever configurations, input chunking strategies, and input sizes, demonstrating substantial performance improvements compared to context-free baselines.

Key insights

• LLMs inherently limit RAG system performance, achieving an average of 7.4/10 even with perfect context, indicating that an ideal RAG system cannot surpass the LLM's intrinsic capabilities.
• The quality of the retriever is paramount for RAG system effectiveness, with significant gains observed from fine-tuned embeddings and multi-stage reranking pipelines, particularly for Brazilian Portuguese.
• Increasing context window size or the number of retrieved chunks does not always improve performance and can introduce "Lost in The Middle" effects; optimizing input size is crucial for efficiency and accuracy.

Summary by alphaXiv.