Reproduce “RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaobo Li, Yirui Zhou, Yuan Xu, Kevin Chen, Daniel Waddington, Swaminathan Sundararaman, Hubertus Franke, Jian Huang
The RAGPerf framework provides an end-to-end evaluation of Retrieval-Augmented Generation systems, simultaneously assessing system performance and output quality under diverse workloads. It identifies bottlenecks and quantifies trade-offs across components, demonstrating LLM generation's latency dominance and critical resource dependencies.
Key insights
- LLM generation consistently accounts for the majority of end-to-end latency (75-91%), while reranking can become a substantial bottleneck in multimodal pipelines (28-87%), especially when retrieving full documents.
- GPU utilization during inference is often memory-bandwidth bound rather than compute-bound, with low SM occupancy, and GPU-accelerated vector indices offer marginal throughput gains that may not justify their high GPU memory cost compared to reserving memory for LLM inference.
- Host memory and GPU memory capacity are critical resource bottlenecks, severely degrading throughput or preventing model loading, with continuous data updates impacting RAG quality and latency if not managed with dynamic indexing strategies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.