Reproduce “CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yeonwoo Jeong, Kyuli Park, Hyunji Cho, Sungyong Park

CaGR-RAG, developed at Sogang University, introduces a system that optimizes disk-based vector search in Retrieval Augmented Generation (RAG) by grouping incoming queries based on their shared cluster access patterns and opportunistically prefetching clusters for subsequent query groups. This approach significantly increased cache hit rates and reduced 99th percentile tail latency by up to 51.55% on the hotpotqa dataset compared to state-of-the-art caching methods.

Key insights

  • Queries, particularly those with similar semantic or syntactic structures, exhibit non-uniform but predictable cluster access patterns that can be leveraged for better cache management.
  • Processing queries sequentially without considering their contextual similarity across a stream leads to inefficient cache utilization and unnecessary disk I/O.
  • While grouping queries improves intra-group cache locality, opportunistic prefetching is crucial for maintaining high cache hit rates during transitions between different query groups, significantly reducing overall and tail latency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.