Reproduce “EdgeRAG: Online-Indexed RAG for Edge Devices” on OpenResearch

About this paper

Korakit Seemakhupt, Sihang Liu, Samira Khan

EdgeRAG enables Retrieval Augmented Generation (RAG) systems to function effectively on resource-constrained edge devices by optimizing the vector database's memory footprint and retrieval latency. It achieves this by selectively storing embeddings, generating others on demand, and employing adaptive cost-aware caching, resulting in an average 1.8x reduction in Time-to-First-Token (TTFT) and up to 3.82x for large datasets that exceed device memory without compromising generation quality.

Key insights

• Storing all second-level embeddings in IVF indices for RAG on edge devices leads to significant memory thrashing and high retrieval latency.
• The computational cost of generating embeddings for different clusters is highly skewed, with a 'long tail' of large clusters incurring disproportionately high latency if generated online.
• A large portion of embedding vectors are not accessed during typical retrieval, indicating potential for memory optimization through selective storage and online generation.

Summary by alphaXiv.