Reproduce “DIRC-RAG: Accelerating Edge RAG with Robust High-Density and High-Loading-Bandwidth Digital In-ReRAM Computation” on OpenResearch

About this paper

Kunming Shao, Zhipeng Liao, Jiangnan Yu, Liang Zhao, Qiwei Li, Xijie Huang, Jingyu He, Fengshi Tian and 4 more

The DIRC-RAG framework, developed by researchers primarily from HKUST, presents a digital in-ReRAM computation architecture designed to accelerate Retrieval-Augmented Generation (RAG) on edge devices. It achieves a 7,800x reduction in retrieval latency and 188,000x improvement in energy efficiency compared to a high-end GPU, while maintaining FP32-level retrieval precision with INT8 quantization.

Key insights

• Hybrid ReRAM-SRAM Computing-in-Memory architectures can effectively balance high memory density with robust computational accuracy for memory-intensive AI workloads like RAG on edge devices.
• A query-stationary dataflow significantly enhances the efficiency of RAG by minimizing data movement when a single query is processed against a large, static document embedding database.
• Integrating hardware-software co-design, such as error-aware bitwise data mapping and an on-chip error detection circuit, is crucial for maintaining high retrieval precision when utilizing low-precision quantization and inherently variant ReRAM technology.

Summary by alphaXiv.