Reproduce “A Memory-Efficient Retrieval Architecture for RAG-Enabled Wearable Medical LLMs-Agents” on OpenResearch

About this paper

Zhipeng Liao, Kunming Shao, Jiangnan Yu, Liang Zhao, Tim Kwang-Ting Cheng, Chi-Ying Tsui, Jie Yang, Mohamad Sawan

Researchers at Westlake University and HKUST developed a memory-efficient retrieval architecture for RAG-enabled Large Language Models (LLMs) on wearable medical devices. This design significantly reduces the memory and energy footprint for retrieval while maintaining high accuracy, enabling privacy-preserving personalized medical AI on edge hardware with ultra-low energy consumption of 177.76 μJ per query.

Key insights

• A coarse initial retrieval using truncated precision (MSB INT4) can dramatically reduce the search space and resource requirements for subsequent high-precision refinement without significant accuracy loss.
• Off-chip data transfer (DRAM access) constitutes over 99% of total energy consumption for RAG retrieval on edge devices, highlighting memory-centric optimizations as critical.
• Maintaining high retrieval accuracy for medical data is feasible on resource-constrained hardware through a judicious combination of quantization and hierarchical processing.

Summary by alphaXiv.