Reproduce “On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models” on OpenResearch

About this paper

Dongyang Li, Junbing Yan, Taolin Zhang, Chengyu Wang, Xiaofeng He, Longtao Huang, Hui Xue, Jun Huang

This research introduces a Generative Expected Calibration Error (GECE) metric to identify long-tail knowledge queries for large language models (LLMs). Integrating GECE into retrieval augmented generation (RAG) pipelines yields an average inference speedup exceeding 4x and enhances task performance across question-answering datasets.

Key insights

• LLMs have knowledge gaps predominantly in long-tail areas, making selective knowledge augmentation more effective than indiscriminate retrieval for both efficiency and performance.
• The GECE metric, by combining generative accuracy, token confidence, and statistical/semantic cues, can reliably distinguish between common and long-tail knowledge from an LLM's perspective.
• Conditional retrieval based on an assessment of knowledge 'long-tailness' significantly reduces computational overhead while simultaneously improving the quality of LLM generations by providing targeted, noise-free information.

Summary by alphaXiv.