Reproduce “Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge” on OpenResearch

About this paper

Heydar Soudani, Evangelos Kanoulas, Faegheh Hasibi

This work empirically compares Fine-Tuning (FT) and Retrieval-Augmented Generation (RAG) for question answering over less popular factual knowledge and introduces Stimulus RAG (SRAG). SRAG consistently achieved higher accuracy than fine-tuned LMs combined with RAG, demonstrating that strategic hint placement can enhance LM performance without the need for costly fine-tuning.

Key insights

• RAG substantially outperforms FT alone for enhancing accuracy on less popular factual knowledge, and combining FT with RAG benefits smaller LMs but can degrade performance in larger LMs.
• The quality of synthetic data generated for fine-tuning (e.g., via prompt-based QA generation) is more crucial than its sheer volume for effective knowledge injection.
• Intelligent prompt design through 'hints' in Stimulus RAG (SRAG) can effectively guide LMs to generate accurate responses, potentially eliminating the need for expensive data augmentation and fine-tuning.

Summary by alphaXiv.