Reproduce “Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning” on OpenResearch

About this paper

Fabian Spaeh, Tianyi Chen, Chen-Hao Chiang, Bin Shen

Researchers at Celonis developed a dynamic few-shot learning framework for agentic Retrieval-Augmented Generation (RAG) systems to suggest answerable queries when an initial user query fails. This method achieved up to 93% answerability and 89% semantic similarity on real-world datasets, outperforming baseline approaches, and demonstrated rapid improvement through a self-learning mechanism.

Key insights

• Agentic RAG systems require a nuanced query suggestion approach that accounts for dynamic workflows, tool constraints, and data availability beyond traditional free-text retrieval.
• Distinguishing between different types of unanswerable queries ('no workflow' vs. 'no knowledge') is crucial for generating targeted and effective suggestions.
• Providing dynamically retrieved, answerability-aware few-shot examples significantly enhances the LLM's ability to generate both semantically similar and executable query suggestions.
• A self-learning methodology, utilizing the RAG agent's own execution results, offers a practical and scalable way to train and improve the query suggestion system in real-world environments.

Summary by alphaXiv.