Reproduce “CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tetsuya Sakai, Ji-Rong Wen and 1 more
The paper introduces CORAL, a large-scale, comprehensive benchmark designed to evaluate multi-turn conversational Retrieval-Augmented Generation (RAG) systems. Experiments on CORAL demonstrate the importance of fine-tuning for improved performance and highlight that intelligent conversation compression can maintain generation quality while enhancing citation accuracy.
Key insights
- Input condensation through various compression strategies surprisingly results in minimal performance decrease for response generation while improving citation labeling accuracy, suggesting that much of the raw dialogue history can be redundant.
- Fine-tuning Large Language Models (LLMs) is crucial for conversational RAG, consistently and significantly outperforming non-fine-tuned counterparts in both response generation and citation labeling.
- Performance scaling varies by task: generation quality plateaus beyond 3B parameters, but citation labeling accuracy continues to improve with models up to 7B parameters, indicating a higher capacity requirement for precise attribution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.