Reproduce “DRAGOn: Designing RAG On Periodically Updated Corpus” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fedor Chernogorskii, Sergei Averkiev, Liliya Kudraleeva, Zaven Martirosian, Maria Tikhonova, Valentin Malykh, Alena Fenogenova
DRAGOn builds a recurring benchmark for evaluating retrieval-augmented generation (RAG) systems on Russian news. It crawls news, generates and filters questions, and ties results to dataset snapshots. Qwen 3 Embedding 8B outperformed FRIDA on retrieval metrics in this benchmark. The workflow supports repeatable comparisons but does not establish reduced contamination or generalize beyond Russian news.
Key insights
- A multi-stage QA generation and filtering process, including Knowledge Graph extraction and LLM-as-Judge evaluation, consistently yields high-quality, context-dependent questions and answers from dynamic news data.
- Evaluating RAG systems against continuously updated corpora effectively addresses data leakage concerns, providing a more realistic and fair assessment of system capabilities in dynamic environments.
- The choice of retrieval model has a substantial impact on end-to-end RAG performance, with specific embedders demonstrating superior effectiveness in retrieving relevant context for generative LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.