Reproduce “CiteEval: Principle-Driven Citation Evaluation for Source Attribution” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yumo Xu, Peng Qi, Jifan Chen, Kunlun Liu, Rujun Han, Lan Liu, Bonan Min, Vittorio Castelli and 2 more

Researchers from AWS AI Labs, Orby.ai, and Google developed CiteEval, a principle-driven framework for evaluating source attribution in Retrieval-Augmented Generation (RAG) systems. They introduced CiteBench, a human-annotated dataset, and CITEEVAL-AUTO, a suite of LLM-based metrics that assess citation quality across a broader context and with finer granularity, demonstrating higher correlation with human judgments than prior NLI-based approaches.

Key insights

  • Evaluating citations against the full range of retrieved documents and explicitly categorizing statements by their true source context (retrieval, user, response, parametric) is crucial for accurate assessment.
  • Fine-grained evaluation criteria and executable edit actions (e.g., delete for redundancy, add for missing evidence) provide a more nuanced understanding of citation quality and enable actionable feedback.
  • Automated metrics can achieve strong correlation with human judgments for citation evaluation when designed to capture contextual awareness and fine-grained criteria, moving beyond simple NLI.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.