Reproduce “Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hailey Onweller, Elias Lumer, Austin Huber, Pia Ramchandani, Vamse Kumar Subbiah, Corey Feld
Researchers at PricewaterhouseCoopers U.S. developed a three-stage framework to extract and evaluate inline citations from LLM-generated deep research reports, moving beyond superficial checks to assess factual accuracy. The work demonstrates that high link accessibility and topical relevance often mask lower factual accuracy, and that increasing search depth can consistently degrade an LLM's capacity for factual synthesis.
Key insights
- A significant disparity exists between surface-level citation quality (link accessibility and topical relevance) and the underlying factual accuracy of attributed claims.
- An inverse relationship was observed between the quantity of citations generated by an LLM provider and the factual accuracy of those citations, suggesting a potential 'attention dilution' effect during synthesis.
- Increasing search depth consistently degrades the factual accuracy of LLM-generated citations, highlighting an information overload challenge in processing and synthesizing larger volumes of source material.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.