Reproduce “Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaohan Wang, Benfeng Xu, Licheng Zhang, Mingxuan Du, Chiwei Zhu, Xiaorui Wang, Zhendong Mao, Yongdong Zhang
The authors introduce Wiki Live Challenge (WLC), a benchmark comparing deep research agents with recently reviewed Wikipedia Good Articles. Its Wiki Eval framework scores writing criteria and checks coverage of reference facts and support from cited webpages. Gemini-3-pro Deep Research leads Wiki Writing at 58.33%, while Gemini-2.5-pro Deep Research leads fact coverage at 30.76%; these LLM-based measures do not prove grounding.
Key insights
- A significant performance gap exists between current DRAs and human expert-level article generation, particularly in achieving comprehensive factual coverage and generating nuanced, specialized details.
- Proprietary DRAs generally outperform open-source frameworks in writing quality and factual accuracy, though all tested models demonstrate substantial limitations in consistently meeting expert-level standards.
- Task difficulty varies across Wikipedia categories, with subjects like History and Mathematics proving more challenging for DRAs, and direct Wikipedia leakage not correlating with higher performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.