Reproduce “DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhuoyang Zou, Abolfazl Ansari, Delvin Ce Zhang, Dongwon Lee, Wenpeng Yin
Penn State University researchers developed DIAGPaper, a multi-agent framework that diagnoses valid and specific weaknesses in scientific papers by dynamically customizing reviewer criteria, engaging in adversarial author-reviewer rebuttals, and prioritizing weaknesses by severity. The system achieves state-of-the-art F1 scores for identifying valid weaknesses and effectively filters invalid critiques, demonstrating up to 40% reduction in initial weaknesses, while also improving the performance of open-source large language models.
Key insights
- Explicitly modeling criterion-specific expertise through dynamic reviewer decomposition and integrating an adversarial author-reviewer rebuttal mechanism significantly improves the validity and specificity of identified weaknesses.
- The rebuttal process effectively filters out a substantial portion (40%-60%) of initially identified, but invalid or weakly supported, weaknesses, mirroring real-world peer review practices.
- Severity-based prioritization, informed by the frequency of weakness categories in meta-reviews, helps surface the most critical issues, providing more actionable feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.