Reproduce “OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maximilian Idahl, Zahra Ahmadi
OpenReviewer introduces a specialized large language model, Llama-OpenReviewer-8B, fine-tuned to produce critical and structured peer reviews for machine learning and AI papers. The system achieves a 55.5% exact match rate with human reviewer recommendations and an average recommendation error of 0.96, demonstrating a significant improvement in aligning with human criticality compared to general-purpose LLMs.
Key insights
- Specialized fine-tuning on domain-specific expert review data enables LLMs to generate critical and structured feedback that accurately matches human judgment, overcoming the inherent positive bias of general-purpose models.
- A structured system prompt, incorporating explicit reviewer guidelines, is essential for conditioning the LLM to adopt the role of an expert reviewer and adhere to academic standards.
- Accurate PDF-to-markdown conversion, capable of handling technical content like equations and tables, is crucial for effectively extracting and presenting scientific paper text to the LLM for review generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.