Reproduce “BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jinge Wu, Hongjian Zhou, Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Ayush Noori, Sean Wu and 3 more

BioMedArena is an open-source toolkit that standardizes the evaluation of biomedical deep research agents by decoupling evaluation components into modular layers. The toolkit, featuring a novel MUTUAL-EVOLVE harness, enabled various LLM backbones to achieve an average performance lift of +15.01 percentage points and surpass prior state-of-the-art across 8 biomedical benchmarks.

Key insights

  • Agent harnesses significantly improve base model performance, with the proposed MUTUAL-EVOLVE harness demonstrating the highest effectiveness among tested strategies, highlighting the value of structured interaction loops.
  • Optimal context management for long biomedical traces is achieved by combining 'Planning' (maintaining working notes) and 'Summarization' (compressing older dialogue), indicating the importance of managing context efficiently.
  • Collaborative features like the Global Workspace, final answer confirmation, and contribution-weighted voting within the MUTUAL-EVOLVE harness are crucial for robust multi-agent performance, preventing premature convergence and leveraging shared findings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.