Reproduce “BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dionizije Fa, Marko Culjak, Bruno Pandza, Mateo Cupic
BioAgent Bench evaluates agents on bioinformatics workflows, scoring required steps and final artifacts rather than answers alone. In Codex CLI, Claude Opus 4.5 averaged 100% completion versus GLM-4.7 at 82.5% across ten tasks. Perturbation tests found corrupted inputs identified in 7/10, but used one trial per task and condition. These grader-based completion results do not establish biological correctness or reliability.
Key insights
- While frontier LLM agents can achieve high pipeline completion for canonical bioinformatics tasks, this performance does not guarantee reliable step-level reasoning or robustness under adversarial inputs.
- The quality of explicit planning correlates with agentic performance, but successful task completion can also occur with less sophisticated planning, indicating a role for domain knowledge.
- Agent robustness is significantly compromised by perturbations like prompt bloat (reducing completion by 28%), erroneous incorporation of decoy files, and failures in detecting or appropriately handling corrupted data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.