Reproduce “DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shubham Agarwal, Akshat Gupta, Mike Heddes, Dilara Soylu, Rui Meng, Patrick Tser Jern Kon, Jiefeng Chen, Mert Cemri and 16 more
DuoDiscover searches for code regions to optimize while building a target-specific, inexpensive evaluator for ranking edits. Workers explore with these yardsticks, test their rankings against full evaluations, and use full evaluations to confirm gains. In a 20-hour, four-agent vLLM comparison, it achieved a 1.92× geometric-mean gain versus 1.07× for Claude Code, a result limited to this setup.
Key insights
- Proxy evaluation can substantially expand the search budget: in sampled tasks, yardsticks ran a median 53× faster than full kernel evaluations and 22× faster than repository evaluations.
- Proxy quality is not stable. The system later invalidated 45% of validated kernel yardsticks and 41% of validated repository yardsticks, making repeated fidelity checks central to reliable search.
- The approach applies across software repositories, GPU and TPU kernels, serving systems, and CPU architecture, but its advantage depends on finding a useful proxy and can be offset by setup overhead in short searches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.