Reproduce “Blind Spots: Where Small Language Models Fall Behind Frontier Models on Terminal Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ashima Suvarna, Ruta Desai, Ahmed Elgohary
The authors evaluate terminal agents by separating solvability, outcome verification, and prospective confidence: finding a successful run, selecting one from completed candidates, and estimating success before acting. On two benchmarks, 32 attempts narrow the TBLITE pass@1 gap between 9B Tmax-9B and GLM-5.2 from 17.7 to 8.0 percentage points. On shared candidates, GLM-5.2 selects better, while all models overestimate success.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.