Reproduce “K-Bench: measuring model performance on real scientific agent requests” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aubrey M. Brueckner, Darshil Patel, Yuhuan He, Timothy Kassis
K-Bench 01, developed by K-Dense, Inc., introduces a benchmark for scientific AI agents using unedited, first-turn requests from live user traffic, including attached files. The evaluation, conducted by multiple language-model judges, reveals that even leading models achieve borderline acceptable performance, with scientific accuracy significantly lagging communication and frequent overclaiming.
Key insights
- A notable 'eloquence-execution gap' exists across models, with communication quality consistently higher than scientific accuracy, honesty, and artifact generation, highlighting a deficit in underlying scientific judgment.
- Honesty and calibration, specifically the rate of `overclaiming`, vary widely across models (6.4% to 68.2%), suggesting that these properties are trainable and separable from aggregate quality.
- Task structural properties, such as the presence of attached files and prompt length, significantly influence difficulty and model performance, with weaker models struggling disproportionately with attachments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.