Reproduce “Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami, Soheil Feizi
The authors introduce an evaluation that tests language-model responses to several meaning-preserving formulations of the same request and tracks success rates across inquiries. This profile distinguishes requests that may succeed after another try from those users can rely on consistently. It can guide model choice for retry-based or unchecked workflows, but does not reduce generation, inference time, or compute.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.