Reproduce “AutoBencher: Towards Declarative Benchmark Construction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, Tatsunori Hashimoto

The AutoBencher framework introduces a declarative approach for automatically constructing evaluation benchmarks for large language models. It significantly improves benchmark quality by reducing rank correlation by 27% (novelty), increasing difficulty by 22%, and achieving a 20% higher Attack Success Rate for safety evaluations compared to existing human-curated datasets.

Key insights

  • Providing an evaluator LLM with 'privileged information' creates an effective information asymmetry, enabling the generation of challenging questions that expose specific model limitations.
  • An adaptive search process, where the evaluator LLM learns from prior model performance feedback, is crucial for systematically discovering novel, difficult, and vulnerable topic categories.
  • Framing benchmark construction as a declarative optimization problem with quantifiable desiderata allows for the systematic and measurable improvement of evaluation dataset quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.