Reproduce “AI Benchmark Democratization and Carpentry” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gregor von Laszewski, Wesley Brewer, Jeyan Thiyagalingam, Juri Papay, Armstrong Foundjem, Piotr Luszczek, Murali Emani, Shirley V. Moore and 23 more

This work introduces a formal specification for AI benchmarks and proposes "AI Benchmark Carpentry" to equip the scientific community with skills for designing and interpreting evaluations. It emphasizes democratizing access to benchmarking tools and knowledge, and integrating energy consumption and hardware variability into evaluation practices to enhance reproducibility and real-world relevance.

Key insights

  • Current static AI benchmarks are largely inadequate for evaluating dynamic AI models, leading to performance disconnects in real-world applications and issues like "memorization."
  • Energy consumption is a paramount, yet often neglected, metric in AI benchmarking, with large language models consuming substantial power during training.
  • Significant performance variability, up to 22% or 1.5x slower for outliers, is observed in GPU-rich systems, impacting benchmark reproducibility and resource efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.