Reproduce “The AI Productivity Index (APEX)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bertie Vidgen, Abby Fennelly, Evan Pinnix, Julien Benchek, Daniyal Khan, Zach Richards, Austin Bridges, Calix Huang and 12 more
Mercor, Harvard Law, and The Scripps Research Institute developed APEX-v1-extended, an expert-driven benchmark evaluating frontier large language models on 400 complex professional tasks. The evaluation revealed that current models, including the top-performing GPT 5 at 67.0%, still possess substantial limitations in executing typical high-value professional work.
Key insights
- Even frontier models, with GPT 5 as the top performer, still exhibit substantial limitations in performing typical professional tasks, achieving a maximum mean score of 67.0%.
- Model performance varies considerably across different professional domains, with tasks in the legal sector generally showing higher proficiency compared to more challenging areas like investment banking.
- The open-sourcing of a development dataset and evaluation harness provides a platform for researchers to iterate and improve models against real-world tasks, guiding AI development towards more practical and valuable capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.