Reproduce “The AI Consumer Index (ACE)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Julien Benchek, Rohit Shetty, Benjamin Hunsberger, Ajay Arun, Zach Richards, Brendan Foody, Osvald Nitski, Bertie Vidgen
The AI Consumer Index (ACE) introduces the first comprehensive benchmark for evaluating frontier AI models on everyday consumer tasks across domains like shopping, DIY, gaming, and food. This work employs a novel LM-based grading methodology that explicitly penalizes ungrounded information, demonstrating that even top models achieve scores barely above 50% and often struggle with factual accuracy and hallucinations in consumer applications.
Key insights
- Even top-performing frontier AI models exhibit a "substantial gap" between their current capabilities and the real-world needs of consumers, with overall scores just above 50%.
- Model performance varies considerably across consumer domains; Food tasks are generally easier, while Shopping tasks are significantly more challenging due to the high demand for precise, grounded information and susceptibility to hallucination.
- Hallucination remains a critical issue for AI models, particularly for factual details such as prices and links, leading to substantial score reductions when ungrounded claims are penalized by the evaluation framework.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.