Reproduce “Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective” on OpenResearch

About this paper

Daniel Franzen, Jan Disselhoff, David Hartmann

Researchers from Johannes Gutenberg University Mainz and Lambda, Inc. achieved 71.6% accuracy on the ARC-AGI benchmark, setting a new state-of-the-art for open-source models and exceeding average human performance. Their method employs a single fine-tuned LLM with extensive task-specific augmentations and a Product of Experts strategy, reducing inference costs to approximately $0.02 per task.

Key insights

• Many perceived limitations of LLMs in abstract reasoning tasks like ARC-AGI are addressable through optimized data representations, custom tokenization, and comprehensive augmentation strategies.
• A "Product of Experts" approach, which aggregates an LLM's confidence across diverse augmented views of a problem, reliably identifies correct solutions by favoring those consistent across multiple perspectives.
• High performance on complex reasoning benchmarks can be achieved with open-source LLMs through efficient candidate generation (e.g., DFS) and robust multi-perspective scoring, offering a cost-effective alternative to larger, proprietary models.

Summary by alphaXiv.