Reproduce “Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Melanie Mitchell, Alessandro B. Palmarini, Arseny Moskvichev
Researchers at the Santa Fe Institute compared human, text-only GPT-4, and multimodal GPT-4V performance on the ConceptARC abstract reasoning benchmark. The study found that while detailed prompting improved text-only GPT-4's accuracy to 33% on full tasks, it remained significantly below human performance (91%), and GPT-4V performed considerably worse (23-25%) than text-only GPT-4 on simpler visual tasks.
Key insights
- Even with sophisticated prompting, text-only GPT-4's abstract reasoning remains substantially below human levels, indicating persistent limitations in true rule induction and generalization.
- The multimodal GPT-4V performed considerably worse than its text-only counterpart when processing visual inputs, suggesting challenges in how current multimodal models integrate and reason with visual information for abstraction.
- Current LLMs appear to rely more on complex pattern matching or 'approximate retrieval' from their training data rather than demonstrating robust, human-like abstraction and flexible generalization on novel tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.