Reproduce “ConceptARC at Scale: Studying Conceptual Understanding in Reasoning Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Philipp Mondorf, Adrien Deliège, Solim LeGris, Claas Beger, Ryan Yi, Barbara Plank, Brenden M. Lake, Todd M. Gureckis
The authors scale ARC-style puzzle evaluation by generating 1,600 tasks across 16 concepts, with each task combining a layout, selector, and action. They assess model answers and compare task representations across these labels. Qwen3.5-27B gets at least one of ten test queries right on 93.21% of tasks, but all ten on 23.17%; representation organization is correlational, not evidence of causal use.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.