Reproduce “Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring "Weight-Space Programming"” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nazmus Sakib Ahmed
The authors audit whether Claude Opus 5.5 can infer neural-network behavior from weights without tools or network execution. Claude recovered local patterns, including update counts and output groupings, but often assigned familiar dataset labels incorrectly. In a label-permuted CIFAR-10 run, it found the grouping but all nine interpreted class pairs were wrong; this one-model study does not show that LLMs cannot trace weights.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.