Reproduce “Prior or Feedback? What an LLM Uses When Adapting Neural Operators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Julian Chan, Javier Mora Jimenez
The authors test LLM tuning of a Fourier neural operator (FNO) using hosted DeepSeek-V4 Pro over 20 trials. It achieves lower held-out test nRMSE in all 36 runs against random search and 35 of 36 against TPE. Its first proposal beats 91.7% of random configurations; replay shows choices respond to score assignments, but not that feedback improves test performance.
Key insights
- The LLM’s initial proposals appear to reflect a useful task-conditioned prior: the first proposal outperformed 91.7% of corresponding random-search configurations by median validation error.
- Reassigning validation scores changed subsequent proposals relative to a notation control, providing evidence that the policy responds to which outcomes are associated with which configurations.
- Decision sensitivity and endpoint performance do not establish that the LLM reasons correctly about physics or that feedback use causes its performance advantage.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.