Reproduce “Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing” on OpenResearch

About this paper

Jiacheng Miao, Jin Mu, Guanhua Chen, James Zou

Fisher-R1 trains an agent for hypothesis testing on synthetic tasks, then uses reinforcement learning. Its reward checks p-value closeness and whether the conclusion matches an answer key, but does not directly score method choice. On P-Bench Hard, Fisher-R1-14B reaches 33.0% ± 1.7% Strict pass@1 versus GPT-5.4’s 30.5% ± 1.6%, a narrow lead on that split and metric.

Key insights

• Targeted training combining supervised fine-tuning and reinforcement learning with an outcome-grounded statistical reward significantly enhances LLM agent reliability for hypothesis testing beyond simply scaling model size.
• There is a notable gap in general-purpose LLMs between providing a correct conclusion (Raw score) and generating a numerically accurate p-value (Strict score), a deficiency addressed by Fisher-R1's training.
• Training on diverse synthetic tasks with controlled perturbations enables the agent to generalize statistical reasoning to real-world, out-of-distribution scientific data, including handling common data quality issues.

Summary by alphaXiv.