Reproduce “AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bingxiang He, Pengqi Sun, Shihao Peng, Shaoxuan Li, Huiyu Xiao, Hongjian Shi, Zixuan Fu, Ziqing Qiao and 13 more
AutoSFT helps an agent improve supervised fine-tuning by changing training data through a structured research loop. A meta-agent selects recipes, while separate execution and analysis agents handle training and propose data changes; synthesis adds verified examples. On Qwen3-1.7B-Base, analysis reaches 67.10 on IFEval versus 60.60 within 30 rounds, in single searches per configuration.
Key insights
- Separating job execution from research analysis reduced context compactions and shell-command activity, though this component alone was not shown to improve benchmark scores.
- Multiple analytical perspectives broadened the interventions: repair and resupervision made up 55% of actions in the analysis run, unlike the mixture-focused single-agent pilot.
- Targeted synthesis can improve capability-level performance, but adding examples is not sufficient for every weakness; one targeted capability remained at zero.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.