Reproduce “From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zijie Zhuang, Changxin Lao, Pengbo Xu, Hanwen Xu, Ruochen Yang, Yingzhi He, Peng Zhang, Jiangxia Cao and 8 more
A framework for industrial machine learning research agents transforms raw experimental trajectories into auditable evidence by implementing bounded artifact verification and post-execution claim qualification. The system successfully generates deployable candidates that achieved positive online lifts in production A/B tests, such as a +0.75% increase in live streaming page duration.
Key insights
- Experimental trajectories are often non-monotonic; later rounds may perform worse than earlier ones, highlighting that the final state is not always the most informative and necessitating explicit evidence qualification across rounds.
- Context-isolated self-verification measurably improves the quality of generated artifacts, such as increasing implementation completeness from 92% to 97% and observability from 67% to 100% at critical workflow handoffs.
- The explicit claim qualification process successfully distinguishes genuinely supported and attributable claims from those lacking sufficient evidence, enabling the creation of a structured, reusable knowledge base.
- The controller's affirmative applicability judgment for reusing records is a bottleneck, exhibiting a tendency for false positives and requiring further improvement for reliable decision-making.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.