Reproduce “Agentic ML Exploration (A-MLE) for Ads Ranking” on OpenResearch

About this paper

Erwin Gao, Vinodh Kumar Sunkara, Jingyi Guan, Qinjin Jia, Hangjun Xu, Xiang Ji, Sherman Wong, Surya Teja Chavali and 31 more

A-MLE frames ads-ranking experimentation as an end-to-end systems problem, combining a tool-using LLM, domain-specific skills, sandboxed code execution, asynchronous training control, statistical evaluation, and shared experiment memory. The system raises domain-task accuracy from 16% for a generic ML agent to 68% and reports up to a 2.56% relative offline error reduction in an open-ended exploration, although many throughput and acceptance-rate claims lack absolute supporting details.

Key insights

• Domain-specific tools and workflow context substantially improve performance: A-MLE reaches 68% on the reported L1 benchmark, compared with 16% for a generic ML agent and 8% for a generic LLM.
• The useful unit of automation is the complete experiment loop rather than a single component; asynchronous waiting, infrastructure-error diagnosis, bounded retries, reruns for high variance, and rolling-baseline comparisons are central to reliable operation.
• Language-model behavior varies by family and prompt: stronger models differ in exploration aggressiveness and conservatism, so model selection and prompting affect open-ended search outcomes beyond basic task competence.

Summary by alphaXiv.