Reproduce “Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Brett Daley, Ilya Gorbachev, Anubhav Natani, Nicholas Brown, Athena Dai, Rachel Song, Marcin Pekalski, Josip Domazet and 2 more
The ABTest Agent helps experimenters retrieve, validate, and diagnose A/B test results while leaving launch decisions to people. Its LLM selects and orders checks, while deterministic code computes reported quantities; fixed-argument tools and template-gated reports constrain outputs. Results & Analysis made up 45% of requests, the largest category, but the paper does not evaluate overall agent accuracy or efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.