Reproduce “Simulation-Based Inference for Adaptive Experiments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Brian M Cho, Aurélien Bibaut, Nathan Kallus
Researchers from Cornell University and Netflix developed a simulation-based inference method for adaptive experiments that ensures nominal Type I error control while substantially reducing confidence interval widths, particularly for undersampled treatment arms. This "simulation with optimism" approach is applicable across a broad class of multi-arm bandit designs, including those incompatible with traditional reweighting techniques.
Key insights
- Simply plugging in unbiased estimates for nuisance parameters during simulation can lead to invalid Type I error control in adaptive settings.
- Adding a decaying positive bias to nuisance arm means during simulation ("optimism") effectively maintains asymptotic Type I error control by producing a more conservative null distribution.
- The chosen test statistic (sample mean of the target arm) ensures the test possesses asymptotic power, leading to the rejection of false null hypotheses as the experiment progresses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.