Reproduce “Optimizer's Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Garud Iyengar, Henry Lam, Tianyu Wang
The Optimizer’s Information Criterion (OIC) offers a general analytical framework to correct the optimistic bias inherent in empirically evaluating data-driven decisions, extending the Akaike Information Criterion to decision selection. It provides asymptotically unbiased performance estimates with low variance, addressing the "Optimizer's Curse" in various optimization settings, including constrained, regularized, and distributionally robust problems, with improved computational efficiency over cross-validation.
Key insights
- The paper provides an explicit $O(1/n)$ characterization of the optimistic bias, linking it to the gradient of the objective and the influence function of the parameter estimator.
- OIC is asymptotically unbiased and introduces minimal additional variance, making it statistically efficient and asymptotically equivalent to LOOCV while being computationally much faster.
- The framework is highly general, extending bias correction to complex scenarios like constrained optimization, regularized objectives, general risk functions, contextual optimization, and allows for quantifying model misspecification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.