Reproduce “PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Avinash Amudala
The PROXIMA framework introduces a method for validating proxy metrics in online controlled experiments, assessing their reliability by combining normalized effect correlation, directional accuracy, and segment-level fragility. It quantifies decision quality and pinpoints user segments where a proxy's signal might contradict the long-term outcome, achieving a 100% win rate for top proxies on an advertising dataset and 96.7% on a recommendation dataset.
Key insights
- A composite reliability score that integrates directional accuracy and segment-level stability offers a more robust and nuanced assessment of proxy trustworthiness than relying solely on aggregate correlation.
- Segment-level fragility analysis is crucial for uncovering hidden "Simpson's Paradox" effects, where a proxy might provide correct aggregate signals but misleading ones in specific user subpopulations.
- Different domains exhibit distinct levels of proxy fragility; for instance, personalized recommendation systems show higher inherent heterogeneity and thus greater proxy fragility compared to online advertising.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.