Reproduce “Offline A/B testing for Recommender Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, Simon Dollé

Researchers from Criteo Research developed improved methods for offline evaluation of recommender systems using historical data. Their novel Pointwise Normalized Capped Importance Sampling (PointNCIS) estimator, which applies localized normalization, achieved the highest correlation with online A/B test results compared to traditional methods, enabling more reliable pre-deployment assessment.

Key insights

  • Traditional counterfactual estimators like capped importance sampling (CIS) exhibit a poor bias-variance trade-off, where increasing the capping parameter dramatically increases variance while significant bias persists.
  • The extreme values of importance weights in recommender system evaluation make traditional methods insufficient for accurate offline assessment.
  • More localized normalization, as implemented in Piecewise Normalized Capped Importance Sampling (PieceNCIS) and Pointwise Normalized Capped Importance Sampling (PointNCIS), significantly improves prediction accuracy by better correcting for bias.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.