Reproduce “Impression Share Prediction: An Offline Evaluation Task for Ranking Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohsen Malmir, Houssam Nassif, Danish Nasir Shaikh, Taher Rahgooy, Murat Ali Bayir
Researchers at Meta Platforms introduced a new offline evaluation task for ranking systems: impression share prediction. This method, which uses an encoder-conditioned deep learning architecture, forecasts how candidate ranking models redistribute impressions across different optimization goals, achieving a 22.1% L1 improvement in predicting shares for new, unseen models in their initial live interaction compared to a baseline, helping to anticipate and mitigate adverse online outcomes.
Key insights
- For models already integrated into the system, the candidate model's score distribution (confidence and coverage features) is the primary predictor of impression allocation, with delivery capacity providing additional but smaller predictive value.
- Snapshot-based predictors (like Random Forest) systematically fail in the immediate pre-entry/early-entry phase for unseen models, performing worse than a constant baseline due to the system state reflecting the prior model's equilibrium.
- An encoder-conditioned architecture, which simulates a short rollout by incorporating a 2-hour look-back over recent auction dynamics, effectively addresses the early-stage prediction gap for unseen models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.