Reproduce “When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sang Su Lee, Vineeth Loganathan, Shishir Dash, Vijay Raghavan
This research introduces a diagnostic protocol for selecting rewards and policies in contextual bandit systems with delayed feedback, revealing that standard offline evaluation often misleads. The protocol integrates checks for reward-north-star alignment, learnability, and robust estimation, enabling practitioners to identify misaligned proxy rewards and justify the complexity of contextual policies, achieving better outcomes in real-world applications.
Key insights
- Static batch evaluations can mis-rank rewards; a denser reward signal allows for significantly faster online learning (N1), a phenomenon not visible in single batch estimates.
- A contextual policy's measured advantage over simpler baselines may stem from its robustness against the unidentifiability of the best single arm rather than from exploiting true per-user personalization (N2).
- Casually computed bootstrap confidence intervals can falsely indicate significance for non-existent lifts (the 'interval trap'), emphasizing the need for robust estimators and correct interval computation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.