Reproduce “Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thomas M. McDonald, Lucas Maystre, Mounia Lalmas, Daniel Russo, Kamil Ciosek

This paper introduces the "Impatient Bandit" approach, a novel method for optimizing recommender systems for long-term user satisfaction despite delayed rewards. It leverages a Bayesian reward model to predict long-term outcomes from progressively revealed intermediate feedback, enabling a Thompson Sampling-based bandit algorithm to rapidly identify and promote content with high sustained engagement.

Key insights

  • Long-term user engagement and satisfaction can be accurately predicted from early, progressively revealed intermediate observations, often within 10 days.
  • Leveraging meta-learning from historical data of established content allows for robust estimation of prior distributions (e.g., temporal correlations) that accelerate learning for new content.
  • Thompson Sampling is effective in balancing exploration and exploitation in settings with progressive, delayed feedback, allowing for quick adaptation and improved long-term optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.