Reproduce “A UCB Bandit Algorithm for General ML-Based Estimators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yajing Liu, Erkao Bao, Linqi Song
Researchers at the University of Minnesota and City University of Hong Kong developed ML-UCB, a generalized Upper Confidence Bound algorithm that integrates arbitrary machine learning models for reward estimation in multi-armed bandit scenarios. This approach derives its exploration bonus from the empirical power-law learning curve of the ML estimator, demonstrating sublinear regret and achieving 40.3% lower cumulative regret compared to LinUCB in a simulated collaborative filtering task.
Key insights
- The empirical power-law decay of an ML estimator's Mean Squared Error (MSE $ $ n^{-s}) can be used as a model-agnostic method to derive a generalized concentration inequality for UCB.
- This approach enables the construction of principled exploration bonuses for arbitrary machine learning models, bypassing the need for model-specific theoretical analysis.
- A higher convergence rate 's' for the ML estimator's learning curve directly correlates with a lower theoretical regret bound for the ML-UCB algorithm.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.