Reproduce “Gradient Ascent for Active Exploration in Bandit Problems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pierre Ménard
Pierre Ménard from the University of Toulouse developed a gradient ascent-based algorithm for fixed confidence active exploration bandit problems, which achieves asymptotic optimality while drastically reducing computational cost per step. This new approach is roughly 100 times faster per step than direct tracking methods, performing comparably in expected sample counts to other competitive heuristics.
Key insights
- Asymptotic optimality in active exploration can be achieved through computationally efficient online sub-gradient ascent, rather than resource-intensive exact optimization at each step.
- The inherent non-smoothness of the objective function in active exploration problems can be effectively handled by online mirror ascent, linking stochastic bandits with online convex optimization.
- Empirically successful heuristic sampling rules often implicitly perform a form of online optimization, suggesting the sufficiency of iterative, gradient-based updates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.