Reproduce “A General Framework for Sequential Decision-Making under Adaptivity Constraints” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nuoya Xiong, Zhaoran Wang, Zhuoran Yang
Researchers from Tsinghua, Northwestern, and Yale introduce a general Eluder-Condition (EC) class and develop algorithms for sequential decision-making under adaptivity constraints. These algorithms achieve optimal Õ(√K) regret with an exponentially reduced, logarithmic number of policy switches (O(log K)) for rare policy switch settings, and provide tight regret bounds for batch learning, extending theoretical guarantees to general function approximation.
Key insights
- A refined "eluder condition" with a logarithmic factor bound on cumulative squared error is key to enabling efficient policy switching under constraints.
- A lazy policy switching strategy, where updates are triggered only by significant discrepancies in in-sample error, allows for exponentially reduced switching costs.
- The proposed EC framework unifies the analysis of diverse RL problems, showing that near-optimal regret can be maintained even with severe adaptivity limitations when using general function approximation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.