Reproduce “Towards Instance-Optimal Offline Reinforcement Learning with Pessimism” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ming Yin, Yu-Xiang Wang

The authors analyze APVI, whose pessimistic value-iteration penalty adapts to return variance and data coverage. Its high-probability suboptimality bound weights uncertainty by optimal-policy visitation and inverse behavior-policy coverage, plus a higher-order term. With optimal-policy coverage, a local minimax lower bound supports similar dependence, though the bounds differ by H divided by minimum positive behavior occupancy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.