Reproduce “Supervised Pretraining Can Learn In-Context Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonathan N. Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, Emma Brunskill
A Decision-Pretrained Transformer (DPT) learns sophisticated in-context reinforcement learning strategies, including online exploration and offline conservatism, through a simple supervised pretraining objective. This approach achieves theoretical equivalence to Posterior Sampling, offering provable sample efficiency in decision-making tasks.
Key insights
- A simple supervised pretraining objective can lead to the emergent learning of sophisticated decision-making behaviors, including effective online exploration and appropriate offline conservatism, without explicit algorithmic design.
- DPT can implicitly learn to exploit latent problem structures, such as linear reward functions, achieving performance that surpasses the quality of its pretraining data source.
- Under specific conditions, DPT is theoretically shown to be equivalent to Posterior Sampling, providing a computationally efficient and practical realization of this provably sample-efficient Bayesian RL algorithm.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.