Reproduce “Random Policy Enables In-Context Reinforcement Learning within Trust Horizons” on OpenResearch

About this paper

Weiqin Chen, Santiago Paternain

State-Action Distillation (SAD) introduces an In-Context Reinforcement Learning approach that leverages data generated by random policies within a defined "trust horizon," eliminating the need for optimal or well-trained policies. This method achieved an average 236.3% offline and 135.2% online performance improvement over the best baselines on ICRL benchmarks when all were restricted to random policies, often approaching the performance of oracle methods.

Key insights

• Effective in-context reinforcement learning is achievable by extracting information from purely random policy interactions, circumventing the need for expert demonstrations or well-trained behavioral data.
• The "trust horizon" provides a theoretical and practical mechanism to determine when actions generated by a random policy are sufficiently reliable to serve as optimal labels for ICRL pretraining.
• Even simple random policies in certain problem types (e.g., sparse-reward grid worlds) can implicitly contain enough information to infer optimal actions, challenging the conventional reliance on high-quality input data.

Summary by alphaXiv.