Reproduce “Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models” on OpenResearch

About this paper

Thibaut Kulak

The paper introduces LDM-v0, a Large Decision Model, which is a multi-task transformer policy trained offline on a diverse collection of thousands of reinforcement learning environments. This model demonstrates that a single shared parameterization can achieve over 80% of reference policy performance in more than 1,600 environments, matching it in approximately 1,000, across varied domains like robotics, autonomous systems, and resource optimization.

Key insights

• A single shared transformer policy can effectively model a wide range of distinct RL behaviors across thousands of heterogeneous environments, often matching the performance of independently trained task-specific policies.
• Automated infrastructure for environment orchestration and reference-policy supervision is critical for integrating and generating high-quality data from diverse and often incompatible RL environments at scale.
• Performance in heterogeneous offline RL sequence modeling improves with increased model capacity, suggesting that scaling laws, similar to those in other large-scale sequence modeling domains, apply to this context.

Summary by alphaXiv.