Reproduce “Synthetic Experience Replay” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cong Lu, Philip J. Ball, Yee Whye Teh, Jack Parker-Holder
SynthER uses a diffusion model to generate full state-action-reward-next-state transitions from collected experience and add them to replay for existing reinforcement learning algorithms. In offline learning, it augments or replaces training data; in online learning, it mixes generated and real replay. On walker2d medium-expert, the authors report near-original performance from 3% of original data, limited to that task.
Key insights
- Modern diffusion models possess sufficient generative power to capture the high-level statistics of RL experience datasets more faithfully than prior generative models (VAEs, GANs).
- High-quality synthetic data can effectively expand small real datasets, preventing overfitting and enabling stable training of larger policy and value networks for improved performance.
- Integrating synthetic experience allows online RL agents to significantly increase their update-to-data ratio, leading to enhanced sample efficiency without requiring complex algorithmic modifications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.