Reproduce “Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marco Bagatella, Thomas Rupf, Georg Martius, Andreas Krause
Soft Forward-Backward (Soft FB) representations extend zero-shot reinforcement learning to optimize general utility functions, which are arbitrary differentiable functions of a policy's occupancy measure. This approach introduces maximum entropy regularization during pre-training to learn a rich set of stochastic policies, allowing for efficient zero-order search at test-time to achieve near-optimal solutions for diverse non-linear objectives.
Key insights
- Maximum entropy regularization during pre-training can effectively capture a broader family of stochastic policies, which is crucial for optimizing diverse general utility functions.
- Reparameterizing task embeddings allows for intuitive control over policy stochasticity and enables Soft FB to implicitly cover a policy space rich enough to approximate optimal solutions for any differentiable general utility.
- Leveraging explicit generative models for successor measures significantly enhances the accuracy of offline policy evaluation and, consequently, overall performance on general utility tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.