Reproduce “Value Gradient weighted Model-Based Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Claas Voelcker, Victor Liao, Animesh Garg, Amir-massoud Farahmand
VaGraM trains model-based reinforcement learning dynamics models with state-wise weights derived from the value function’s gradient, directing limited model capacity toward prediction errors that affect control performance. Across Pendulum and Hopper experiments, it improves stability over direct value-aware losses and retains higher returns than maximum-likelihood training under reduced model capacity and task-irrelevant distractor dimensions, while showing sensitivity to value-estimation quality in larger MuJoCo tasks.
Key insights
- Model learning should prioritize errors according to their effect on value estimates, not merely their statistical size; dimensions with small value gradients can be modeled less precisely without directly harming control.
- Direct VAML can exploit unreliable value-function extrapolation, whereas VaGraM preserves the task-aware weighting while providing more predictable optimization behavior for neural transition models.
- The benefit is largest in under-parameterized settings: when a model has enough capacity to fit all dynamics, VaGraM and MLE become similar, but with irrelevant or difficult dimensions, value-based allocation of capacity becomes useful.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.