Reproduce “Direct Advantage Estimation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hsiao-Ru Pan, Nico Gürtler, Alexander Neitz, Bernhard Schölkopf

This paper presents Direct Advantage Estimation (DAE), a method that interprets the advantage function as a causal effect and directly estimates it by minimizing the variance of a transformed return. The Max Planck Institute for Intelligent Systems and DeepMind researchers show DAE consistently outperforms Generalized Advantage Estimation (GAE) in terms of sample efficiency and final policy performance across MinAtar and Atari environments.

Key insights

  • The advantage function can be causally interpreted as the effect of an action on the expected return, suggesting it is more stable and local than the Q-function.
  • The advantage function is the unique function that minimizes the variance of a transformed return, providing a principled basis for its direct estimation.
  • A practical loss function allows for n-step bootstrapping and effective learning of the advantage function alongside a value function within a single network.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.