Reproduce “A Theoretical Justification for Asymmetric Actor-Critic Algorithms” on OpenResearch

About this paper

Gaspard Lambrechts, Damien Ernst, Aditya Mahajan

This research offers a theoretical justification for asymmetric actor-critic algorithms, demonstrating their improved performance and faster convergence in Partially Observable Markov Decision Processes (POMDPs). The analysis shows these benefits stem from the elimination of error terms such as aliasing bias and inference error, which typically affect symmetric learning approaches.

Key insights

• Asymmetric critics eliminate the `ε_alias` error term by conditioning on the true environment state during training, thereby directly overcoming the bias introduced by agent state aliasing.
• The overall asymmetric actor-critic algorithm achieves zero 'inference error' (`ε_inf,asym = 0`), which represents discrepancies between true and approximate beliefs, a common challenge in POMDPs for symmetric methods.
• The benefits of reduced aliasing and inference errors in the asymmetric approach generally outweigh potential increased approximation errors in the natural policy gradient that may arise from using full state information.

Summary by alphaXiv.