Reproduce “A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Felix Leibfried, Sergio Pascual-Diaz, Jordi Grau-Moya
Researchers from PROWLER.io introduced a unified Bellman optimality principle combining extrinsic reward maximization with intrinsic empowerment, providing theoretical proofs for the existence, uniqueness, and convergence of optimal values. Their Empowered Actor-Critic (EAC) and Actor-Critic with Intrinsic Empowerment (ACIE) algorithms showed improved initial learning performance across various MuJoCo robotics tasks compared to model-free baselines.
Key insights
- It is possible to unify extrinsic reward maximization and intrinsic empowerment into a single Bellman optimality principle, offering a comprehensive theoretical foundation for combining diverse motivations in RL.
- The inherent drive for control (empowerment) can be integrated into advanced deep RL algorithms, providing a mechanism for enhanced exploration and faster acquisition of foundational behaviors, especially in complex environments.
- The proposed framework generalizes various existing Markov Decision Process (MDP) formulations, including standard value iteration and different forms of information-theoretic regularization, by adjusting specific parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.