Reproduce “Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jordi Grau-Moya, Felix Leibfried, Tim Genewein, Daniel A. Braun
Researchers from the Max Planck Institute developed a unified variational free energy framework for planning in Markov Decision Processes, simultaneously accounting for information-processing constraints on policies and model uncertainty. The framework introduces a generalized Bellman recursion and a convergent iterative algorithm, showing how agents can balance reward maximization with computational costs and model ambiguity through tunable parameters.
Key insights
- A mathematical congruence exists between information-theoretic control (bounded rationality) and principles for managing model uncertainty (biasing beliefs using KL-divergence).
- The unified framework introduces two interpretable parameters, alpha (policy stochasticity) and beta (model uncertainty attitude), allowing for flexible modeling of agent behaviors ranging from perfectly rational to highly bounded, and optimistic to pessimistic.
- The generalized framework recovers established MDP planning paradigms, such as standard value iteration, Bayesian MDP planning, and robust planning, as specific limit cases by adjusting the alpha and beta parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.