Reproduce “Reward Maximisation through Discrete Active Inference” on OpenResearch

About this paper

Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, Ryan Smith

The authors show when active inference matches reward maximization and Bellman-optimal planning. Standard active inference evaluates complete action sequences and is Bellman optimal only for a one-step horizon. Recursive sophisticated inference evaluates later actions as state-contingent choices, matching backward induction at any finite horizon with known transitions and rewards, zero-temperature preferences, and exact state inference in partially observed problems.

Key insights

• Active Inference can achieve Bellman optimality by encoding rewards as specific preference distributions, particularly in the "zero-temperature limit."
• A recursive "sophisticated active inference" scheme is necessary and sufficient to recover multi-step Bellman optimal solutions, generalizing dynamic programming's backward induction.
• Active Inference inherently integrates intrinsic exploration (maximizing entropy of future states) and ambiguity resolution (minimizing uncertainty) with extrinsic reward maximization.

Summary by alphaXiv.