Reproduce “A Bellman Optimality Equation for Plasticity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeremy Lucas, Doina Precup
Jeremy Lucas and Doina Precup formulate plasticity as observation-to-action information and derive a discounted Bellman optimality equation for maximizing it in finite Markov decision processes. The resulting contraction-based value iteration and knapsack dynamic program work in tabular control, while experiments show that plasticity can favor noisy, observation-sensitive behavior that differs from empowerment-oriented policies.
Key insights
- Plasticity and empowerment optimize different information directions: plasticity rewards actions that reveal distinctions in observations, whereas empowerment rewards actions that preserve control over future observations.
- In the control-gated MDP, the plasticity-optimal policy is deterministic and reaches about 0.92 bits of plasticity but 0 bits of empowerment, demonstrating that the objectives can strongly diverge.
- In the two-room gridworld, the learned policy prefers a noisy region because environmental variation creates more opportunities for observation-dependent actions, even though deterministic regions may offer greater controllability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.