Reproduce “Modelling transition dynamics in MDPs with RKHS embeddings” on OpenResearch

About this paper

Steffen Grunewalder, Guy Lever, Luca Baldassarre, Massi Pontil, Arthur Gretton

The authors learn conditional next-state expectations directly from sampled transitions, using kernel embeddings in a reproducing kernel Hilbert space, and use them in value iteration instead of fitting a transition density and integrating it. On a pendulum value-estimation task, the method runs 50–110 times faster than NPDP and has lower error; guarantees do not extend generally to continuous-state problems.

Key insights

• Representing conditional expectation operators directly as RKHS inner products transforms intractable integration problems into computationally efficient kernel computations.
• The approximate Bellman optimality operator, constructed using RKHS embeddings, is a sup-norm contraction mapping, guaranteeing the convergence of value iteration to a unique fixed point.
• The approach allows for robust reinforcement learning in complex domains where only raw sensor data is available, without requiring explicit system dynamics models or predefined state representations.

Summary by alphaXiv.