Reproduce “Hilbert Space Embeddings of POMDPs” on OpenResearch

About this paper

Yu Nishiyama, Abdeslam Boularias, Arthur Gretton, Kenji Fukumizu

The paper presents kPOMDP, a sample-based controller for swing-up cart balancing when only pendulum angle is observed and angular velocity is hidden. It embeds beliefs, transitions, and observations in RKHSs, updates beliefs with Kernel Bayes’ Rule, and performs value iteration. In simulation, it nearly reached the maximum score of 100 and outperformed tested histogram policies, under the stated setup.

Key insights

• Leveraging RKHS embeddings allows for a nonparametric approach to POMDPs that handles complex, continuous state and observation spaces without explicit discretization.
• The computational complexity of kPOMDPs scales primarily with the number of training samples ($n$), rather than the intrinsic dimensionality of the state or observation spaces, offering advantages in high-dimensional settings.
• The empirical kernel Bellman operator can be enforced to be isotonic and contractive through normalization of weight vectors, which is crucial for the convergence and stability of the value iteration process.

Summary by alphaXiv.