Reproduce “Online Causal Kalman Filtering for Stable and Effective Policy Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuo He, Lang Feng, Xin Cheng, Lei Feng, Bo An

Researchers from Nanyang Technological University developed Online Causal Kalman Filtering for Policy Optimization (KPO), a method that processes token-wise importance sampling ratios autoregressively to smooth local fluctuations and preserve within-sequence local coherent structure. This approach achieved superior performance and enhanced training stability on six challenging mathematical reasoning tasks compared to strong baselines.

Key insights

  • Token-level importance sampling (IS) ratios exhibit high variance and structural inconsistency (rapid, short-lived fluctuations) in local off-policy derivation, significantly destabilizing policy optimization.
  • Despite global heterogeneity, off-policy deviation should ideally display local homogeneity within semantically similar segments, necessitating a temporally coherent smoothing approach.
  • Causal Kalman filtering effectively transforms these rapidly switching, high-frequency token dynamics into long, coherent segments, leading to more stable policy gradients.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.