Reproduce “Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Sampling” on OpenResearch

About this paper

Min Zeng, Yichen Zhang, Xiaofeng Shao

This paper develops self-normalized inference for constant-stepsize Temporal-Difference (TD) learning under Markovian sampling, providing asymptotically pivotal confidence regions for the stationary target and the true projected Bellman solution. The approach utilizes Richardson–Romberg extrapolation for bias reduction and enables online, memory-efficient computation without explicit long-run covariance estimation.

Key insights

• Functional Central Limit Theorems (FCLTs) for constant-stepsize TD learning under Markovian sampling must explicitly account for the multiplicative component of the TD matrix and iterate error to accurately characterize long-run covariance.
• Richardson–Romberg extrapolation, when applied to parallel recursions driven by the same Markov trajectory, effectively reduces the bias of constant-stepsize estimates, yielding a higher-order approximation to the true Bellman solution.
• Self-normalized inference allows for constructing asymptotically pivotal statistics, which sidesteps the need for explicit estimation of complex long-run covariance matrices or tuning parameters, simplifying practical uncertainty quantification.

Summary by alphaXiv.