Reproduce “Value Forcing: Aligning Return Distributions at All Noise Levels” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chongyi Zheng, Zhiyuan Zhou, Qiyang Li, Benjamin Eysenbach, Sergey Levine

Value Forcing learns distributions of possible returns from offline transitions using flow matching, enforcing Bellman consistency at every noise level rather than only on final predictions. In a fixed-policy return-estimation test on puzzle-4x4, its mean 1-Wasserstein distance was 22 ± 5, versus 118 ± 6 for PCBF, an 81% reduction reported by the authors. This measures distribution accuracy, not policy success.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.