Reproduce “KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruicheng Zhang, Kaixi Cong, Jun Zhou, Zhizhou Zhong, Zunnan Xu, Shuiyang Mao, Wei Liu, Xiu Li
KVPO introduces an ODE-native online Group Relative Policy Optimization framework for aligning streaming autoregressive video generators with human preferences. The method leverages causal-semantic exploration through Key-Value (KV) cache manipulation and models a surrogate policy in the velocity-field space, resulting in improved visual quality, motion quality, and text-video alignment; for instance, it boosted Visual Quality by 28.4% on LongLive for multi-prompt long videos.
Key insights
- Manipulating the historical Key-Value cache, instead of injecting noise, provides an effective and on-manifold mechanism for causal-semantic exploration, enabling diverse high-level semantic variations in autoregressive video generation.
- Modeling a surrogate policy directly within the ODE's native velocity-field space offers a more faithful representation of the generative model's intrinsic preferences compared to using latent Euclidean distances, significantly enhancing preference optimization.
- The gradient of the TVE-based policy optimization objective can be shown to reduce to a reward-weighted contrastive flow-matching objective, effectively steering the ODE dynamics towards preferred trajectories by minimizing velocity residuals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.