Reproduce “QF3: Fast Flow RL with Filtered Q-Gradients” on OpenResearch

About this paper

Chung Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Gaurav Singh, Jinkun Cao, Ken Goldberg, Pieter Abbeel and 2 more

QF3 addresses robot learning by improving flow policies from replayed experience. It anchors policy updates to replay actions through flow matching, then applies a critic gradient to a one-step action prediction, clipping velocity corrections that stray too far from the replay target. In tested settings, clipping prevents collapse; on G1 dance tracking, training reaches full episodes about 10 times faster than FPO++.

Key insights

• The velocity-space clip acts as a local trust region: it limits critic influence where value estimates may be unreliable, without requiring backpropagation through the full flow sampler.
• On four MuJoCo tasks, QF3 was competitive with tested off-policy flow-RL methods; a single hyperparameter setting achieved at least 79% of task-specific best return on each task.
• QF3 supports both learning humanoid controllers from scratch and reward-based fine-tuning of pretrained manipulation policies, though the manipulation evaluations were conducted in simulation.

Summary by alphaXiv.