Reproduce “Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning” on OpenResearch

About this paper

Thanh Nguyen, Tri Ton, Hongbin Choe, Tung M. Luu, Chang D. Yoo

Researchers at KAIST developed Bootstrapped Flow Q-Learning (BFQ), an Offline Reinforcement Learning framework enabling fast and highly expressive policy learning through accurate single-step action generation. BFQ achieved an average normalized score of 92.8 on D4RL MuJoCo tasks, trained in 7.8 hours (33% faster than 5-step DQL), and inferred actions at 851.2 Hz.

Key insights

• Directly modeling the displacement vector along a flow path, rather than instantaneous marginal velocity fields, can enable highly expressive single-step action generation in generative policies.
• A recursive decomposition and 'bootstrapping' of short-range displacements can effectively learn complex, long-range noise-to-action mappings with a single, shared policy network.
• Architectural preconditioning and a balanced boundary conditioning ratio are crucial for stabilizing optimization and anchoring the learned policy to local flow dynamics, promoting robust and high-performing single-step generation.

Summary by alphaXiv.