Reproduce “Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learning” on OpenResearch

About this paper

Ziyi Zhang, Li Shen, Deheng Ye, Yong Luo, Huangxuan Zhao, Meng Liu, Wei Yu, Lefei Zhang

Researchers at Wuhan University, Sun Yat-sen University, Tencent, and Xiaomi introduce MVC-ZigAL, a reinforcement learning framework designed to enhance few-step Text-to-Multiview (T2MV) diffusion models. This framework significantly improves both per-view fidelity and cross-view consistency, achieving superior performance over state-of-the-art baselines on benchmarks like MATE-3D while maintaining fast generation speeds.

Key insights

• Reformulating the T2MV denoising process as a multiview-aware MDP with a joint-view reward function is crucial for optimizing multiview consistency.
• Leveraging 'self-reflection' through zigzag sampling to create advantage signals from refined trajectories effectively addresses the issue of weak learning signals in few-step generation regimes.
• A constrained optimization approach, rather than simple weighted sum of rewards, more effectively balances the often conflicting objectives of individual view quality and overall multiview geometric consistency.

Summary by alphaXiv.