Reproduce “CamPilot: Improving Camera Control in Video Diffusion Model with Efficient Camera Reward Feedback” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenhang Ge, Guibao Shen, Jiawei Feng, Luozhou Wang, Hao Lu, Xingye Tian, Xin Tao, Ying-Cong Chen

CamPilot introduces a reward feedback learning framework for video diffusion models that enhances camera control by employing an efficient, camera-aware 3D decoder. This approach significantly improves alignment with desired camera trajectories, achieving superior visual fidelity (FID 11.22) and temporal coherence (FVD 81.35) while enabling precise 3D scene reconstruction.

Key insights

  • Directly decoding video latents into 3D representations (3D Gaussians) provides an efficient and intrinsically 3D-aware mechanism for computing rewards in camera control tasks, circumventing the overhead of RGB decoding.
  • Misalignment between input video latents and camera poses leads to geometrically inconsistent and blurry 3D Gaussian renderings, which can be effectively leveraged as a strong negative reward signal.
  • A visibility-aware reward objective stabilizes training by focusing supervision on deterministic regions of the generated content, preventing over-penalization of stochastically generated or hallucinated elements in generative models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.