Reproduce “Geometry-Instructed Video Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chirui Chang, Xiaoyang Lyu, Yi-Hua Huang, Haoru Tan, Shizhen Zhao, Yikang Ding, Jianmin Bao, Xin Tao and 2 more

A new framework, Geometry-Instructed Video Editing (GIVE), enables precise object-level geometric manipulations in videos by leveraging lightweight geometry streams for before and after object states. This approach consistently updates geometry-dependent secondary effects, yielding up to an 85.92% user preference WinRate in blinded studies and outperforming baselines on synthetic benchmarks.

Key insights

  • Lightweight geometry streams, specifically depth-box (for placement and extent) and orientation-box (for appearance-agnostic heading), are sufficient to provide explicit 3D object-state guidance to a generative model without full 3D reconstruction.
  • Procedural data generation from a graphics engine is a viable and effective strategy to overcome the lack of controlled, paired supervision for complex geometric video editing tasks.
  • Aligning Rotary Positional Encoding (RoPE) indices across input streams while adding a state offset effectively distinguishes pre-edit and post-edit conditions, maintaining spatiotemporal coherence during the edit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.