Reproduce “ReRoPE: Repurposing RoPE for Relative Camera Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chunyang Li, Yuanbo Yang, Jiahao Shao, Hongyu Zhou, Katja Schwarz, Yiyi Liao
ReRoPE enables camera-controlled image-to-video and video-to-video generation by encoding relative camera relationships in selected low-frequency temporal channels of a pretrained video model’s positional representation, while retaining other temporal and spatial channels. It fine-tunes self-attention without adding a camera encoder. On 200 SDG-1.5M videos, RTE is 0.0629 versus 0.2434 for ReCamMaster, but it does not lead every quality measure.
Key insights
- Pre-trained video diffusion models possess underutilized representational capacity within the low-frequency components of their Rotary Positional Embeddings, which can be repurposed for new control signals.
- Injecting relative camera pose embeddings into these identified redundant RoPE bands allows for robust and shift-invariant camera control, overcoming limitations of absolute pose encoding.
- A targeted and minimal modification strategy that preserves high-frequency RoPE components and spatial encodings prevents catastrophic forgetting and maintains the generative capabilities of the original model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.