Reproduce “ScaleVid: Geometry-Aware Video Object Scaling with Mesh-Free Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youze Huang, Penghui Ruan, Bojia Zi, Xianbiao Qi, Shihao Zhao, Rong Xiao
ScaleVid enables geometry-aware video object scaling, allowing precise 3D resizing of objects in videos based on user-specified factors without requiring explicit 3D reconstruction during inference. The system achieves superior geometric alignment, background preservation, and perceptual quality on various benchmarks while maintaining efficient processing.
Key insights
- Decoupling geometric transformation learning from realistic video synthesis using pseudo-source reconstruction allows learning precise 3D control from synthetic paired data while leveraging real videos for appearance refinement and temporal consistency.
- A progressive 2D-to-3D training strategy, starting with planar transformations and moving to 3D deformation guidance, effectively builds a robust system for complex geometric scaling.
- Incorporating bidirectional training for the Deformer and an inverse deformation step during Main Model training helps maintain transformation consistency and significantly improves background preservation and foreground fidelity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.