Reproduce “ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Omar El Khalifi, Thomas Rossi, Oscar Fossey, Thibault Fouque, Ulysse Mizrahi, Philip Torr, Ivan Laptev, Fabio Pizzati and 1 more
ActCam introduces a training-free method for simultaneously controlling character motion and camera trajectory in video generation, which uses camera-aligned conditioning signals and a two-phase denoising schedule. The approach achieves superior control fidelity and visual quality, demonstrated by higher overall VBench scores of 0.8497 compared to 0.8370 for a fine-tuned baseline, and a lower Sampson Error of 0.4546 versus 0.5665.
Key insights
- Developing geometrically consistent, camera-aligned conditioning representations is crucial for robust joint motion and camera control, resolving ambiguities inherent in 2D signals.
- A two-phase conditioning schedule, transitioning from depth+pose to pose-only guidance, effectively balances global scene structure adherence with fine-grained detail generation, preventing coarse depth artifacts from propagating.
- Explicitly removing the static reference character from the reconstructed background depth prevents interference with dynamic character motion and ensures consistent compositing in the generated video.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.