Reproduce “Generative Blocks World: Moving Things Around in Pictures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vaibhav Vavilala, Seemandhar Jain, Rahul Vasanth, D.A. Forsyth, Anand Bhattad
Generative Blocks World introduces a method for semantic image editing that leverages a 3D primitive-based scene representation to manipulate images. It achieves superior geometric adherence (AbsRel_src of 0.072 vs. LooseControl's 0.143) and texture preservation (PSNR of 18.7 vs. LooseControl's 6.65) for random camera moves, allowing precise control over object manipulation and camera movements.
Key insights
- Representing scenes as editable 3D convex polytopes provides an intuitive and geometrically consistent interface for complex image manipulations, including camera movements.
- Projecting source image textures using 3D point correspondences is crucial for maintaining object identity and visual fidelity when generating edited scenes.
- Powerful 2D diffusion models can be effectively conditioned on abstract 3D primitive geometry without requiring extensive retraining or fine-tuning of the generator itself.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.