Reproduce “GeoDiffuser: Geometry-Based Image Editing with Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rahul Sajnani, Jeroen Vanbaar, Jie Min, Kapil Katyal, Srinath Sridhar

GeoDiffuser introduces a zero-shot, optimization-based method for precise geometric image editing, enabling 2D and 3D object transformations and removal within a single framework. It quantitatively outperforms state-of-the-art baselines in edit adherence (e.g., Mean Distance 5.579 for 2D edits vs. 38.070 for Dragon Diffusion) and maintains high content preservation.

Key insights

  • Geometric transformations can be effectively integrated into the attention mechanism of diffusion models by transforming reference queries before attention computation.
  • Combining a reference diffusion process (for style preservation) with an edit diffusion process (for new content generation) via masked blending allows for realistic object transformation and background inpainting.
  • A zero-shot, optimization-based approach, guided by specific loss terms, can achieve precise and realistic geometric edits without requiring additional model training or fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.