Reproduce “2D Instance Editing in 3D Space” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhuan Xie, Aoxuan Pan, Ming-Xian Lin, Wei Huang, Yi-Hua Huang, Xiaojuan Qi
Researchers from The University of Hong Kong introduced a "2D-3D-2D" framework for interactive 2D image editing by lifting a user-selected object into a manipulable 3D representation. This approach enables physically plausible, large-scale 3D deformations and precise object placement while consistently preserving object identity and producing high-fidelity outputs, outperforming existing purely 2D interactive editing methods.
Key insights
- Leveraging an explicit 3D intermediate representation (3DGS) allows for physically plausible and identity-preserving object manipulation, addressing a core limitation of purely 2D pixel-level methods.
- Combining robust 2D instance segmentation, efficient single-view 3D reconstruction, and real-time 3D deformation under local rigidity assumptions creates a powerful and generalizable editing pipeline.
- Sparse control points with a topology-aware graph and iterative optimization enable efficient, interactive deformation of complex 3D objects, making the system practical for user control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.