Reproduce “Instructive3D: Editing Large Reconstruction Models with Text Instructions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kunal Kathare, Ankit Dhiman, K Vikas Gowda, Siddharth Aravindan, Shubham Monga, Basavaraja Shanthappa Vandrotti, Lokesh R Boregowda
Instructive3D from Samsung R&D Institute India enables fine-grained, text-conditioned editing of 3D objects generated by Large Reconstruction Models (LRMs) by applying a diffusion process within the LRM's triplane latent space. This approach allows users to modify textures, colors, and patterns while preserving object geometry, outperforming baselines in perceptual quality and realism.
Key insights
- Fine-grained 3D editing can be effectively achieved by performing text-conditioned diffusion directly in the compact latent space of a 3D reconstruction model's triplane representation.
- Leveraging pre-trained 2D image editing models (InstructPix2Pix) and 3D reconstruction models (LRMs) enables efficient data curation, circumventing the need for large, manually-edited 3D datasets.
- Concatenating the original latent with the noisy latent as input to the denoiser helps preserve the fundamental 3D geometry during the editing process.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.