Reproduce “TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingyu Zhuang, Di Kang, Yan-Pei Cao, Guanbin Li, Liang Lin, Ying Shan
TIP-Editor introduces a 3D scene editing framework that integrates text prompts, image prompts, and a specified 3D bounding box to enhance user control and output fidelity. The system leverages 3D Gaussian Splatting and a stepwise 2D personalization strategy to achieve precise appearance and location manipulation, outperforming existing text-only methods.
Key insights
- Integrating multi-modal inputs (text, image, and 3D bounding box) significantly improves user control over both the visual appearance and precise spatial placement of 3D edits.
- A stepwise 2D personalization approach using DreamBooth-like fine-tuning for the scene and LoRA layers for novel concepts, combined with an attention-based localization loss, effectively prevents concept interference and ensures accurate appearance and location capture.
- Leveraging 3D Gaussian Splatting's explicit and flexible data structure, alongside targeted optimization strategies (global/local SDS, pixel refinement), enables high-quality, localized editing while preserving the integrity of unedited scene parts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.