Reproduce “InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junqi You, Chieh Hubert Lin, Weijie Lyu, Zhengbo Zhang, Ming-Hsuan Yang
University of California Merced and collaborators develop InstaInpaint, a feed-forward framework that performs 3D scene inpainting in 0.4 seconds using a Large Reconstruction Model adapted through masked fine-tuning, achieving 1000x speedup over optimization-based methods while maintaining state-of-the-art quality with best FID scores (84.535 on SPIn-NeRF, 198.54 C-FID on LLFF) through a novel self-supervised training strategy that generates multi-view consistent masks using object segmentation, geometric projection, and random sampling without requiring paired before-and-after datasets, enabling seamless object removal, insertion, and multi-region editing by teaching the transformer to deduce cross-view consistent geometry from reference views while handling inconsistent masked regions in other viewpoints.
Key insights
- Self-supervised masked fine-tuning can successfully adapt reconstruction models to generative inpainting tasks without paired training data
- Combining multiple mask types (25% object, 25% geometric, 50% random) prevents overfitting to object segmentation biases while maintaining geometric consistency
- Reference-based inpainting works best when the reference view doesn't receive explicit mask tokens, allowing the model to learn seamless integration between reconstructed and inferred regions
- Large Reconstruction Models can be repurposed for complex generative tasks beyond their original reconstruction purpose
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.