Reproduce “FaithFill: Faithful Inpainting for Object Completion Using a Single Reference Image” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rupayan Mallick, Amr Abdalla, Sarah Adel Bargal
FaithFill enables faithful object completion from a single reference image by synthesizing multiple views of the object using a diffusion-based Neural Radiance Field, then finetuning a Stable Inpainting model on this generated multi-view dataset. The method produces visually and semantically accurate completions, outperforming existing methods in human and AI evaluations while preserving object characteristics and scene context.
Key insights
- Synthetically generating multiple 2D views of an object from a single reference image, using a diffusion-based Neural Radiance Field (NeRF), effectively transforms a one-shot inpainting problem into a multi-view learning scenario.
- Creating a synthetic dataset with diverse occlusion patterns from these generated views allows for robust finetuning of a diffusion-based inpainting model, overcoming the underfitting issues common with single-image finetuning.
- Employing Low-Rank Adaptation (LoRA) for finetuning significantly reduces computational overhead while maintaining the performance of the inpainting module.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.