Reproduce “Towards Interactive Image Inpainting via Sketch Refinement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chang Liu, Shunxin Xu, Jialun Peng, Kaidong Zhang, Dong Liu
Researchers at the University of Science and Technology of China developed SketchRefiner, a two-stage interactive image inpainting system that refines imperfect user-drawn sketches to robustly guide image restoration. The system effectively corrects sketch misalignment and incoherence, leading to superior quantitative metrics such as PSNR 22.16, SSIM 0.769, and FID 9.531 on ImageNet, and enhanced qualitative results for interactive image editing tasks compared to existing methods.
Key insights
- Explicitly refining imperfect user-drawn sketches for misalignment and incoherence is crucial for generating high-quality, structurally consistent inpainting results without artifacts.
- Integrating sketch guidance through multi-dimensional feature encoding and adaptive modulation (PSE and SFA blocks) is more effective than simple concatenation, preventing "feature diminishing" and leveraging semantic understanding.
- A combination of pixel-wise (L1) and region-level (Cross-Correlation) loss functions is essential for robust sketch refinement, effectively addressing structural inconsistencies.
- Synthetically generated sketches that mimic real human drawing characteristics (misalignment, incoherence, abstraction) can effectively address data scarcity and improve model generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.