Reproduce “HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention” on OpenResearch

About this paper

Shuang Chen, Amir Atapour-Abarghouei, Hubert P. H. Shum

HINT, a Transformer-based model developed at Durham University, excels at reconstructing missing image regions by integrating a Mask-aware Pixel-shuffle Down-sampling module and a Spatially-activated Channel Attention Layer. It achieves superior qualitative and quantitative inpainting performance, producing more coherent structures and sharper textures with practical inference times, outperforming many state-of-the-art methods including diffusion models on various benchmarks.

Key insights

• Preserving visible information consistently during downsampling through mask-aware operations is crucial for accurate inpainting, especially when dealing with large and irregular missing regions.
• It is possible to efficiently integrate spatial awareness into channel self-attention, enabling effective global context modeling without incurring the quadratic computational cost of full spatial attention.
• An FFN-SCAL-FFN "Sandwich" block structure can significantly enhance feature extraction and the effectiveness of attention mechanisms when processing inputs with missing information.

Summary by alphaXiv.