Reproduce “RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting” on OpenResearch

About this paper

Hejun Wang, Jinxi Li, Junwei Jiang, Shiwei Mao, Hu Cheng, Shouwang Huang, Bo Yang

RelightFormer, a feed-forward generative Transformer, directly synthesizes relit images from single or multi-view inputs and a target illumination, bypassing traditional inverse rendering. It achieved state-of-the-art performance across various relighting benchmarks, including an sPSNR of 23.80 for single-image and 25.07 for 32-view relighting, and introduced the large-scale Laval Objaverse Dataset (LOD).

Key insights

• Directly synthesizing relit images with a generative Transformer provides a more efficient and robust alternative to ill-posed and computationally expensive inverse rendering.
• Deeply integrating target illumination and multi-view features via a cross-attention mechanism, inspired by rendering equations, is crucial for high-fidelity and physically consistent relighting.
• Employing permutation-invariant encoding allows for symmetrical processing of unordered multi-view inputs, which is critical for robust multi-view reasoning and generalization.

Summary by alphaXiv.