Reproduce “Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformers” on OpenResearch

About this paper

Federico Stella, Fei Jiang, Zhongshi Jiang, Zohar Barzelay, Emanuel Garbin, Amin Jourabloo, Liuhao Ge

Researchers from Meta and École polytechnique fédérale de Lausanne developed a method for photorealistic novel view synthesis of human faces at 512x512 resolution using next-scale transformers, producing consistent multiple views from single or multi-view inputs. The approach achieved superior image quality, perceptual fidelity, and cross-view consistency on the PSGS dataset compared to diffusion-based baselines, particularly for non-frontal views, and enabled high-quality 3D head reconstruction when integrated with a Gaussian lifting model.

Key insights

• Next-scale autoregressive transformers can be effectively adapted for high-resolution, multi-view human face synthesis by extending their architecture to predict multiple outputs concurrently, leading to improved cross-view consistency.
• A progressive, multi-stage training approach, starting with resolution adaptation on general-purpose data and then specializing on smaller, realistic human datasets, is crucial for achieving high-quality results in data-scarce domains without relying on extensive 2D pre-training.
• Fine-tuning the VQ-VAE on the target dataset and avoiding classifier-free guidance were empirically found to be beneficial for specialized tasks with limited human face data, improving reconstruction quality and identity preservation.

Summary by alphaXiv.