Reproduce “GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Youngjoong Kwon, Yao He, Heejung Choi, Chen Geng, Zhengmao Liu, Jiajun Wu, Ehsan Adeli

Researchers at Stanford University developed GenFusion, a feed-forward method that captures human performance and synthesizes novel views from monocular video by progressively updating a canonical appearance space and employing probabilistic diffusion-based rendering. This approach achieves perceptually sharper and temporally consistent results compared to prior deterministic and per-frame probabilistic methods.

Key insights

  • Continuously accumulating appearance information into a canonical space using visibility-weighted averaging effectively builds a comprehensive temporal context for reconstructing unobserved regions.
  • Formulating novel view rendering as a probabilistic regression problem with a diffusion model enables the generation of high-fidelity, perceptually realistic images that are robust to minor misalignments and dynamic deformations, avoiding blurry outputs.
  • The synergistic combination of progressive temporal context updates and diffusion-based probabilistic rendering is essential for achieving both high perceptual quality and temporal consistency in feed-forward human performance capture.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.