Reproduce “Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shengze Wang, Michael Stengel, Tianye Li, Seonwook Park, Amrita Mazumdar, Koki Nagano, Alex Trevithick, Shalini De Mello

Researchers at NVIDIA developed Grounded-Exo2Ego, a framework for robustly generating egocentric video from a single exocentric input by integrating a dual-branch diffusion model with structured semantic grounding and improved data pipelines. The system achieves superior image, video, and object metric performance, with PSNR improvements of up to +2.59 dB and LPIPS reductions of 30.5% compared to prior methods on the EgoExo4D dataset.

Key insights

  • Relying solely on geometric conditioning is insufficient for exocentric-to-egocentric video generation, necessitating structured semantic grounding to plausibly synthesize unobserved regions.
  • Resolving the misalignment between ground-truth egocentric camera poses and monocular 3D reconstructions via re-localization is critical for providing consistent training signals and improving output quality.
  • Spatially gating semantic object tokens with reprojected masks significantly improves the accuracy of object localization and overall scene reconstruction in generated egocentric videos.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.