Reproduce “EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junho Park, Andrew Sangwoo Ye, Taein Kwon

The EgoWorld framework translates a single exocentric image into a high-fidelity egocentric view by integrating diverse multi-modal observations. This approach demonstrates improved realism and generalization, consistently reducing Fréchet Inception Distance (FID) by 30-39% and increasing Peak Signal-to-Noise Ratio (PSNR) by 3-5 dB across various unseen scenarios and real-world datasets compared to existing methods.

Key insights

  • Integrating a diverse set of complementary exocentric observations (geometric via depth/pose/point clouds and semantic via text) is crucial for accurate, detailed, and generalized egocentric view synthesis.
  • A dedicated 3D egocentric hand pose estimator, trained to predict the first-person hand configuration, provides a robust spatial anchor for transforming the scene and ensuring realistic hand-object interactions in the target view.
  • Latent diffusion models, when effectively conditioned by multi-modal inputs, can bridge the substantial visual and geometric gap between exocentric and egocentric perspectives from a single input image.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.