Reproduce “EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Libo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Rui Xu, Xiaoxiao Long, Wenjia Wang, Jingbo Wang and 5 more

EgoReAct introduces the first autoregressive framework to generate 3D human reactions in real-time from streaming egocentric video, enforcing causality and 3D spatial alignment, and it includes a novel Human Reaction Dataset. The approach achieves a Frechet Inception Distance of 0.456 for realism and a first-frame latency of 21.33 ms, outperforming prior methods on metrics like realism, diversity, and spatial consistency.

Key insights

  • Strict causality and autoregressive processing are essential for enabling real-time, streaming generation of human reactions from egocentric video.
  • Integrating 3D dynamic cues, such as metric depth and head velocity, significantly enhances spatial alignment and geometric consistency in generated reactions.
  • The creation of a large-scale, spatially aligned dataset like HRD is crucial for training robust models that accurately map egocentric perception to 3D human movements.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.