Reproduce “StreamME: Simplify 3D Gaussian Avatar within Live Stream” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Luchuan Song, Yang Zhou, Zhan Xu, Yi Zhou, Deepali Aneja, Chenliang Xu

StreamME enables on-the-fly, high-fidelity 3D head avatar reconstruction directly from live monocular video streams, achieving state-of-the-art image quality and compact models with rapid inference, facilitating real-time interactive applications. It operates by removing reliance on complex deformation networks and introducing geometry-driven motion-aware point management.

Key insights

  • A geometry-driven approach that directly links Gaussian points to a tracked mesh, coupled with motion-aware point management, can achieve rapid facial deformation adaptation without heavy MLPs.
  • Real-time, "on-the-fly" training is feasible for 3D head avatars, dramatically reducing reconstruction time from hours to minutes or even seconds.
  • Intelligent simplification of Gaussian points through motion awareness and surface regularization effectively reduces model complexity and computational load while preserving rendering quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.