Reproduce “CLIPGaussian: Universal and Multimodal Style Transfer Based on Gaussian Splatting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kornel Howil, Joanna Waczyńska, Piotr Borycki, Tadeusz Dziarmaga, Marcin Mazur, Przemysław Spurek

CLIPGaussian introduces a plug-in framework for universal and multimodal style transfer across 2D images, videos, 3D objects, and 4D dynamic scenes, leveraging Gaussian Splatting. The method enables both text- and image-guided stylization while jointly optimizing appearance and geometry, achieving state-of-the-art results in text-guided 3D stylization and strong temporal coherence in videos.

Key insights

  • Direct optimization of all Gaussian parameters within existing GS architectures, in a plug-in manner, enables comprehensive style transfer with geometric changes while preserving model size and efficiency.
  • Leveraging multimodal CLIP embeddings alongside patch-based processing facilitates detailed and locally consistent style transfer, accommodating diverse input styles from text or images.
  • The inherent nature of Gaussian primitives spanning multiple frames in dynamic scenes naturally ensures superior temporal consistency in video style transfer, effectively mitigating flickering issues.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.