Reproduce “TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lisa Weijler, Muhammad Jehanzeb Mirza, Leon Sick, Can Ekkazan, Pedro Hermosilla

TTT-KD introduces the first Test-Time Training method for 3D semantic segmentation, leveraging knowledge distillation from frozen 2D foundation models as a self-supervised objective. This approach adapts 3D backbones at test-time, achieving up to 45% mIoU improvement on out-of-distribution datasets and up to 8.5% mIoU on in-distribution data.

Key insights

  • Robust visual features from powerful 2D foundation models can effectively guide the test-time adaptation of 3D models for dense prediction tasks through knowledge distillation.
  • Adapting a 3D backbone at test-time using a self-supervised auxiliary objective allows for effective generalization to domain shifts without requiring labeled target data or large batches.
  • The initial joint training with the KD loss acts as a beneficial regularizer, improving the model's feature representations and generalization even before test-time adaptation begins.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.