Reproduce “3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seonho Lee, Jiho Choi, Inha Kang, Jiwook Kim, Junsung Park, Hyunjung Shim

KAIST researchers developed Geometric Distillation, a lightweight and annotation-free framework that fine-tunes 2D-trained Vision-Language Models to acquire robust 3D spatial understanding by distilling geometric cues from 3D foundation models. The method achieves state-of-the-art performance on various 3D perception and 3D vision-language benchmarks while being significantly more computationally efficient than prior approaches.

Key insights

  • Transferring human-inspired geometric cues, such as sparse correspondences and relative depth, is an effective strategy to impart 3D understanding to 2D-trained VLMs.
  • Relative depth supervision is more robust and yields better generalizable geometric representations compared to using absolute depth values.
  • Powerful 3D foundation models can serve as effective "teachers" to distill rich, annotation-free geometric priors into 2D VLMs without explicit 3D reconstruction.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.