Reproduce “Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifan Zhang, Junhui Hou
OLIVINE is a method for fine-grained image-to-LiDAR contrastive distillation that enhances 3D representation learning by addressing the "self-conflict" problem and data imbalances. It integrates Visual Foundation Models for weak semantic labels, semantic-guided consistency regularization, and a density/category-aware sampling strategy, achieving 50.09% mIoU in linear probing on nuScenes-lidarseg, a 5.14% improvement over the previous state-of-the-art.
Key insights
- Weak semantic labels from Visual Foundation Models can effectively mitigate the "self-conflict" problem by explicitly defining positive relationships for semantically similar, non-corresponding cross-modal pairs.
- Enforcing consistency among semantic features via vMF distribution regularization stabilizes the learning process and creates a more structured and reliable feature space.
- Adaptive sampling that considers both spatial density and category frequency is crucial for achieving balanced and robust representation learning across diverse and imbalanced 3D datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.