Reproduce “COLIP-2: Olfaction-Vision-Language Embeddings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kordel Kade France
COLIP-2 integrates olfaction, vision, and language into a unified embedding space, enabling robotic systems to probabilistically localize aroma sources from molecular structures or gas-sensor readings within visual scenes. The model achieves 0.90 Recall@5 for olfaction-descriptor retrieval and a 0.90 top-1 accuracy for gas-sensor source substance classification.
Key insights
- Olfaction-vision grounding can emerge effectively through a shared text geometry of odor descriptors, reducing the need for extensive paired scent-image datasets during training.
- Carefully guarded finetuning of the vision-language backbone (e.g., using LoRA with a `lambda` guardrail) improves olfaction-language alignment without degrading existing olfaction-vision relationships.
- Aggressive, unguarded finetuning of the text encoder leads to catastrophic collapse of metrics routed through that component, emphasizing the importance of controlled adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.