Reproduce “See & Sniff: Learning Visuo-Olfactory Representations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak

See & Sniff introduces a scalable way to learn shared visual–olfactory representations by pairing real gas-sensor recordings with semantically matched web images. Its local contrastive model improves smell classification and cross-modal retrieval, while producing smell-conditioned spatial maps that reach 0.8362 mAP and 0.6456 mIoU for locating odor sources in images.

Key insights

  • Visual supervision improves olfactory representations even when only smell input is used at inference: smell-only retrieval reaches 69.44% Recall@1, versus 52.54% for the SmellNet baseline.
  • Dense local alignment is more useful than global alignment for difficult top-ranked retrieval and spatial grounding, while lightweight modality-specific aligners further improve localization.
  • Synthetic semantic pairing is effective for initial representation learning, but performance remains uneven for visually or chemically confusable ingredients and may not capture real odor co-occurrence.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.