Reproduce “NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yupeng Zhang, Ruize Han, Zhiwei Chen, Wei Feng, Liang Wan

NoOVD introduces a framework for Open-Vocabulary Object Detection that utilizes frozen Vision-Language Models to directly discover and embed novel object categories during training. The method prevents misclassification of latent novel objects as background and enhances their recall, leading to a 2.8% increase in novel category AP on OV-LVIS compared to the F-ViT baseline.

Key insights

  • The primary challenge in OVD for two-stage detectors is the misclassification of unlabeled novel-category objects as background during training, which degrades novel category recall and performance.
  • Leveraging the zero-shot capabilities of frozen Vision-Language Models through a parameter-free feature extractor (K-FPN) is effective for retaining and transferring broad semantic knowledge to the detector.
  • Knowledge self-distillation, by aligning features of implicitly discovered novel objects with their robust VLM representations, can embed novel category understanding without requiring additional labeled data or noisy pseudo-labels.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.