Reproduce “How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang, Kyusong Lee, Qing Wang

A new benchmark, OVDEval, is introduced to provide a comprehensive, fine-grained evaluation of Open-Vocabulary Detection (OVD) models across six linguistic aspects, utilizing meticulously designed hard negatives. This work also proposes NMS-AP, a refined metric addressing the "Inflated AP Problem" in traditional Average Precision, revealing that state-of-the-art OVD models show strong general object detection but poor fine-grained linguistic comprehension.

Key insights

  • Current state-of-the-art OVD models exhibit proficient general object detection but significantly lack deep fine-grained linguistic understanding, commonsense reasoning, and robustness against subtle semantic distinctions.
  • The performance of OVD models on specific fine-grained tasks is highly dependent on the particular types of data encountered during their pre-training, indicating a struggle to generalize knowledge across diverse linguistic domains without explicit exposure.
  • Traditional AP indeed inflates reported OVD model performance, particularly for complex, fine-grained tasks, by allowing models to gain credit from multiple, spatially overlapping, yet semantically distinct predictions for the same object region.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.