Reproduce “Open-Vocabulary DETR with Conditional Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy
OV-DETR, developed by researchers from Nanyang Technological University, introduces the first end-to-end Transformer-based open-vocabulary object detector by employing a novel conditional binary matching mechanism and leveraging CLIP-derived multi-modal queries. It achieved a 1.3 mAP improvement on novel classes over ViLD on OV-LVIS and 1.8 mAP on OV-COCO, demonstrating enhanced generalization capabilities.
Key insights
- DETR's bipartite matching can be successfully adapted for open-vocabulary detection by reformulating it into a conditional binary matching problem, addressing the "classification cost matrix" limitation for novel classes.
- Leveraging pre-trained vision-language models like CLIP allows for flexible multi-modal queries (text or image) and enables generalization to unseen categories in an end-to-end framework.
- An end-to-end Transformer-based approach inherently avoids the "proposal bottleneck" that limits the performance of RPN-based open-vocabulary detectors on novel objects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.