Reproduce “FMG-Det: Foundation Model Guided Robust Object Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Darryl Hannan, Timothy Doster, Henry Kvinge, Adam Attarian, Yijing Watkins
Researchers at Pacific Northwest National Laboratory developed FMG-Det, an object detection algorithm that leverages foundation models (SAM and CLIP) for pre-processing and an instance interpolation module to robustly handle high levels of bounding box noise. The approach achieved state-of-the-art performance, with a 6.6 MAE improvement on PASCAL VOC and 6.8 MAE improvement on MS-COCO over prior methods, maintaining functionality in scenarios where other techniques failed.
Key insights
- Foundation models like SAM and CLIP can effectively perform zero-shot pre-processing to correct noisy bounding box annotations, providing a substantial first step towards robust object detection.
- Dynamically interpolating between original noisy and foundation model-corrected bounding boxes during training improves performance, especially under high noise, by allowing the model to leverage both sources of information.
- The offline Foundation Model Correction (FMC) pipeline alone contributes the largest portion of the overall performance improvement, highlighting the critical role of intelligent data curation prior to training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.