Reproduce “Automated Model Evaluation for Object Detection via Prediction Consistency and Reliability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seungju Yoo, Hyuk Kwon, Joong-Won Hwang, Kibok Lee

Researchers at Yonsei University and ETRI developed Prediction Consistency and Reliability (PCR), an automated model evaluation framework that estimates object detector performance on unlabeled data by analyzing pre-NMS bounding box consistency and reliability. PCR achieved superior mAP estimation with an average RMSE of 4.61 for vehicle detection and 3.57 for pedestrian detection, outperforming existing baselines, and introduced a new corruption-based meta-dataset for realistic evaluation.

Key insights

  • Internal pre-NMS bounding box predictions, typically discarded, contain rich, underexploited information valuable for assessing localization and classification quality without extra computational overhead.
  • Low-confidence final predictions that are highly consistent with their merged pre-NMS counterparts often indicate false positives or detection failures.
  • High-confidence final predictions that receive strong support from numerous other high-confidence pre-NMS boxes in the same region are indicative of reliable and correct detections.
  • Realistic image corruptions, rather than artificial augmentations, are crucial for creating effective meta-datasets that truly reflect real-world performance degradation in object detection.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.