Reproduce “BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation” on OpenResearch

About this paper

Andrey Moskalenko, Danil Kuznetsov, Irina Dudko, Anastasiia Iasakova, Nikita Boldyrev, Denis Shepelev, Andrei Spiridonov, Andrey Kuznetsov and 1 more

Researchers at Lomonosov Moscow State University and FusionBrain Lab evaluated the robustness of promptable segmentation models against realistic bounding box inputs, developing a method called BREPS to generate adversarial prompts grounded in human drawing behavior. Their findings reveal that state-of-the-art models exhibit an average 30% drop in IoU performance when faced with these challenging, yet plausible, bounding box variations compared to idealized inputs.

Key insights

• Human-drawn bounding boxes exhibit substantial variability, with mobile users providing less precise annotations than desktop users, leading to an average inter-user variance of 15% IoU in model performance for the same instance.
• The geometric quality of a user-drawn bounding box (CIoU-Loss) shows weak correlation (≤ 0.4 SRCC) with the resulting segmentation model's IoU, indicating model sensitivity to subtle prompt variations not captured by simple geometric metrics.
• Promptable segmentation models display high sensitivity to even minor bounding box shifts, with an exhaustive search revealing abrupt drops in segmentation quality and localized instabilities from single-pixel coordinate changes.

Summary by alphaXiv.