Reproduce “CoordFormer: Give Me Any Coordinates and I Will Give You Labels” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Iacopo Curti, Pierluigi Zama Ramirez, Alioscia Petrelli, Luigi Di Stefano

COORDFORMER is a coordinate-based architecture that performs semantic segmentation on very-high-resolution images by predicting labels at arbitrary spatial locations. It achieved state-of-the-art results on the MaSS13K benchmark with an mIoU of 92.53 and BIoU of 53.24, outperforming previous methods with fewer parameters and demonstrating robust scalability up to 64 Mpx without out-of-memory errors.

Key insights

  • Directly querying labels at arbitrary coordinates allows for adaptive inference strategies, balancing accuracy, efficiency, and memory usage based on task requirements or hardware constraints.
  • The Localized Cross-Attention mechanism efficiently integrates global semantic context with local coordinate-aware features, reducing computational complexity compared to attending to all global tokens.
  • Focusing inference on semantic boundaries via SEFI significantly reduces the number of processed coordinates (e.g., from 100% to 14%) with minimal impact on performance, substantially improving runtime efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.