Reproduce “SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, Yanwei Pang

SED is an encoder-decoder framework for open-vocabulary semantic segmentation that uses a hierarchical image encoder, a gradual fusion decoder, and a category early rejection scheme. It achieves state-of-the-art accuracy, such as 18.6% mIoU on the PC-459 dataset, while providing up to 4.7x faster inference compared to previous methods.

Key insights

  • Hierarchical backbones like ConvNeXt are more effective than plain Vision Transformers for dense prediction tasks in OVSS, improving local spatial information capture and computational efficiency.
  • Carefully fine-tuning the hierarchical encoder with a small learning rate multiplier while stopping gradient back-propagation from decoder skip-layers to the encoder is crucial for boosting performance without degrading VLM generalization.
  • Dynamically rejecting non-present categories early in the decoder substantially accelerates inference (e.g., 4.7x faster) without compromising segmentation accuracy, addressing a key practical bottleneck.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.