Reproduce “Zero-shot Hierarchical Plant Segmentation via Foundation Segmentation Models and Text-to-image Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junhao Xing, Ryohei Miyakawa, Yang Yang, Xinpeng Liu, Risa Shinoda, Hiroaki Santo, Yosuke Toda, Fumio Okura

ZeroPlantSeg, a zero-shot framework, performs hierarchical plant segmentation by integrating foundation segmentation models with vision-language models and unsupervised clustering. It effectively delineates individual leaves and whole plants from top-view images of rosette plants, demonstrating strong cross-domain generalization and outperforming supervised methods on unseen species and environments.

Key insights

  • Integrating foundation segmentation models with vision-language models provides a powerful zero-shot approach, leveraging generic segmentation capabilities with semantic reasoning for complex, domain-specific tasks like hierarchical plant segmentation.
  • Text-to-image cross-attention in VLMs can effectively identify biologically relevant keypoints (e.g., leaf bases) within individual leaf instances, which is crucial for inferring the hierarchical structure of rosette-shaped plants.
  • A greedy, iterative clustering strategy based on these extracted keypoints can robustly group individual leaf instances into whole plant individuals without requiring any prior labeled training data for the specific plant species or environment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.