Reproduce “Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM” on OpenResearch

About this paper

Pingping Zhang, Tianyu Yan, Yang Liu, Huchuan Lu

The Dual-SAM framework from Dalian University of Technology significantly enhances the Segment Anything Model (SAM) for high-performance marine animal segmentation, addressing challenges unique to underwater imagery. By integrating domain-specific adaptations, including a dual-branch encoder and novel prompt and connectivity prediction mechanisms, the model achieves state-of-the-art results with improvements of 3-5% over CNN-based approaches and 2-3% over Transformer-based methods on diverse marine datasets like MAS3K.

Key insights

• Effective adaptation of large foundation models to challenging, specialized domains benefits from a multi-faceted approach, combining lightweight fine-tuning (adapters) with domain-specific pre-processing (gamma correction) and architectural enhancements.
• Explicitly modeling multi-directional pixel connectivity (C3P) provides a more robust and topologically sound representation for object boundaries compared to traditional pixel-wise classification, crucial for precise segmentation.
• Dual-branch processing of input images, combined with mutual supervision (PMS), fosters complementary feature learning and enhances overall robustness and accuracy by leveraging diverse image representations.

Summary by alphaXiv.