Reproduce “FishDetector-R1: Unified MLLM-Based Framework with Reinforcement Fine-Tuning for Weakly Supervised Fish Detection, Segmentation, and Counting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yi Liu, Jingyu Song, Vedanth Kallakuri, Katherine A. Skinner

FishDetector-R1, a unified MLLM-based framework, performs fish detection, instance segmentation, and counting in underwater imagery using only weak point-level supervision. The framework achieves an mIoU of 93.1% (3B variant) and 93.8% (7B variant) on DeepFish, effectively matching and surpassing fully supervised baselines, and demonstrates robust generalization to unseen marine environments.

Key insights

  • Reinforcement learning from verifiable rewards, when applied to MLLMs with weak supervision, effectively bridges the performance gap, enabling high-quality instance segmentation comparable to fully supervised methods.
  • The unified "detect-to-count" prompt coupled with RL fine-tuning establishes a strong internal consistency between detection, segmentation, and counting, mutually reinforcing these related tasks.
  • Adapting powerful MLLMs and segmentation foundation models through targeted RL fine-tuning allows for robust zero-shot generalization, enabling accurate analysis in diverse and previously unseen underwater environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.