Reproduce “SASVi - Segment Any Surgical Video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ssharvien Kumar Sivakumar, Yannik Frisch, Amin Ranem, Anirban Mukhopadhyay

The SASVi method integrates an object detection Overseer model with SAM2, enabling automated, temporally consistent semantic segmentation of surgical videos. This approach significantly improves temporal consistency metrics compared to methods relying solely on initial prompts and facilitates the generation of comprehensive large-scale annotations from sparsely labeled datasets.

Key insights

  • Combining a lightweight domain-specific object detector (Overseer) with a general foundation model (SAM2) effectively addresses the challenge of limited annotated data for complex tasks.
  • Automated, conditional re-prompting allows foundation models to maintain temporal consistency and semantic understanding in dynamic environments without continuous manual intervention.
  • This methodology enables the creation of large-scale, fully annotated surgical video datasets from minimal existing annotations, significantly enriching resources for surgical AI research.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.