Reproduce “Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenjun Hou, Yi Cheng, Kaishuai Xu, Yan Hu, Wenjie Li, Jiang Liu

The SCAN framework enhances Surgical Visual Question Answering by employing a memory-augmented multimodal large language model that autonomously generates question-specific hints and broader scene context directly from surgical images. This approach achieves state-of-the-art performance, with accuracy improvements of up to +3.9% on EndoVis-18-VQA, +2.5% on EndoVis-17-VQLA, and +0.3% on Cholec80-VQA datasets compared to baseline MLLMs.

Key insights

  • Autonomous generation of contextual memory directly from input images eliminates the need for external, potentially error-prone resources, thereby improving model robustness and generalization capabilities.
  • Indirect Memory, which provides comprehensive contextual information about the broader surgical scene, plays a more critical role in overall performance than question-specific Direct Memory.
  • The synergistic combination of both Direct and Indirect memory types is crucial for achieving optimal performance, demonstrating their complementary contributions to understanding complex surgical scenarios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.