Reproduce “MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiyi Zhang, Yuchen Yuan, Ying Zheng, Jialun Pei, Jinpeng Li, Zheng Li, Pheng-Ann Heng
A new framework named MEJO addresses inter-task and intra-task optimization conflicts in surgical triplet recognition by integrating MLLM-derived expert knowledge and employing disentangled representation learning. This approach achieved state-of-the-art performance on the CholecT45 dataset with a 40.1% AP_IVT and on CholecT50, surpassing previous methods by up to 4.4% and 1.6% AP_IVT, respectively.
Key insights
- Explicitly disentangling task-shared and task-specific representations using a Temporal-Spatial Prompt (TSP) Adapter effectively resolves ambiguities and negative transfer in multi-task surgical action recognition.
- A coordinated gradient rebalancing strategy enables more synchronous optimization across head and tail classes, significantly improving the learning of rare surgical actions.
- Integrating MLLM-derived fine-grained expert knowledge dynamically enhances visual features, leading to a deeper and more nuanced understanding of instrument-centric surgical cues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.