Reproduce “Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models” on OpenResearch

About this paper

Shuo Wang, Yan Zhu, Xiaoyuan Luo, Zhiwei Yang, Yizhe Zhang, Peiyao Fu, Manning Wang, Zhijian Song and 3 more

The EndoKED framework automatically extracts and distills knowledge from large-scale, unannotated image-text colonoscopy records, generating high-quality pixel-level polyp annotations. This approach enables training AI models that achieve performance comparable to or exceeding state-of-the-art methods on expert-annotated benchmarks, while demonstrating remarkable data efficiency for downstream tasks such as optical biopsy.

Key insights

• General-purpose LLMs can accurately abstract report-level diagnostic information from complex free-text clinical records without further adaptation.
• A progressive distillation approach, synergistically combining MIL and SAM-guided WSSS, effectively transforms coarse report-level supervision into high-quality pixel-level annotations from entirely unannotated data.
• AI models trained on automatically distilled knowledge from large-scale unannotated records can achieve performance comparable to or surpass models trained on expert-annotated datasets, while also exhibiting superior data efficiency for new tasks.

Summary by alphaXiv.