Reproduce “In-context learning enables multimodal large language models to classify cancer pathology images” on OpenResearch

About this paper

Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S.M. El Nahhas, Gustav Müller-Franzes and 3 more

A study by the Kather Group at Technical University Dresden demonstrated that multimodal large language models like GPT-4V can classify cancer pathology images using in-context learning. This approach achieved up to 90% accuracy on binary tasks, often matching or exceeding specialized models trained with identical, small datasets, and provided transparent text-based reasoning.

Key insights

• In-context learning significantly improves multimodal large language models' (LVLMs) classification accuracy on medical images, transforming poor zero-shot performance into high few-shot accuracy.
• The contextual relevance of in-context examples is crucial, with kNN-based sampling consistently outperforming random sampling for ICL in medical image tasks.
• LVLMs employing ICL with minimal data can achieve performance competitive with or superior to specialized, fine-tuned deep learning models on certain histopathology tasks, challenging traditional training paradigms.

Summary by alphaXiv.