Reproduce “FunBench: Benchmarking Fundus Reading Skills of MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qijie Wei, Kaiheng Qian, Xirong Li
Researchers from Renmin University of China developed FunBench, a hierarchical benchmark for assessing multimodal large language models' ability to read fundus images, which showed that existing MLLMs have limited proficiency, particularly in foundational anatomical recognition, and that their performance correlates strongly with the linguistic understanding of their LLM modules.
Key insights
- Current MLLMs exhibit limited fundus reading skills, performing poorly in higher-level tasks and struggling with foundational medical knowledge like laterality recognition, often scoring near-to-chance.
- The overall performance of MLLMs in fundus reading correlates highly with their Large Language Model (LLM) modules (Spearman correlation of 0.917), suggesting strong reliance on linguistic reasoning over visual feature extraction.
- Vision Encoders (VEs) like CLIP-ViT, widely used in MLLMs, are less effective for fine-grained fundus disease differentiation compared to robust VEs like DINOv2-large, highlighting a need for ophthalmic-specific VEs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.