Reproduce “μ-Bench: A Vision-Language Benchmark for Microscopy Understanding” on OpenResearch

About this paper

Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, Serena Yeung-Levy

Stanford University researchers introduced μ-BENCH, an expert-curated vision-language benchmark featuring 17,235 diverse microscopy images and 22 tasks, to address the lack of standardized evaluation for vision-language models in biomedical understanding. Benchmarking revealed that even state-of-the-art models like GPT-4o achieve only moderate accuracy (e.g., 62.6% on coarse-grained perception) and specialist fine-tuning often leads to catastrophic forgetting, which can be mitigated by weight interpolation.

Key insights

• State-of-the-art Vision-Language Models (VLMs) exhibit substantial error rates on μ-BENCH, with the best model achieving 62.6% accuracy on coarse-grained perception, highlighting ample room for improvement.
• Specialist biomedical VLMs frequently underperform generalist models on broader microscopy tasks, suggesting current narrow fine-tuning methods lead to catastrophic forgetting of generalized biological knowledge.
• A weight interpolation strategy effectively mitigates catastrophic forgetting, allowing merged models to retain broad knowledge while enhancing specialist capabilities and improving overall perception performance.

Summary by alphaXiv.