Reproduce “PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery” on OpenResearch

About this paper

Runlong He, Mengya Xu, Adrito Das, Danyal Z. Khan, Sophia Bano, Hani J. Marcus, Danail Stoyanov, Matthew J. Clarkson and 1 more

UCL researchers developed PitVQA-Net, an image-grounded text embedding LLM, and introduced the PitVQA dataset for visual question answering in endonasal pituitary surgery. The model achieved an 8% higher balanced accuracy on the PitVQA dataset and a 9% higher balanced accuracy on the EndoVis18-VQA dataset compared to prior state-of-the-art models.

Key insights

• Specialized, large-scale, and richly annotated datasets are crucial for advancing VQA capabilities in complex medical domains, providing the necessary data for fine-grained surgical understanding.
• Advanced vision-language fusion, particularly through image-grounded text embeddings with cross-attention and leveraging pre-trained large models, significantly improves contextual alignment and understanding in surgical VQA.
• A lightweight classification head incorporating an Excitation Block with gated-attention mechanisms enhances feature discrimination and proves effective for accurate classification in highly imbalanced surgical VQA tasks.

Summary by alphaXiv.