Reproduce “Radiology-Aware Model-Based Evaluation Metric for Report Generation” on OpenResearch

About this paper

Amos Calamida, Farhad Nooralahzadeh, Morteza Rohanian, Koji Fujimoto, Mizuho Nishio, Michael Krauthammer

RadEval is introduced as a specialized model-based evaluation metric for automatically generated radiology reports, addressing the limitations of general-purpose NLG metrics by incorporating domain-specific clinical knowledge. Developed by researchers from the University of Zürich, University Hospital of Zürich, and Kyoto University, it demonstrates improved alignment with human radiologist judgment and a more nuanced assessment of clinical accuracy compared to traditional methods.

Key insights

• Domain-adapted neural evaluation metrics, like RadEval, can significantly improve correlation with human judgment in specialized fields such as radiology by incorporating domain-specific knowledge.
• The way comparative pairs are generated for training (e.g., 'Top 10%' similar reports versus 'Best Match') influences the evaluation metric's performance, with broader similarity sets yielding better results.
• Even with domain-specific language models like BioClinical BERT, a general large language model like XLM-RoBERTa can sometimes provide better performance in this evaluation context.

Summary by alphaXiv.