Reproduce “Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge” on OpenResearch

About this paper

Md Tahmid Rahman Laskar, Israt Jahan, Elham Dolatabadi, Chun Peng, Enamul Hoque, Jimmy Huang

This research enhanced the reliability of LLMs acting as judges for biomedical relation extraction by implementing structured output formatting and domain adaptation via transfer learning. The approach improved evaluation accuracy and enabled smaller, fine-tuned open-source models to achieve competitive performance, offering a scalable alternative to human assessment.

Key insights

• The unstructured nature of LLM-generated outputs significantly impedes the evaluation capabilities of LLM-Judges in biomedical relation extraction, leading to low initial performance.
• Requiring structured output (JSON) from LLM-Generators consistently improves LLM-Judge accuracy across various models and datasets, demonstrating an average gain of approximately 15%.
• Domain adaptation through fine-tuning allows smaller, open-source LLM-Judges to achieve competitive evaluation performance, sometimes outperforming larger zero-shot proprietary models by transferring knowledge from existing biomedical data.

Summary by alphaXiv.