Reproduce “BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunsoo Kim, Yusuf Abdulle, Honghan Wu
A new benchmark, BioHopR, challenges large language models with multi-hop, multi-answer reasoning over the structured PrimeKG biomedical knowledge graph. Evaluations revealed that current state-of-the-art LLMs exhibit limited capabilities in these complex tasks, particularly for two-hop queries, with proprietary models achieving higher precision than open-source alternatives.
Key insights
- Multi-hop reasoning, particularly the implicit identification of intermediate 'bridge' nodes and alignment of reasoning chains, remains a significant bottleneck for all current LLMs.
- Specialized open-source biomedical LLMs, while fine-tuned for medical contexts, do not consistently generalize well to complex, structured multi-hop reasoning tasks.
- LLMs exhibit substantial difficulty in generating exhaustive sets of correct answers for questions with multiple ground truth outcomes, as evidenced by significantly lower performance with multi-answer prompting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.