Reproduce “Strong Reasoning Isn't Enough: Evaluating Evidence Elicitation in Interactive Diagnosis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhuohan Long, Zhijie Bao, Zhongyu Wei
Researchers at Fudan University developed an evaluation framework, EviMed benchmark, and the Information Coverage Rate (ICR) metric to assess large language models' evidence elicitation in interactive medical diagnosis. Their work identified information collection as a bottleneck, and a feedback-driven strategy, REFINE, improved diagnostic accuracy and evidence gathering, particularly through heterogeneous model collaboration.
Key insights
- Strong diagnostic reasoning in static settings does not consistently translate to effective interactive information acquisition, with insufficient evidence collection acting as a primary bottleneck for LLMs.
- Strategies that decouple information gathering from diagnosis (e.g., Summarized-Conversation) or use reasoning-based feedback (REFINE) significantly improve both information coverage and diagnostic success in interactive scenarios.
- Heterogeneous model collaboration, by pairing models strong in inquiry with those strong in reasoning, enhances overall performance and suggests a cost-effective deployment strategy for medical AI agents.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.