Reproduce “Almanac: Retrieval-Augmented Language Models for Clinical Medicine” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Kevin Alexander, Euan Ashley and 6 more

Almanac, a retrieval-augmented language model framework from Stanford Medicine, improves the factuality and safety of large language models for clinical medicine by grounding responses in verifiable external knowledge. It achieved an 18% absolute increase in factual accuracy and 95% safety against adversarial prompts compared to a baseline LLM.

Key insights

  • Explicitly grounding LLM responses with dynamic, verifiable external knowledge significantly enhances factuality and safety, particularly against adversarial prompts, in clinical contexts.
  • The RAG approach effectively addresses knowledge staleness and improves explainability by providing real-time information retrieval and citations, increasing trustworthiness.
  • Physician preference for a more conversational style (like ChatGPT's) over higher factual accuracy and safety (like Almanac's) highlights a critical user experience challenge for clinical AI tools.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.