Reproduce “MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lukas Thede, Yash Kumar Atri, David Chen, Danielle Bitterman, Matthias Bethge, Tom Hartvigsen, Zeynep Akata
MedKIT introduces a benchmark for testing whether language models can use updated oncology evidence beyond a direct fact-recall question. Across 12 update strategies and five models, methods often improved direct answers but showed weaker transfer to reversed comparisons and open-ended tasks; sequential updates, locality, and general capability retention also exposed trade-offs.
Key insights
- Success on the direct update question does not reliably translate into broader use: performance tends to decline from paraphrases to reversed comparisons and then to open-ended tasks.
- Continual post-training had the strongest average transfer among method families, but later updates could erode earlier gains; replay improved retention without fully preventing forgetting.
- Retrieval and localized memory can preserve existing model behavior, but access to evidence alone does not ensure that a model applies it correctly in open-ended responses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.