Reproduce “Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models” on OpenResearch

About this paper

Hannah Rosa Friesacher, Emma Svensson, Susanne Winiwarter, Lewis Mervin, Adam Arany, Ola Engkvist

This research from AstraZeneca, KU Leuven, and Chalmers University empirically demonstrates that real-world pharmaceutical data exhibits significant temporal distribution shifts, which impact the reliability of QSAR classification models. The study reveals that the effectiveness of uncertainty quantification methods varies greatly depending on the assay type and magnitude of these shifts, necessitating continuous model re-evaluation.

Key insights

• Real-world pharmaceutical data is not i.i.d.; significant temporal distribution shifts are prevalent, particularly in dynamic, project-specific Target-Based assays.
• The effectiveness of uncertainty quantification methods is highly sensitive to the type and magnitude of temporal shifts, especially large label shifts, which can diminish their advantage over baseline models.
• Model performance and calibration are generally unstable over time in drug discovery projects, underscoring the necessity for continuous re-evaluation and adaptation of models.

Summary by alphaXiv.