Reproduce “When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs” on OpenResearch

About this paper

Bogdan Zagribelnyy, Ivan Ilin, Maksim Kuznetsov, Nikita Bondarev, Mathieu Reymond, Roman Schutski, Thomas MacDougall, Rim Shayakhmetov and 5 more

Researchers from Insilico Medicine AI developed a novel benchmarking framework for single-step retrosynthesis, introducing the ChemCensor metric to evaluate chemical plausibility rather than exact-match accuracy. This framework, combined with the large-scale CREED dataset and an out-of-domain URSA-expert-2026 benchmark, facilitated the development of C3LM models that showed improved plausible reaction predictions for novel target molecules, with ChemCensor's plausibility classification agreeing 87% with expert chemists.

Key insights

• Plausibility-based metrics like ChemCensor are essential for robustly evaluating retrosynthesis models, providing a more chemically relevant assessment than traditional exact-match accuracy.
• Training LLMs on large, diverse datasets like CREED, which contains multiple plausible reaction options, significantly improves their ability to generate chemically sound and diverse retrosynthetic disconnections for novel, out-of-domain molecules.
• Performance on in-distribution benchmarks does not reliably predict generalization capabilities to out-of-domain targets, highlighting the necessity of rigorous OOD evaluation sets like URSA-expert-2026.

Summary by alphaXiv.