Reproduce “SureRoute: Toward a Hallucination-Free Self-Improving Platform for Retrosynthesis” on OpenResearch

About this paper

Jieli Zhou, Naiwu Chen, Longzhang Liu, Peiyu Zhang

Researchers at XtalPi Inc. developed SureRoute, a platform for retrosynthesis that combines an ensemble of generative models with ChemHarness, a rule-based chemical intuition engine, to mitigate "Chemical Hallucination." The platform achieved 74.3% recall@1 and a 4.6% Chemical Hallucination rate on a benchmark of 350 industrial molecules, reducing hallucination by 4 to 6 times compared to individual SOTA models and frontier LLMs.

Key insights

• Defining and quantifying 'Chemical Hallucination' reveals a critical shortcoming of existing retrosynthesis models, where plausible-looking routes are chemically unsound and not captured by standard recall metrics.
• An explicit, rule-based 'executable verifier' like ChemHarness can function as a model-agnostic plug-in, significantly reducing chemical hallucination rates across diverse generative models, including LLMs.
• Anchoring self-improving AI systems with a trustworthy, auditable chemical verifier is essential to prevent the amplification of errors ('misevolution') and guide learning loops toward genuinely reliable chemical outcomes.

Summary by alphaXiv.