Reproduce “Reliable End-to-End Material Information Extraction from the Literature with Source-Tracked Multi-Stage Large Language Models” on OpenResearch

About this paper

Xin Wang, Anshu Raj, Matthew Luebbe, Haiming Wen, Shuozhi Xu, Kun Lu

Researchers from the University of Alabama, University of Oklahoma, and Missouri University of Science and Technology introduce a four-stage, source-tracked pipeline using large language models to reliably extract 47 distinct material features across composition, processing, microstructure, and properties from scientific literature. The system achieves an F1-score of 0.959, with zero false positive materials and a 3.3% material miss rate, demonstrating a robust method for curating high-quality experimental materials datasets.

Key insights

• The source-tracking mechanism is critical for achieving high recall and virtually eliminating false positives, significantly outperforming multi-stage processing alone which otherwise showed decreased recall.
• Microstructure information, historically challenging due to its complexity and scattered reporting, saw the most substantial improvements in precision and recall with the source-tracked multi-stage approach.
• LLMs require careful guidance to distinguish between legitimate inferences and inappropriate theoretical modeling or 'completion bias' to maintain strict adherence to experimentally reported data.

Summary by alphaXiv.