Reproduce “Exploring LLMs for Scientific Information Extraction Using The SciEx Framework” on OpenResearch

About this paper

Sha Li, Ayush Sadekar, Nathan Self, Yiqi Su, Lars Andersland, Mira Chaplin, Annabel Zhang, Hyoju Yang and 5 more

Virginia Tech researchers developed SciEx, a modular framework leveraging large language models (LLMs) and retrieval-augmented generation (RAG) for extracting structured information from scientific papers by integrating text, tables, and figures. Evaluating the framework with GPT-4o and Gemini-2.5-Flash on diverse scientific datasets demonstrated moderate F1-scores up to 0.380, with GPT-4o generally outperforming, while revealing persistent challenges in multi-modal comprehension and document quality handling.

Key insights

• A modular and composable framework design can effectively address the complexities of multi-modal, long-context scientific information extraction, enhancing adaptability and maintainability for evolving requirements.
• LLMs can perform schema-guided extraction across text, tables, and figures, but their performance is significantly influenced by dataset complexity, document quality, and the robustness of cross-modal reasoning capabilities.
• Implementing iterative retrieval, self-verification, and multi-modal contextual reasoning within an extraction pipeline is crucial for improving the completeness and correctness of extracted scientific information.

Summary by alphaXiv.