Reproduce “MeXtract: Light-Weight Metadata Extraction from Scientific Papers” on OpenResearch

About this paper

Zaid Alyafeai, Maged S. Al-Shaibani, Bernard Ghanem

MeXtract introduces a family of lightweight Large Language Models (LLMs) (0.5B to 3B parameters) specifically fine-tuned for extracting structured metadata from long scientific papers. The approach achieved state-of-the-art performance within its size class, with the 3B model reaching an F1 score of 73.23% on the MOLE benchmark, and demonstrated strong generalization to unseen metadata schemas.

Key insights

• Lightweight LLMs, when carefully fine-tuned with schema-driven instruction tuning and Direct Preference Optimization (DPO), can achieve highly competitive metadata extraction performance, often surpassing larger, general-purpose models.
• The explicit separation of schema definitions and natural language guidelines for metadata attributes enhances model adaptability and generalization capabilities to unseen or out-of-domain metadata schemas.
• Direct Preference Optimization (DPO) is effective in training LLMs to generate structured output that consistently adheres to complex format, type, and length constraints, significantly improving output quality.

Summary by alphaXiv.