Reproduce “How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs” on OpenResearch

About this paper

Yapei Chang, Kyle Lo, Mohit Iyyer, Luca Soldaini

Researchers from the Allen Institute for AI and the University of Maryland developed "How2Everything," a comprehensive framework for mining web-scale procedural data, creating a robust evaluation benchmark, and improving LLMs' ability to generate step-by-step instructions. The framework extracted over 350,000 procedures and demonstrated over 10-point performance gains in procedural generation via reinforcement learning without causing regressions on other tasks.

Key insights

• Web-scale mining, combined with rigorous LLM-based filtering, can successfully yield a large, diverse, and high-quality dataset of real-world procedures.
• A robust evaluation protocol based on "critical failures" can be reliably implemented by frontier LLMs, and then distilled into smaller, open-source models for scalable and reproducible assessment.
• The developed How2Bench benchmark is sensitive enough to track clear scaling trends across a wide range of model sizes and training stages, decoupling from superficial metrics like perplexity or output formatting.
• Targeted RL training with a specific procedural validity reward can significantly enhance LLM capabilities without causing regressions on general knowledge or reasoning tasks.

Summary by alphaXiv.