Reproduce “AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ren Ma, Jiantao Qiu, Chao Xu, Pei Chu, Kaiwen Liu, Pengli Ren, Yuan Qu, Jiahui Peng and 21 more
Researchers from Shanghai Artificial Intelligence Laboratory developed MinerU-HTML, a model-based HTML extraction pipeline, to create AICC, a 7.3-trillion token AI-ready corpus. Models pretrained on AICC achieved a 1.08 percentage point higher average accuracy on 13 diverse benchmarks compared to models trained on heuristically extracted data with identical filtering.
Key insights
- HTML-to-text extraction is a critical, under-addressed factor in web corpus construction, with its quality directly influencing downstream LLM performance.
- A model-based approach to HTML extraction can effectively overcome the limitations of heuristic methods by semantically understanding and preserving complex document structures.
- High-fidelity preservation of structured content (code, formulas, tables) in training data is essential for enhancing LLMs' capabilities in specialized and technical domains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.