Reproduce “Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rian Touchent, Nathan Godey, Eric de la Clergerie
Biomed-Enriched presents a biomedical dataset curated using a two-stage LLM-driven paragraph-level annotation process to extract high-value content, including 2 million clinical case paragraphs. This approach enables more data-efficient continual pre-training, achieving comparable performance on biomedical benchmarks with one-third fewer training tokens than baseline methods.
Key insights
- LLM-driven paragraph-level annotation effectively identifies and extracts high-value content, such as clinical cases and high-educational-quality passages, which are often missed by coarser article-level filtering.
- Strategic data curation through quality filtering and domain-specific upsampling at a fine granularity significantly improves data efficiency for continual pre-training, reducing token requirements substantially.
- Explicitly encoding paragraph-level metadata, such as predicted domain and educational quality, via prefixing can provide models with structured context, aiding comprehension in downstream tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.