Reproduce “FLUX: Data Worth Training On” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gowtham, Sai Rupesh, Sanjay Kumar, Saravanan, Venkata Chaithanya

The FLUX preprocessing pipeline enhances large language model training data by jointly optimizing token retention and quality, outperforming prior methods. It yields 25% more usable tokens and achieves a 32.14% MMLU accuracy with 34.4% compute savings for equivalent performance on 3B-parameter models.

Key insights

  • The long-standing trade-off between maximizing token retention and ensuring high data quality can be systematically overcome through a multi-stage, jointly optimized preprocessing pipeline.
  • Prioritizing line-level content excision over whole-document rejection effectively preserves valuable content embedded in otherwise noisy documents, enhancing data utility.
  • A dual-bin model-based classification framework, leveraging independently calibrated classifiers for general knowledge and benchmark-targeted relevance, refines data quality without severe token loss.
  • Whole-document language identification proves more effective than line-level approaches in retaining tokens and improving downstream model quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.