Reproduce “Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Chen Qi, Peng Cheng
Velocitune introduces a velocity-based dynamic domain reweighting method for continual pre-training of large language models. It leverages learning velocity and scaling laws to dynamically adjust domain weights, achieving average performance improvements of 1.6% on math benchmarks and 3.8% on coding benchmarks for CodeLlama-7B, and up to 4.9% on system command generation for Llama3-8B.
Key insights
- Dynamic data ordering and sequential presentation during training significantly contribute to performance gains, beyond just achieving an optimal static data mixture.
- The accurate estimation of a realistic target loss for each domain is critical for effective dynamic reweighting; without it, training becomes unstable and performance degrades.
- Velocitune achieves faster stabilization of domain weights compared to distance-based reweighting methods, implying more efficient convergence to optimal data proportions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.