Reproduce “BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Javier de la Rosa, Eduardo G. Ponferrada, Paulo Villegas, Pablo Gonzalez de Prado Salas, Manu Romero, Marıa Grandury
The BERTIN project introduced an efficient pre-training methodology for monolingual Spanish RoBERTa-base models, utilizing perplexity sampling on the mC4-es dataset. This approach achieved competitive performance on several downstream tasks with five times less data and half the training steps compared to typical pre-training, making high-quality language model development more accessible for Spanish.
Key insights
- Intelligent data-centric approaches, specifically perplexity sampling, can significantly reduce data requirements and training times without compromising final model performance.
- Competitive, state-of-the-art monolingual language models can be trained with substantially fewer resources, democratizing LLM development for smaller teams and less privileged institutions.
- Re-initializing the optimizer when increasing sequence length during pre-training is crucial for maintaining or improving model performance, particularly under resource constraints and simplified learning rate schedules.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.