Reproduce “Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Özgür Uğur, Mahmut Göksu, Mahmut Çimen, Musa Yılmaz, Esra Şavirdi, Alp Talha Demir, Rumeysa Güllüce, İclal Çetin and 1 more
NewmindAI developed Mecellem models, a suite of specialized language models for the Turkish legal domain, through from-scratch pre-training of encoders and continual pre-training of decoders. These models demonstrate strong performance on a newly introduced Turkish legal benchmark, with the 155M parameter encoder achieving a 47.52 Legal Score and 92.36% production efficiency.
Key insights
- The optimization of Masked Language Modeling (MLM) pre-training loss does not linearly correlate with superior downstream embedding task performance, particularly for morphologically rich languages like Turkish, necessitating downstream task evaluation for optimal checkpoint selection.
- Purpose-built encoder architectures, when pre-trained from scratch on domain-specific data and further post-trained with advanced contrastive learning methods like GISTEmbed, offer high parameter efficiency and competitive performance for specialized legal retrieval tasks.
- A custom tokenizer optimized for Turkish morphology and a meticulous data curation pipeline incorporating morphology-driven quality filtering are critical for achieving robust domain adaptation and high performance in specialized Turkish NLP.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.