Reproduce “LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, Ilias Chalkidis
LEXTREME introduces the first multi-lingual and multi-task benchmark specifically for the legal domain, featuring 11 datasets across 24 European languages and multiple NLU tasks. It establishes a challenging evaluation platform for current language models, showing that even top baselines achieve modest scores and revealing the superior performance of specialized monolingual models in many cases.
Key insights
- LEXTREME presents a significant challenge for current language models, with even the highest-performing baselines achieving modest scores, indicating substantial room for improvement in legal Natural Language Understanding (NLU).
- Monolingual general-purpose models (NativeBERT) often outperform their multilingual counterparts, achieving the highest language aggregate score and suggesting that language-specific pre-training offers advantages.
- ChatGPT (gpt-3.5-turbo) consistently underperformed much smaller, supervised models in a zero-shot setting on complex legal NLU tasks, highlighting limitations without task-specific fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.