Reproduce “Contextual morphologically-guided tokenization for Latin encoder models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marisa Hudspeth, Patrick J. Burns, Brendan O'Connor
Researchers from UMass Amherst and NYU demonstrated that incorporating expert-curated morphological knowledge into tokenization for Latin RoBERTa models improves morphological alignment and boosts performance across several NLP tasks. This approach yielded up to a 4.4% increase in morphological accuracy and 13.2% in Named Entity Recognition F1 on out-of-domain Latin texts.
Key insights
- Explicit hard constraints through MorphPreTokenization are significantly more effective for aligning tokenizers with linguistic morphological boundaries than softer 'seeding' approaches.
- Leveraging high-quality, expert-curated linguistic resources offers a powerful alternative for improving language model performance in low-resource settings where vast raw text corpora are unavailable.
- Morphologically-guided tokenization substantially improves language model generalization capabilities, particularly on out-of-domain texts and for rare words, indicating deeper linguistic understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.