Reproduce “MixLID: Token-Level Language Identification in Mixed-Language Text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim
MixLID identifies the language of individual tokens in code-switched text and can aggregate token predictions to label a sentence’s predominant language. It trains mmBERT-small first on monolingual sentence labels projected onto words, then on naturally labeled and synthetically augmented mixed-language examples. Across eleven ACS pairs, it averages 93.2% accuracy and 88.8% macro F1; evaluation covers only tested pairs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.