Reproduce “ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Long Phan, Hieu Tran, Hieu Nguyen, Trieu H. Trinh

VietAI Research, Case Western Reserve University, and New York University introduced ViT5, a Transformer-based encoder-decoder model built on the T5 framework, specifically for Vietnamese text generation. The model established new state-of-the-art performance on abstractive summarization and achieved competitive results on Named Entity Recognition by reformulating tasks into a text-to-text format.

Key insights

  • Longer pretraining context lengths (1024 tokens) significantly improve performance on downstream tasks involving long documents, such as summarization.
  • Monolingual encoder-decoder models like ViT5 generally surpass multilingual counterparts (e.g., mT5, mBART) for specific language NLP tasks.
  • Pretraining on a diverse, general-domain corpus (CC100) provides better generalizability and performance compared to more domain-specific datasets, even for domain-specific tasks.
  • The unified text-to-text paradigm of the T5 architecture effectively reformulates and addresses diverse NLP tasks, including classification tasks like NER.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.