Reproduce “Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nguyen Quang Duc, Le Hai Son, Nguyen Duc Nhan, Nguyen Dich Nhat Minh, Le Thanh Huong, Dinh Viet Sang
Researchers from Hanoi University of Science and Technology and Vietnam National University HCMC developed a comprehensive suite of Vietnamese datasets and two continually pre-trained LLaMA2-7b models, aimed at advancing Retrieval-Augmented Generation and Large Language Models for Vietnamese. Their custom tokenizers reduce the token count for Vietnamese text by up to 70% compared to the original LLaMA2 tokenizer, and a bi-encoder achieved 73.28% Accuracy@1 on a legal text retrieval task.
Key insights
- Combining large-scale data curation with synthetic data generation techniques can effectively address resource scarcity for less-resourced languages like Vietnamese.
- Custom tokenizers designed specifically for Vietnamese can dramatically reduce the token count (up to 70%) required for text encoding, improving processing efficiency.
- Continual pre-training of established LLM architectures, such as LLaMA2-7b, on extensive target-language corpora proves effective for developing high-quality language understanding and generation capabilities in new languages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.