Reproduce “ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seonwu Kim, Yohan Na, Kihun Kim, Hanhee Cho, Geun Lim, Mintae Kim, Seongik Park, Ki Hyun Kim and 2 more
Researchers at LG Uplus developed ixi-GEN, a methodology applying Domain Adaptive Continual Pretraining (DACP) to small language models (sLLMs) to enhance their performance in specific industrial domains like Telco and Finance. This approach yielded average performance improvements of 50-69% on Telco benchmarks and 31-49% on Finance benchmarks, while largely preserving general language capabilities, thus offering a cost-efficient alternative to larger models.
Key insights
- A DACP-based recipe can consistently boost sLLM performance in target domains (Telco, Finance) while effectively preserving general linguistic capabilities across various foundation models.
- Larger sLLMs (7B and above) generally demonstrate better preservation of general capabilities during DACP compared to smaller models (3B and below).
- Domain-adapted sLLMs can outperform larger general-domain LLMs in specialized tasks, offering a cost-efficient alternative for industrial applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.