Reproduce “Task Oriented In-Domain Data Augmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiao Liang, Xinyu Hu, Simiao Zuo, Yeyun Gong, Qiang Lou, Yi Liu, Shao-Lun Huang, Jian Jiao
The TRAIT framework addresses data scarcity and lack of task-awareness in Large Language Model (LLM) domain adaptation by augmenting scarce in-domain data with intelligently selected general corpus data and task-oriented synthetic passages. This approach significantly boosts LLM performance, achieving an average 8.0% improvement in few-shot ad tasks and a 7.5% average accuracy gain across math benchmarks, while preserving general capabilities.
Key insights
- Explicitly making training data "task-oriented" with synthetic passages, especially through an "enlightenment paragraph," is crucial for teaching LLMs how to apply domain knowledge effectively.
- Intelligently selecting in-domain data from general corpora (e.g., Fineweb, MathPile) not only augments scarce datasets but also facilitates "knowledge replay" to preserve the LLM's general capabilities.
- A two-stage training approach, separating domain knowledge acquisition from task alignment, is more effective than a single-stage approach for optimal performance and task-awareness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.