Reproduce “System Message Generation for User Preferences using Open-Source Models” on OpenResearch

About this paper

Minbyul Jeong, Jungho Cho, Minsoo Khang, Dawoon Jung, Teakgyu Hong

Upstage AI researchers developed SYSGEN, an automated pipeline that generates high-quality system messages and aligned assistant responses from existing supervised fine-tuning datasets, improving open-source LLMs' adherence to user instructions and system roles. The method significantly enhances performance on single-turn and multi-turn alignment benchmarks while enabling system role capabilities in models lacking native support.

Key insights

• Automated generation of diverse, structured system messages and their aligned assistant responses can significantly enhance LLM control and adaptability without extensive manual annotation or proprietary data.
• Knowledge distillation using SYSGEN-generated data successfully enables system role capabilities in open-source models that initially lack native support, broadening their applicability.
• The synergistic training with both generated system messages and their newly aligned assistant responses is crucial for comprehensive performance improvements across various benchmarks, minimizing degradation on general tasks.

Summary by alphaXiv.