Reproduce “TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi
This research from the University of Washington and Allen Institute for AI introduces TURNWISEEVAL, a benchmark specifically designed to measure the gap between single- and multi-turn language model capabilities, and TURNWISEDATA, a scalable synthetic data generation pipeline. Training with this multi-turn data improved multi-turn conversational performance by up to 12.8 points on TURNWISEEVAL-Self without degrading single-turn abilities.
Key insights
- A quantifiable performance gap exists between LLMs' single-turn and multi-turn conversational abilities, evident even in advanced models like GPT-4, which showed degraded multi-turn performance.
- Synthetic multi-turn data, particularly from the TURNWISEDATA pipeline, effectively enhances multi-turn conversational performance and helps bridge the identified gap.
- Preference-tuning with multi-turn data can significantly improve multi-turn abilities without negatively impacting a model's existing single-turn instruction-following or knowledge capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.