Reproduce “Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel D'souza, Julia Kreutzer, Adrien Morisot, Ahmet Üstün, Sara Hooker
Cohere Labs researchers developed a method utilizing "Treasure Markers" during instruction fine-tuning to enhance large language model controllability and performance on long-tail data. This approach achieved an absolute 5.7% increase in Win Rates for open-ended generation, with a 9.1% gain in underrepresented domains, and reduced length constraint violations from 36.58% to 1.25%.
Key insights
- Explicitly training LLMs with a rich taxonomy of internal markers during fine-tuning provides a robust, intuitive mechanism for multidimensional control over generation.
- Marker dropout during training is crucial for models to generalize and infer desired output characteristics from natural language prompts, reducing reliance on explicit marker input.
- Treating markers as natural language tokens within the model's vocabulary allows for seamless integration and more predictable adherence to specified attributes compared to external control methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.