Reproduce “The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza
A collaborative effort including researchers from Lahore University of Management Sciences and the University of Michigan developed multi-model workflows to automate the generation of synthetic Preference Optimization (PO) datasets. This approach, leveraging iterative response generation and LLM-based evaluation, showed improved performance over single-model baselines.
Key insights
- Strong LLMs, such as GPT-4o, demonstrate superior and consistent performance as a judge, particularly when provided with all responses in a single 'Combined Scoring' context.
- Multi-model feedback loops for response generation, utilizing one LLM for generation and another for iterative revision, produce higher-quality outputs compared to single-model baselines.
- While multi-LLM jury and debate systems can offer benefits like bias reduction, a single, highly capable LLM acting as a judge can sometimes be more effective and computationally efficient for certain evaluation tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.