Reproduce “Alignment Data Map for Efficient Preference Data Selection and Diagnosis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seohyeong Lee, Eunwon Kim, Hwaran Lee, Buru Chang
The Alignment Data Map characterizes LLM preference data by response quality and variability to enable efficient data selection and diagnosis. Training with just 33% of data identified as high-quality and low-variability achieved performance comparable to or better than full datasets, while also effectively flagging mislabeled samples.
Key insights
- Preference data with consistently high response quality and low variability (i.e., subtle distinctions among good options) provides the most effective learning signals for LLM alignment.
- Solely focusing on reward margin is insufficient; the absolute quality of responses is a critical dimension that, when combined with variability, yields superior data selection.
- The Alignment Data Map can serve both as an efficient data selection mechanism to reduce annotation costs and as a robust diagnostic tool to identify and correct mislabeled preference data in existing datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.