Reproduce “Crowd-SFT: Crowdsourcing for LLM Alignment” on OpenResearch

About this paper

Alex Sotiropoulos, Sulyab Thottungal Valapu, Linus Lei, Jared Coleman, Bhaskar Krishnamachari

Crowd-SFT organizes alignment feedback into groups that fine-tune competing model variants, selects candidates through repeated evaluation, and allocates points to contributors according to group performance. In experiments using an emotion-vector metric, selection reduced target distance across all 28 emotions; simulations found that the best settings for model convergence differed from those that most closely tracked individual contributions.

Key insights

• Choosing among several fine-tuned candidates can improve a target-alignment measure compared with training one model on all available examples, even when adding more examples to single-model fine-tuning yields modest gains.
• The strategy that best improves the model need not best estimate individual contributions: L2-based evaluation favored convergence, while dot-product evaluation with random or interleaved grouping often produced stronger reward-to-Shapley correlations.
• Reward attribution became less accurate as the simulated user population grew; using more groups generally helped but did not remove this decline.

Summary by alphaXiv.