Reproduce “Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Justin Zhao, Flor Miriam Plaza-del-Arco, Benjamin Genchel, Amanda Cercas Curry

The Language Model Council (LMC) introduces a democratic framework for benchmarking Foundation Models on highly subjective tasks, using all council members to create tests, generate responses, and collectively judge. This approach achieved a 0.92 Spearman correlation with human evaluations for emotional intelligence, demonstrating superior alignment with human preferences and robustly mitigating individual LLM biases.

Key insights

  • Aggregating judgments from a diverse ensemble of LLMs effectively neutralizes individual LLM biases, leading to more robust and fair evaluations.
  • LLM judges, when evaluating emotional intelligence, primarily prioritize actionability, clarity, and structure in responses, providing insights into their evaluative criteria.
  • While increasing council size and test set size improves ranking stability and separability, the incremental benefits diminish significantly beyond approximately 9 judges and 50 examples, suggesting an 'optimal zone' for resource allocation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.