Reproduce “Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jessica Huynh, Alfredo Gomez, Athiya Deviyani, Renee Shelby, Jeffrey P. Bigham, Fernando Diaz
Research quantifies how modifications to evaluation rubrics affect agreement between human ratings and large language model (LLM) autoraters. The study found that providing examples and additional context in analytic rubrics generally improved human-autorater agreement, while the efficacy of rubric decomposition depends on task complexity and the method used to aggregate analytic scores.
Key insights
- Edited analytic rubrics, which included representative examples and contextual information, generally led to improved human-autorater agreement for complex tasks like AES.
- Reducing positional bias by evaluating analytic sub-criteria in separate API calls was often more impactful than adding examples, particularly for Llama on AES and both models on IF.
- The effectiveness of analytic rubrics versus holistic rubrics is not universal, depending on task complexity and the aggregation method used, which can significantly influence observed agreement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.