Reproduce “Comparing Developer and LLM Biases in Code Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donahue, Ameet Talwalkar, Wayne Chi and 1 more
TRACE compares LLM judges with human preferences across three coding settings and analyzes rubric criteria linked to disagreements. It measures pairwise accuracy and uses regression to estimate how humans and judges weight shared criteria. In filtered chat-based coding, OpenAI GPT-5 mini reaches 57.6% accuracy, while majority humans agree with the original user on a 30-example overlap set. TRACE diagnoses associations; it does not establish causes or improve judges.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.