Reproduce “Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei-Hsiang Lin, Sheng-Lun Wei, Hen-Hsen Huang, Hsin-Hsi Chen
Lin et al. (2025) investigates the relationship between an LLM's answer generation and evaluation capabilities, revealing a weak correlation under standard conditions. The study introduces a self-reference-guided evaluation strategy that significantly strengthens this correlation, making a model's generation accuracy a more reliable indicator for judge selection.
Key insights
- Under standard Chain-of-Thought (CoT) prompting, LLM generation and judgment abilities are intrinsically weakly correlated (partial correlation below 0.3 in 25 out of 33 cases).
- LLM judgment is heavily influenced by the correctness of the evaluated response and exhibits a bias towards predicting 'Correct', often more than by the judge's own knowledge.
- The self-reference-guided method shifts the primary determinant of judgment performance from the agent's answer label to the judge's own internal knowledge, facilitating a more active verification process.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.