Reproduce “Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shanshan Xu, T.Y.S.S Santosh, Oana Ichim, Barbara Plank, Matthias Grabmair
Researchers introduced SV-ECHR, a dataset enriched with judge split vote distributions from the European Court of Human Rights, to investigate human disagreement and model calibration in legal case outcome classification, revealing that while models perceive split-vote cases as difficult, they struggle to align their uncertainty with the nuanced degree of human judicial disagreement.
Key insights
- Judicial disagreement, particularly split votes, is significantly correlated with 'Specific Legal Context,' such as cases involving vulnerable applicants or high-reputation countries.
- While models can differentiate the overall difficulty of split-vote cases from unanimous ones, they show a weak correlation with the *degree* of judge disagreement (entropy).
- Achieving true 'human calibration'—where a model's uncertainty aligns with the nuanced distribution of human judgments—remains a significant challenge, even with soft-label training or temperature scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.