Reproduce “Data-Centric Machine Learning in the Legal Domain” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hannes Westermann, Jaromir Savelka, Vern R. Walker, Kevin D. Ashley, Karim Benyekhlef

This research empirically quantifies how dataset properties, including size, train/test splits, and labeling errors, influence deep learning classifier performance across various legal text classification tasks. It identifies 'semantic homogeneity' as a key factor explaining why some legal categories are inherently more difficult for models to learn, demonstrating that data quality and characteristics profoundly impact AI system reliability in the legal domain.

Key insights

  • Performance improvements in legal machine learning can be substantially achieved by enhancing data properties, often rivaling or exceeding gains from model architecture tweaks.
  • Systematic labeling errors are far more detrimental to classifier performance than random errors, underscoring the critical need for robust human annotation guidelines and quality control.
  • The 'semantic homogeneity' of a class directly correlates with its learning difficulty, with more homogeneous classes being easier to learn and more robust to data variations and errors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.