Reproduce “FreaAI: Automated extraction of data slices to test machine learning models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Samuel Ackerman, Orna Raz, Marcel Zalmanovici

IBM Research developed FreaAI, a system that automatically extracts statistically significant and explainable data slices where machine learning models exhibit under-performance. This system consistently identified localized model weaknesses across various open datasets, providing targeted insights beyond aggregate performance metrics.

Key insights

  • Uncovering under-performing data slices is critical for ensuring ML solutions deliver their expected business value, addressing specific conditions where a model might fail.
  • Automated extraction of feature models and statistically rigorous validation are essential to efficiently and reliably identify these hidden model weaknesses.
  • Combining simple, interpretable heuristics with statistical significance testing allows for the discovery of actionable and explainable model failure conditions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.