Reproduce “Selecting Sub-tables for Data Exploration” on OpenResearch

About this paper

Kathy Razmadze, Yael Amsterdamer, Amit Somech, Susan B. Davidson, Tova Milo

The SubTab framework from Tel Aviv University, Bar-Ilan University, and the University of Pennsylvania introduces a two-phase method for generating small, informative sub-tables from large datasets, designed for initial data exploration. It employs table embedding and centroid-based selection to capture prominent data patterns and diversity, enabling users to derive an average of 4 correct insights per dataset compared to 1.2 for random baselines, with interactive response times.

Key insights

• Formally defining 'informativeness' for sub-tables based on 'cell coverage' (how well prominent association rules are captured) and 'diversity' (how non-repetitive the sub-table's rows are) provides a robust criterion for data exploration.
• Directly optimizing sub-table selection against these informativeness metrics is computationally intractable for large datasets, necessitating an indirect, approximation-based approach.
• Table embeddings derived from cell value co-occurrences can effectively encode data patterns (analogous to association rules) and facilitate efficient, centroid-based selection of representative rows and columns that balance coverage and diversity.

Summary by alphaXiv.