Reproduce “Cross-table Synthetic Tabular Data Detection” on OpenResearch

About this paper

G. Charbel N. Kindji, Lina Maria Rojas-Barahona, Elisa Fromont, Tanguy Urvoy

Researchers from Orange Labs and Université de Rennes explored the detection of synthetic tabular data in scenarios where both the generative model and table structure are unknown, termed "cross-table shift". The study found that while detection is effective in controlled environments (AUC up to 0.91), current table-agnostic baseline methods achieved performance barely better than random chance (AUC 0.51-0.58) under cross-table shift, highlighting the difficulty for real-world deployment.

Key insights

• Synthetic tabular data contains detectable patterns in controlled environments where generators and datasets are known, as evidenced by the Table-Transformer's AUC of 0.91 for TVAE detection in a "No Shift" scenario.
• The "cross-table shift" poses a significant challenge, causing a drastic performance drop for all baseline detection models, indicating fundamental limitations in current table-agnostic approaches.
• Textual representations of tabular data, even simple character trigrams, demonstrated marginally better generalization compared to more complex table-based transformer models under severe cross-table shifts, although overall performance remained low.

Summary by alphaXiv.