Reproduce “Data Cleaning Using Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuo Zhang, Zezhou Huang, Eugene Wu
A Columbia University-developed system, Cocoon, introduces a two-level decomposition strategy for data cleaning that integrates large language models (LLMs) with statistical methods to achieve higher accuracy and recall than existing systems, consistently outperforming baselines on 4 out of 5 standard datasets.
Key insights
- Decomposing complex data cleaning problems into smaller, manageable sub-tasks enables LLMs to effectively leverage their semantic understanding despite context window limitations.
- Combining traditional statistical methods for efficient detection and LLM-driven semantic reasoning allows for robust identification and correction of errors that purely statistical or purely LLM-based approaches miss.
- Generating cleaning operations as commented SQL queries ensures scalability, interpretability, and seamless integration into existing data pipelines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.