Reproduce “Wrangling Messy CSV Files by Detecting Row and Type Patterns” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gerrit J. J. van den Burg, Alfredo Nazabal, Charles Sutton
Automatic detection of CSV file dialects is achieved using a data consistency measure that combines pattern and type scoring, enabling the method to correctly parse over 97% of files in diverse real-world corpora like GitHub and UKdata.
Key insights
- Correct CSV parsing yields highly consistent tabular structures and recognizable data types, which can be quantified through pattern and type scores.
- Mimicking how human analysts identify correct parsing by looking for structural regularity and consistent data types is an effective strategy for automated dialect detection.
- Combining a pattern score (row length consistency) with a type score (data type uniformity across columns) provides a robust measure to identify the most suitable CSV dialect.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.