Reproduce “Principles for Open Data Curation: A Case Study with the New York City 311 Service Request Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Tussey, Jun Yan
This work empirically investigates extensive data quality challenges within the New York City 311 Service Request dataset and proposes six tailored data curation principles to enhance the reliability and utility of government open data. The analysis revealed pervasive issues such as structural inconsistencies, undocumented fields, and date anomalies, which informed the development of practical guidelines for improving data integrity.
Key insights
- Even prominent government open datasets exhibit pervasive data quality issues, including undocumented fields, inconsistent missing data representations, and significant date-time anomalies (e.g., negative/zero service request durations due to processing errors or Daylight Saving Time changes).
- Redundant and sparse fields are common, indicating inefficient data structures and a potential for significant file size reduction without loss of critical information.
- Official data dictionaries are frequently outdated or incomplete, leading to a mismatch between documented and actual data characteristics, which complicates data interpretation and use by stakeholders.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.