Reproduce “Data Pipeline Quality: Influencing Factors, Root Causes of Data-related Issues, and Processing Problem Areas for Developers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harald Foidl, Valentina Golendukhina, Rudolf Ramler, Michael Felderer
This research systematically investigated factors influencing data pipeline quality and identified common developer issues, establishing a comprehensive taxonomy of 41 influencing factors across five themes. Empirical studies revealed data type problems as a primary root cause and highlighted data integration, ingestion, and compatibility as key challenges faced by developers.
Key insights
- A comprehensive, domain-agnostic taxonomy of 41 influencing factors, organized into 14 categories and 5 themes, provides a foundational understanding of data pipeline quality.
- Data type issues are the most frequent root cause of data-related problems, often manifesting in the cleaning, ingestion, and integration stages of pipelines.
- Compatibility emerges as a distinct and significant problem area for developers in data processing, separate from traditional pipeline stages, frequently concerning software or hardware interoperability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.