Reproduce “Optimization Opportunities for Cloud-Based Data Pipeline Infrastructures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Johannes Jablonski, Georg-Daniel Schwarz, Philip Heltweg, Dirk Riehle
Researchers from Friedrich-Alexander-Universität Erlangen-Nürnberg conducted a systematic literature review to identify and synthesize application- and domain-independent optimization opportunities for cloud-based data pipeline infrastructures. The study integrates optimization goals, contexts, and implementation methods into a concept matrix, providing a structured overview of existing research and highlighting key areas for future work.
Key insights
- The concept matrix developed provides a structured 'theory' mapping the interconnections between optimization goals, their contextual factors, and the implementation methods used in cloud data pipelines.
- Scheduling is the most prevalent optimization method, frequently employed for makespan-related objectives, while parameters like task dependencies and VM pricing are widely utilized.
- Significant research gaps exist in optimizing for multi-tenant environments, 'regular' (non-big) data, truly large-scale data, and the need for more industry-based evaluations and advanced ML/AI-driven approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.