Reproduce “Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Philip Groet, Joost Hoozemans, Andreas Grapentin, Felix Eberhardt, Zaid Al-Ars, H. Peter Hofstee
This paper presents a system that enables zero-copy, zero-serialization data sharing across compute nodes by integrating Apache Arrow with memory disaggregation techniques. The approach achieves up to a 40x speedup in data transfer for large datasets, addressing memory underutilization in data centers.
Key insights
- Extending Apache Arrow allows for zero-copy memory sharing by directly mapping data across nodes, eliminating data duplication.
- Transferring only table descriptors (metadata) via Apache Arrow's modified IPC protocol enables zero-serialization data transfer, avoiding costly serialization/deserialization.
- The system provides transparent memory disaggregation, allowing applications to use the familiar Apache Arrow API without awareness of the underlying distributed memory architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.