Reproduce “PANDA: Extreme Scale Parallel K-Nearest Neighbor on Distributed Architectures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md. Mostofa Ali Patwary, Nadathur Rajagopalan Satish, Narayanan Sundaram, Jialin Liu, Peter Sadowski, Evan Racah, Suren Byna, Craig Tull and 3 more
PANDA introduces a fully distributed K-Nearest Neighbor (KNN) algorithm based on a global kd-tree, designed to scale to massive scientific datasets exceeding a trillion data points. The system achieves high performance and near-linear speedups by exploiting multi-level parallelism and communication-computation overlap, processing billions of queries in seconds.
Key insights
- A global distributed kd-tree spatially partitions data across thousands of compute nodes, allowing for logarithmic search complexity for KNN queries on massive datasets.
- Multi-level parallelism is leveraged, from SIMD vector instructions for distance calculations to thread-level and cluster-level distribution, optimizing performance across modern supercomputing architectures.
- Efficient distributed kd-tree construction utilizes sampling to approximate medians for balanced splits, and the querying process dynamically adjusts search radii to balance accuracy and communication efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.