Reproduce “Scikit-dimension: a Python package for intrinsic dimension estimation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonathan Bac, Evgeny M. Mirkes, Alexander N. Gorban, Ivan Tyukin, Andrei Zinovyev
Researchers introduced `scikit-dimension`, an open-source Python package that unifies 19 intrinsic dimension estimators under a scikit-learn compatible API. Extensive benchmarking on 521 real-life and synthetic datasets characterized the performance and inter-relationships of these methods, identifying a strong "consensus" intrinsic dimension.
Key insights
- A strong "consensus" intrinsic dimension underlies the diverse estimates, with the first principal component of z-scored ID values explaining 42.6% of the variance across methods.
- Intrinsic dimension estimators cluster into distinct groups, primarily separating PCA-based methods from non-linear methods, which further sub-cluster based on their operating principles.
- Certain commonly used PCA-based ID estimators, such as "broken stick" and Kaiser criteria, are highly sensitive to redundant features, leading to inflated ID estimates in such scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.