Reproduce “Measuring Disentanglement: A Review of Metrics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon
Researchers from Ubisoft - La Forge and École de technologie supérieure (ETS) systematically reviewed 13 supervised disentanglement metrics, proposing a new taxonomy and evaluating their performance under controlled conditions. Their analysis clarifies what each metric truly measures, highlighting the robustness of predictor-based methods (like DCI with Random Forest) and the limitations of others, particularly concerning discretization, noise, and non-measured factors.
Key insights
- Predictor-based metrics, especially DCI implemented with a Random Forest, offer superior robustness to noise, non-linear relations, and non-measured factors, effectively assessing modularity and compactness.
- Discretization, a common component in many information-based and intervention-based metrics, significantly impairs their accuracy when dealing with non-linear factor-code relationships or noise.
- Metrics that rely on computing a 'gap' between top information scores (e.g., SAP, MIG-RMIG) often yield binary-like results, failing to capture the continuous degree of disentanglement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.