Reproduce “PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis” on OpenResearch

About this paper

Yan Wu, Esther Wershof, Sebastian M Schmon, Marcel Nassar, Błażej Osiński, Ridvan Eksi, Zichao Yan, Rory Stark and 2 more

PerturBench introduces a standardized framework for benchmarking machine learning models that predict cellular responses to genetic and chemical perturbations. This framework curates diverse datasets, defines biologically relevant tasks, and incorporates novel rank-based metrics to detect critical model failures like mode collapse, revealing that simpler models often outperform complex published architectures in predicting perturbation effects.

Key insights

• No single model universally outperforms others across all datasets and metrics; simpler baseline models (e.g., Latent Additive) are often highly competitive or superior to more complex published architectures, especially on larger datasets and with rank metrics.
• Mode collapse is a pervasive issue in many generative models, and traditional fit-based metrics are insufficient for its detection, underscoring the necessity of the newly introduced rank-based diagnostics.
• Ablation studies reveal that complex architectural components, such as adversarial classifiers in CPA* or sparsity constraints in SAMS-VAE*, can sometimes hinder performance or are difficult to optimize effectively, suggesting potential for simpler designs.

Summary by alphaXiv.