Reproduce “Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift” on OpenResearch

About this paper

Robin Holzinger, Riccardo Colletti

University of California, Berkeley researchers empirically investigated how neural network architectures influence robustness to temporal distribution shift across image and text data. Their work found that models achieving the highest in-distribution accuracy often experienced the fastest performance degradation over time, advocating for a re-evaluation of model selection criteria in dynamic environments.

Key insights

• A systematic trade-off exists: architectural inductive biases that enable superior in-distribution accuracy often render a model more vulnerable to temporal distribution shift.
• Models that exploit highly discriminative, localized features for peak performance on current data tend to degrade faster as data distributions evolve, as these features are often 'period-specific'.
• Architectures with weaker inductive biases or those leveraging stable, coarse-grained representations from large-scale pre-training generally exhibit more gradual degradation over time, even if their initial in-distribution performance is lower.

Summary by alphaXiv.