Reproduce “Neural Coherence : Find higher performance to out-of-distribution tasks from few samples” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Simon Guiroy, Mats Richter, Sarath Chandar, Christopher Pal
Neural Coherence introduces a principle for model selection that identifies optimal pre-trained checkpoints for out-of-distribution tasks using a minimal number of unlabeled target samples. The method analyzes the evolving statistics of neural activations, demonstrating high statistical efficiency by closing significant generalization gaps across various architectures and tasks.
Key insights
- Analyzing the dynamic trajectories of neural activations across all layers provides richer, more robust signals for predicting OOD performance than static, final-layer metrics.
- The optimal point for OOD generalization is not necessarily the best source-validation performance, but rather where the model's internal representations maintain coherence between source and target before diverging.
- Characterizing activation distributions with aggregated moments is statistically efficient, enabling effective model and data selection with as few as 1-5 unlabeled target examples.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.