Reproduce “Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thomas T. Zhang, Alok Shah, Yifei Zhang, Vincent Zhang, Nikolai Matni, Max Simchowitz
A new optimizer, Double Preconditioning (DoPr), enhances deep learning models' resilience to Test-Time Feedback by promoting isotropic feature learning. This approach improves downstream performance metrics across various applications, often independently of traditional validation loss improvements.
Key insights
- Downstream performance in Test-Time Feedback scenarios is often decoupled from standard training or validation loss, indicating that optimizing solely for loss may not suffice for real-world utility.
- Activation Preconditioning (AP) addresses non-isotropic feature learning by whitening gradients, promoting a more uniform contraction of feature subspace distances and robust error mitigation under TTF.
- DoPr serves as a flexible "plug-in" to existing gradient-based optimizers and remains compatible with hyperparameter scaling frameworks like "P, simplifying practical adoption.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.