Reproduce “Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Piyush Sao
This work identifies complex singularities, termed "ghosts of softmax," as a fundamental geometric constraint limiting safe step sizes in cross-entropy optimization. A computable step-size reliability boundary is introduced, demonstrating that a normalized step size below 1 prevents training collapse, and an adaptive controller based on this boundary allows training to survive extreme learning rate spikes.
Key insights
- The reliability of local Taylor models for cross-entropy loss is fundamentally limited by the distance to complex singularities in the loss landscape, rather than solely by real-line curvature.
- A normalized step size, r = τ/ρa, where τ is the step distance and ρa is the derived radius, serves as a universal, optimizer-agnostic coordinate for predicting training stability, with r < 1 indicating safe operation.
- As model predictions become more confident, the safe step size ρa tends to shrink due to an increasing logit-derivative spread (Δa), explaining why training can become fragile even when the real loss surface appears smooth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.