Reproduce “Geometry-Guided Layerwise FFN Width Allocation in Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timur Mudarisov, Mikhail Burtsev, Radu State
A method quantifies "geometric work" of Feed-Forward Network layers in Transformers using activation-derived profiles to non-uniformly allocate FFN width, leading to improved validation loss compared to uniform width and hand-designed tapering schedules. This approach leverages normalized geometric metrics to adapt FFN capacity dynamically across layers, enhancing efficiency and performance.
Key insights
- Normalized geometric work metrics (spherical and hyperbolic) consistently reveal a "front-loaded" FFN capacity profile, indicating higher demand in earlier layers, which contrasts with scale-sensitive raw geometry metrics that increase towards later layers.
- The assumptions of local additive loss and an intrinsic error-width power law hold empirically, supported by perturbation diagnostics and intrinsic dimension estimates of residual clouds, justifying their use in the allocation surrogate.
- Geometry-derived schedules provide a data-dependent, adaptable alternative to fixed tapering, capable of identifying non-monotone capacity profiles and improving upon traditional uniform width allocation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.