Reproduce “ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huy Tran, Max Milkert, David Hyde
ASAP (Amortized Doubly-Stochastic Attention via Sliced Dual Projection) converts trained iterative Sinkhorn attention layers into efficient, non-iterative operators for inference, achieving a 5.3x speedup on IMDb with minimal performance degradation. The method preserves the performance of high-fidelity Sinkhorn teachers while maintaining their resource-efficient training.
Key insights
- Doubly-stochastic attention, by imposing column balance, acts as a beneficial inductive bias for Transformers, mitigating issues like attention sinks.
- Complex, iterative optimal transport dual potentials (from Sinkhorn) can be accurately approximated by a lightweight parametric map of simple sliced geometric features.
- A "train-then-compile" paradigm can combine the training-time resource efficiency of Sinkhorn attention with the inference efficiency of non-iterative methods, preserving the performance of the trained teacher.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.