Reproduce “Quantum Doubly Stochastic Transformers” on OpenResearch

About this paper

Jannis Born, Filip Skogh, Kahn Rhrissorrakrai, Filippo Utro, Nico Wagner, Aleksandros Sobczyk

A hybrid classical-quantum Transformer, termed QDSFormer, replaces the conventional softmax function in self-attention with a variational quantum circuit (QontOT) to enforce doubly stochastic attention. This approach enhances training stability and performance, particularly for Vision Transformers on small-scale data, achieving a 30% accuracy improvement and significantly earlier "Eureka moments" on a compositional object recognition task.

Key insights

• The QontOT quantum circuit generates a significantly more diverse and complete set of DSMs than classical methods, effectively preserving input information and inducing higher entropy.
• Enforcing parametric doubly stochastic attention via QontOT consistently improves Transformer training stability, reduces performance variance, and accelerates critical learning phases (e.g., "Eureka moments").
• Quantum hardware noise, surprisingly, tends to increase the entropy of generated DSMs, which may act as a beneficial regularization effect by mitigating vanishing gradients while largely preserving the relative ordering of attention values.

Summary by alphaXiv.