Reproduce “Quantum-Inspired Self-Attention in a Large Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikita Kuznetsov, Niyaz Ismagilov, Ernesto Campos
This research introduces Classical Quantum-Inspired Self-Attention (QISA) and a quantum-deployable variant (QISA-A), integrating them into a GPT-1 autoregressive language model. QISA achieved a 13x improvement in cross-entropy loss, a 15.5x better character error rate, and a 4.7x better word error rate compared to classical self-attention, with a 2.6x increase in inference time.
Key insights
- The quantum-inspired architectural design of QISA's value layer leads to improved performance even when parameter counts are equivalent to classical self-attention in single-head configurations.
- The increased complexity from the quantum-inspired value layer enhances the model's ability to perform more effective transformations, contributing to better language modeling.
- Shallow quantum circuits (1-3 layers of the Hardware Efficient Ansatz) proved sufficient for the tested configurations, showing minimal additional benefit from deeper circuits.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.