Reproduce “Complex Transformer: A Framework for Modeling Complex-Valued Sequence” on OpenResearch

About this paper

Muqiao Yang, Martin Q. Ma, Dongyu Li, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov

Researchers from Carnegie Mellon University developed the Complex Transformer, an encoder-decoder framework that natively processes complex-valued sequences by expanding the attention mechanism for complex operations. The model achieved 74.22% Average Precision Score on music transcription and 59.94% accuracy on IQ signal classification, demonstrating improved performance over real-valued and existing complex-valued baselines.

Key insights

• Directly modeling complex-valued sequences enables the capture of both magnitude and phase information, potentially leading to richer representations than processing only real parts or concatenating real/imaginary components.
• The attention mechanism's ability to capture global context and long-range dependencies is beneficial for complex-valued sequence modeling, overcoming limitations of recurrent architectures in this domain.
• Decomposing complex attention operations into interacting real and imaginary multi-head attention terms allows for a principled way to integrate complex numbers into the Transformer architecture.

Summary by alphaXiv.