Reproduce “CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, Jong-Seon No

CrossMPT introduces a transformer architecture for error correcting codes that utilizes masked cross-attention guided by the parity-check matrix to process distinct magnitude and syndrome inputs. This approach improves decoding performance and significantly reduces computational complexity, memory usage, and inference time, making transformer-based decoders practical for longer codes.

Key insights

  • Explicitly separating and processing magnitude and syndrome information with cross-attention, rather than concatenated self-attention, significantly improves decoding efficiency and performance.
  • Integrating the code's parity-check matrix as a mask for cross-attention mechanisms provides a theoretically grounded and computationally efficient way to leverage domain knowledge in neural decoders.
  • Less important magnitude-magnitude and syndrome-syndrome relationships identified in previous transformer decoders can be effectively removed, allowing for more targeted and efficient attention computation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.