Reproduce “Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features” on OpenResearch

About this paper

Kaichen Xu, Yihang Du, Mianpeng Liu, Zimu Yu, Xiaobo Sun

The Causality-Aware Position Encoder (CAPE) integrates explicit causal structures into transformer models, addressing their reliance on sequential order for positional encoding in non-sequential data. CAPE identifies causal relationships, embeds them in hyperbolic space, and converts them into rotary positional encodings, yielding improved performance in diverse multi-omics tasks.

Key insights

• Causal relationships can be effectively learned from non-sequential data and geometrically embedded into hyperbolic space to preserve key causal properties like strength and specificity.
• Integrating causality-aware positional encodings into transformer self-attention leads to attention attenuation that scales with causal distance and generality.
• The use of hyperbolic geometry within a rotary positional encoding framework enhances the transformer's ability to process non-sequential data by endowing it with robustness to positional disturbances and sensitivity to subtle causal semantics.

Summary by alphaXiv.