Reproduce “Rotary Masked Autoencoders are Versatile Learners” on OpenResearch

About this paper

Uros Zivanovic, Serafina Di Gioia, Andre Scaffidi, Martín de los Rios, Gabriella Contardo, Roberto Trotta

RoMAE (Rotary Masked Autoencoder) extends standard Transformer models to process diverse data modalities, including irregular time-series, images, and audio, by integrating continuous, multi-dimensional Rotary Positional Embeddings (RoPE). The framework achieves competitive or superior performance against specialized models across classification, regression, and interpolation tasks, notably reaching 84.7% accuracy on ESC-50 audio classification and improving F-scores on the ELAsTiCC irregular time-series challenge.

Key insights

• The inclusion of a learned `[CLS]` token within RoMAE's input sequence enables the Encoder to reconstruct absolute positional information, effectively serving as an anchor for relative positional embeddings.
• RoMAE demonstrates that a generalized Transformer architecture can effectively process diverse modalities like images, audio, and irregular time-series without specialized modifications, by leveraging continuous RoPE and MAE pre-training.
• The design of p-RoPE (truncated RoPE) ensures that a portion of the embedding space remains unmodified by rotational transformations, providing a direct data channel that enhances robustness to varying sequence lengths and challenges the notion of strong long-term decay.

Summary by alphaXiv.