Reproduce “Investigating Recurrent Transformers with Dynamic Halt” on OpenResearch

About this paper

Jishnu Ray Chowdhury, Cornelia Caragea

This research provides an empirical comparison of depth-wise and chunk-wise recurrent Transformer architectures for algorithmic and generalization tasks, introducing Gated Universal Transformer (GUT) and Gated Universal Temporal Latent Bottleneck (GUTLB). The findings highlight distinct performance trade-offs in adaptive computational depth, information flow control, and robustness across varying sequence complexities.

Key insights

• Incorporating explicit inductive biases like global dynamic halting and explicit gating into depth-wise recurrent Transformers can improve performance on near-IID algorithmic tasks.
• Chunk-wise recurrent Transformers offer superior robustness and length generalization by bounding the attention receptive field, despite potentially underperforming on IID hierarchical tasks.
• Simply adding depth-wise adaptive computation within chunks (GUTLB) does not consistently yield benefits over baseline chunk-wise models, indicating complexities in combining multi-level recurrence.

Summary by alphaXiv.