Reproduce “CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability” on OpenResearch

About this paper

Chad A. Capps

CART (Context-Anchored Recurrent Transformer) introduces an architecture that uses a unique-layer prelude to generate a frozen K,V context anchor for a shared-weight cross-attending recurrent core, incorporating a learned Linear Time-Invariant (LTI) gate for stability. At the largest tested scale (d=1024), CART did not achieve parameter efficiency compared to standard stacked transformers, showing 1-2% higher perplexity than a parameter-matched Dense 7-layer model and approximately 10% higher perplexity than a Dense 12-layer model.

Key insights

• The learned LTI gate's spectral radius consistently settled within a narrow band of [0.79, 0.83] during full training, indicating an aggressive-discard regime and adaptability to model width and recurrence depth.
• Initial representational depth provided by the prelude (P) consistently contributed more significantly to model performance than the number of recurrent iterations (R) in the shared core.
• The benefit of increasing recurrence depth (R) reversed during full training, with R=6 performing optimally among tested depths for larger models (d "greater than or equal to" 512), rather than higher R values.

Summary by alphaXiv.