Reproduce “Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling” on OpenResearch

About this paper

Changze Lv, Zhenghua Wang, Yiran Ding, Yixin Wu, Tianlong Li, Zhibo Xu, Muling Wu, Tianyuan Shi and 4 more

Fudan University and Westlake University researchers developed Layer-Specific Positional Embedding Scaling (LPES), a method that applies unique scaling factors to Rotary Position Embeddings (RoPE) in each transformer layer. This approach, optimized by a Bézier-constrained genetic algorithm, consistently improved LLMs' ability to utilize long contexts by reducing positional bias without introducing additional inference latency, achieving up to 2.42x speedup compared to multi-pass baselines.

Key insights

• Modeling layer-specific RoPE scaling factors with Bézier curves dramatically reduces the combinatorial search space and introduces a beneficial inductive bias of smoothness, preserving layer-wise representational structure.
• The optimization process for these scaling factors is training-free and fast, enabling efficient adaptation of deployed LLMs without fine-tuning model parameters or risking catastrophic forgetting.
• Optimal layer-specific scaling effectively mitigates positional bias and enhances context utilization in LLMs while maintaining general capabilities and exhibiting strong transferability across diverse long-context tasks.

Summary by alphaXiv.