Reproduce “Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation” on OpenResearch

About this paper

Zichong Li, Chen Liang, Liliang Ren, Tuo Zhao, Yelong Shen, Weizhu Chen

Researchers from Georgia Institute of Technology and Microsoft developed RoPE-Perturbed Self-Distillation, a training method that enhances large language models' ability to process long contexts by making them less sensitive to the exact position of information. This approach improves long-context performance and positional robustness, leading to a 12.04% average RULER score improvement for Llama-3-8B at 64K context and better length extrapolation.

Key insights

• Positional brittleness in LLMs likely stems from an over-reliance on brittle, fine-grained phase information encoded by higher-frequency components of RoPE indices.
• Enforcing predictive consistency between standard and RoPE-perturbed input views encourages models to depend more on robust semantic signals and coarser, more stable positional structures.
• The specific design choices of the self-distillation objective, including reverse KL divergence with a stop-gradient teacher and stochastic per-sample perturbation, are crucial for optimal performance.

Summary by alphaXiv.