Reproduce “HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position” on OpenResearch

About this paper

Kechi Zhang, Ge Li, Huangzhao Zhang, Zhi Jin

HiRoPE introduces a training-free method to extend the context length of large language models for code by integrating its intrinsic hierarchical structure into Rotary Position Embedding. This approach significantly improves performance on long code language modeling, achieving up to a hundredfold increase in recall on code symbol understanding tasks and demonstrating potential for exponential length extrapolation.

Key insights

• Explicitly leveraging the hierarchical structure of source code in positional encoding dramatically enhances LLMs' ability to model long-range dependencies.
• Distributing positional information across hierarchical levels effectively mitigates out-of-distribution (O.O.D) issues, enabling robust length extrapolation beyond pre-training limits.
• A training-free, 'plug-and-play' modification to RoPE can yield substantial context extension benefits while maintaining stable performance on both short and ultra-long sequences.

Summary by alphaXiv.