Reproduce “Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs” on OpenResearch

About this paper

Mikhail Menschikov, Alexander Kharitonov, Maiia Kotyga, Vadim Porvatov, Anna Zhukovskaya, David Kagramanyan, Egor Shvetsov, Evgeny Burnaev

This study investigates position bias in multilingual Large Language Models, demonstrating it is highly model-specific (e.g., some prefer late positions) and that explicit relevance scoring in prompts can unexpectedly degrade accuracy. It also reveals that models can be "confidently wrong" when failing to utilize middle-context information, as accuracy drops are not reflected in increased output entropy.

Key insights

• Position bias is primarily a model-driven phenomenon, with models like DeepSeek-7B, Mistral-7B, and Qwen2.5-7B exhibiting a strong late-position preference, contrary to the common assumption of universal early-token bias.
• Explicit prompt-based guidance using relevance scores unexpectedly degraded accuracy across most languages and models compared to omitting scores, challenging current prompt engineering best practices.
• LLMs exhibit a disconnect between positional disadvantage and expressed uncertainty, remaining "confidently wrong" when failing due to middle-context bias, as accuracy drops are not mirrored by increased output entropy.

Summary by alphaXiv.