Reproduce “Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics” on OpenResearch

About this paper

Zhengheng Li, Panrui Li, Xuyang Liu, Puzhi Xia

Researchers at Southeast University found that the test query's spatial position in Diffusion Large Language Models' In-Context Learning is a critical variable, with optimal placement being task-dependent (e.g., trailing for sequential reasoning, prefix for global perception). They developed "Average Confidence" as a reliable metric for decoding stability, which underpins Auto-ICL, a training-free adaptive routing strategy that achieved near-oracle performance across various tasks.

Key insights

• Query position is a first-order variable in dLLM ICL, with performance fluctuations due to placement being comparable to those from changing demonstration content.
• Optimal query placement is highly task-dependent, requiring trailing positions for sequential reasoning tasks but prefix positions for global-perception tasks.
• Positional sensitivity in dLLMs stems from a "Recency Effect" in attention, where the query anchors reasoning to adjacent examples, and from task-dependent reshaping of token-level decoding trajectories (e.g., strict diagonal for trailing, "V-shape" for prefix).
• "Average Confidence," aggregating activation probabilities across all decoding steps, serves as a robust proxy for generation stability and correlates strongly with task accuracy in dLLMs, unlike traditional single-step confidence.

Summary by alphaXiv.