Reproduce “Mitigating Copy Bias in In-Context Learning through Neuron Pruning” on OpenResearch

About this paper

Ameen Ali, Lior Wolf, Ivan Titov

Researchers introduced a method to mitigate "copying bias" in In-Context Learning (ICL) by identifying and pruning specific neurons responsible for this behavior within Large Language Models (LLMs). This neuron-level intervention, guided by a synthetic task, consistently improved ICL accuracy across diverse LLM architectures and tasks, outperforming prompt-engineering baselines.

Key insights

• Copying bias in ICL can be attributed to a specific, task-agnostic subset of "copying neurons" within LLMs, which can be identified using a carefully constructed synthetic task.
• Directly intervening at the neuron level through pruning offers a complementary and effective strategy for improving ICL performance and reliability, moving beyond external prompt manipulations.
• Pruning these copying neurons not only reduces errors but also enhances the quality of the model's "task vectors," suggesting an improved ability to represent and infer task-relevant information.

Summary by alphaXiv.