Reproduce “Improving Chain-of-Thought for Logical Reasoning via Attention-Aware Intervention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nguyen Minh Phuong, Dang Huu Tien, Naoya Inoue
Researchers at JAIST developed Attention-Aware Intervention (AAI), a non-interactive, end-to-end method that enhances large language models' logical reasoning by directly reweighting specific attention heads based on structural prompt information. AAI improved accuracy by up to +2.83 on ProofWriter and +3.0 on LogicalDeduction with Qwen-3 32B, offering competitive performance with negligible computational overhead.
Key insights
- Specific attention heads within Transformer models exhibit distinct activation patterns (anchor, aggregation, copy) that are functionally aligned with different logical reasoning operations.
- Directly modulating attention scores during inference, by reinforcing relevant rule-content links and suppressing irrelevant ones, is an efficient way to inject prior knowledge and improve reasoning fidelity.
- Targeted intervention in a small subset of attention heads can lead to substantial improvements in logical reasoning performance, validating the functional specialization of these heads.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.