Reproduce “A Preliminary Study on Explaining Risk of Code Changes using LLM-Based Prediction Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yalin Liu, Kosay Jabre, Rui Abreu, Zachariah J. Carmichael, Vijayaraghavan Murali, Akshay Patel, Jun Ge, Weiyan Sun and 5 more
A method is presented for explaining code change risk predictions from large language models by using attention weights to highlight potentially problematic code segments. Evaluation showed that highlighting the top 2 hunks achieved 53.85% coverage of outage root-cause lines with a 26.28% average review percentage.
Key insights
- LLM attention weights can provide low-latency, targeted explanations for code change risk by highlighting specific code segments, addressing the "black box" problem without generating potentially inaccurate natural language rationales.
- Hierarchical mapping of token-level attention to developer-centric code units (lines, hunks, files) and tailored aggregation functions are crucial for generating actionable explanations.
- A pragmatic balance exists between the coverage of outage root-cause lines and the average percentage of code developers must review, with hunk-level highlighting offering a suitable trade-off.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.