Reproduce “If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xietao Wang-Lin, Anton Isopoussu, Louis Mahon
Iterative LLM bug-fixing can behave as an unstable dynamical process: models often modify correct programs, and the resulting damage can outweigh successful repairs. Experiments with Gemini 2.5 Flash-Lite and Qwen2.5-7B-Instruct show that localized search/replace edits create more cycles and invalid changes than whole-file rewriting, while activation analysis reveals an internal signal that regulates how readily the model labels code as buggy.
Key insights
- A model’s ability to find bugs does not imply that continued intervention is safe: pseudo-bug detection can make correct code fail tests or stop compiling.
- Atomic edits are operationally convenient but tend to produce more invalid applications, compensatory changes, and recurring cycles than complete-file rewrites.
- The internal bugginess signal is controllable but not selective: suppressing it preserves correct code while also blocking repairs, whereas amplifying it increases both useful edits and hallucinated or harmful changes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.