Reproduce “A Multi-Agent Approach to Fault Localization via Graph-Based Retrieval and Reflexion” on OpenResearch

About this paper

Md Nakhla Rafi, Dong Jae Kim, Tse-Hsun Chen, Shaowei Wang

Researchers at Concordia University and DePaul University introduce LLM4FL, a multi-agent framework leveraging large language models for fault localization in large-scale software projects. The system employs graph-based code navigation, a divide-and-conquer strategy to manage token limits, and verbal reinforcement learning for iterative refinement, achieving superior fault localization accuracy on the Defects4J benchmark compared to most existing LLM-based and supervised methods.

Key insights

• A multi-agent architecture combined with structured, graph-based code navigation effectively addresses LLM token limitations and enhances their ability to reason about inter-procedural dependencies in complex codebases.
• Integrating an iterative self-critique mechanism, inspired by verbal reinforcement learning, significantly improves LLM fault localization accuracy by enabling intrinsic refinement of rankings and reasoning.
• The initial ordering of input artifacts provided to LLMs has a substantial impact on fault localization performance, highlighting the importance of preprocessing strategies for optimal LLM utilization.

Summary by alphaXiv.