Reproduce “FlexFL: Flexible and Effective Fault Localization with Open-Source Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chuyang Xu, Zhongxin Liu, Xiaoxue Ren, Gehao Zhang, Ming Liang, David Lo
FlexFL is a two-stage, LLM-based framework for fault localization that flexibly leverages bug reports and/or test cases. It demonstrates competitive performance using open-source LLMs like Llama3-8B, outperforming traditional and many proprietary LLM-based baselines while mitigating data privacy and cost concerns.
Key insights
- Combining diverse bug information (bug reports and test cases) within a two-stage LLM-based refinement process enhances fault localization accuracy.
- Open-source LLMs, when equipped with a custom tool interaction pipeline and robust postprocessing, can achieve competitive fault localization performance compared to proprietary models, addressing privacy and cost concerns.
- The iterative 'Reason-Act' mechanism and precise FQN matching enable open-source LLMs to effectively navigate and extract information from code repositories, overcoming inherent limitations like context length and lack of native function calling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.