Reproduce “Less Is More: Measuring How LLM Involvement affects Chatbot Accuracy in Static Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Krishna Narasimhan
A study systematically evaluates how varying degrees of Large Language Model (LLM) involvement affect the accuracy of natural language-to-static analysis query translation, specifically for Joern's CPGQL. It finds that constraining LLM output to a structured intermediate representation yields superior accuracy and efficiency compared to direct query generation or agentic, tool-augmented approaches.
Key insights
- Constraining LLM output to a structured intermediate representation (A2) significantly outperforms direct query generation (A1) and tool-augmented agentic generation (A3) in accuracy for natural language-to-CPGQL translation.
- Agentic generation (A3) delivered the lowest accuracy across all models while incurring substantially higher computational costs (e.g., 8x more tokens than A2) and slower execution times, providing no complementary problem-solving capability.
- Model scale interacts with approach: large LLMs excel at generating schema-compliant structured intermediates, while smaller LLMs face a 'schema compliance' bottleneck, struggling more with valid JSON generation despite the benefits of the A2 approach.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.