Reproduce “Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanggyu Lee, Suchae Jeong, Jihie Kim
This research from Dongguk University developed a method to improve Large Language Model classification of logical errors by integrating a refined taxonomy of error types and their relationships into prompt engineering. The approach significantly increased classification accuracy, with GPT-4 achieving 86%, and demonstrated a methodology for augmenting logical error datasets from correct code.
Key insights
- Integrating explicit domain knowledge, such as a refined taxonomy of logical error types and their causal/coincidental relationships, significantly enhances LLM performance in complex classification tasks.
- Advanced prompt engineering techniques like Chain-of-Thought and Tree-of-Thought are effective in guiding LLMs to reason through ambiguous classifications, especially when combined with structured knowledge.
- LLMs can be leveraged to augment specialized datasets by generating examples of specific logical errors from correct code, helping address data scarcity in niche domains like programming education.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.