Reproduce “Anka: A Domain-Specific Language for Reliable LLM Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Saif Khalfan Saif Al Mazrouei
Anka, a new domain-specific language, improves the reliability of code generated by large language models for data transformation tasks. The language's constrained syntax reduces systematic errors in complex multi-step code generation, achieving 100% accuracy on such tasks compared to 60% with Python.
Key insights
- LLMs can acquire novel DSLs entirely from in-context prompts, achieving a 99.9% parse success rate for Anka without any prior fine-tuning on the new language.
- Designing a target language with explicit constraints and unambiguous structures effectively mitigates common systematic errors made by LLMs in complex code generation.
- The accuracy benefits of DSL constraints are most pronounced for multi-step and complex tasks, demonstrating that explicit guidance is crucial as task complexity increases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.