Reproduce “Toward Robust Legal Text Formalization into Defeasible Deontic Logic using LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Elias Horner, Cristinel Mateis, Guido Governatori, Agata Ciabattoni
The research explores the use of Large Language Models to formalize legal provisions into Defeasible Deontic Logic (DDL), demonstrating their capability with prompt engineering and a two-stage refinement pipeline. This method achieves higher F1-scores in term identification and rule correspondence compared to previous rule-extraction approaches, offering an efficient solution to the legal knowledge representation bottleneck.
Key insights
- Iterative refinement of Chain-of-Instructions (CoI) prompting proved the most effective baseline for guiding LLMs toward semantically valid and logically coherent DDL rules.
- A two-stage pipeline incorporating a dedicated refinement step, particularly using Claude Sonnet 4 with Extended Thinking, modestly improved overall success scores by consolidating atom names and enhancing consistency.
- Attempts to process multiple law snippets simultaneously or incorporate extensive formalization history generally yielded limited or negative performance, suggesting that too much context can hinder LLM focus.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.