Reproduce “Legal Prompting: Teaching a Language Model to Think Like a Lawyer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fangyi Yu, Lee Quartey, Frank Schilder
Researchers from Thomson Reuters Labs demonstrated that guiding large language models with prompts explicitly designed around established legal reasoning frameworks, such as TRRAC, substantially improves performance on complex legal textual entailment tasks. This approach set new state-of-the-art accuracies for the COLIEE competition, achieving an 81.48% accuracy on the 2021 dataset.
Key insights
- Domain-specific legal reasoning prompts, particularly TRRAC, were found to be exceptionally effective, substantially guiding LLMs to perform complex legal textual entailment and outperforming general prompting methods.
- Even minor changes in prompt wording can significantly impact LLM performance in specialized legal domains, highlighting the models' sensitivity to input formulation.
- Few-shot prompting consistently improved performance over basic zero-shot, while Chain-of-Thought (CoT) prompting showed inconsistent effectiveness for legal reasoning, sometimes degrading results.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.