Reproduce “Improving the Accuracy and Efficiency of Legal Document Tagging with Large Language Models and Instruction Prompts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Emily Johnson, Xavier Holt, Noah Wilson
Researchers at the University of Massachusetts, Amherst, developed Legal-LLM, an instruction-tuned Large Language Model for legal multi-label classification that reinterprets the task as structured generation. This approach achieved micro-F1 scores of 0.83 on POSTURE50K and 0.80 on EURLEX57K, along with macro-F1 scores of 0.76 and 0.71, respectively, outperforming baselines and demonstrating enhanced accuracy for less frequent legal categories through a weighted loss function.
Key insights
- Reframing multi-label classification as a structured generation problem effectively leverages the instruction-following and generative capabilities of LLMs, surpassing traditional classification paradigms.
- The implementation of a weighted loss function during fine-tuning significantly improves the model's performance on low-frequency labels, directly mitigating label imbalance issues in legal datasets.
- Specific phrasing of instruction prompts can subtly influence the model's output and overall performance, highlighting the importance of prompt engineering in instruction-tuned LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.