Reproduce “NitiBench: A Comprehensive Study of LLM Framework Capabilities for Thai Legal Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Tasawong, Thitiwat Nopparatbundit, Keerakiat Pratai, Sarana Nutanong
NitiBench introduces a comprehensive benchmark and evaluation framework for Thai legal question answering, featuring two datasets (NitiBench-CCL, NitiBench-Tax) and multi-label retrieval metrics. The research reveals that Retrieval-Augmented Generation (RAG) systems, especially with hierarchy-aware chunking, outperform Long-Context LLMs for complex Thai legal tasks, while identifying current retrievers as a significant performance bottleneck.
Key insights
- Integrating domain knowledge through hierarchy-aware chunking consistently improved retrieval and end-to-end performance in RAG systems, preserving legal section integrity.
- Current retrieval models represent a significant bottleneck for Thai legal QA, particularly for complex, multi-label queries, and show poor generalization to out-of-domain data.
- RAG-based systems, when incorporating tailored components, significantly outperformed Long-Context LLMs as standalone QA systems for complex Thai legal tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.